Choosing the right Central Log Management (CLM)
Purpose
A type of logging solution system that consolidates all of your log data and pushes to one central, accessible and easy-to-use interface. Centralize logging is designed to make your life easier. Not only does CLM provide multiple features that allow you to easily collect log information, but it also helps you consolidate, analyze, and view that information quickly and clearly. CLM gives you tons of capabilities including:
- Storing log data from multiple sources in a central location
- Enforcing retention policies on your logs so they are available for a specific time period
- Easily searching inside the logs for important information
- Generating alerts based on metrics you define on the logs
- Sharing your dashboard and log information with others simply and quickly
- Low costs and increased storage and backup for historical data
- Setting up security alerts and granting login access to particular users without granting server root access
The Options
For this things, we scoring based on
Graylog
Graylog is a leading centralized log management solution built to open standards for capturing, storing, and enabling real-time analysis of terabytes of machine data. We deliver a better user experience by making analysis ridiculously fast and efficient using a more cost-effective and flexible architecture.
Minimum Setup
Bigger Production Setup
| Details | |
|---|---|
| Licence Type | Open Source with Enterprise |
| Language | Java ( >= 8 ) |
| Components | Elasticsearch (6.x or 7.x) MongoDB (4.0 or 4.2) |
| OS Packages | Ubuntu, Debian, RHEL/CentOS, SLES |
| Containers | Docker |
| Data Formats | Syslog (TCP, UDP, AMQP, Kafka) GELF(TCP, UDP, AMQP, Kafka, HTTP) AWS - AWS Logs, FlowLogs, CloudTrail Beats/Logstash CEF (TCP, UDP, AMQP, Kafka) JSON Path from HTTP API Netflow (UDP) Plain/Raw Text (TCP, UDP, AMQP, Kafka) |
| Data Transport | Apache Kafka, RabbitMQ, REST API |
Advantages
- Quick Setup
- Authentication / Authorization included
- Parsing, Alerting, Archiving
Challenges
- Graph / Dashboard is basic
- Fewer plugins available
Elastic Stack
ELK is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a stash like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.
The Elastic Stack is the next evolution of the ELK Stack.
Standard Setup
Additional Popular Setup
| Details | |
|---|---|
| Licence Type | Open Source with Subscription option or using Elastic Cloud |
| Language | Java ( >= 8 ) |
| Components | Elasticsearch Kibana Logstash Security |
| OS Packages | Ubuntu, Debian, RHEL/CentOS, SLES, Windows, MacOS |
| Containers | Docker |
| Data Formats | Dozen of input Supported |
| Data Transport | REST API |
Advantages
- Starting version 7.11 Security (Authentication / Authorization) is free
- Parsing, Alerting, Archiving
- Graph / Dashboard is amazing
- Many plugins available
Challenges
- Medium Complex to setup
- Elastic Stack need big resource, like Elasticsearch it’s must have min 3 nodes.
- Logstash resource usage is heavy
Splunk
A scalable and reliable data platform for investigating, monitoring, analyzing and acting on your data.
| Details | |
|---|---|
| Licence Type | Free for 500MB data per day Splunk Cloud Splunk Enterprise start $2700/year Splunk Data Stream Processor |
| Language | Java, C++, Python |
| Components | Splunk Server Splunk Forwarder |
| OS Packages | All Linux, FreeBSD 11, Solaris 11, AIX, MacOS |
| Containers | Docker |
| Data Formats | Dozen of input Supported |
| Data Transport | Multiple |
Advantages
- Fantastic querying data feature
- Can ingest and chart ANYTHING thrown at it. It’s magical
- Machine learning plugins available
- Thousand plugins available
Challenges
- Expensive
- Searching billion data can be slow, must have some tunning