What is syslog-ng?

Syslog-ng: An overview

Syslog-ng, also known as syslog-new generation. is a robust, open-source logging utility. It functions as a flexible and powerful daemon that collects, parses, transforms, and forwards log messages from various sources to multiple destinations. It is well known for its capabilities that go beyond traditional syslog tools, such as support for TCP, TLS encryption, and content-based filters.

Evolution of syslog-ng

The history of system logging began with the basic syslog protocol and the syslogd daemon. Syslogd, the traditional daemon, primarily used UDP for log transmission. This created risks of the log message being lost during the transmission. Further, its complex and rigid configuration limits its message processing capabilities.

Syslog-ng, the new-generation daemon, was built on the syslogd framework with enhancements that focus on reliable log transport. This was made possible by introducing TCP and enhanced filtering capabilities, fundamentally changing the way syslog collection worked. Syslog-ng includes advanced content-based filtering; real-time log correlation; and support for various syslog formats, such as JSON and Journald. It also comes with capabilities for logging in to databases, ensuring high portability.

One of the contemporaries of syslog-ng is rsyslog, an open-source, high-performance logging daemon. It supports the unique Reliable Event Logging Protocol (RELP) in addition to TCP/TLS for highly reliability and log forwarding without data loss. It is also known for its rocket-fast speed, modular architecture, and powerful filtering and routing capabilities.

Significance of syslog-ng

The operational efficiency of syslog-ng is rooted in its highly modular architecture and advanced message processing pipeline.

Reliable transmission: Unlike the original UDP-only syslogd, syslog-ng introduced TCP for reliable, connection-oriented log transmission, ensuring loss-free transmission.

Enhanced security: Syslog-ng incorporates TLS/SSL encryption, enabling secure transmission of sensitive log data across untrusted networks to a central log server.

Advanced log parsing: Syslog-ng can not only collect logs but also actively process them. It supports log parsing, which involves breaking down unstructured log messages into structured name-value pairs, crucial for data normalization.

Syslogd vs. syslog-ng vs. rsyslog

The following table differentiates syslogd, sylog-ng, and rsyslog based on some of the important criteria.

How syslog-ng works

The operation of syslog-ng is based on a structured, modular pipeline that processes log messages in a defined sequence.

Step 1: Message ingestion at source

The process begins when syslog-ng reads log messages from a configured source. Source drivers are modules responsible for reading specific message types. Common drivers include:

system(): For reading native operating system logs.

network(): For receiving remote messages via UDP/TCP.

Step 2: Parsing and classification

Once the log data is received from a source, the message enters the processing pipeline. The message is sent to a parser, which segments the log message based on timestamp, hostname, facility, severity, and the message body. For complex, unstructured log messages, advanced parsers extract critical fields and convert the message into a structured format.

Step 3: Filtering and rewriting

Before forwarding, the message is checked against configured filters and rules. Filters are logical expressions that filter logs based on facility, severity, program, hostname, or content of the message body. It determines if a message is relevant for a specific log path.

After filtering, if necessary, the messages can be modified using the Rewrite rule option. This option enables adding a tag, sanitizing data, or changing the hostname field to ensure better integration with the destination system.

Step 4: Routing

The process of routing starts with the log statement, which acts as the connector that defines the entire log path. It links the source to the destination after applying the filters and necessary rules. A single message can be routed through multiple, independent log paths. For example, a message could be routed to an encrypted file and forwarded over the network to a SIEM system, enabling flexible routing.

Step 5: Message output at the destination

The final processed message is delivered to the configured destination drivers, which are basically the modules that handle the output message. Common destination drivers include:

file(): For writing to a local file.

network(): For forwarding logs to a remote log server.

sql(): For directly storing logs on databases like MySQL and PostgreSQL.

Benefits of syslog-ng

Syslog-ng provides significant advantages over traditional logging methods, making it the preferred choice for log management in complex network environments.

Optimized performance and scalability

Syslog-ng is designed with a multi-threaded architecture, allowing it to handle massive volumes of log data concurrently and efficiently. This prevents bottlenecks and ensures the logging system can scale horizontally to accommodate growth in IT infrastructure.

Guaranteed log delivery

By supporting TCP/TLS and implementing disk buffering, the daemon ensures that log messages are safely delivered. If a destination is temporarily unavailable, the messages are stored locally and forwarded later, preventing critical data loss.

Enhanced security and compliance

The use of TLS encryption secures log messages in transit, protecting sensitive data. Centralized, tamper-proof logging is a fundamental requirement for many compliance standards (like the PCI DSS and HIPAA), which syslog-ng facilitates by providing a single, auditable log repository.

Data normalization and enrichment

Its advanced parsing and rewriting capabilities allow for the normalization of log data from different vendors and applications into a consistent, structured format. This data enrichment makes the logs more valuable and easier for analysis.

Flexible integration with big data ecosystems

Syslog-ng includes a wide array of destination drivers that enable direct integration with modern big data stores and message queues (e.g., Elastic Stack, Splunk, and Kafka). This capability positions it as an efficient log transport pipeline for advanced SIEM systems.

Syslog-ng and ManageEngine EventLog Analyzer

While syslog-ng can function as a full-fledged syslog server, in large enterprise networks, it often acts as a secure and reliable data pipeline that feeds critical syslog data to security monitoring and analysis tools like ManageEngine EventLog Analyzer.

In such enterprise setups, syslog-ng plays the role of a log forwarder that collects, filters, and transmits logs from various syslog sources. EventLog Analyzer, in turn, acts as the syslog server that indexes and analyzes the data for threat detection, investigation, and response, transforming raw syslog data into security intelligence in a centralized platform.

This kind of modern log infrastructure allows enterprises to leverage the powerful synergy of a secure log transporter like syslog-ng and a sophisticated log management tool like EventLog Analyzer. While syslog-ng handles reliable delivery, EventLog Analyzer adds:

• Correlation: Connects related events across sources to surface attack patterns.
• Threat detection: Applies over 700 prebuilt rules mapped to the MITRE ATT&CK framework.
• Compliance reporting: Auto-generates audit reports for the PCI DSS, HIPAA, SOX, and more.
• Forensic search: Queries months of indexed data in seconds.

This architecture addresses separates concerns where syslog-ng ensures no log is lost in transit while EventLog Analyzer ensures no threat is lost in the noise.

Frequently asked questions