Syslog forwarding: 10 best practices to forward logs securely

What is syslog forwarding?

Syslog forwarding is the mechanism of transmitting log messages from system software, applications, and network devices from the source machine to a centralized syslog server or log collector. It uses the syslog protocol to standardize the messages generated by various sources, allowing efficient parsing and normalization at the server level to monitor the activities on the source machines.

Significance of syslog forwarding

Syslog forwarding is considered a crucial process in security monitoring for the following reasons:

Log security and integrity

In the event of a local system compromise, audit logs are the primary sources of threat detection, leading attackers to often tamper with these logs to evade detection. By forwarding these critical logs to a separate server, the logs are preserved in their original formats, ensuring their integrity for future forensic analysis.

Reliable data storage and backup

Log forwarding allows critical audit logs to be stored in a secure dedicated server over a period of time to ensure compliance. It also facilitates reliable backup storage, preventing data loss in case of a host system failure, which is crucial for centralized security monitoring and troubleshooting.

System performance optimization

Forwarding logs reduces the overhead on the local system's CPU and disk, enabling efficient log management and storage optimization. This reduces the strain on the local system, thus improving its performance and operational efficiency.

How does a syslog forwarder work?

A log forwarder typically runs as a lightweight service or agent on the host system, which might be a server, a workstation, or a network device, and performs the following functions:

Log collection

The forwarder's primary role is to aggregate log data from various sources. It handles various log formats, including unstructured plaintext and structured formats like JSON.

• Log reception: The forwarder is configured to monitor specific log files (e.g., /var/log/messages or access.log), read events from operating system channels (like the Windows Event Log or Linux journald), or receive logs directly from applications via an API.
• Log listening: The forwarder constantly tails these sources in real time. As soon as a new log entry is written, the forwarder captures it, preventing gaps in the data stream.

Log processing

After collection, the forwarder performs essential processing tasks based on its configuration.

• Filtering: It applies rules to determine which logs should be forwarded. For example, it might be configured to drop low-priority debug messages and only forward high-priority alerts and errors to the server, optimizing the bandwidth of the server.
• Parsing: It parses unstructured logs and transforms them into a standardized, structured format like JSON or a CEF message. This streamlines data formats, making it easier for the central receiver to index and analyze the data.
• Enrichment: The forwarder may enrich the log data by adding metadata such as the host's IP address, the hostname, or a custom application tag for enhanced security analytics.

Log transmission

This is the final step, where the forwarder ensures that the logs reach the central collector without any compromise.

• Buffering and resiliency: To prevent data loss during network outages or when the central server is temporarily unavailable, the forwarder uses an internal buffer in the memory or on disk. Once connectivity is restored, it automatically resumes forwarding the buffered logs, preventing data loss in transit.
• Protocol selection: It encapsulates the log message using the configured network protocol, such as UDP, TCP, or TLS, and securely sends it to the destination IP address and port.
• Security: If configured for secure transmission over TLS, the forwarder handles the encryption and authentication process to protect sensitive log data while it is transferred over the network.

Types of syslog forwarders

Syslog forwarders are classified primarily by their function within a logging environment as follows:

1. Syslog agents

These agents or devices originate and aggregate syslog messages from their own local source (e.g., operating system, application, or network device) and send them directly to a central syslog collector. They typically offer minimal filtering or processing before transmission.

Traditional logging daemons like syslogd or systemd-journald on a Linux host often act as syslog agents.

2. Dedicated relays

Relays are software components or machines that act as intermediaries or log brokers. They continuously receive logs from multiple sources, and then process and forward them to final destinations. They can filter, process, and reformat logs and allow for load balancing by sending the same logs to multiple destinations for redundancy or log analysis.

Advanced syslog daemons like rsyslog or syslog-ng are specifically configured for relaying.

3. Syslog servers

These are specialized applications or syslog monitoring tools that receive logs in various formats, such as standard syslog or CEF, for standardized processing. Their primary role is to parse, normalize, and filter the collected data for effective ingestion into a specific log analysis tool or a third-party SIEM solution.

Dedicated log management solutions that include collection, processing, and forwarding capabilities, such as ManageEngine EventLog Analyzer, not only act as syslog servers but also forwarders.

Syslog forwarding best practices

Here are 10 best practices for streamlined and seamless syslog forwarding.

1. Standardize the syslog format

Always configure your log sources and forwarders to use the modern RFC 5424 standard or a structured format like JSON. This format provides consistent structure, shows high-precision timestamps, and ensures reliable parsing.

2. Use suitable protocols depending on the data type

To guarantee log delivery and protect sensitive data, always use TCP instead of UDP. Furthermore, implement TLS/SSL encryption for all log traffic, especially across untrusted networks, to ensure integrity and confidentiality. However, UDP can be used for high-volume, non-critical data that requires faster speeds, and similarly, RELP can be employed for highly assured delivery.

3. Implement buffering and queuing at the source

Configure syslog agents such as rsyslog or syslog-ng with a disk-assisted or in-memory queue. This is crucial for seamless operation if the central collector is down or unreachable. It ensures the logs are safely buffered locally and automatically forwarded when the connection is restored, preventing data loss.

4. Filter and prioritize logs at the source

Implement filtering at the source agent or the first relay to discard low-value, high-volume messages (e.g., debug or trace logs) that are not needed for security or operational analysis. This reduces network load and ingestion costs.

5. Deploy syslog relays

In large or distributed environments, utilize intermediate syslog relays which aggregate logs and then reliably forward it to the central collector. This reduces the number of connections the central collector must manage, facilitating scalability.

6. Ensure accurate time synchronization

All log sources, relays, and collectors must use the Network Time Protocol (NTP) to synchronize their clocks. Without consistent timestamps, log correlation during an incident becomes impossible,so it's crucial to synchronize the timezones.

7. Configure redundancy and failover

Configure all agents and relays to forward logs to a primary and at least one secondary central collector that acts as a backup. Implementing this failover mechanism ensures continuous log collection even when the main collector experiences an outage.

8. Monitor syslog flow and collector health

Actively monitor the health and performance of both the central collector (CPU, disk I/O, and ingestion rate) and the forwarding agents (queue size and last successful transmission). Set up alerts for sudden drops in log volume or backlog growth to proactively identify broken forwarders.

9. Dedicate separate network and storage resources

Dedicate specific network interfaces for log traffic to isolate high-volume data from critical application traffic. On the central collector, utilize fast storage SSDs and ensure that the disk is capable of handling the high volume of incoming logs, and set it apart from the disks used for long-term storage or querying.

10. Maintain a consistent host identity

Log processing tools rely on an accurate hostname or IP address to verify the sender. Ensure that the forwarding path is configured to preserve the original sender device's identity and not the relay's identity. This guarantees accurate logging and reporting at the central destination.

What's next?

Frequently asked questions