Popular tool for centralizing syslogs - Rsyslog, syslog-ng, logstash, fluentd, EventLog Analyzer

Why should you centralize syslogs

Syslogs are generated from Linux/Unix systems and other network devices from across your network. You have to monitor your syslogs continuously as they contain vital information that can help in identifying any malicious activity in your network. Searching through the logs and analyzing them can be done easily if the syslogs are aggregated in a central repository.

Below are certain tools that help in monitoring Syslogs efficiently

Rsyslog

Rsyslog is the abbreviation for "rocket fast system for log processing" and uses the standard BSD protocol. It supports logs from various log sources across your network.

It uses the TCP, RELP, SSH, and TLS protocols to ensure reliability and security during storage and transmission of the syslogs. Rsyslog daemons can interpret every log message and send alerts to IT admins through email. However, it can't classify, tag or correlate log messages to provide in-depth analysis.

It uses a mail module called ommail, which uses the SMTP protocol in direct mode only. It reports even minor defects like "disk failure on <hostname>" through mails which are sent in pre-configured time intervals. Ensure you specify the time interval carefully while configuring ommail as these mails could spam your inbox and you might accidentally miss an important alert mail.

Syslog-ng

Syslog-ng is a syslog management solution that supports universal log collection. It can effectively collect logs from network devices and send them to servers at local or remote destinations without deploying many agents in hosts. It can also collect Windows event logs using the Windows Event Collector (WEC) tool.

It is safe and reliable as it uses TCP, RLTP, SSL, and TLS for storage and transmission of log files. Syslog-ng can process, normalize and correlate the log data and forward them to platforms like Hadoop, MangoDB, and Elastic Search.

Log management solutions analyze and interpret the correlated logs to generate reports and alerts. Syslog-ng doesn't have a built-in log analysis functionality. It can only support a log management solution in performing faster analysis by providing structured data as input to it.

Logstash

Logstash, like syslog-ng is a log management solution that can collect, parse and filter log data from various sources like syslogs from devices, Apache Logs, Windows Event Logs, AWS platform logs and more. It can index and parse log data irrespective of format and complexity using built-in parsers and plug-in modules which are customizable.

Logstash can also find geo locations of various hosts using the IP addresses.The indexed and normalized logs can be sent to a search engine like Elastic Search to perform search on the logs using simple queries or a log management solution to analyze the logs and interpret them to identify anomalies and impending threats.

Fluentd

Fluentd is a log management solution, with similar functionalities of Logstash. It also collects, parses and filters log data from various sources. It offers unified logging mechanism by converting the log data into JSON format. The indexed and normalized log data are sent to destinations like a search tool (Elastic Search), log analytics tool (Nagios) or a storage utility (Amazon S3).

Fluentd supports numerous plug-ins which can allow you to customize the input sources of log data and their output destinations. However, this tool is dependent on a third-party analytics tool to analyze the logs and raise alerts for anomalies.

ManageEngine EventLog Analyzer

EventLog Analyzer is an all-in-one log management solution. It can collect logs from log sources across your network like switches, firewalls, routers, servers, databases, applications, cloud platforms, and devices. The logs are aggregated, parsed, indexed and normalized for easier interpretation. Search can be performed by simple text querying. EventLog Analyzer performs log correlation to identify logs generated from different network devices that correspond to the same event.

EventLog Analyzer can generate intuitive reports for all the activities taking place across your network. It uses User and Entity Behavior Analytics (UEBA) to predict abnormal user behavior in the network. It uses Advanced Threat Analytics to identify and raise alerts on malicious IPs. EventLog Analyzer also checks if your network adheres to all IT compliance mandates all the time.

It can send real-time alerts using built-in and customizable alert profiles via email/SMS to notify IT security admins about an impending attack. Click here to know more.