Log collection software

What is log collection, and why does it matter for security?

Log collection is the process of gathering log data (i.e., records of events, transactions, and system states) from every device, application, and service in an IT environment and moving it into a central platform where it can be searched, correlated, and retained.

It is crucial for security monitoring and threat detection because each event, whether a brute-force attempt against a domain controller, a suspicious query against a payment database, or a firewall rule change made at 2am, is recorded in a log file on a different machine. Without a collector pulling those events into one place, an analyst has to log on device by device to find them, manage logs, correlate them, and then flag the attack. By then, the damage is already done.

Log collection challenges

Log collection can be a challenging task because some systems such as firewalls, intrusion detection systems (IDSs), and intrusion prevention systems (IPSs) have many events per second that generate large amounts of log data. To collect and process log data in real time, regardless of the volume of log data and the number of devices in the network, organizations need a robust log collection mechanism.

Every network has different systems and environments that generate various log formats, such as event logs, syslog messages, and other application logs. The information gained from a router log differs from that gained from a firewall. Also, some logs cannot be collected directly, such as those in DMZs. All in all, log collectors need to be flexible enough to accommodate all network devices and applications.

ManageEngine EventLog Analyzer solves both problems (the volume and the format) by processing up to 25,000 logs per second across more than 750 supported sources from a single console.

Universal log collection and sources

EventLog Analyzer collects logs from over 750 log sources out of the box. The four categories below cover the vast majority of enterprise deployments. The coverage breadth is a differentiator because a single console replaces the four or five single-purpose tools most log management solutions end up stitching together.

Windows event logs

The solution collects security, system, and application logs from every Windows server and workstation in the network. More details on the setup, Windows Event Forwarding and Windows Event Collector architecture, and Event Viewer specifics are on the dedicated Windows event log collection page.

Syslog and network device logs

It collects syslog messages from routers, switches, firewalls (from Cisco, Palo Alto Networks, Fortinet, SonicWall, Sophos, Check Point, WatchGuard, Barracuda, H3C, Huawei, and more), IDSs and IPSs, VPN concentrators, and other network devices. EventLog Analyzer's built-in syslog server capabilities enable efficient management and monitoring of syslog events.

Application and database logs

The solution collects logs from IIS and Apache web servers, Microsoft SQL Server, Oracle Database, MySQL, DHCP servers, terminal servers, and print servers.

Cloud and virtualization logs

It collects AWS CloudTrail and EC2 instances (via APIs), VMware ESX and ESXi logs, and IBM AS/400 logs. On-premises and cloud logs land in the same index, so a hybrid workload can be searched with one query rather than two (Fig. 1).

Custom log collection

EventLog Analyzer supports custom log collection, meaning it can collect events from text files on both Windows and Linux computers. Some applications don't follow the standard logging services and record log information as text files instead. When these logs are collected, they are parsed into custom fields created for that particular log data.

Log collection methods

EventLog Analyzer can collect logs from multiple log sources. Windows devices don't require agents to collect logs, while syslog devices require them mostly for load balancing purposes. Thus, EventLog Analyzer is designed to support both agent-based and agentless collection mechanisms to cater to all devices and applications in the network (Fig. 2).

Below is a summary of every collection method the product supports and when each one applies.

• Agent-based collection: A lightweight service installed on the source host reads events locally, buffers them, and ships them to the server. For more details, refer to the agent-based log collection page.
• Agentless collection: The EventLog Analyzer server reaches out and pulls events from source machines using native protocols with no software on the endpoint.
• Syslog collection: The built-in syslog listener receives UDP and TCP syslog messages from routers, switches, firewalls, and Linux hosts on port 514 (configurable).
• SNMP trap collection: The SNMPTrapServer receives asynchronous trap messages from network devices that don't speak syslog natively.
• WMI-based collection: WMI queries pull events from remote Windows machines when direct forwarding is not viable.
• API-based collection: REST and cloud APIs bring in logs from AWS, cloud services, and SaaS applications that produce event streams instead of files.
• File importing: For offline forensic work or EVTXs, the log files can be uploaded directly to EventLog Analyzer. Refer to the log import page for more details.

Enterprise-scale log collection with a distributed architecture

A single-server deployment works fine for a hundred sources. Ten thousand sources spread across three continents is a different problem. EventLog Analyzer's distributed edition addresses this: Remote collector nodes are deployed at each site, gather logs locally, and forward compressed, encrypted streams to a central archive over SSH. The results are dramatically less WAN bandwidth used and no dependence on the central server being reachable from every endpoint.