Log and NetFlow collection and processing.

Log data, generated by every device and application in the network, along with NetFlow data, which monitors network traffic, both provide insights into network activities, making them the main sources of input to security information and event management (SIEM) solutions.

A SIEM solution collects, stores, and analyzes this information to gain deeper insights into network behavior, detect threats, and proactively mitigate attacks. This article will discuss some techniques used for collecting and processing this information.

How are logs collected?

Logs are collected from all devices such as databases, routers, firewalls, servers, IDS/IPS devices, domain controllers, workstations, and applications.

Log collection can be done two ways:

  • Agentless log collection
  • Agentless log collection is the predominant method SIEM solutions use to collect logs. In this method, the log data generated by the devices is automatically sent to a SIEM server securely. There is no need for an additional agent to collect the logs, which reduces the load on the devices.

  • Agent-based log collection
  • Agent-based log collection demands an agent be deployed on every device that can generate logs. This method can help filter out logs while collecting them, based on defined parameters. Agents also take up less bandwidth and resources, and help provide filtered and structured log data. This method is employed when the devices are in a secure zone where communication is restricted, and it is difficult to send logs to a SIEM server.

How is NetFlow data collected?

The NetFlow collection corresponds to information about IP traffic, mainly:

  • Source IP address
  • Destination IP address
  • Ports accessed
  • Services carried out
  • And more.

Gathering NetFlow data is done with a NetFlow collector, which also records timestamps, the packets requested, entry and exit interfaces of the IP traffic, and more. The process of NetFlow collection includes gathering the NetFlow data and analyzing the data for bandwidth speed, resource utilization, transmission, and reception in a network by NetFlow collectors. The main functions of NetFlow collectors include collecting flow data transmitted using the User Datagram Protocol (UDP) from NetFlow-enabled devices, and filtering the collected data to reduce its volume. The Simple Network Management Protocol (SNMP) collects information about traffic at every observation point in a network. This method of data collection is enabled on all devices that have an ethernet port, and the data is monitored and analyzed to detect anomalies and prevent threats.

Learn more

Log processing

The different techniques used for log processing are listed below.

Log parsing

A parser can take unstructured raw log data and format it by grouping similar data under relevant attributes. Parsing makes the retrieval and searching of logs easier. Every SIEM solution includes multiple parsers to process the collected log data.

Log normalization

Normalization is the process of mapping only the necessary log data under relevant attributes, which can be configured by the IT security admin. To monitor the important activities in a network, logs have to be normalized. Log normalization can help distingush between regular and irregular activities in a network.

Log indexing

Normalized log data is separated and stored in files that contain indexed log information; admins can apply queries to the indexed data to accelerate the searching process. SIEM solutions can be configured by the network admin to record the data under a particular index for easier retrieval and interpretation.

Log analysis

Correlating log data helps in identifying if the different log sources correspond to one particular event that threatens network security. Forensic reports help verify where the network was compromised and how an attack was carried out. Log analysis plays a vital role in understanding user behavior and detecting threats, and also helps in preventing an attack before it occurs.


Chapter 2

Different functions of SIEM

Learn about the different capabilities of an ideal SIEM solution.


Chapter 3

Component of SIEM Architecture


Chapter 4

Log Management

Learn about log management and why it is necessary.


Chapter 5

Incident Management

Learn about security incidents and how they are handled.


Chapter 6

Threat intelligence

Learn about security audits, real-time monitoring, and correlation and how they are useful to mitigate cyberthreats.


Chapter 7

Cloud security

Learn why it is important to secure data that is stored online on cloud computing platforms.


Chapter 8

User Entity and Behavior Analytics

Learn why UEBA is critical to maximize cybersecurity.


Chapter 9

Data protection

Learn why it is important to adhere to compliance regulations.