Parsing Apache logs

Apache logs contain voluminous and critical data about web server activities. You can convert the unstructured Apache logs into a structured format through parsing, indexing and normalizing techniques. Parsing converts Apache logs into a structured format which makes interpretation and analysis of the log easier.

Parsing Apache logs using grep, awk and other commands.

Apache logs can be parsed using various commands that can be executed in your Command Line Interface (CLI) such as grep, awk, and more. This section discusses the different commands that can be used to parse and analyze Apache logs.

Grep command.

You can parse Apache logs using the grep command. This command can extract fields from log data by using the "exact match" criteria for regex and basic string queries.

Example: To extract and view the web activity logs of a particular IP address, say 192.168.10.10, in your access log file, you can run the following command in your

Terminal:

This command will return all the logs from the Apache access log file that have the same IP address as mentioned in the query.

Cat and cut commands.

The cat command can be used to concatenate all the log data in a file and display it a line by line format in the terminal or write them in a destination file. This command is used along with cut command to help you extract a particular field from a log by specifying the position of the field and the delimiters on each side of the field.

For instance, if a 'HTTP status codes' field is at the 8th position in your Apache access log, you can use the below command to extract all the HTTP status codes from the Apache access logs:

Note:

1. -d: denotes delimiter <space> enclosed within " ".
2. -f: denotes field position (8 in this case).

Awk command.

Awk is a terminal command that helps in searching Apache web server logs effectively. This command can search through the logs for a specific field using a perfect match query. The awk command can include logical operators, conditional statements, begin/end blocks and looping statements to enhance the log search.

Consider these Apache access logs,

1. 127.0.0.1 Jane [22/Feb/2020:11:22:30 -0700] "GET /natural-image.jpg HTTP/2" 404 479
2. 192.168.10.10 Maria [24/Feb/2020:01:12:20 -0700] "GET /amazon-forests-image.png HTTP/2" 200 1456

From these logs, if you want to extract the access logs that have a "404" HTTP status code, you can run the following awk command in your terminal,

Note: Here $5 denotes the fifth field to be checked.

Tail, sort and uniq commands.

1. Tail command is used to extract a specified number of logs from the bottom of a log file. For instance, to extract the last 100 lines from your access logs, use this command in your terminal:
   $ tail -100 /var/log/apache2/access.log
2. Sort command is used to sort the logs in a specified order after extracting them using grep, tail or awk commands. For instance, if you want to sort the 100 logs you extracted using the tail command from your log file, run the below command in your terminal:
   $ tail -100 /var/log/apache2/access.log | sort -n

   Note: -n is used to sort the data numerically, you can use any sorting notation depending on your requirement.

3. Uniq command is used to display only unique log data by eliminating redundant logs. For instance, if you want to remove the redundant log data from your result after extracting the logs from your Apache access log file, you can use this command in your CLI as given below:
   $ tail -100 /var/log/apache2/access.log | uniq -d
   Note: -d prints repeated lines only once. You can use any uniq notation based on your requirement.

Parsing Apache logs using a log management solution.

Parsing the Apache logs using the above commands can be very tedious and time-consuming. Using the above techniques, getting a holistic picture of an operational or security issue would be impossible. A better option to analyze Apache logs would be to use a log management solution. A log management solution can ease your work by automating the process of collecting, parsing, indexing, normalizing and analyzing the Apache logs.

EventLog Analyzer is a comprehensive log management solution that can analyze Apache access and error logs from all versions of your Apache servers, including Apache tomcat. It generates intuitive reports like Apache Server Trend, Top Users, Top URLs, Top Browsers, Top Visitors, Top Errors, Top Referrers and Apache Reports Overview based on users' web activities. Furthermore, this solution also alerts you in real-time via email and SMS in case of a threat. Click here to know more on EventLog Analyzer's Apache log management capabilities.