Import Log Files

EventLog Analyzer helps you collect and analyze logs from different sources such as servers, network devices, and applications. The solution provides actionable intelligence that helps security teams stay on top of security threats in the organization.

This solution provides you the capability to import log files. The supported log formats include Windows and syslog device formats, application log formats and archived files log formats.

Windows and syslog device log formats

• Windows Eventlog (EVTX format)
• IBM AS/400
• Linux/Unix Syslog format (RFC 5424 and 2131)

Note: To import .evt logs (Windows XP and Windows 2003), you will need to convert the .evt to .evtx using the command wevtutil export-log application.evt application.evtx /lf in your EventLog Analyzer installation.

Application log formats

• Apache access logs
• DHCP Linux logs
• DHCP Windows logs
• IBM Maximo logs
• IIS W3C FTP logs
• IIS W3C Web Server logs
• MSSQL Server logs
• MySQL logs
• PostgreSQL Logs

Archived files log formats

• Cisco archive files
• Syslog archive files
• Windows archive files

Steps to import log files

Navigate to the Import Configuration page using any one of the following menu options:

• +Add >Import Logs
• Settings > Configurations > Import Log Data
• Home > Applications > Imported Logs
• Home > Applications > Actions > +Import

Importing log files from different locations

EventLog Analyzer allows you to import:

• Log files from a local path.
• Log files from a shared path.
• Log files from a remote path.
• Log files from cloud storage.

Log file import from a local path

With this option, you can import log files from any device that has access to EventLog Analyzer.

Note: Log import cannot be scheduled to run at regular time intervals.

1. From the File Location option, select Local Path.
2. Click on Browse to select the necessary file(s) from your local device. Alternatively, you can enter the device name (or) IP address of the device (or) specify the full UNC path, then click on Open. The necessary file(s) is selected.
3. If you know the log format of the log file, select the log format from the given drop-down. If you do not know the log format select Automatically Identify.

   Note: You can view a preview of the selected log file and extract the desired fields, by clicking on the View symbol of the attached log file and enabling the pop-up window option in your browser.

4. Click on the + button and OK to select the device that the log file is associated to. You can also enter the name of the device or select the device from the pop-up that appears.
5. If you wish to store the imported logs for 2 days, enable the Store logs for a short term option. By default, the log storage time-period is 32 days.
6. Click on Import.

Log file import from a shared path or UNC path

The log file import via Universal Naming Convention (UNC) path allows you to access shared network folders on a local area network (LAN).

1. From the File Location option, select Shared Path.
2. Enter the device name or IP address from which you wish to upload the log file. Alternatively, you can click on Browse to select the Windows device.
3. Select the desired file from the device and click OK. The necessary file is selected.
4. If you know the log format of the log file, select the log format from the given drop-down. If you do not know the log format select Automatically Identify.

   Note: You can view a preview of the selected log file and extract the desired fields, by clicking on the View symbol of the attached log file and enabling the pop-up window option in your browser.

5. Click on the + button and OK to select the device that the log file is associated to. You can also enter the name of the device or select the device from the pop-up that appears.
6. If you wish to store the imported logs for 2 days, enable the Store logs for a short term option. By default, the log storage time-period is 32 days.
7. If you want to automate a log file import at regular time intervals, enable the Schedule log import option.
8. With the Schedule drop-down menu you can customize the time interval between each log file import.
9. Additionally, you can build a file name pattern for the imported log files, using the time format options given. The name of the file stored at the specified time is updated in accordance to the file name pattern.
10. Click on Import.

Log file import from a remote path

Importing log files from a remote path in EventLog Analyzer needs authentication. This authentication can be achieved in two ways:

1. Username and password
2. SSH private key file sharing (Specific to SFTP protocol)

Authentication type: Password

1. From the Browse Files option, select Remote Path.
2. Enter the device name from which you wish to import the log file. Alternatively, you can click on the + icon to browse and select the Windows device.
3. Choose the required protocol (Ethernet, FTP and SFTP) and enter the port number.
4. Select the desired file from the device and click OK.
5. Provide the Username of the remote device and select Authentication Type as Password.
6. Enter the password in the field below.
7. Browse and select the Associated Device.
8. The Store Logs for Short-term option will store the imported log data in EventLog Analyzer for a brief period of two days. If the option is left unchecked, the logs will be stored as per your database retention configuration.
9. You can choose to schedule the log import at specific time intervals.

Authentication type: SFTP-based SSH private key file sharing

1. Select Remote Path from the Browse Files options listed.
2. Enter the device name from which you wish to import the log file. Alternatively, you can click on the + icon to browse and select the Windows device.
3. Choose SFTP as the protocol and enter the port number. (Default port value is 22)
4. Provide the username and choose Key File as the Authentication Type.
   Note: EventLog Analyzer supports OpenSSH key file format only.
5. Browse and select the key file from the device. You can refer to this link to learn how to generate a key file with ssh-keygen, a standard component of Secure Shell protocol.
6. If the key file is passphrase protected, select the Use Passphrase checkbox and enter the phrase in the field below.
7. Browse and select the Associated Device.
8. The Store Logs for Short-term option will store the imported log data in EventLog Analyzer for a brief period of two days. If the option is left unchecked, the logs will be stored as per your database retention configuration.
9. If you would like to automate a log file import at regular time intervals, enable the Schedule Log Import option.
10. With the Schedule drop-down menu, you can customize the time interval between each log file import.
11. Additionally, you can build a Filename Pattern for the imported log files using the time format options given. The name of the file stored at the specified time will be updated in accordance to the file name pattern.
12. Click on Import to save the configuration.

Log file import from cloud storage

To import logs from AWS S3 buckets, you first need to create an IAM user with access to the S3 bucket(s). You can also grant users access to only specific S3 buckets by following the steps given in this link.

To configure AWS S3 buckets for importing logs,

• In the Cloud tab, click the link displayed to configure the AWS account.

• Enter the Display Name, Access Key, and Secret Key of the AWS account and click Add.

• Once the AWS account gets added, it will be displayed in the drop-down list available in the Cloud tab.
• From the drop-down list, select the AWS account and then the S3 bucket from which logs are to be imported.
• Click Import to initiate log importing.

MySQL Logs

EventLog Analyzer supports only error logs and general logs from MySQL. MySQL logon failures are taken into account from MySQL general query logs.

To enable logging in MySQL,

• Open the my.cnf file (in case of Linux) or my.ini file (in case of Windows) and add the below entries to the file.
• For error logs: log_error=<error-log-file-name>
• For general logs:
  • >= v5.1.29:
    general_log_file=<general-log-file-name>
    general_log=1 (or) ON
  • < v5.1.29:
    log=<log-file-name>
• Restart the MySQL instance for the changes to take effect.

To import MySQL logs in EventLog Analyzer,

• You can import MySQL log files from a local path, a shared path , or a remote path.
• To import MySQL log files, you need to manually choose the log format. Once you've selected the right file, select MySQL Logs from the Log Format drop-down list in the Selected File(s) section.
• Click Import to initiate the log importing process.

PostgreSQL Logs

Log format of PostgreSQL logs is determined by log_line_prefix parameter, set in postgresql.conf file.

The default format of PostgreSQL logs is '%m [%p] ' which logs a time stamp and the process ID.

This format is supported by default in EventLog Analyzer.

Importing additional fields in EventLog Analyzer

If the user wants to add additional fields, log_line_prefix parameter in the postgresql.conf file must be changed.

The log_line_prefix parameter must follow the format(key- value pair) given below in the postgresql.conf file.

log_line_prefix format:

log_line_prefix = 'time_stamp=%m or %t process_id=%p application_name=%a database_name=%d connection_from_with_port=%r connection_from=%h session_id=%c transaction_id=%x user_name=%u command_tag=%i sql_state_code=%e session_start_time=%s '

SAP ERP Audit Logs

To add the SAP ERP application for monitoring, the audit logs have to be enabled.

To enable the SAP ERP audit logs:

To the DEFAULT.PFL file in the location <SAP_installed path>\sys\profile, add

• rsau/enable = 1
• rsau/local/file = <log location>/audit_00

DB2 Audit Logs

Db2 database systems allow auditing at both the instance and database levels. The db2audit tool is used to configure the auditing process. The tool can also be used to archive and extract audit logs, from both instance and database levels. The audit facility can be configured by following these six steps.

1. Configuring db2audit data path, archive path, and scope.
2. Creating an audit policy for database auditing.
3. Assigning the audit policy to the database.
4. Archiving the active logs.
5. Extracting the archived logs.
6. Importing the logs to EventLog Analyzer.

EventLog Analyzer also supports diagnostic logs. Click here to learn how to generate the diagnostic logs report. 

1. Configuring db2audit data path, archive path, and scope

The configure parameter modifies the db2audit.cfg configuration file in the instance's security subdirectory. All updates to this file will occur even when the instance is stopped. Updates occurring when the instance is active will dynamically affect the auditing being done by the Db2 instance. To know more on all possible actions on the configuration file, refer source

• Open DB2 Command Line Processor with administrator privilege.
• Run the following command:

• Run the following command:

• Run the following command:

Now the logs will be generated for the DB2 instance in the given data path.

2. Creating an audit policy for database auditing

• Open DB2 Command Line Processor with administrator privilege.
• Run the following command to connect to a database:

• Run the following command to create an audit policy for the database:

• Run the following command to commit:

Now the audit policy has been created.

3. Assigning the audit policy to the database

• Open DB2 Command Line Processor with administrator privilege.
• Run the following command to assign a policy to the database:

• Run the following command to commit:

Now the created audit policy is assigned to the database.

4. Archiving the active logs

You can archive the active logs from both instance and database. The logs will be archived to the archive path that you configured in the first step.

• Open DB2 Command Line Processor with administrator privilege.
• Run the following command to archive the active database logs:

• Run the following command to archive active instance logs:

Now the logs will be archived to a new file with a timestamp appended to the filename. An example of the filename is given below.

• Instance Log file: db2audit.instance.log.0.20060418235612
• Database Log file: db2audit.db.your_database.log.0.20060418235612

Both files have to be extracted into a human-readable format to be imported into EventLog Analyzer.

5. Extracting the archived logs

• Open DB2 Command Line Processor with administrator privilege.
• Run the following command to extract the archived instance logs:

• Run the following command to extract archived database logs:

Both files will be extracted to the given archive path and can be imported into EventLog Analyzer.

6. Importing the logs to EventLog Analyzer

Now you will have to import the extracted database and instance log files into EventLog Analyzer. Here is a comprehensive guide on how to import log files in EventLog Analyzer.

Diagnostic Logs

EventLog Analyzer also provides a report for diagnostic logs. To generate the diagnostic logs report, follow the given steps.

• Run the following command to find the location of the diagnostic log file.

or or

• Navigate to the specified path and import the file named db2diag.txt to EventLog Analyzer. Here is a comprehensive guide on how to import log files in EventLog Analyzer.

Import Troubleshooting tips

If you are unable to import a log file, ensure the following:

1. The credentials used are valid and have the necessary permissions.
2. The device is reachable.
3. The specified file exists and is accessible.
4. The log file format selected from the drop-down matches the log format of the chosen file.

Field extraction from logs

1. You can create a custom field by clicking on the tools icon at the top right corner of your log message. Follow the steps given in this page to use custom patterns for logs.

a. Now custom fields are also displayed in the left pane.

b. Click on the Save button.

List of imported log files

You can view a list of all imported log files in your EventLog Analyzer installation. This is the default page that appears when the import log option is selected. This page provides details of the imported log file including, filename, device, monitoring interval, time taken to import the log file, log format, and size of the log file.

Apache Overview Dashboard: Parsing Additional fields by modifying the log format

The Combined Log Format is one of the log formats commonly used with Apache logs.

The Combined Log format is:

While importing the log files in the Combined log format, the log files will not include the values for the fields response time and bytes received.

The following widgets in the Apache Overview dashboard can display their values accurately only if the response time and bytes received fields are parsed.

1. Bytes Transferred
2. Top 20 Slowest URLs
3. Web Activity Trend
4. Top 10 Slowest Servers

In order to parse these additional fields, the log format has to be modified. The values for the additional fields can be obtained once the logs are configured with the parameters "%{ms}T" and "%I".

Eventlog Analyzer can parse the modified log format by default.

The modified log format containing the parameters for response time and bytes received is:

%{ms}T - time taken to serve the request (in milliseconds)
%I - bytes received, including headers

The modified log has 2 directives in addition to the commonly used Combined Log Format. These directives are present at the end of the format, therefore, the combined log format will continue to be parsed as it was parsed in the previous versions.

Procedure to change the Apache log format

1. Define a new log format and assign a label to it.
   Copy to Clipboard

   LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %{ms}T %I" modified

2. The label can be used to reference the new format string as the customLog directive.
   Copy to Clipboard

   CustomLog logs/access.log modified

3. The new format will go into effect when the webserver is restarted.

   After the log files have been imported, the updated Apache Overview dashboard has been displayed below: