Extract New Fields to Parse and Index Logs


Network Administrators are always in need of more information and insights from their log data. There are times when an IT administrator identifies some log information which is useful and would like to have it indexed automatically as a new field. Having more fields being indexed makes your log data more useful, especially when conducting log forensic analyses and creating network security reports.

EventLog Analyzer allows administrators to extract or create custom/new fields from raw logs by using the interactive field extraction UI. This tool helps create regular expression (regex) patterns which EventLog Analyzer can identify, parse, and index as custom fields from all new logs it receives from the network.

 

 

 

Procedure to extract new fields

In the Search screen, use the log type filter and optionally the device filter and search query text box in order to return the logs which contain the new field you wish to extract. Click Go to obtain the results of the search. Hover the mouse over any log entry which contains the new field, and click on the wrench icon that appears.

 

Step 1: View the log type and its existing fields

Extract Field - Step1

  1. The log type is displayed. For non-default log types, the display name will be the name of the imported log file or device name/port.

  2. View the details of the existing fields and values for this log type with the help of the Show and Show All links.

 

Step 2: Select the required custom field value(s) and specify the details

Extract Field - Step2

 

  1. Select and click the field value in the message, to be extracted as a new field.

  2. Provide a name for this field. Specify the prefix and suffix to the field value.

  3. Click Create Pattern to generate a parser rule or regex (regular expression) pattern.

Note: The word(s)/character(s) preceding and succeeding the selected field value are called its prefix and suffix. These are used as identifiers to extract the fields exactly. The prefix and suffix can have a static value (exact match) or dynamic value (similar pattern) or they can be ignored.

 

Example of Static and Dynamic values

Message : Successful Network Logon: User Name: sylvian Domain: ADVENTNET Logon ID: (0x0,0x6D51131) Logon Type: 3 Logon Process: NtLmSsp Authentication Package: NTLM Workstation Name: SYLVIAN Logon GUID: - Caller User Name: - Caller Domain: - Caller Logon ID: - Caller Process ID: - Transited Services: - Source Network Address: 192.168.113.97 Source Port: 0 22873

In this log, a value such as Logon Type can be considered static as most logs refer to the type of logon as Logon Type.

A value like Source Network Address can be considered dynamic as it has several alternatives, such as Source IP Address or Source Address. All alternatives, however, follow a similar pattern.

If the prefix and suffix are defined as static values, the field extraction process becomes precise.

 

Step 3: Validate the pattern and save it to extract the new field

Extract Field - Step3

 

  1. A parser rule pattern is created using the field definition.

  2. Validate is used to test the generated pattern against the previous search results. You can manually check the suitability of the pattern by analyzing the Matched Log Messages and Unmatched Log Messages displayed.

Extract Field - Step4

 

  1. If the matched log messages are too few or do not meet your field definition requirement, the pattern needs to be modified. You can edit the generated pattern manually, if you are familiar with regular expressions (regex). Otherwise, simply click Choose another Pattern to choose from a list of patterns generated by the application.

Extract Field - Step5

 

You can add an open attribute to this new field. The procedure is shown in the below screen shot.

Add Open Attribute

 

  1. Optionally, you can define any existing field matching criteria to apply the pattern for this specific log type.

  2. Save the pattern to enable EventLog Analyzer to identify, parse, and index these custom fields from the new logs it receives from network systems and applications. While saving the pattern, a pop-up window asks whether you wish to save this pattern as an identity rule for this user defined log type.

Save the pattern as Identity Rule for this user defined Log Type

The custom pattern (parser rule) can be marked and unmarked as Identity rule for that user defined log type. Multiple custom patterns can be marked as Identity rules for a particular user defined log type.

The saved pattern will be listed in the Settings tab under Custom Pattern Settings section.

  1. Use the Ask Support button if you face any problem and get assistance from the EventLog Analyzer team.

Get download link