Search Engine Management

Elasticsearch is a distributed, RESTful search and analytics engine. When configured in Log360 it distributes data between the nodes that are added thereby optimizing disk space and also improving the performance of Log360.

• Search engine Settings
• Actions on nodes
• Prerequisites
• Setting up Elasticsearch
• Configuring Elasticsearch in Log360

Search engine Settings

How to disable or enable ES auto restart

• If Auto Restart is enabled, starting Log360 will restart the child nodes and allow the child nodes to join Log360's cluster.
• Auto Restart can only be used when EventLog Analyzer is integrated with Log360. Also, the feature is enabled by default and it can be turned off if required.

Steps to disable/enable auto restart:

1. Open Admin → Search Engine Management → Settings

   

   

2. Disable Auto restart by disabling Enable auto-restart for the configured product's node option in Settings and click Save.

   

3. Once you click Save, auto restart is disabled. You can follow the same steps to enable auto restart.

   

Actions on nodes

• Adding a node: Helps in the distibution of log storage as data will be split and stored between the nodes.
• Starting a node: The Elasticsearch service is started in the added node and the node then connects to the Log360 server.
• Stopping a node: The Elasticsearch service running in the machine is stopped and data present in the node will not be accessible when the node isn't connected.
• Deleting a node: Data is removed from the node and the node is deleted.

Prerequisites

1. Increase file descriptors

Make sure to increase the limit on the number of open files descriptors for the user running Elasticsearch to 65,536 or higher. For the .zip and .tar.gz packages, set ulimit -n 65536 as root before starting Elasticsearch, or set nofile to 65536 in /etc/security/limits.conf.

Note: This is applicable only for Linux and macOS.

2. Ensure sufficient virtual memory

Elasticsearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions.

You can increase the limits by running the following command as root in Linux: sysctl -w vm.max_map_count=262144

3. Disable swapping

Usually Elasticsearch is the only service running on a box, and its memory usage is controlled by the JVM options. There should be no need to have swap enabled.

On Linux systems, you can disable swap temporarily by running: sudo swapoff -a

On Windows, the equivalent can be achieved by disabling the paging file entirely by going to System Properties > Advanced > Performance > Advanced > Virtual memory.

4. Ensure sufficient threads

Elasticsearch uses many thread pools for different types of operations. It is important that it can create new threads whenever needed. Make sure that the number of threads that the Elasticsearch user can create is at least 4096.

This can be done by setting ulimit -u 4096 as root before starting Elasticsearch, or by setting nprocto 4096 in /etc/security/limits.conf.

5. JVM DNS cache settings

Elasticsearch runs with a security manager in place. With a security manager in place, the JVM defaults to caching positive host name resolutions indefinitely. If your Elasticsearch nodes rely on DNS in an environment where DNS resolutions vary with time, then you might want to modify the default JVM behavior.  This can be modified by adding networkaddress.cache.ttl=<timeout> to your Java security policy.

6. Port availability

Ensure that port 9322 is available on the machine that will run Elasticsearch.

7. Sharing of <Installation Dir>/EventLog Analyzer/ES/repo

Ensure that the folder <Installation Dir>/EventLog Analyzer/ES/repo is shared with the service account of the Log360 server. This folder will be used to create snapshot from Elasticserch to save archives. If the Log360 server is not in AD, it will be an open share or else make sure that the user has the permission to share the folder and follow the steps below.

1. Share the folder <Installation Dir>/EventLog Analyzer/ES/repo manually with the Log360 server.
2. Copy the shared path of <Installation Dir>/EventLog Analyzer/ES/repo directory.
3. Navigate to <Installation Dir>/EventLog Analyzer/ES/config/dae.properties file and specify the copied path as the value for node.repo.sharedlocation.
4. Restart the EventLog Analyzer server.

Setting up Elasticsearch

By default, uses self-signed certificates Elasticsearch security i.e authentication and encryption. If you want to use your own certificates for security, follow the steps below.

• First make sure you have a client, node, and root certificate in the PEM format.
• Rename the certificates and their corresponding keys as follows.
  • Client certificate to client.pem and its key to client.key
  • Node certificate to localnode.pem and its key to localnode.key
  • Root certificate to root_ca.pem and its key to root_ca.key
• Now, go to /ES/config and open the dae.properties file.
• Change the value of the parameter use_custom_certificates to true.
• In /ES/config/certificates, check if the following files exist. If they do exist, delete them.
• client.key
• client.pem
• localnode.key
• localnode.pem
• root_ca.key
• root_ca.pem
• Then, copy your certificates to <Log360_Home>/ES/config/certificates
• Now, go to <Log360_Home>/ES/bin and run the verifyCertificates.bat file.
• If you receive a message saying Certificate Validation Done, start the server. If you do not get the message, contact support at log360-support@manageengine.com

Setting up certificates for existing nodes

Follow the steps below to replace the certificates in the existing nodes:

• Go to the machine and then stop the elasticsarch service by opening the taskmanager>services.
• Move the certificates to <INSTALLAITON DIR>\ES\config\certificates
• Navigate to <INSTALLAITON DIR>\ES\config, open the elasticsearch.yml file and replace the following line with the respective details in both the nodes.dn and admin_dn CN=*.node,OU=none,O=none,L=none,ST=US,C=US
• Restart the service.

Configuring Elasticsearch in Log360

To configure Elasticsearch in Log360, follow the steps mentioned below.

1. Login to Log360.
2. Navigate to Admin > Administration > Search Engine Management.
3. Click on Add Server.
4. In the Add Server drop box, enter the server details and the path to installation directory along with TCP port (optional).
5. Click Save.