High Availability (HA) Configuration (Professional Edition - MySQL)

Before configuring the HA setup, please ensure that you have read this document completely.

This document contains the following sections;

  1. How does the HA Set up Work?

  2. Prerequisites

  3. Steps to Configure High Availability

    1. Backup

    2. Restore

    3. Installing MySQL as Windows Service

    4. Steps to be Performed in Primary server

    5. Steps to be Performed in Standby server

    6. Configuring Notifications when Standby Server Takes Over the Role of Primary

  1. Additional Configurations

  2. FAQs

1. How does the HA Set up Work?

A server's availability is measured by the percentage of time, it is providing service, in the total time since it is deployed. In general, the servers are expected to be available for 24 hours a day, 7 days a week, and 365 days a year. However, a part or all of the server may be down, during a planned or unplanned downtime. Now, here comes the need for the High Availability, which provides fault tolerance and improved scalability.

IT360 supports the HA environment, with the Primary and Standby servers installed on different machines.

The below points explain the working of the HA setup, in detail;

Top

2. Prerequisites

  1. Network connectivity should be reliable between the Primary and Standby servers.
  1. The MySQL database should run on the same server, where IT360 is running.

  2. Both the Primary and the Standby servers should have the same IT360 Build version.

  3. The Primary and Standby servers should be in the same time-zone and time.

  4. If license is applied in the Primary Server, the same should to be applied in the Standby Server also. Click here for the steps to apply the license in both the servers.

  5. Before configuring the HA setup, take a backup of the IT360 database.

  6. Receiving Netflows in HA Enabled IT360 Setup:

    1. Exporting flows to both the Servers: Configure both the Primary and the Standby servers' ipaddresses in the router, to export the flows. In this case, the Standby server will be in hot-standby mode and will not be providing any service, until it takes over the role of Primary.

    Note: This process consumes additional bandwidth, since the flow is exported to the Standby server also.
    1. Exporting flows only to the current Primary server: In order to avoid the additional bandwidth utilization, manual intervention is made to change the Primary server`s ipaddress in the routers, when the Standby server takes over the role of the Primary. This in turn results in export of the flows, only to the current Primary server.

  1. Receiving Traps in HA Enabled IT360 Setup:

    1. Exporting traps to both the Servers: Configure both the Primary and the Standby servers' ipaddresses in the router, to export the traps. In this case, the Standby server will be in hot-standby mode and will not be providing any service, until it takes over the role of Primary.

    Note: This process consumes additional bandwidth, since the trap is exported to the Standby server also.
    1. Exporting traps only to the current Primary server: In order to avoid the additional bandwidth utilization, manual intervention is made to change the Primary server`s ipaddress in the routers, when the Standby server takes over the role of the Primary. This in turn results in export of the traps, only to the current Primary server.

Top

3. Steps to Configure High Availability

In order to configure HA, first you need to backup the data from the Primary server and restore the same in the Standby server. This is to make sure that the data in both the servers are in sync. Next is to install MySQL as Windows service. After this, its is required to activate MySQL data replication between the Primary and Standby databases. The replication process enables the data from one MySQL database server (the master) to be replicated to one or more MySQL database servers (the slaves). The steps required for data replication are listed below for the Primary and the Standby servers, separately.

3.1 Backup

  1. Stop the IT360 service in the Primary server, to take a backup.
  1. Ensure that the processes mysqld-nt.exe, java.exe and javaw.exe, related to the IT360's installation directory are not runnning in the Task Manager.

  2. Execute the FailoverBackup_Mysql.bat script, available under the <PrimaryIT360_Home>/bin/ directory, to start the data backup.

  3. Once the backup is over, a directory backup is created under the <PrimaryIT360_Home> directory, and the backup file IT360backup_DateTimeStamp.bak is placed in this directory.

  4. Copy the backup file IT360backup_DateTimeStamp.bak to the Standby server (in a directory, e.g. C:\IT360backup_DateTimeStamp.bak). This will be useful while executing the FailoverRestore_Mysql.bat script during restoration.

3.2 Restore

  1. You can restore the data over a Standby installation of IT360.

  2. For this, stop the IT360 service in the Standby server.

  3. Ensure that the processes mysqld-nt.exe, java.exe and javaw.exe, related to the IT360's installation directory are not runnning in the Task Manager.

  4. Execute the FailoverRestore_Mysql.bat script, available under <StandbyIT360_Home>/bin/ directory, with the backup file (this is the file copied from the Primary to the Standby Central server - refer here) name as argument. (Provide the absolute path of the backup file)

Example: C:\ManageEngine\IT360\bin>FailoverRestore_Mysql.bat ,C:\IT360backup_DateTimeStamp.bak.

Top

<IT360_HOME> stands for the IT360 installation directory.  For e.g., if IT360 is installed in D:\ManageEngine, then <IT360_HOME> refers to D:\ManageEngine\IT360.

3.3 Installing MySQL as Windows Service

Note: This process of Installing MySQL as Windows Service has to be carried out in both the Primary and the Standby servers.
  1. Note down the mysql port number from <IT360_HOME>/applications/conf/AMServer.properties, against the property am.mysql.portUpdate that MySQL Port number in the <IT360_HOME>/mysql/my-defaults.ini file, as shown below;

    • Include port=<value from am.mysql.port> (e.g., port=32266) as a first entry under [mysqld] section in <IT360_HOME>/mysql/my-defaults.ini file.

  2. Go to <IT360_HOME>/mysql/bin directory.
  1. Execute the below mentioned command to install MySQL as a service. (Note: While installing the MySQL service, please use the absolute path of mysqld-nt.exe as mentioned below to install the MySQL service)

Top

3.4 Steps to be Performed in Primary server

Follow the below instructions;

  1. Ensure that the Primary IT360 services are stopped and the mysql process is not runnning, in the Task Manager.

  2. Change DB_HOME in ServiceDesk/startDB.bat file, by following the below steps;

  1. Execute the following batch files. Here, the scripts have to be executed with Administrator privilege. For e.g. on Windows 2008 Server, right click on the Command Prompt icon and Select the "Run as Administrator" option'. Now, follow the below steps;

    1. Execute the script FailoverSetup.bat, under <IT360_HOME>/bin.
  1. The script will prompt you to configure the database details.
  1. Refer the below screenshot for the complete steps:
The Secondary Server / Database mentioned in the below wizard is the Standby Server / Database.

Note: If the User Name and Port differ from the default ones, then configure their values, by referring the <IT360_HOME>/applications/working/conf/database_params.conf file, which is available in the Standby server.
  1. Click here to receive notifications, when the Standby server takes over as Primary.

Top

3.5 Steps to be Performed in Standby server

Follow the below instructions;

  1. Ensure that the IT360 services are stopped and the mysql process is not runnning, in the Task Manager.

  2. Change DB_HOME in ServiceDesk/startDB.bat file, by following the below steps;

  1. Execute the following batch files. Here, the scripts have to be executed with Administrator privilege. For e.g. on Windows 2008 Server, right click on the Command Prompt icon and Select the "Run as Administrator" option'. Now, follow the below steps;

    1. Execute the script FailoverSetup.bat under <IT360_HOME>/bin.

    2. The script will prompt you to configure the database details.

    3. Refer the below screenshot for the complete steps:
The Secondary Server / Database mentioned in the below wizard is the Standby Server / Database.

Note: If the User Name and Port differ from the default ones, then configure their values, by referring the <IT360_HOME>/applications/working/conf/database_params.conf file, which is available in the Standby server.
  1. Stop user-started MySQLs: Stop the MySQLs running on both the Primary and Standby servers, by following the below steps;
  1. Start MySQL as Windows Service: Follow the below steps;

    1. Go to services.msc.
    2. Start the MySQLs (MySQLIT360), installed as Services.
  1. Start the Primary server. Make sure that the Primary server startup process is complete and then start the Standby Server.

  2. Click here to receive notifications, when the Standby server takes over as Primary.

Top

3.6 Configuring Notifications when Standby Server Takes Over the Role of Primary

You will receive notifications, in the form of emails, when the Standby server takes over the functions that were being performed by the Primary. For this, you need to manually add the email id (s) in the file: IT360_HOME>\applications\working\conf\FailOver.xml.

Sample entries are as follows;

<FAILOVER>

<Primary HEART_BEAT_INTERVAL="60" />

        <Standby FAIL_OVER_INTERVAL="60" RETRY_COUNT="1">

   <BACKUP ENABLED="TRUE" BACKUP_INTERVAL="600"/>

            <SEND_EMAIL

                SMTP_SERVER="smtp"

                TO_ADDRESS="xyz@zohocorp.com"

                FROM_ADDRESS="admin@zohocorp.com"

                SUBJECT="Primary Server Failed"

                BODY="Primary Server is failed and taken over by the Standby Server"/>

            </Standby>

   </FAILOVER>

Note: Stop and start the ManageEngine IT360 service, for the above changes to take effect.

The optimal value for the RETRY_COUNT is 10.

Top

4. Additional Configurations

  1. If  MIBS are added in the Primary server, the same MIBS should be added in the Standby server also, provided, the Standby has taken over the role of the Primary. (Click here to know about MIB upload procedure for Networks and click here to know about the MIB upload procedure for Applications & Servers)

  2. Once the Standby server has taken over the role of the Primary, follow the steps below, to copy the attachment files to the new Primary server from the old one.

Copying SDP attachments files to the new Primary Server

  1. Go to the Primary server`s installation directory.

  2. Execute backUpData.bat script, available under <PrimaryServerIT360_Home>/Servicedesk/bin directory, with the argument: backUpData.bat --confiles.

  3. Once the backup is over, a directory confbackup is created (if it is not already available), under <PrimaryServerIT360_Home>/servicedesk directory and the backup file backup_7513_conbackup_DateTimeStamp.data is placed in this directory.  

  4. Copy that backup_7513_conbackup_DateTimeStamp.data file into the <StandbyServerIT360HOME>/servicedesk directory.

  5. Rename that .data extension to .zip (e.g., if the file name is backup_7513_conbackup_DateTimeStamp.data, then rename it to backup_7513_conbackup_DateTimeStamp.zip)

  6. Extract the backup_7513_conbackup_DateTimeStamp.zip file, in the <StandbyServerIT360HOME>/servicedesk directory.
Note: As of now, this is a limitation in our implementation and will be addressed in one of our upcoming hot fix releases.

Top

5. FAQs

  1. When does the MySQL data replication failure occur? What is the recommended solution for the same?

When the network connectivity between the Primary and Standby servers breaks, each server will start running as a Standalone Primary server. In this scenario, the Administrator will receive an email notification, indicating the failure in replication. Upon receiving this email, he has to restore the HA mode, by performing the HA configuration, once again.

  1. How to verify that the Mysql replication has started?

    To check whether the DB replication has started for both the Primary and Standby server DBs, do the following;

    1. Open the file <machinename>.err [IT360_HOME/mysql/data], in both the Primary and Standby servers.

    2. Check for the entry replication started. If the entry is available, then replication is successfully done.

  1. How to give multiple email ids, while configuring the notifications?

Type in the email ids, separated by commas.

  1. What is the recommended value for the HEART_BEAT_INTERVAL and the FAILOVER_INTERVAL, while configuring the notifications?

It is recommended to have the HEART_BEAT_INTERVAL and the FAILOVER_INTERVAL, above 20 seconds. Having lesser values may lead to unexpected behaviour.

  1. What is the purpose of setting the RETRY_COUNT, while configuring the notifications? Is the value set for it in the above sample entries is fixed or changeable?

In general, the Standby will go through the parameters, specified in the 'FailOver.xml' file, available under the '<IT360_HOME>/applications/working/conf' directory.  By specifying the RETRY_COUNT value in this configuration file, before starting the service, you can instruct the Standby server to try as many times as the count says, before taking over the role of the Primary. 

By default, this attribute is configured as 1, which means that it will try only once, before taking the role as Primary. Subsequent retry attempts will happen with an interval of 60 secs. For instance, if you need the Standby server to wait for 5 mins before taking over the role as Primary, you can set the RETRY_COUNT value as 5. Please note that the optimal value for the RETRY_COUNT is 10.

Top



Copyright © 2014, ZOHO Corp. All Rights Reserved.