High Availability (with MySQL database)
(Feature available only in Premium Edition. Procedure applicable only for builds 6302 and later. For earlier versions, click here)
In mission-critical environments, one of the crucial requirements is to provide un-interrupted access to passwords. PMP provides the 'High Availability' feature just to ensure this.
PMP has provision to use either MySQL or MS SQL Server as backend. By default, PMP uses the MySQL database, which comes bundled with the product. This document is applicable for configuring High Availability with MySQL. If you are using MS SQL and wish to configure High Availability, refer to this document.
How does High Availability work?
- There will be redundant PMP server and database instances
- One instance will be the Primary providing read/write access to the users. All users will be connected with primary only
- The other instance will act as Secondary
- At any point of time data in both Primary and Secondary will be in sync with each other. PMP leverages MySQL's database replication technique for data synchronization. The data replication happens through a secure, encrypted channel
- When Primary server goes down, the Secondary will offer 'Read Only' access to the users, until the fully-functional primary server is brought back to service. The changes made in the database in the intervening period will be automatically synchronized upon connection restoration
Scenario 1 - Primary & Secondary in different geographical locations and WAN Link failure happens between the locations
Assume that the Primary Server is in one geographical location 'A' and Secondary is deployed in another location 'B'. The users in both the locations will be connected to the Primary and will be carrying out password management activities. At any point of time data in both Primary and Secondary will be sync with each other. Assume there happens loss of network connectivity between the two locations. In such a scenario, users in location 'A' will continue to remain connected with the primary and will be doing all operations. Users in location 'B' will be able to get emergency read-only access to the passwords from Secondary. Once the network between the two locations is up again, data in both the locations will be synchronized.
Scenario 2 - Primary & Secondary within the same network & Primary goes down
In case, the Primary crashes or goes down, the users in location 'A' & 'B' can rely upon the emergency read-only access to the passwords from the Secondary.
What happens to Audit Trails?
In the high availability scenarios mentioned above, audit trails will be recorded as usual. In scenario 1, as long as there is network connectivity between the two locations, the audit trails will be printed by the primary. When users connect to the Secondary, it will print operations such as 'password retrieval', 'login' and 'logout'. When the two locations get back network connectivity, the audit data will be synchronized. In scenario 2, when the primary crashes, the 'password retrieval', 'login' and 'logout' done by the users in secondary will be audited. Other audit records will already be in sync at the Standby.
How to set up High Availability?
Setting up high availability in PMP consists of the following four simple steps:
Step 1: Primary & Secondary Setup
You can use your current PMP installation as primary server and install another instance of PMP as secondary server in a separate workstation. To install PMP as secondary, during installation process, you need to choose the option "Configure this server as High availability secondary server (Read Only)". After installation, the PMP Secondary server should not be started.
Step 2: Create Data Replication Pack for High Availability in Primary
- Stop Primary and Secondary Servers, if running. Ensure that the mysqld process of PMP is NOT running
- Open a command prompt and navigate to <PMP_Primary_Installation_Folder>/bin directory
- Run the script HASetup.bat <FQDN of PMP Primary Server> <FQDN OF PMP Secondary Server > (Windows) / HASetup.sh <FQDN of PMP Primary Server> <FQDN OF PMP Secondary Server >(Linux)
- This will create a replication package named 'HAPack.zip' under <PMP_Primary_Installation_Folder>/replication folder. This zip contains the database package for secondary
- Copy the HAPack.zip. This has to be put in the PMP Secondary installation machine as detailed in Step 3 below.
To run this script, you need to pass the fully qualified domain names of the host where PMP primary and secondary servers are installed as commandline arguments. For Example, if the primary server is running at (say) primary-server in the domain zohocorpin.com and the secondary server is running at (say) secondary-server in the domain zohocorpin.com, you need to execute the above script as follows:
In Windows: HASetup.bat primary-server.zohocorpin.com secondary-server.zohocorpin.com
In Linux: sh HASetup.sh primary-server.zohocorpin.com secondary-server.zohocorpin.com
Step 3: Put the HA Data Replication Pack in Secondary
Put the HAPack.zip file copied from the PRIMARY Installation (as detailed in the previous step) in to the <PMP_Secondary_Installation_Folder> and unzip it. Take care to extract the files under <PMP_Secondary_Installation_Folder> only. It will overwrite the existing data files.
Step 4: Specify the Location of Encryption Master Key
After extracting HAPack.zip in PMP Secondary Server, navigate to <PMP_Secondary_Installation_Folder>/conf folder, edit manage_key.conf and specify the location of pmp_key.key (encryption master key). PMP requires the pmp_key.key file accessible with its full path when it starts up every time. After a successful start-up, it does not need the key anymore and so the device with the key file can be taken offline.
The High Availability configuration is ready now. To get it up and running, start PMP Secondary server.
Verify High Availability setup
After carrying out the above steps, you can verify if the High Availability setup is working properly by looking at the message in "Admin >> General >> High Availability" page of Primary server. If the setup is proper, you will see the following:
High Availability Status: Alive
Replication Status: Alive
If both the above messages show "Alive", it indicates that high availability is working fine. In case, if the status turns 'Failed', it indicates failure of the setup.
What does 'High Availability Status' Indicate (Alive/Failed)?
Constant replication of data between Primary and Standby server is the technology underlying high availability. The status 'Alive' indicates perfect data replication and data synchronization. If there happens any disruption like network problems between Primary and Standby (in turn between the databases), the status will get changed to 'Failed'.
This may happen when there is no communication/connection between the database of primary server and that of the standby server. When the connection gets reestablished, data synchronization will happen and both databases will be in sync with each other. During the intervening period, those who have connected to the primary and standby will not face any disruption in service.
In short, this status is only an indication of the connection/communication between databases and does not warrant any troubleshooting.
What does 'Replication Status' Indicate (Alive/Failed)?
As mentioned above, high availability leverages the MySQL replication feature. The database of Primary server acts as the Master and the one with Standby acts as the 'Slave'. When the data gets replicated properly, the status will be 'Alive'. In case, there happens any error in updating the data or query failure, the replication status become 'Failed'.
Once the status becomes failed, PMP High Availability setup also breaks down. That means, you will have to configure high availability setup all over again.
If you find the status marked as 'Failed' even after re-configuring High Availability, you may have to contact PMP support with the following log files:
PMP In Windows: <PMP Installation Folder>/mysql/data/<hostname>.err file
PMP In Linux: <PMP Installation Folder>/mysql/data/tmp/ .out file
Alerting Mechanism for Status Failure
Since the above two conditions assume importance in high availability setup, it is important to receive real-time alerts when the status turns Alive to Failed and vice-versa. To configure alerts, go to Audit >> Resource Audit >> Configure User Audit >> General Operations and select the mode of alert (email/SNMP trap/Syslog message) for the events 'High Availability Alive' and 'High Availability Failed'.
Note 1 : In case, the Primary Server crashes, when carrying out disaster recovery, please ensure the following:
- Go to the <PMP_Home>/mysql/data of secondary server and copy the files ibdata1, passtrix
- Install another instance of PMP afresh (same version as that of the PMP secondary from which you copied the above files)
- In the new installation, go to <PMP_Home>/mysql/data and overwrite the ibdata1, passtrix files
- Start the new PMP installation
- After configuring high availability, if you change the port of the Primary PMP server, the high availability setup will not work. It has to be re-configured with suitable changes.
If you have configured TFA
- Whenever you enable TFA or when you change the TFA type (PhoneFactor or RSA SecurID or One-time password) AND if you have configured high availability, you need to restart the PMP secondary server once.