Configuring High Availability Servers

High Availability (HA) Servers maximize system availability and eliminate any single point of failure.
HA works in Enterprise Server and Peer Server as follows:

FES HA: If there is a failure in the Active Server while it was launching an application, the Secondary Server assumes the Active role and starts launching the Event Processes again.
FPS HA: If there is a failure in the Active Server while it was processing the messages from the components, the Secondary Server assumes the Active role and continues to process the messages.

Fiorano provides complete flexibility to administrators with the below options to implement High Availability on FES HA and FPS HA:

• Database Replication: From active to passive serverIn scenarios where it is not possible to share the database, administrators can still use Fiorano's High Availability Enterprise Systems using the in-built replication support.
• Shared Database: Between active and passive server

HA Locking Mechanism

HA locking mechanism is employed by the servers in Replicated as well as Shared modes to determine the server state in case a server of the pair is unavailable or if the network fails. This mechanism works by hosting a file on a machine and sharing it with the servers. This file, provided with Read and Write permissions, is termed as "LockFile" and the machine hosting the LockFile is referred as the Gateway Machine.
A server can switch to Active state only if it holds a lock over the LockFile. For example, if two servers are configured with a shared LockFile, whichever server gets the first access to the lockfile becomes the Active Server. This mechanism makes the state-switching of an HA server more deterministic.

Example1:
If the directory containing the lock file on Windows, is shared using the samba protocol, this directory should be mapped to a network drive on the Windows machine hosting the server. Let us say the shared directory is mapped to a drive letter 'Z:/' and the lock file is fes.txt, the lock file path would be 'Z:/fes.txt'. This path should be used to set the LockFile parameter while configuring the profile via eStudio. 

On Non-Windows Operating System, the directory on the Gateway Machine containing the LockFile should be mounted on the machine hosting the server.

Example2:
If the lock file is present in a windows samba share and is mounted at /home/user/db on the machine hosting the server, then the lock file path on the machine hosting the server would be /home/user/db/fes.txt. This absolute path should be used to set the LockFile parameter while configuring the profile via Studio.

Working with a LockFile

Creating and Sharing the LockFile

To share a LockFile on a separate system:

1. Create a folder with an identifiable name in the designated system.
2. Right-click the folder and do either one of the below options:
   • Go to Share with and click Specific people.
   • Click Properties, go to Sharing tab in the dialog box, and click the Share button.
3. In the text field, type the name of the system where the file has to be mounted and click Add; the name appears in the list below.
4. Click the system name and select Read/Write under Permission Level
5. In this shared folder, create a file with an identifiable name. Examples: fes.txt, fps.txt, fesShared.txt.

Mounting the LockFile

To mount the shared LockFile on the system where operations are carried out:

1. Right-click My Computer and click Map network drive.
2. In the Map network drive dialog box:
   • Choose any drive from the Drive drop-down.
   • Click Browse and choose the location of the shared drive orIn the Folder text field, provide the IP address of the Gateway Machine and the shared folder name in the following format:
     IP Address\Shared Folder
   • Select Reconnect at logon check box to avoid mounting the file after every log in.
     
     

Database Replication

The central concept of backchannel replication is that the Active Server (the server which is in the Active State) replicates its data store and state to the Passive Server, thus keeping both servers in sync. This replication channel is supported on a private network dedicated to the synchronization of the broker state and messaging data.
Note:If servers are running in Replicated HA mode, it becomes essential to have a reliable network for database synchronization calls.
The secondary server accepts no client connection while in its hot-standby (passive) role, but is prepared to immediately transition to the Active role as soon as it detects that the Active Server is unavailable. If the primary fails, all Fiorano applications fail over from the primary and reconnect to the designated secondary backup broker. This Hot-failover process is immediate and is completely transparent to all client applications. The Secondary Server in the active role is sensitive to re-establishment of the replication channel. Once the primary comes up again, it assumes the role of the Secondary Server (since the original Secondary Server assumes to be the Primary Server).
The primary and secondary broker-pair use the replication channel to routinely seek the heartbeat of the other and watch for any interruption in the data flow or connection to switch states. A locking mechanism (explained below) is employed to determine the state of the servers.

Default Replication Server Profiles

The default FES and FPS profiles for Primary and Secondary Servers are mentioned in the below table.

Configuring Replication Servers

On a broad perspective, the steps involved in the configuration is:

• Setting up the LockFile
• Configuring the profile

Setting up LockFile

Refer Working with a LockFile section to set up the LockFile.

Configuring FES HA Servers

1. To load Primary FES, open Profile Management perspective, right-click Profiles node and navigate to: Load Profile > Fiorano ESB > haprofile1 > primary > FES. 
   
   
   
   
2. Load Secondary FES by navigating to: Load Profile > Fiorano ESB > haprofile1 > secondary > FES
   
   
   
   
3. Primary FES HA Server appears as FESHAPrimary and Secondary FES HA Server appears as FESHASecondary under Profiles node.
   
   
   
   
4. Under Profiles node, select FESHAPrimary and FESHASecondary (use CTRL or SHIFT); right-click and select FES Replicated HA
   
   
   
   
5. In the FES Replicated HA properties dialog box, set the basic configuration for the parameters as below:
   • BackupServerIp: Specifies the backup server IP. Secondary Server acts as the backup server for Primary Server and vice-versa and hence provide the respective IP addresses.
   • BackupHAIPAddress: IP Address of backup peer server in HA mode.This parameter is mandatory to run HA.
   • LockFile: Full path of the file which will be used for determining the HA states. This file should be present in a different machine which is always available to the HA servers with all permissions.
   • GatewayServerIPAddress: IPAddress used to detect network failures. The IP specified must be of a machine that is always available on the network and this machine must be used to save the LockFile (above). Note: The IP addresses of all the above properties need to be changed from 'localhost' to the appropriate one accordingly.
     
     
     
     
6. Click Close and then save the profile to apply the changes.

Configuring FPS HA Servers

1. Launch Primary FPS and Secondary FPS from the default profile: haprofile1 by following the same path as mentioned in Configuring FES HA Servers section (choose "FPS" in place of "FES"). Both the servers appear as "HAFPS".
   
   
   
   
2. Under Profiles node, select both HAFPS profiles (use CTRL or SHIFT); right-click and select FPS Replicated HA
   
   
   
   
3. In the FPS Replicated HA properties dialog box, set the basic configuration for the parameters as below:
   • BackupServerIp: Specifies the backup server IP. Secondary Server acts as the backup server for Primary Server and vice-versa and hence provide the respective IP addresses.
   • BackupHAIPAddress: IP Address of backup peer server in HA mode.This parameter is mandatory to run HA.
   • LockFile: Full path of the file which will be used for determining the HA states. This file should be present in a different machine which is always available to the HA servers with all permissions.
   • GatewayServerIPAddress: IPAddress used to detect network failures. The IP specified must be of a machine that is always available on the network and this machine must be used to save the LockFile (above).
   • PrimaryURL: URL of Enterprise server from which configuration should be loaded.
   • BackupURLs: Semi-colon separated backup URLs which are used when server at PrimaryURL is not available. Note: The IP addresses of all the above properties need to be changed from 'localhost' to the appropriate one accordingly.
     
     
     
     
4. Click Close and then save the profile to apply the changes.

Starting Server with the Replicated FES/FPS Server Profile

To load the server on one of the profiles mentioned in Default Replicated Server Profiles section, use the following command at the location: "$Fiorano_home/esb/server/bin":

• FES

Windows: server.bat -mode fes -profile <ProfileName>/<Primary or Secondary>. Linux: ./server.sh -mode fes -nobackground -profile <ProfileName>/<Primary or Secondary> Example:./server.sh -mode fes -nobackground -profile haprofile1/primary (Linux)

• FPS

Windows: server.bat -profile <ProfileName>/<Primary or Secondary>. Linux: ./server.sh -nobackground -profile <ProfileName>/<Primary or Secondary> Example:./server.sh -nobackground -profile haprofile1/primary (Linux)

Server State

Possible States of the Replicated HA Servers as displayed in the Command Prompt:

• Active
• Passive
• StandaloneSame as active. Indicates that the backup server is down or not present in network.
• DeadIndicates that the server is down or not present in network.
• ActiveSync/Active Transition
• PassiveSync/Passive Transition
• WaitingSame as passive. Indicates that the databases of the Active and Passive Servers need to be synchronized.

Shared Database

In this mode of High availability, the primary-secondary broker pair shares a common database and do not replicate data over the network. If the primary fails, all Fiorano applications fail over from the primary and reconnect to the designated secondary backup broker. The primary and secondary broker-pair use the network channel between them to routinely seek the heartbeat of the other and watch for any break in connection to switch States.
Locking Mechanism is employed to determine the state of the servers. The database which is common to both the servers is referred as Shared Database.

Default Shared Server Profiles

The default FES and FPS profiles for Primary and Secondary Servers are mentioned in the below table.

Configuring Shared HA Servers

On a broad perspective, the steps involved in the configuration is:

• Setting up the LockFile
• Setting up the Shared Database
• Configuring the profile

Setting up the LockFile

Refer Working with a LockFile section to set up the LockFile.

Setting up the Shared Database

Follow the same method as mentioned in Working with a LockFile section (ignoring the LockFile part) to set up and mount a Database directory.

Configuring FES Shared HA Servers

1. To load Primary FES, open Profile Management perspective, right-click Profiles node and navigate to: Load Profile > Fiorano ESB > haprofile_shared > primary > FES.
   
   
   
   
2. Load Secondary FES by navigating to: Load Profile > Fiorano ESB > haprofile_shared > secondary > FES
   
   
   
   
3. Primary FES Shared HA Server appears as FES_SharedHAPrimary and Secondary FES Shared HA Server appears as FES_SharedHASecondary under Profiles node.
   
   
   
   
4. Under Profiles node, select FES_SharedHAPrimary and FES_SharedHASecondary (use CTRL or SHIFT); right-click and select FES Shared HA
   
   
   
   
5. In the FES Shared HA properties dialog box, set values for the parameters: BackupServerIp, BackupHAIPAddress, LockFile and GatewayServerIPAddress. (Refer Step#5 in Configuring FES HA Servers section for description of these properties).
   
   
   
   
6. Click Close and then save the profile to apply the changes.

Configuring FPS Shared HA Servers

1. Launch Primary FPS and Secondary FPS from the default profile: haprofile_shared by following the steps in Configuring FES Shared HA Servers section (choose "FPS" in place of "FES"). Both the servers appear as "hafps_shared".
   
   
   
   
2. Under Profiles node, select both hafps_shared profiles (use CTRL or SHIFT); right-click and select FPS Replicated HA
   
   
   
   
3. In the FPS Shared HA properties dialog box, set values for the parameters: BackupServerIp, BackupHAIPAddress, LockFile, GatewayServerIPAddress, PrimaryURL and BackupURLs. (Refer Step#5 in Configuring FPS HA Servers section for description of these properties).
   
   
   
   
4. Click Close and then save the profile to apply the changes.

Starting Server with the Shared FES/FPS Server Profile

Since the shared HA pair use a common database, the location of the database has to specified while starting up each server. The "-dbPath" command line option is used for specifying the location of the shared database.
To load the server on one of the above profiles (after configuring for HA), use the following command at the location: "$Fiorano_home/esb/server/bin":

• FES

Windows: server.bat -mode fes -profile <ProfileName>/<Primary or Secondary> -dbpath <Shared DB location>.

Linux: ./server.sh -mode fes -nobackground -profile <ProfileName>/<Primary or Secondary> -dbpath <Shared DB location>

Example: ./server.sh -mode fes -nobackground -profile haprofile_shared/primary -dbPath /root/shareddb10/fes (Linux Primary FES Server)

• FPS

Windows: server.bat -profile <Primary or Secondary> -dbpath <Shared DB location>.

Linux: ./server.sh -nobackground -profile <Primary or Secondary> -dbpath <Shared DB location>

Example: ./server.sh -nobackground -profile haprofile_shared/primary -dbPath /root/shareddb10/fps (Linux Primary FPS Server)

Server State

Possible States of a server in Shared Database as displayed in the Command Prompt:

• Active
• Passive
• ActivatingServer switches to Activating State as soon as it acquires the lock. Once all its services are activated, it switches to Active or Passive State accordingly.