...

The central concept of backchannel replication is that the Active Server server (the server which is in the Active Statestate) replicates its data store and state to the Passive Server, thus keeping both servers in sync. This replication channel is supported on a private network dedicated to the synchronization of the broker state and messaging data.

The secondary server accepts no client connection while in its hot-standby (passive) role, but is prepared to immediately for immediate transition to the Active role as soon as it detects that the Active Server is unavailable. If the primary fails, all Fiorano applications fail over from the primary and reconnect to the designated secondary backup broker.
The primary and secondary broker-pair use the replication channel to routinely seek the heartbeat of the other and watch for any interruption in the data flow or connection to switch states. A locking mechanism (explained below) is can also be employed to determine the state of the servers.

This Hot-failover process is immediate and is completely transparent to all client applications. The Secondary Server in the active role is sensitive to re-establishment of the replication channel. This reconnection may come from a recovery of the Primary Server or from a replacement Primary Server. Once the primary comes up again, it assumes the role of the Secondary Server (since the original Secondary Server is now the Primary Server). Important:

HA Locking Mechanism

HA locking mechanism is employed by the servers in replicated mode to determine the server state in case, a case a server of the pair is unavailable or if the network fails. A file with read and write permissions file is shared on a machine this . This file is referred as the LockFile. The machine hosting the LockFile is referred to as the gateway Gateway machine. A server can switch to Active state only if it holds a lock over the LockFile.

In HA implementation prior to the locking mechanism, a network link failure between the servers could have led to both servers switching to the Standalone state. Since the lock can be held by only one server at a time, it prevents both servers from switching to Active/Standalone state.

The locking mechanism makes the state switching State Switching of a an HA server more deterministic.

...

• Active
• Passive
• Standalone - Same as active. Indicates that the backup server is down /or not present in networkthe network.
• Dead - Indicates that the server is down /or not present in networkthe network.
• Active - Sync/Active Transition
• Passive - Sync/Passive Transition
• Waiting - Same as passivePassive. Indicates that the databases of the Active and Passive Servers need to be synchronized.

The following diagram explains the transition to various states:
Image Modified
Figure 2: Transition to various states

• On Upon startup, the Server enters into WAITING state. In this state, the server is waiting waits for its backup server to connect to it. This is the initial synchronization state, which is required to sync up the primary server with the secondary to avoid avoiding any message loss. This server will change changes state if one of the following occurs.:
  • Switch to PASSIVE SYNC state: If the HA channel is established and the other server is in STANDALONE state.
  • Switch to PASSIVE(STANDBY) SYNC or ACTIVE SYNC state: If the HA channel is established and the other server is also in WAITING state, then the servers assumes themselves as being in Active or Passive roles depending on the Repository Timestamps (whichever server has the latest timestamp is assumed to be the primary).
    If the repository does not have Timestamps (the server-store is cleaned up), then the server which is configured as primary turns to ACTIVE SYNC.

• When the Server is actively serving clients and its backup server is not running or if the HA transport channel is broken and it has the lock over the lockfile, the state of the server is STANDALONE. If the server in STANDALONE state establishes the HA channel and the other server is in WAITING state, then the STANDALONE server shifts to ACTIVE SYNC state. A passive (standby) server can switch to STANDALONE if the other server is not running or if the transport channel is broken and it acquires the lock over the LockFile.
• When the Server is in ACTIVE SYNC state, the server starts synchronizing its data with the backup server which is in PASSIVE SYNC. The Server in ACTIVE SYNC continues to serve its clients. Completion of the Runtime Synchronization Protocol causes a transition of the backup server to the ACTIVE state and the server in PASSIVE SYNC state moves to PASSIVE state.
• Once the Primary Server completes the synchronization, it enters into the ACTIVE state and begins actively transmitting state information and Call Replications onto the PASSIVE Server. At this point, if there is a failure of the ACTIVE server, the Hot Standby PASSIVE Server is ready to move into the STANDLONE state and starts accepting requests from the clients.
• An active server can switch to the WAITING state if the transport channel is broken and it loses the lock over the LockFile. A STANDALONE server can switch to the WAITING state if it loses the lock over the lock file.
• Whenever there is a change in the server state, it broadcasts the present and previous state to the Backup Server. The Servers Server transition is a function of its own state, the present and previous state of the Backup Server and whether or not it holds the lock over the LockFile.

...

The Fiorano 10 installer comes with prebuilt preconfigured profiles for replicated mode which are preconfigured and ready to run on a single machine.

Table 1: The default profiles to be used for Primary and Secondary Serversservers are listed below.

To launch the server on one of these profiles, the user can use the script server.bat/sh in at the location: $Fiorano_home/esb/server/bin.

server.bat -mode fes -profile haprofile1/primary - to launch the primary server of Fiorano Enterprise 

server.bat -mode fps -profile haprofile1/secondary - to launch the secondary server of Fiorano Peer HA -profile1

server.bat -mode fps -profile haprofile2/primary - to launch the primary server of Fiorano Peer HA -profile2

Configuration Steps

• Setting up the lockfileLock file
• Configuring the profile

Anchor
Setting up the LockFile Setting up the LockFile

Setting up the LockFile

A file is created and the directory containing it is shared with readRead/write permissionsWrite permission. The lock file, if present on a machine having UNIX/Solaris operating system, should be shared by using the NFS protocol - version 4. If the lock file is present on Windows, it should be shared using the Samba Protocol . (The lock file can be shared on Windows using NFS – version 4, if it supports it.).

The table below gives some of the possible combinations of operating systems Operating Systems in the HA Setup:

On Windows Operating System, the directory on the gateway machine containing the LockFile should be mapped to a network drive.

Example:
If the directory containing the lock file on Windows , is shared using the samba protocol, this directory should be mapped to a network drive on the Windows machine hosting the server. Let us say the A shared directory is mapped to a the drive letter 'C:/' and with the lock file is "lock.lck , " will have the lock file path now becomes - 'C:/lock.lck'. This path should be used to set the LockFile parameter while configuring the profile via Fiorano StudioeStudio.

On a non-Windows Operating System, the directory on the gateway machine containing the LockFile should be mounted on the machine hosting the server.

Example:
If the lock file is present in a windows Windows samba share & and is mounted at /home/user/db on the machine hosting the server, then the lock file path on the machine hosting the server would be /home/user/db/lock.lck. This absolute path should be used to set the LockFile parameter while configuring the profile via Fiorano StudioeStudio.

Example Examples for Mount Command

Example 1:
'mount -t cifs –o rw //<gatewayIP>/<sharename>  <path on local machine where the directory has to be mounted.>'mounted>

If the lock file 'lock.lck' is shared using NFS v4 and the directory "db" is mounted at /home/user/db on the machine hosting the server, then the lock file path on the machine hosting the server would be /home/user/db/lock.lck. This absolute path should be used to set the LockFile parameter while configuring the profile via Fiorano StudioeStudio.

Example Mount Command2:
'mount -t nfs4 -o rw <gatewayIP>:/  <path on local machine where the directory has to be mounted.>'

...

mounted>

File Sharing

Configuring and Mapping mapping Network Drives

...

between Unix-based Machines

...

(Solaris and Linux, Linux and Linux, Solaris and Solaris

...

) is explained in the sections below.

"A" and "B" in Machine A and Machine B referred in the following sections can be Linux or Solaris.

Shared DB in Machine A

1. Create directory "dirA" which you want is to be shared and Login as root.

2. See Check the status of the Network File System server .using the command below:

   Code Block
   $ /etc/init.d/nfs status

3. If it hasn't the server has not been started yet, start it .using the command below:

   Code Block
   $ /etc/init.d/nfs start

4. Open NFS Server Configuration.

   Code Block
   $system-config-nfs

5. Add path of directory the directory to be shared, permissions, and host(s).

   Code Block
   $chmod o+rwx path_of_dirA -R

Shared DB in Machine B

1. Create directory "dirB" where you want contents of MachineA:<path_of_dirA> need to be stored.

2. Login as root and open the File Systems Table.

   Code Block
   $vi /etc/fstab

3. Add entry in this the format below:

   Code Block
   MachineA: path_of_dirA path_of_local_dirB nfs defaults

   Code Block
   title Example
   192.168.1.209:/home/user1/Desktop/dirA/ /home/user2/Desktop/dirB/ nfs defaults

4. Run mount command.:

   Code Block
   $mount -a

Shared DB in

...

the third Machine

In the case where When DB is stored in a third machine, instead of either the primary or secondary, two machines have to be allowed to access this shared resource. Then while While adding the hosts to the NFS server configuration, add the corresponding netmask.

Example: if If Machine B (HA Primary) and Machine C (HA Secondary) have to access Machine A .

...

with Machine A

...

IP

...

address 192.168.1.209 and Machine B

...

IP

...

address 192.168.1.45

...

, type 192.168.1.0/24 to the hosts’ field

...

and repeat the steps mentioned for Machine B for C as well.

...

Windows and UNIX based Machines

Sharing from Windows

...

machine:

Create directory the directory that needs to be shared. In the sharing Sharing and security Security menu, change permissions to “Allow network users to change my files”. Give a name to the shared directory, sayfor example, winshare.

Mounting on UNIX based Machine:

1. Create the directory where you want the contents of the Windows shared directory need to be availablesaved.

2. Run following the mount command.:

   Code Block
   mount //windows_machine_ip/winshare local_directory_path -o,rw,username=windows_user_name

3. Then when you are prompted for the password type; the Provide the Windows password when prompted.

Required The required directory has been will be mounted.

Sharing from UNIX-based Machine:

1. Create

...

1. the directory that needs to be shared and login as root.
2. Edit the samba configuration file

...

1. (open /etc/samba/smb.conf to edit).

...

2. Add the sharing details at end of the file.

Template is provided herebelow, followed by examplean example:

Template: [<Profiles,

...

name

...

for the shared directory>]

• path = <path of shared directory>
• browseable = yes
• writable = yes
• guest ok = yes

Example:

Say our For a UNIX machine has with IP address: 192.168.1.22. Now, If if the shared directory is : “/home/fmq/Desktop/share”. Then one can give the profiles name as , profile name can be “share”, which will appear while mounting on windows on Windows system. So, following lines can be added:

...

Save the above file and restart the samba service. i.e.: Execute command “

Execute the commands below to restart when the Samba server is:

/etc/init.d/smb

...

 restart

/etc/init.d/smb

...

 start

Mounting on Windows based Machine:

1. Open My Computer > Tools > Map Network Drive, the .
2. In the Map Network Drive dialog box appears.Specify  dialog box that gets displayed, specify the drive name in the Drive field.

3. In the Folder field, type

   ,

   "\\UNIX_machine_IP\profilename

   
   For our example, this would be

   "
   Example: \\192.168.1.22\share

   Note
   Make sure to select the Reconnect at Logon

   is checked

   option.

4. Browse for the location where you want meant to store the database files and click the Finish button.
5. Now you can see the The mapped drive can be seen in the My Computer node.

Between Windows machines

...

Windows Machine A:

Create directory the directory that needs to be shared. In the sharing theSharing and security Security menu, change permissions to “Allow network users to change my files”. Give a name to the shared directory, sayfor example, winshare.

Windows Machine B:

1. Open My Computer > Tools > Map Network Drive, the .
2. In the Map Network Drive dialog  dialog box appears.Specify that appears, specify the drive name in the Drive field.

3. In the Folder field, type

   in

   "\\MachineA_IP\winshare".

   Note
   Make sure to select the Reconnect at Logon

   is checked

   option.

4. Browse for the location where you want to store the database files and click the Finish button.
5. Now you can see the The mapped drive can be seen in the My Computer node.

Anchor
Configuring the FES HA Profile Configuring the FES HA Profile

Configuring the FES HA Profile

Fiorano 10 gives the ability to configure the HA through Fiorano Studio eStudio to simplify your the configuration process in an offline mode.

To configure FES HA, perform the following steps:

1. Open the HA profile (replicated).
2. Right-click the profile and select click Open from pop-up menu.
   
   
   Figure 3: Selection of FES Replicated HA
    
3. The FES Replicated HA… dialog box appears as shown in Figure 6HA dialog box gets displayed.
   
   
   Figure 4: FES Replicated HA… HA dialog box

Description of Properties (refer to Figure 6):

• BackupServerIp - Specifies the Backup Server IP. This IP is configured in backup URL for default connection factories.
• BackUpServerPort - Specifies the Backup Server port. This port is configured in backup URL for default connection factories.

...

1. 1. Open the Backup Server profile in the Fiorano Studio eStudio Profile Manager.
   2. Navigate to <ProfileName> > Fiorano > socketAcceptors > port-1 > ConnectionManager > Properties of ConnectionManager and change the port Port parameter.

The Figure 7 figure below illustrates the configuration of the Fiorano ConnectionManager.

Figure 5: Configuration of the Fiorano ConnectionManager

• BackupHAIPAddress - IP Address of backup server in HA mode. This parameter is mandatory to run HA.
• BackupHAPort - Port of the Backup server on which peer is listening for status requests send by another server. [This parameter is same as the 'Port' parameter but it specifies the port used by the backup server]
• Port - This is the port on which the HA Manager is going to listen for connections from its backup server. Once the connection is established, it starts serving as the back channel for broadcasting the state of the servers to the backup server whenever there is a state transition.
• BackupRMIServerPort - Port used by backup server to bind the Mx4J RMI Connector.
• RMIServerPort – Port used by the server to bind the Mx4j Rmi Connector.
• LockFile - Full path of the file which will be used for determining the HA states. This file should be present in the third machine and always available to the HA servers with all permissions. This parameter is mandatory to run HA. [Also, See 'GatewayServerIPAddress' parameter while configuring.]
• GatewayServerIPAddress -IP address of Gateway machine. This is used to detect network failures. It is recommended that the IP specified should be of a machine that is always available on the network. It is mandatory to specify this parameter. This parameter should always be the IP Address of the machine hosting the LockFile.
• GatewayServerPort - In Replication HA mode, network failure is detected by using the gateway server machine. Specifies the port on which gateWay machine is listening for incoming requests
• PingInterval - Time interval (in ms) after which the remote server is pinged in Replication HA mode.
• ActiveLockReAcquistionInterval - This parameter indicates the wait Interval between each attempt to acquire the lock for active server. This value should be in multiples of pingInterval/2 otherwise server may try to acquire the lock on the next multiple of pingInterval/2.
• PassiveLockAcquistionInterval - This parameter indicates the wait Interval for the passive server to acquire the lock when the link between active and passive server is down. This value should be greater than '2*ActiveLockReacquisitionInterval', otherwise an exception will be thrown. Server won't start if this value is not set properly.
• AppRepositoryPath - Path of the enterprise server application repository. FES stores event process information in this directory. Bydefault its value is $FIORANO_INSTALL_DIR/server/repository/applications.
• Component RepositoryPath - Specifies the Component Repository Path. FES stores services' information in this directory. By default its value is $FIORANO_ INSTALL_DIR/server/repository/components
• FPSRepositoryPath - Path of the Fiorano Peer server Configurations repository. FES stores peer server configurations in this directory.By default its value is $FIORANO_INSTALL_DIR/runtimedata/EnterpriseServers/<ProfileName>/FES/peers.
• ApplicationSyncType - Sync Type for Application Repository 0 - FULL SYNC - The Active Application Repository is replaced on the Passive Application Repository 1 - PARTIAL_SYNC - Only the new Applications in the Active Application Repository is updated on the Passive Application Repository 2 - NO_SYNC - No Synchronization will happen between the Active and Passive Repository. Assumes that there is no Application Edited/Removed/Added.
• ComponentSyncType - Sync Type for Component Repository 0 - FULL SYNC - The Active Component Repository is replaced on the Passive Component Repository 1 - PARTIAL_SYNC - Only the new Components in the Active Component Repository is updated on the Passive Component Repository 2 - NO_SYNC - No Synchronization will happen between the Active and Passive Repository. Assumes that there is no Component Edited/Removed/Added.The user can configure both Primary and Secondary FES from a single screen.

You can also configure both Both the Primary and Secondary FES can be configured from a single screen. For this, perform the following actions:

1. Open both the profiles and select Primary and Secondary FES profiles. Select both of them and from popup select , right-click and click the FES Replicated HA as shown in the Figure below option.
   
   
   Figure 6: FES Replicated HA option
    
2. The dialog box that gets opened contains column properties for both FES HA Primary and FES HA Secondary as shown in Figure 9.
    
   
   Figure 7: FES Replicated HA dialog box
    
3. Save the profile and we are ready to start the serverafter making the changes required.

To configure FPS HA in a similar way, perform the following steps:

1. Open the HA profile (replicated).
2. Right-click the profile and select click the FPS Replicated HA.. from the pop-up menu. The FPS Replicated HA.. dialog appears as shown in the figure below. option.
   
   
   Figure 8: FPS Replicated HA dialog box
    
3. Make the changes required in the FPS Replicated HA dialog gets displayed
4. Save the profile and we are ready to start the server.

Description of FPS Replicated HA Properties

• BackupServerIp - Specifies the backup server ip. This IP is configured in backup-url for default connection factories.
• BackUpServerPort - Specifies the backup server port. This port is configured in backup-url for default connection factories.
  This can be changed by modifying the backup server profile.
  1. Open the Backup Server profile in the Fiorano Studio Profile Manager.
  2. Navigate to <ProfileName> > Fiorano > socketAcceptors > port-1 > ConnectionManager > Properties of ConnectionManager
  and change
  2. Change the
  port
  1. Port parameter.

• BackupHAIPAddress - IP Address of backup peer server in HA .
• BackupHAPort - Port of the Backup Peer server on which peer is listening for status requests send by another server. This parameter is same as the 'Port' parameter but it specifies the port used by the backup server
• Port - This is the port on which the HA Manager is going to listen for connections from its backup server. Once the connection is established, it starts serving as the back channel for broadcasting the state of the servers to the backup server whenever there is a state transition.
• BackupRMIServerPort - Port used by backup server to bind the Mx4J RMI Connector.
• RMIServerPort – Port used by the server to bind the Mx4j Rmi Connector.
• LockFile - Full path of the file which will be used for determining the HA states. This file should be present in the third machine and always available to the HA servers with all permissions. This parameter is mandatory to run HA. [Also, See GatewayServerIPAddress parameter while configuring.]
• GatewayServerIPAddress - IP address of Gateway machine. This is used to detect network failures. It is recommended that the IP specified should be of a machine that is always available on the network. It is mandatory to specify this parameter. This parameter should always be the IP Address of the machine hosting the LockFile.
• GatewayServerPort - In Replication HA mode, network failure is detected by using the gateway server machine. Specifies the port on which gateWay machine is listening for incoming requests
• ActiveLockReAcquistionInterval - This parameter indicates the wait Interval between each attempt to acquire the lock for active server. This value should be in multiples of pingInterval/2 otherwise server may try to acquire the lock on the next multiple of pingInterval/2.
• PassiveLockAcquistionInterval - This parameter indicates the wait Interval for the passive server to acquire the lock when the link between active and passive server is down. This value should be greater than '2*ActiveLockReacquisitionInterval', otherwise an exception will be thrown. Server won't start if this value is not set properly.
• PingInterval - Time interval (in ms) after which the remote server is pinged in Replication HA mode.
• PrimaryURL – The primary URL of MQ server [i.e the FES] from which configuration should be loaded
• BackupURL(s) – The backup URL(s) of MQ server [i.e the FES] from which configuration should be loaded.

Verifying HA Setup

On Upon starting the Fiorano 10 Server that is part of an HA pair, the server prints debug information about its own state (ACTIVE, PASSIVE, and WAITING). It also prints information about its backup server 's state whenever it detects a change. Example

[Sat Feb 07 14:03:50 IST 2009xxxx] Old status of remote server = DEAD
[Sat Feb 07 14:03:50 IST 2009xxxx] New status of remote server = WAITING
[Sat Feb 07 14:03:50 IST 2009xxxx] Old status of remote server = DEAD
[Sat Feb 07 14:03:50 IST 2009xxxx] New status of remote server = WAITING
[Sat Feb 07 14:03:50 IST 2009xxxx] Old status of remote server = WAITING
[Sat Feb 07 14:03:50 IST 2009xxxx] New status of remote server = PASSIVE TRANSITION STATE

The Console includes statements such as shown below:
' Primary Server switched to ACTIVE' or and 'Secondary Server switched to PASSIVE', which indicate that the pair has successfully connected. Also, a statement gets printed when the lock is successfully acquired over the lockfile on the active servers console of the Active server. Example: Successfully acquired lock on:C:\lock.txt.

The figure below illustrates a successfully started Fiorano HA Peer Server:.

Figure 9: Fiorano HA Peer Server console

Shutting down the HA Server

For details on To know how to shutdown the servers, please refer to sections Shutting Down ESB Server and Shutting Down the Peer Server.