Load Balancing using FioranoMQ Dispatcher Services

Question 1: What is the objective of clustering? What are the software/hardware requirements for clustering?

Server clustering, allows, clients connected on different FioranoMQ servers to exchange information, without being connected to each other's server. Fiorano's load balancing architecture involves the use of Dispatcher-enabled MQServer, to route, incoming client connections, to the least-loaded server in a cluster. (MQ Administrator can set up an instance of MQServer to act as Dispatcher, by using the Fiorano Admin Studio). The dispatcher component of FioranoMQ Server is connected to multiple FioranoMQ servers, which can run on multiple machines, or, the same machine. All of these servers, become part of the cluster that is serviced by the dispatcher.
Clustering requires no additional software/hardware as such other than what is required for FioranoMQ server.

Question 2: What is the best recommended approach for configuring FioranoMQ so as to minimize the single point of failure problem?

This can be achieved through clustering (The FioranoMQ administrator can set up multiple FioranoMQ servers in the cluster to provide dispatcher facilities). This prevents single point of failure.

Question 3: Are client connections automatically distributed to the least used JMS Server in a multiple server cluster?

FioranoMQ's load balancing architecture involves the use of a special Dispatcher application, to route incoming client connections to the least-loaded server in a cluster. Any MQServer can be launched as a dispatcher. All of these servers become part of the cluster that is serviced by the dispatcher. Connecting the servers as neighbors allow topics to be shared transparently, between the clients and servers, to exchange information seamlessly across the servers.

Question 4: Can multiple instances of Dispatcher be started to prevent the dispatcher from being the single point of failure?

You can run multiple instances of the Dispatcher Tool, to have N-MQ Servers (server farms) associated with each Dispatcher. In addition, you can have the same MQServer as a part of multiple clusters.

Question 5: Can I launch multiple instances of dispatcher on the same machine?

Multiple instances of the dispatcher server can be started on the same machine on different ports.

Question 6: Is there any option available for setting backup/failover URLs for FioranoMQ clients?

The failover support of FioranoMQ is enhanced by the use of N-Failover URLs, with which clients can transparently move to a backup server if the primary server goes down.
For more information refer to section N Failover URLs Support in the FioranoMQ Handbook.

Question 7: Intermittently, the following exception trace can be viewed, which is causing loss of data in the production system: LEAST_LOADED_SERVER_CALCULATION_ERROR :: JMSException occurred while executing lookupLeastLoaded method of ConnectionFactoryProxy. at fiorano.jms.runtime.pubsub.FioranoTopicConnectionFactoryProxy.lookupLeast-Loaded (Unknown Source) at fiorano.jms.runtime.pubsub.FioranoTopicConnectionFactoryProxy.createConnec-tion (Unknown Source) at fiorano.jms.runtime.pubsub.FioranoTopicConnectionFactory.createConnection-Proxy (Unknown Source) at fiorano.jms.runtime.pubsub.FioranoTopicConnectionFactory.createTopicConnec-tion (Unknown Source) What is the reason for this??

This exception occurs due to the fact that the dispatcher server was unable to route or load balance the new connection request to a member server of its cluster.
This can occur due to the following reasons:

1. The dispatcher was unable to retrieve the least loaded server from the cluster due to some error.
2. The dispatcher was unable to connect to the least loaded server or the preferred server.
3. The dispatcher was able to connect to the least loaded server of the cluster but that server does not have the Connection Factory object being used to create the connection from the client side. If all the servers are running on the same LDAP Admin store then this should not occur. To ensure that the connection factory object exists on all the member servers, run the FioranoMQ AdminTool on all the servers and check the list of Topic Connection Factories under the TCF tab.

For case 2 the reason could be that the concerned member server of the cluster is not active and running.
In order to know the exact cause of the exception, use the following catch block ,where this exception is being thrown ,at client side:
catch(FioranoException e) { e.printCompleteStackTrace() }
If you are using the ExceptionListener, you can additionally use the e.printCompleteStackTrace() call in the onException() method.
In addition, check the mqout and mqerr logs, for any exception that might have occurred on the FioranoMQ server side.
To gain information about these exceptions at the server side, increase the Log levels of the following parameters on the dispatcher server:
FMQDispatcherServices=10 Dispatcher=10
Increase these Trace levels using the Log Manager tab of the Fiorano Studio.

Question 8: How does FioranoMQ provide for High Availability of its data-store?

FioranoMQ runs over a proprietary file-based data store where all persistent information is stored in files. To add support for High Availability to FioranoMQ servers running in a cluster, use RAID/SAN hard disks for storage of persistent data.
In case of RAID, the information stored on one disk is replicated on one or more disks. Thus, if a primary server writing on say, disk1 crashes, the secondary server would access, all information from the replicated disk2. Setting this up for a cluster provides High Availability support within that cluster of FioranoMQ servers.

Message Replication

Question 9: How does FioranoMQ support Message Replication across different instances of FioranoMQ server?

FioranoMQ supports Message Replication, over topics, using the FioranoMQ repeater and server-to-server communication over queues, using the FioranoMQ bridges. These modules, coupled with the FioranoMQ dispatcher can be used, to set up a fully loaded, balanced cluster of FioranoMQ servers with support for failover.

Server-to-Server Communication - FioranoMQ Neighboring

Question 10: What does server- to- server communication mean? How can I make use of it?

Server-to-server communication, allows two or more JMS servers, to share information in a controlled manner. This allows clients on one server to exchange information with clients connected to another server in the cluster, without each client having to connect explicitly to each server, allowing large numbers of concurrent client connections in the cluster.
Server- to- server communication, can be used, to logically distribute the system domain into closed systems with controlled boundaries for inter-system communication.

Question 11: What is a repeater?

Repeater is a JMS Client Application, which is responsible for server -to -server communication by repeating messages from one topic to the other. The repeater is started as a stand-alone Java Application and can run in secure and non-secure mode. For more information, refer to the Repeater section. 

Question 12: How different is the repeater from the dispatcher?

Repeater and Dispatcher are completely different components of the FioranoMQ Enterprise suite. The Dispatcher manages a cluster of servers, and is responsible for routing incoming client requests to the least-loaded server in the cluster. The Repeater on the other hand, is a JMS Client Application that is responsible for server- to- server communication by repeating messages from one topic to the other.

Question 13: Can I run a single instance of Repeater to connect multiple machines in a cluster?

You can run a single instance of Repeater, to connect multiple machines in the cluster.

Question 14: What information does a Repeater propagate?

Repeater propagates only Topic related information. In addition, FioranoMQ supports the concept of a distributed queue using Bridges. For more information on configuring bridges, refer the Bridge section. 

Question 15: How do I have an application on one host send point-to-point messages to an application on another host?

For more information, refer the Bridge section.

Question 16: How do I send Object Messages from Publisher on one MQ Server to a Subscriber on another MQ Server?

To publish and subscribe to Object Messages in a server cluster, ensure that all the intermediate routers have a reference, to the published Object in its CLASSPATH.

Question 17: Do network failures in a server cluster stall publishing of messages to the local server?

No. An application can continue to publish to the local server even in case of network failures between remote servers. The local JMS server stores messages and forwards them to remote nodes as soon as the network comes up again.

Question 18: How do I connect two servers?

FioranoMQ architecture allows multiple FioranoMQ Servers to be connected together, allowing clients connected on one server to exchange information with clients connected on any other MQ server. Servers can be connected (both over LAN and WAN), using FioranoMQ Repeater. All server-to-server communication is handled transparently by FioranoMQ software and the client application does not need to be modified it, in any way. MQ Servers can be part of the same LAN or can be spread across multiple WANs. Repeater is enabled for HTTP/HTTPS as well. For more information, refer to the Repeater section.

Question 19: How do I perform load balancing in a cluster?

FioranoMQ's load balancing architecture involves the use of Dispatcher-enabled MQServer to route incoming client connections to the least-loaded server in a cluster. FioranoMQ Sever can also act, as the dispatcher, to route incoming client connections to least loaded server in a cluster. MQ Administrator can set up an instance of MQServer in, from the Admin Tool.

Question 20: How do I integrate FioranoMQ with MSMQ, MQSeries, TibRv or other JMS servers?

FioranoMQ Bridge,can be used to forward requests and responses between two different message queuing systems. Communication to any other JMS vendor is done using the standard JMS API. Communication to MSMQ is done, using MSMQ Java APIs and communication to TibRv is done using TibRv Java APIs. In addition, the FioranoMQ Bridge sends messages as JMS Messages to the target JMS vendor. The FioranoMQ Bridge is configured using XML based configuration files that allow a single instance of the Bridge to "bridge" any number of messaging servers. For more information, refer the Bridge section.

Concurrent Connections

Question 21: How does the FioranoMQ Server achieve scalability?

For details on Fiorano's Scalable Connection Manager, refer to the Connection Management section.

Question 22: The following exceptions were thrown while running FioranoMQ. What do these exceptions mean and why are they thrown?

Exception 1
java.net.SocketException: Too many open files at java.net.PlainSocketImpl.accept(Compiled Code) at java.net.ServerSocket.implAccept(Compiled Code) at java.net.ServerSocket.accept(Compiled Code) at fioranomq.t3.srvr.ListenThread.run(Compiled Code)

Exception 2
java.io.IOException: Too many open files
at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.(UNIXProcess.java:54) at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.(UNIXProcess.java:54) at java.lang.Runtime.execInternal(Native Method) at java.lang.Runtime.exec(Runtime.java:551) at java.lang.Runtime.exec(Runtime.java:477) at java.lang.Runtime.exec(Runtime.java:443)

Cause of the problem

Both the exceptions thrown above,indicate a problem, related to resources of operating systems being used. This problem usually occurs, when several users are connected to the server at the same time. Java opens many files, in order to read in the classes required to run your application. High volume applications can use a lot of file descriptors. This could lead to a lack of new file descriptors. The number of file differs from one operating system to another.

Windows

By default, open file handles (file descriptors) limit is set to 16,384.

Linux

The Admin user can set the file descriptors limit in the etc/security/limits.conf configuration file, as shown in following example.
soft nofile 1024 hard nofile 4096

A system-wide file descriptor limit can also be set, by adding the following three lines to the /etc/rc.d/rc.local startup script:

Increase system-wide file descriptor limit. echo 4096 > /proc/sys/fs/file-max echo 16384 > /proc/sys/fs/inode-max
Solaris
You can modify maximum value defined in rlim_fd_max. By default, the value is set to 65,536.
HP-UX
Nfile, defines the maximum number of open files. This value is usually determined by the formula: ((NPROC*2)+1000) where NPROC is usually: ((MAXUS-ERS*5)+64). For a MAXUSERS of 400, this works out to 5128. You can usually set it higher, maxfiles is the Soft file limit per process and maxfiles_lim is the hard file limit per process.
AIX
The file descriptors limit is set in the /etc/security/limits file and its default value is 2000. This limit can be changed by the ulimit command or the setrlimit subroutine. The maximum size is defined by the constant OPEN_MAX.
You need to first monitor file descriptors and get a status of open files and other potential issues. This, then needs to be followed by increasing, the number of file descriptors according to the operating system.
Known FioranoMQ Server Issues
Increasing the number of File Descriptors usually fixes this kind of problem, but you also need to make sure, that the FioranoMQ Server as an application does not use too many files and that open files are getting closed properly so that file descriptors are released.
All issues reported to Fiorano Customer Support have involved a lack of file descriptors or an overflow of the descriptor table. This problem always occurs when the OS notifies the java process that no new file descriptor can be allocated. In this case, you need to increase the number of file descriptors.

Crash/Disaster Recovery

Question 23: What recovery mechanism does FioranoMQ have in case of disaster/crash of the server?

FioranoMQ provides for recovery and extraction of its data store in case of any crash or disaster using the Analyzer/Extractor Toolkit. With this toolkit, all persistent information can be analyzed and extracted to a new location for further use.
For more information, refer to the Working with the DB Recovery Tool section.

Other lssues

Question 24: What throughput degradation would occur if all messages were replicated to queue copies on several servers in a cluster?

In most cases, clustering for messages is achieved using store and forward mechanism, where the messages are moved around in the network. If a copy is created for each queue, then receivers needs to identify the message to the point, where they have already consumed messages, to avoid receiving duplicate messages on fail-over. Queue, forwarding works as a background daemon, which may increase end-to-end time delay (between send and receive) by up to 40%. However, the throughput of the cluster (defined as the number of messages handled by the server cluster) remains largely unaffected by clustering.