| # Apache Tomcat 6.0 |

Links Docs Home FAQ User Guide 1) Introduction 2) Setup 3) First webapp 4) Deployer 5) Manager 6) Realms and AAA 7) Security Manager 8) JNDI Resources 9) JDBC DataSources 10) Classloading 11) JSPs 12) SSL 13) SSI 14) CGI 15) Proxy Support 16) MBean Descriptor 17) Default Servlet 18) Clustering 19) Load Balancer 20) Connectors 21) Monitoring and Management 22) Logging 23) APR 24) Advanced IO 25) Additional Components Reference Release Notes Configuration Javadocs JK 1.2 Documentation Apache Tomcat Development Building Changelog Status Developers Architecture Functional Specs. |
# Apache Tomcat 6.0 ## Clustering/Session Replication HOW-TO | print-friendly version
Important Note
This document is pending an update to the latest implementation. You can also check the configuration reference documentation.
For the impatient
> Simply add
>     <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
> to your `<Engine>` or your `<Host>` element to enable clustering. Using the above configuration will enable all to all session replication using the `DeltaManager` to replicate session deltas. Here are some of the important default values: 1. Multicast address is 228.0.0.4 2. Multicast port is 45564 3. The IP broadcasted is `java.net.InetAddress.getLocalHost().getHostAddress()` 4. The TCP port listening for replication messages is the first available server socket in range `4000-4100` 5. Two listeners are configured `ClusterSessionListener` and `JvmRouteSessionIDBinderListener` 6. Two interceptors are configured `TcpFailureDetector` and `MessageDispatch15Interceptor` The following is the default cluster configuration:
>             <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>                      channelSendOptions="8">

>               <Manager className="org.apache.catalina.ha.session.DeltaManager"
>                        expireSessionsOnShutdown="false"
>                        notifyListenersOnReplication="true"/>

>               <Channel className="org.apache.catalina.tribes.group.GroupChannel">
>                 <Membership className="org.apache.catalina.tribes.membership.McastService"
>                             address="228.0.0.4"
>                             port="45564"
>                             frequency="500"
>                             dropTime="3000"/>
>                 <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>                           address="auto"
>                           port="4000"
>                           autoBind="100"
>                           selectorTimeout="5000"
>                           maxThreads="6"/>

>                 <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>                   <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>                 </Sender>
>                 <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>                 <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>               </Channel>

>               <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>                      filter=""/>

>               <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>                         tempDir="/tmp/war-temp/"
>                         deployDir="/tmp/war-deploy/"
>                         watchDir="/tmp/war-listen/"
>                         watchEnabled="false"/>

>               <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>               <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>             </Cluster>
Cluster Basics
To run session replication in your Tomcat 6.0 container, the following steps should be completed: All your session attributes must implement `java.io.Serializable` Uncomment the `Cluster` element in server.xml Uncomment the `Valve(ReplicationValve)` element in server.xml If your Tomcat instances are running on the same machine, make sure the `tcpListenPort` attribute is unique for each instance. Make sure your `web.xml` has the `<distributable/>` element or set at your `<Context distributable="true" />` If you are using mod_jk, make sure that jvmRoute attribute is set at your Engine `<Engine name="Catalina" jvmRoute="node01" >` and that the jvmRoute attribute value matches your worker name in workers.properties Make sure that all nodes have the same time and sync with NTP service! Make sure that your loadbalancer is configured for sticky session mode. Load balancing can be achieved through many techniques, as seen in the Load Balancing chapter. Note: Remember that your session state is tracked by a cookie, so your URL must look the same from the out side otherwise, a new session will be created. Note: Clustering support currently requires the JDK version 1.5 or later.
Overview
> To enable session replication in Tomcat, three different paths can be followed to achieve the exact same thing: Using session persistence, and saving the session to a shared file system (PersistenceManager + FileStore) Using session persistence, and saving the session to a shared database (PersistenceManager + JDBCStore) Using in-memory-replication, using the SimpleTcpCluster that ships with Tomcat 5 (server/lib/catalina-cluster.jar) In this release of session replication, Tomcat performs an all-to-all replication of session state. This is an algorithm that is only efficient when the clusters are small. For large clusters, the next release will support a primary-secondary session replication where the session will only be stored at one or maybe two backup servers. Currently you can use the domain worker attribute (mod_jk > 1.2.8) to build cluster partitions with the potential of very scaleable cluster solution. In order to keep the network traffic down in an all-to-all environment, you can split your cluster into smaller groups. This can be easily achieved by using different multicast addresses for the different groups. A very simple setup would look like this:
>             DNS Round Robin
>                    |
>              Load Balancer
>               /           \
>           Cluster1      Cluster2
>           /     \        /     \
>       Tomcat1 Tomcat2  Tomcat3 Tomcat4
What is important to mention here, is that session replication is only the beginning of clustering. Another popular concept used to implement clusters is farming, ie, you deploy your apps only to one server, and the cluster will distribute the deployments across the entire cluster. This is all capabilities that can go into with the FarmWarDeployer (s. cluster example at `server.xml`) In the next section will go deeper into how session replication works and how to configure it.
Cluster Information
> Membership is established using multicast heartbeats. Hence, if you wish to subdivide your clusters, you can do this by changing the multicast IP address or port in the `<Membership>` element. The heartbeat contains the IP address of the Tomcat node and the TCP port that Tomcat listens to for replication traffic. All data communication happens over TCP. The `ReplicationValve` is used to find out when the request has been completed and initiate the replication, if any. Data is only replicated if the session has changed (by calling setAttribute or removeAttribute on the session). One of the most important performance considerations is the synchronous versus asynchronous replication. In a synchronous replication mode the request doesn't return until the replicated session has been sent over the wire and reinstantiated on all the other cluster nodes. Synchronous vs asynchronous is configured using the `channelSendOptions` flag and is an integer value. The default value for the `SimpleTcpCluster/DeltaManager` combo is 8, which is asynchronous. You can read more on the send flag. During async replication, the request is returned before the data has been replicated. async replication yields shorter request times, and synchronous replication guarantees the session to be replicated before the request returns.
> Bind session after crash to failover node
> > As you configure more then two nodes at same cluster for backup, most loadbalancer send don't all your requests after failover to the same node. The JvmRouteBinderValve handle tomcat jvmRoute takeover using mod_jk module after node failure. After a node crashed the next request going to other cluster node. The JvmRouteBinderValve now detect the takeover and rewrite the jsessionid information to the backup cluster node. After the next response all client request goes direct to the backup node. The change sessionid send also to all other cluster nodes. Well, now the session stickyness work directly to the backup node, but traffic don't go back too restarted cluster nodes! As jsessionid was created by cookie, the change JSESSIONID cookie resend with next response. You must add JvmRouteBinderValve and the corresponding cluster message listener JvmRouteSessionIDBinderListener. As you add the new listener you must also add the default ClusterSessionListener that receiver the normal cluster messages.
> >     <Cluster className="org.apache.catalina.tcp.SimpleTcpCluster" >
> >     ...
> >          <Valve className="org.apache.catalina.cluster.session.JvmRouteBinderValve"
> >                    enabled="true" sessionIdAttribute="takeoverSessionid"/>
> >          <ClusterListener className="org.apache.catalina.cluster.session.JvmRouteSessionIDBinderListener" />
> >          <ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener" />
> >     ...
> >     <Cluster>
> Hint: With attribute sessionIdAttribute you can change the request attribute name that included the old session id. Default attribuite name is org.apache.catalina.cluster.session.JvmRouteOrignalSessionID. Trick: You can enable this mod_jk turnover mode via JMX before you drop a node to all backup nodes! Set enable true on all JvmRouteBinderValve backups, disable worker at mod_jk and then drop node and restart it! Then enable mod_jk Worker and disable JvmRouteBinderValves again. This use case means that only requested session are migrated.
Configuration Example
>             <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>                      channelSendOptions="6">

>               <Manager className="org.apache.catalina.ha.session.BackupManager"
>                        expireSessionsOnShutdown="false"
>                        notifyListenersOnReplication="true"
>                        mapSendOptions="6"/>
>               <!--
>               <Manager className="org.apache.catalina.ha.session.DeltaManager"
>                        expireSessionsOnShutdown="false"
>                        notifyListenersOnReplication="true"/>
>               -->
>               <Channel className="org.apache.catalina.tribes.group.GroupChannel">
>                 <Membership className="org.apache.catalina.tribes.membership.McastService"
>                             address="228.0.0.4"
>                             port="45564"
>                             frequency="500"
>                             dropTime="3000"/>
>                 <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>                           address="auto"
>                           port="5000"
>                           selectorTimeout="100"
>                           maxThreads="6"/>

>                 <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>                   <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>                 </Sender>
>                 <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>                 <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>                 <Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
>               </Channel>

>               <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>                      filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>

>               <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>                         tempDir="/tmp/war-temp/"
>                         deployDir="/tmp/war-deploy/"
>                         watchDir="/tmp/war-listen/"
>                         watchEnabled="false"/>

>               <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>             </Cluster>
Cluster Architecture
> Component Levels:
>              Server
>                |
>              Service
>                |
>              Engine
>                |  \
>                |  --- Cluster --*
>                |
>              Host
>                |
>              ------
>             /      \
>          Cluster    Context(1-N)
>             |             \
>             |             -- Manager
>             |                   \
>             |                   -- DeltaManager
>             |
>          ---------------------------
>             |                       \
>           Channel                    \
>         ----------------------------- \
>          |          |         |        \
>        Receiver    Sender   Membership  \
>                                          -- Valve
>                                          |      \
>                                          |       -- ReplicationValve
>                                          |       -- JvmRouteBinderValve
>                                          |
>                                          -- LifecycleListener
>                                          |
>                                          -- ClusterListener
>                                          |      \
>                                          |       -- ClusterSessionListener
>                                          |       -- JvmRouteSessionIDBinderListener
>                                          |
>                                          -- Deployer
>                                                 \
>                                                  -- FarmWarDeployer
How it Works
To make it easy to understand how clustering works, We are gonna take you through a series of scenarios. In the scenario we only plan to use two tomcat instances `TomcatA` and `TomcatB`. We will cover the following sequence of events: `TomcatA` starts up `TomcatB` starts up (Wait that TomcatA start is complete) `TomcatA` receives a request, a session `S1` is created. `TomcatA` crashes `TomcatB` receives a request for session `S1` `TomcatA` starts up `TomcatA` receives a request, invalidate is called on the session (`S1`) `TomcatB` receives a request, for a new session (`S2`) `TomcatA` The session `S2` expires due to inactivity. Ok, now that we have a good sequence, we will take you through exactly what happens in the session repliction code `TomcatA` starts up Tomcat starts up using the standard start up sequence. When the Host object is created, a cluster object is associated with it. When the contexts are parsed, if the distributable element is in place in web.xml Tomcat asks the Cluster class (in this case `SimpleTcpCluster`) to create a manager for the replicated context. So with clustering enabled, distributable set in web.xml Tomcat will create a `DeltaManager` for that context instead of a `StandardManager`. The cluster class will start up a membership service (multicast) and a replication service (tcp unicast). More on the architecture further down in this document. `TomcatB` starts up When TomcatB starts up, it follows the same sequence as TomcatA did with one exception. The cluster is started and will establish a membership (TomcatA,TomcatB). TomcatB will now request the session state from a server that already exists in the cluster, in this case TomcatA. TomcatA responds to the request, and before TomcatB starts listening for HTTP requests, the state has been transferred from TomcatA to TomcatB. In case TomcatA doesn't respond, TomcatB will time out after 60 seconds, and issue a log entry. The session state gets transferred for each web application that has distributable in its web.xml. Note: To use session replication efficiently, all your tomcat instances should be configured the same. `TomcatA` receives a request, a session `S1` is created. The request coming in to TomcatA is treated exactly the same way as without session replication. The action happens when the request is completed, the `ReplicationValve` will intercept the request before the response is returned to the user. At this point it finds that the session has been modified, and it uses TCP to replicata the session to TomcatB. Once the serialized data has been handed off to the operating systems TCP logic, the request returns to the user, back through the valve pipeline. For each request the entire session is replicated, this allows code that modifies attributes in the session without calling setAttribute or removeAttribute to be replicated. a useDirtyFlag configuration parameter can be used to optimize the number of times a session is replicated. `TomcatA` crashes When TomcatA crashes, TomcatB receives a notification that TomcatA has dropped out of the cluster. TomcatB removes TomcatA from its membership list, and TomcatA will no longer be notified of any changes that occurs in TomcatB. The load balancer will redirect the requests from TomcatA to TomcatB and all the sessions are current. `TomcatB` receives a request for session `S1` Nothing exciting, TomcatB will process the request as any other request. `TomcatA` starts up Upon start up, before TomcatA starts taking new request and making itself available to it will follow the start up sequence described above 1) 2). It will join the cluster, contact TomcatB for the current state of all the sessions. And once it receives the session state, it finishes loading and opens its HTTP/mod_jk ports. So no requests will make it to TomcatA until it has received the session state from TomcatB. `TomcatA` receives a request, invalidate is called on the session (`S1`) The invalidate is call is intercepted, and the session is queued with invalidated sessions. When the request is complete, instead of sending out the session that has changed, it sends out an "expire" message to TomcatB and TomcatB will invalidate the session as well. `TomcatB` receives a request, for a new session (`S2`) Same scenario as in step 3) `TomcatA` The session `S2` expires due to inactivity. The invalidate is call is intercepted the same was as when a session is invalidated by the user, and the session is queued with invalidated sessions. At this point, the invalidet session will not be replicated across until another request comes through the system and checks the invalid queue. Phuuuhh! :) Membership Clustering membership is established using very simple multicast pings. Each Tomcat instance will periodically send out a multicast ping, in the ping message the instance will broad cast its IP and TCP listen port for replication. If an instance has not received such a ping within a given timeframe, the member is considered dead. Very simple, and very effective! Of course, you need to enable multicasting on your system. TCP Replication Once a multicast ping has been received, the member is added to the cluster Upon the next replication request, the sending instance will use the host and port info and establish a TCP socket. Using this socket it sends over the serialized data. The reason I choose TCP sockets is because it has built in flow control and guaranteed delivery. So I know, when I send some data, it will make it there :) Distributed locking and pages using frames Tomcat does not keep session instances in sync across the cluster. The implementation of such logic would be to much overhead and cause all kinds of problems. If your client accesses the same session simultanously using multiple requests, then the last request will override the other sessions in the cluster.
Monitoring your Cluster with JMX
> Monitoring is a very important question when you use a cluster. Some of the cluster objects are JMX MBeans Add the following parameter to your startup script with Java 5:
>     set CATALINA_OPTS=\
>     -Dcom.sun.management.jmxremote \
>     -Dcom.sun.management.jmxremote.port=%my.jmx.port% \
>     -Dcom.sun.management.jmxremote.ssl=false \
>     -Dcom.sun.management.jmxremote.authenticate=false
> 3. Activate JMX with JDK 1.4: Install the compat package Install the mx4j-tools.jar at common/lib (use the same mx4j version as your tomcat release) Configure a MX4J JMX HTTP Adaptor at your AJP Connector
>        <Connector port="${AJP.PORT}"
>           handler.list="mx"
>           mx.enabled="true"
>           mx.httpHost="${JMX.HOST}"
>           mx.httpPort="${JMX.PORT}"
>           protocol="AJP/1.3" />
> Start your tomcat and look with your browser to http://${JMX.HOST}:${JMX.PORT} With the connector parameter `mx.authMode="basic" mx.authUser="tomcat" mx.authPassword="strange"` you can control the access! List of Cluster Mbeans
> Name | Description | MBean ObjectName - Engine | MBean ObjectName - Host
> --- | --- | --- | ---
> Cluster | The complete cluster element | `type=Cluster` | `type=Cluster,host=${HOST}`
> ClusterSender | Configuration and stats of the sender infrastructure | `type=ClusterSender` | `type=ClusterSender,host=${HOST}`
> ClusterReceiver | Configuration and stats of the recevier infrastructure | `type=ClusterReceiver` | `type=ClusterReceiver,host=${HOST}`
> ClusterMembership | Configuration and stats of the membership infrastructure | `type=ClusterMembership` | `type=ClusterMembership,host=${HOST}`
> IDataSender | For every cluster member it exist a sender mbeans. It exists speziall MBeans to all replication modes | `type=IDataSender, senderAddress=${MEMBER.SENDER.IP}, senderPort=${MEMBER.SENDER.PORT}` | `type=IDataSender,host=${HOST}, senderAddress=${MEMBER.SENDER.IP}, senderPort=${MEMBER.SENDER.PORT}`
> DeltaManager | This manager control the sessions and handle session replication | `type=Manager,path=${APP.CONTEXT.PATH}, host=${HOST}` | `type=Manager,path=${APP.CONTEXT.PATH}, host=${HOST}`
> ReplicationValve | This valve control the replication to the backup nodes | `type=Valve,name=ReplicationValve` | `type=Valve,name=ReplicationValve,host=${HOST}`
> JvmRouteBinderValve | This is a cluster fallback valve to change the Session ID to the current tomcat jvmroute. | `type=Valve,name=JvmRouteBinderValve, path=${APP.CONTEXT.PATH}` | `type=Valve,name=JvmRouteBinderValve,host=${HOST}, path=${APP.CONTEXT.PATH}`
FAQ
Please see the clustering section of the FAQ.
Copyright © 1999-2006, Apache Software Foundation