HornetQ reactivation/reconnect attempts
thebravedave May 18, 2012 2:42 AMHello, I am having this problem where remote ejbs/JMS aren't working correctly when the client server is
started after the host server. I have 2 servers on seperate boxes. One is the JMS host server, the
other is the JMS client server.
Here is how we have our setup working. We have our server A acting as the hornetQ JMS host. There are topics that are available on it. We have our server B acting as the client.
The client server gets a remote EJB RemoteConnectionFactory from the host server and publishes a topic message to it. We then have a message driven bean on the client (server B) server that is listening to that topic,
and should get triggered when a message is published to the topic.
Im configuring everything in standalone.xml, and not using jboss-ejb-clients.properties.
We are using jboss 7.1.1.Final, which uses jboss-ejb-client-1.0.5.Final, so the fix should work in this version.
I want to test that the fix is working correctly.
To do so I will start up the client server and then afterwards start up the host server and see if the client server reconnects to the host server after the host server has completed its startup.
I start up the client server, and get this error in the logs. This error is to be expected as the JMS host server is not up at this point.
2012-05-17 10:22:18,320 INFO [org.hornetq.ra.inflow.HornetQActivation] Attempting to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@39477a84 destination=areavaultResourceupdate destinationType=javax.jms.Topic ack=Auto-acknowledge durable=false clientID=null user=Theuser password=**** maxSession=15)
2012-05-17 10:22:18,322 ERROR [org.hornetq.ra.inflow.HornetQActivation] Unable to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@39477a84 destination=areavaultResourceupdate destinationType=javax.jms.Topic ack=Auto-acknowledge durable=false clientID=null user=sococouser password=**** maxSession=15): HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:769)
at org.hornetq.ra.inflow.HornetQActivation.setupSession(HornetQActivation.java:362)
at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:294)
at org.hornetq.ra.inflow.HornetQActivation.handleFailure(HornetQActivation.java:566)
at org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:609)
at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:212)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:821)
at java.lang.Thread.run(Thread.java:662)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
I then start up the server that is the JMS Host. I see some communication between the client and the
host on port 5445. I think this is when the client starts communicating with the host. All of this traffic (right below) happens pretty much all at once and right after the
JMS host finishes starting up.
Heres the tcpdump packet info (restricted to port 5445 only):
10:40:32.347045 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [S], seq 3659829354, win 14600, options [mss 1460,sackOK,TS val 262568567 ecr 0,nop,wscale 7], length 0
10:40:32.347128 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [S.], seq 267315287, ack 3659829355, win 14480, options [mss 1460,sackOK,TS val 1902849926 ecr 262568567,nop,wscale 6], length 0
10:40:32.347255 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 1, win 115, options [nop,nop,TS val 262568567 ecr 1902849926], length 0
10:40:32.378593 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 1:22, ack 1, win 115, options [nop,nop,TS val 262568599 ecr 1902849926], length 21
10:40:32.378660 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [.], ack 22, win 227, options [nop,nop,TS val 1902849958 ecr 262568599], length 0
10:40:32.378812 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 1:22, ack 22, win 227, options [nop,nop,TS val 1902849958 ecr 262568599], length 21
10:40:32.378947 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 22, win 115, options [nop,nop,TS val 262568599 ecr 1902849958], length 0
10:40:32.379892 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 22:43, ack 22, win 115, options [nop,nop,TS val 262568600 ecr 1902849958], length 21
10:40:32.380010 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 22:43, ack 43, win 227, options [nop,nop,TS val 1902849959 ecr 262568600], length 21
10:40:32.382852 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 43:61, ack 43, win 115, options [nop,nop,TS val 262568603 ecr 1902849959], length 18
10:40:32.384551 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 61:143, ack 43, win 115, options [nop,nop,TS val 262568605 ecr 1902849959], length 82
10:40:32.384625 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [.], ack 143, win 227, options [nop,nop,TS val 1902849964 ecr 262568603], length 0
10:40:32.390737 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 43:60, ack 143, win 227, options [nop,nop,TS val 1902849970 ecr 262568603], length 17
10:40:32.398461 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 143:156, ack 60, win 115, options [nop,nop,TS val 262568618 ecr 1902849970], length 13
10:40:32.398858 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 60:77, ack 156, win 227, options [nop,nop,TS val 1902849978 ecr 262568618], length 17
10:40:32.438497 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 77, win 115, options [nop,nop,TS val 262568659 ecr 1902849978], length 0
The only problem is that after this when I perform a task that should trigger a JMS message on
the client, the message driven bean is not triggered. In fact I am using tcpdump to listen to all traffic to the JMS client
and when I perform a task that should trigger the message driven bean (which is configured to listen to a JMS topic from the JMS host server)
I don't get an traffic from the client server to the host server (or host to client).
If I start the host up first, then afterwards start up the client server, everytime I perform
a task that should trigger the message driven bean, I see the traffic occur on port 5445 (from client->host and host->client) and the
message driven bean is triggered correctly.
I do see some intermitten packets being sent on port 5445 as such
...
10:54:32.389758 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 767, win 115, options [nop,nop,TS val 263408610 ecr 1903689969], length 0
10:55:02.389588 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [P.], seq 822:843, ack 767, win 115, options [nop,nop,TS val 263438610 ecr 1903689969], length 21
10:55:02.389782 IP userdeveug01.eug.mycompany.net.5445 > asdeveug02.eug.mycompany.net.50224: Flags [P.], seq 767:788, ack 843, win 227, options [nop,nop,TS val 1903719969 ecr 263438610], length 21
10:55:02.389900 IP asdeveug02.eug.mycompany.net.50224 > userdeveug01.eug.mycompany.net.5445: Flags [.], ack 788, win 115, options [nop,nop,TS val 263438610 ecr 1903719969], length 0
....
But none of these packets are being sent when I trigger the event that should publish a message to the topic and hence the message driven bean. When I start up the JMS host first
and the client server afterwards, and trigger the even that should trigger the topic, it is very obvious that the packets being sent on port 5445 are coming from the triggering event.
I find this confusing, because I think that our message utility class is getting the RemoteConnectionFactory class correctly, due to the large chunk of packets sent on port 5445 when the host server starts up as well
as the fact that when I kill java running on the host server I get a
HornetQException[errorCode=2 message=Channel disconnected]
error, which makes me think that the client server was actually connected to the host server.
Please help me. As it is we have to always do startup in a certain order, and since we have automatic weekly restarts of the client server, if the host server were down at that time for some reason (or being restarted) then the
JMS would just not work, which would cause bugs in our software.
Let me know if you need any more information.
Thank you!