13 Replies Latest reply on Mar 1, 2009 10:05 PM by gaohoward

    JBoss messaging client timeout

      Hi,

      I am using JBoss 4.2.2GA and Messaging 1.4.2 with remoting version 2.2.2.SP10. We have setup JBM cluster with oracle as database. The jboss server is accessed by a remote client located beyond a firewall with a one hour timeout for any idle client. We have a JMS listener which listnes for messages on JBM through the firewall with remoting bisocket configured for 4458 secondary port.To get over the idle timeout issue we are refreshing the message listener connection every half an hour.But even after doing that, after 12 hours of idle time we are seeing that the messages coming on JBM queues are getting stuck for a few messages. Do I need to change any paramater in remoting bisocket to get over this issue ? Any help would be appreciated.

        • 1. Re: JBoss messaging client timeout
          gaohoward

          Hi,

          you need to increase the validatorPingTimeout and clientLeasePeriod in the bisocket configure file. also you should be aware of this issue:

          https://jira.jboss.org/jira/browse/JBMESSAGING-1482

          it will go in next release 1.4.3 GA. or you can try remoting SP11.

          • 2. Re: JBoss messaging client timeout

            Thanks Howard for the suggestion. We downloaded SP11 and increased validatorPingTimeout and clientLeasePeriod to 50000 in remoting bisocket file. We need to keep our dev server running overnight to see if the error goes away.
            Meanwhile on the client side as I mentioned before that we have a idle client timeout of 1 hour in the firewall setup. We refresh the client listner connections every half an hour for that. Now on the client side log on every hour(but not on th 1/2th hour) we are seeing this message for every client connection refreshed

            [Finalizer] WARN org.jboss.remoting.LeasePinger - LeasePinger[SocketClientInvoker[c6bb6c, bisocket://cawcremdd6:4457](a5e5p4y-ksiwrb-frkxz15x-1-frkyc7jf-7t)] tried to remove lease for client with session ID a5e5p4y-ksiwrb-frkxz15x-1-frl29p2s-kq, but no such lease was found

            But messages are picked up by the listener after that for the time being.We do not see anything on the server logs.Do you think there could be any other problem for the messages getting stuck in JBM queues?

            • 3. Re: JBoss messaging client timeout

              I should add one more thing here. Earlier we migrated from JBoss MQ to JBoss Messaging. There was no cluster for JBoss Messaging and the client side failover error handling was through the JMS ExceptionListener.

              We refreshed the connection every half an hour to get over the firewall freeze problem and it worked perfectly without any problem.

              • 4. Re: JBoss messaging client timeout

                This sounds similar to problems that we've been seeing - silly question... where did you get the 2.2.2SP11 download from? Or did you build from source?

                Thanks,
                Joseph

                • 5. Re: JBoss messaging client timeout

                  I downloaded jboss-remoting SP11 from http://repository.jboss.com/jboss/remoting/2.2.2.SP11-brew/lib/ and made changes suggested by Howard. We had given an idle time of 14 hours for our listener. Now some of the messages sent to the JBM queue are getting stuck as the listener is not able to pick up the message.I am seeing the following exception in the JBoss server log for those failed message deliveries. We are using JMS based listener,please help. We never had this problem when we used JBM without clustering.

                  ERROR [ServerInvokerCallbackHandler] Error handling callback
                  org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://10.***.205.***:634256266/callback?callbackServerHost=10.***.205.***&callbackServerPort=634256266&callbackServerProtocol=bisocket&clientMaxPoolSize=1&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&datatype=jms&guid=a5e5p4y-ksiwrb-frkxz15x-1-frlpylal-2lx&isCallbackServer=true&onewayThreadPool=org.jboss.jms.server.remoting.DirectThreadPool&serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper]
                  at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:579)
                  at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422)
                  at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:133)
                  at org.jboss.remoting.Client.invoke(Client.java:1640)
                  at org.jboss.remoting.Client.invoke(Client.java:554)
                  at org.jboss.remoting.Client.invokeOneway(Client.java:604)
                  at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
                  at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
                  at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1452)
                  at org.jboss.jms.server.endpoint.ServerSessionEndpoint.replicateDeliveryResponseReceived(ServerSessionEndpoint.java:1066)
                  at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$3.run(MessagingPostOffice.java:1346)
                  at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:120)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                  at java.lang.Thread.run(Thread.java:595)
                  Caused by: java.io.IOException: Timed out trying to create socket
                  at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:508)
                  at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:827)
                  at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569)
                  ... 14 more

                  Thanks,
                  Pratim

                  • 6. Re: JBoss messaging client timeout

                    Howard,
                    while we are testing failover I am seeing this in the logs

                    WARN org.jboss.remoting.Client - unable to remove remote callback handler: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://***:4457/?JBM_clientMaxPoolSize=500&clientLeasePeriod=50000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&connectionWait=60000&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&numberOfRetries=30&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&validatorPingPeriod=10000&validatorPingTimeout=50000]

                    As you suggested I had increased clientLeasePeriod to 50000 and validatorPingTimeout to 50000. Those seems to be passed to the client now which we see in the logs. But still we are seeing this socket connection timeout for a few messages after a long idle time.

                    • 7. Re: JBoss messaging client timeout
                      gaohoward

                      Sorry I gave you incorrect version of remoting. Actually from remoting part the issue should be fixed in SP12, Not SP11, see this
                      https://jira.jboss.org/jira/browse/JBREM-1082

                      So I think you can try JBM 1.4.0.SP3-CP07 which fix this from JBM side (working with remoting 2.2.2.SP10 or later)

                      • 8. Re: JBoss messaging client timeout

                        Howard,

                        I am little bit curious if JBM 1.4.2-GA-SP1 fixes the issue as well.We are using 1.4.2 GA,so would like to upgrade to the next higher version.
                        Once again thanks for your help.

                        • 9. Re: JBoss messaging client timeout
                          gaohoward

                          Yes you are right. SP1 includes the fix. You can try it also.

                          • 10. Re: JBoss messaging client timeout

                            Thanks a lot Howard. I tried with 1.4.0 SP03-CP07 and it works. No timeout no messages stuck in the queue!!
                            However during building the messaging part I found etc/ folder missing under src/ so I copied the whole etc folder from output and it did the trick.
                            Another team is trying with 1.4.2 GA SP1,I'll update you on that also.

                            Thanks once again.
                            Pratim

                            • 11. Re: JBoss messaging client timeout

                              Messaging 4.2.2 SP1 also works fine.

                              • 12. Re: JBoss messaging client timeout

                                my bad, JBM 1.4.2 SP1 works fine.

                                • 13. Re: JBoss messaging client timeout
                                  gaohoward

                                  That's great! Thanks for the updating.