1 Reply Latest reply on Sep 23, 2010 1:46 PM by whitingjr

    Socket timed out on ResourceManager.commitLocal

    whitingjr

      Hi,

      I am running the SPECjms2007 benchmark against JBM with a MySQL database. There is one resource manager in the SUT and the database is configured using local transaction.

       

      The benchmark pushes a variety of message type/size and load. The demand on the messaging server is at (low) BASE 35 for the horizontal topology.

      The benchmark is not completing during warmup due to an socket timeout when processing a message that is sent transactionally.

       

      I have not investigated yet but initial thoughts are there is a deadlock when the server is updating the database. Am I on the right track do you think ?

       

      HQ_Interaction3DR_1_EHID_1: Uncaught exception.
      org.jboss.jms.exception.MessagingTransactionRolledBackException
          at org.jboss.jms.tx.ResourceManager.commitLocal(ResourceManager.java:225)
          at org.jboss.jms.client.container.SessionAspect.handleCommit(SessionAspect.java:582)
          at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleCommit_2058061115.invoke(SessionAspect_z_handleCommit_2058061115.java)
          at org.jboss.jms.client.delegate.ClientSessionDelegate$commit_8461082169793485964.invokeNext(ClientSessionDelegate$commit_8461082169793485964.java)
          at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
          at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
          at org.jboss.jms.client.delegate.ClientSessionDelegate$commit_8461082169793485964.invokeNext(ClientSessionDelegate$commit_8461082169793485964.java)
          at org.jboss.jms.client.delegate.ClientSessionDelegate.commit(ClientSessionDelegate.java)
          at org.jboss.jms.client.JBossSession.commit(JBossSession.java:164)
          at org.spec.jms.eventhandler.hq.HQ_Interaction3DR.oneIteration(HQ_Interaction3DR.java:134)
          at org.spec.perfharness.WorkerThread.pace(WorkerThread.java:339)
          at org.spec.jms.agents.SPECWorkerThread.pace(SPECWorkerThread.java:979)
          at org.spec.jms.agents.SPECWorkerThread.run(SPECWorkerThread.java:750)
          at org.spec.jms.eventhandler.hq.HQ_Interaction3DR.run(HQ_Interaction3DR.java:98)
      Caused by: org.jboss.jms.exception.MessagingNetworkFailureException
          at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:245)
          at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
          at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
          at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$sendTransaction$aop(ClientConnectionDelegate.java:225)
          at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(Cli
      entConnectionDelegate$sendTransaction_N3268650789275322226.java)
          at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
          at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
          at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
          at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
          at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
          at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
          at org.jboss.jms.client.delegate.ClientConnectionDelegate.sendTransaction(ClientConnectionDelegate.java)
          at org.jboss.jms.tx.ResourceManager.commitLocal(ResourceManager.java:206)
          ... 13 more
      Caused by: org.jboss.remoting.InvocationFailureException: Socket timed out.  Waited 300000 milliseconds for response while calling on InvokerLocator [bisocket://hornetq:4457/?JBM_clientMaxPoolSize=1000&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=300000&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]; nested exception is:
          java.net.SocketTimeoutException: Read timed out
          at org.jboss.remoting.transport.socket.SocketClientInvoker.handleException(SocketClientInvoker.java:137)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.handleOtherException(MicroSocketClientInvoker.java:1079)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:941)
          at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(B
      isocketClientInvoker.java:461)
          at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
          at org.jboss.remoting.Client.invoke(Client.java:2034)
          at org.jboss.remoting.Client.invoke(Client.java:877)
          at org.jboss.remoting.Client.invoke(Client.java:865)
          at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
          ... 24 more
      Caused by: java.net.SocketTimeoutException: Read timed out
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(SocketInputStream.java:146)
          at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
          at java.io.FilterInputStream.read(FilterInputStream.java:83)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1342)
          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:895)
          ... 30 more

       

      os: RHEL 5.5  x86_64

      database: MySQL 5.1.50 community

      JBM: 1.4.7.GA

       

      Regards,

      Jeremy

        • 1. Re: Socket timed out on ResourceManager.commitLocal
          whitingjr

          This turned out to be nothing to do with transactions.

          It was a mistake on my part deploying identical versions of the messaging libraries on the client and server.
          The error message in the logs on the server side indicated the reason for the read timeout. The boolean field member called checkForDuplicates was not being transported across the wire by the client.

           

          2010-09-23 12:47:05,002 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#63[10.0.1.1:56283]) WorkerThread#63[10.0.1.1:56283] failed
          java.io.IOException: Read timed out
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.read(SocketInputStream.java:146)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
              at java.io.DataInputStream.readBoolean(DataInputStream.java:242)
              at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.read(ConnectionSendTransactionRequest.java:69)
              at org.jboss.jms.wireformat.JMSWireFormat.read(JMSWireFormat.java:298)
              at org.jboss.remoting.transport.socket.ServerThread.versionedRead(ServerThread.java:902)
              at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:754)
              at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
              at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
              at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
          2010-09-23 12:47:05,002 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#13[10.0.1.1:56123]) WorkerThread#13[10.0.1.1:56123] failed
          java.io.IOException
              at java.io.DataInputStream.readBoolean(DataInputStream.java:244)
              at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.read(ConnectionSendTransactionRequest.java:69)
              at org.jboss.jms.wireformat.JMSWireFormat.read(JMSWireFormat.java:298)
              at org.jboss.remoting.transport.socket.ServerThread.versionedRead(ServerThread.java:902)
              at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:754)
              at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
              at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
              at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

           

          Jeremy