13 Replies Latest reply on Aug 15, 2012 8:30 AM by ericrrobinson

    java.lang.IndexOutOfBoundsException when using a stomp client

    ericrrobinson

      I am getting the following stack trace from hornetq when running in JBoss 7.1.1

       

      11:49:10,133 WARN  [org.hornetq.core.server.impl.QueueImpl] (Old I/O server worker (parentId: 222337051, [id: 0x0d40981b, /0.0.0.0:5445])) removing consumer which did not handle a message, consumer=or

      g.hornetq.core.server.impl.ServerConsumerImpl@605c90b4, message=Reference[1323]:RELIABLE:ServerMessage[messageID=1323,priority=4, bodySize=210,expiration=0, durable=true, address=jms.topic.testTopic2,

      properties=TypedProperties[null]]@1713639557: java.lang.IndexOutOfBoundsException

              at org.jboss.netty.buffer.AbstractChannelBuffer.setIndex(AbstractChannelBuffer.java:67) [netty-3.2.6.Final.jar:]

              at org.hornetq.core.buffers.impl.ChannelBufferWrapper.setIndex(ChannelBufferWrapper.java:497) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.message.impl.MessageImpl.forceCopy(MessageImpl.java:960) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.message.impl.MessageImpl.encodeToBuffer(MessageImpl.java:891) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.message.impl.MessageImpl.getEncodedBuffer(MessageImpl.java:486) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.protocol.core.impl.wireformat.SessionReceiveMessage.encode(SessionReceiveMessage.java:73) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:183) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:162) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.protocol.core.impl.CoreSessionCallback.sendMessage(CoreSessionCallback.java:76) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:798) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:313) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2195) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.server.impl.QueueImpl.deliverDirect(QueueImpl.java:2140) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.server.impl.QueueImpl.addTail(QueueImpl.java:450) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.postoffice.impl.PostOfficeImpl.addReferences(PostOfficeImpl.java:1255) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.postoffice.impl.PostOfficeImpl.access$200(PostOfficeImpl.java:79) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.postoffice.impl.PostOfficeImpl$1.done(PostOfficeImpl.java:1077) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:188) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.persistence.impl.journal.JournalStorageManager.afterCompleteOperations(JournalStorageManager.java:447) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1066) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:678) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:580) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1437) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1164) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:441) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:508) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:556) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:517) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:533) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73) [hornetq-core-2.2.13.Final.jar:]

              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:534) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287) [netty-3.2.6.Final.jar:]

              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-2.2.13.Final.jar:]

              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-2.2.13.Final.jar:]

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:367) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:100) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) [netty-3.2.6.Final.jar:]

              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181) [netty-3.2.6.Final.jar:]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_03]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_03]

              at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_03]

       

      I have attached my standalone.xml as well as example code that can reproduce the issue.

       

      You need to start the StompClient, MyProducer, then the MyMessageConsumer in that order and one right after the other.

       

      I have changed the forceCopy method in MessageImpl.java to the following to prevent the exception, but not sure if it is just hiding the larger issue.

       

         private void forceCopy()

         {

            // Must copy buffer before sending it

            buffer = buffer.copy(0, buffer.capacity());

           

            if (endOfBodyPosition != -1)

            {

                buffer.setIndex(0, endOfBodyPosition);

            }

       

            if (bodyBuffer != null)

            {

               bodyBuffer.setBuffer(buffer);

            }

       

            bufferUsed = false;

         }

        • 1. Re: java.lang.IndexOutOfBoundsException when using a stomp client
          clebert.suconic

          Another user reported me this as well. There's an issue with not protecting the buffer creation on stomp. If you do a synchronized (message) { message.forceCopy()} it will probalby fix it.

           

           

          We should be doing some major work with Stomp soon and I will make sure this will be fixed. I should have a fix for you on monday / tuesday.

          • 2. Re: java.lang.IndexOutOfBoundsException when using a stomp client
            ericrrobinson

            Thanks for the reply Clebert. I will give that a try and let you know how it goes.

            • 3. Re: java.lang.IndexOutOfBoundsException when using a stomp client
              ericrrobinson

              I have been looking over the hornetq code and I guess I am not sure where you think the synchronized(message) should go. The only place I see the forceCopy being called is in MessageImpl.java

              • 4. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                ericrrobinson

                This does not look to be a synchronization issue. I believe the issues is because the StompSession is changing the Buffer indexes when its creating the frame. This calls bodyChanged on the message which invalidates the buffer and sets the endOfBodyPosition to -1. The invalid buffer then causes forceCopy to be called for the next consumer that calls encodeToBuffer. In forceCopy it tries to set the indexes of the new buffer to endOfBodyPosition which is -1 and causes the indexOutOfBoundsException.

                 

                Looks like a possible fix is to check the endOfBodyPosition for -1 in forceCopy and if its -1 save off the buffer.writeIndex before doing the buffer.copy so we can set the new buffer.writeIndex with the original position.

                • 5. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                  ericrrobinson

                  Here is what I finally changed the forceCopy in MessageImpl.java

                   

                  private void forceCopy()

                     {

                        //Buffer could of been invalidated causing the endOfBodyPosition to be set to -1

                        //So save off the writerIndex before the copy cause it will be lost on the copy

                        if (endOfBodyPosition == -1) {

                           endOfBodyPosition = buffer.writerIndex();

                        }

                       

                        // Must copy buffer before sending it

                        buffer = buffer.copy(0, buffer.capacity());

                       

                        //Set the indexes of the new copied buffer.

                        buffer.setIndex(0, endOfBodyPosition);

                       

                   

                   

                        if (bodyBuffer != null)

                        {

                           bodyBuffer.setBuffer(buffer);

                        }

                   

                   

                        bufferUsed = false;

                     }

                  • 6. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                    jbertram

                    I'm attempting to reproduce your error with the test.zip you attached.  I got everything running, but I haven't seen the IndexOutOfBoundsException yet.  I've run the producer until it stops a few times and the only exception I see is this:

                     

                    12:29:06,553 ERROR [org.jboss.security.authentication.JBossCachedAuthenticationManager] (Old I/O server worker (parentId: 9143267, [id: 0x008b83e3, /0.0.0.0:61613])) Login failure: javax.security.auth.login.FailedLoginException: Password Incorrect/Password Required
                              at org.jboss.security.auth.spi.UsernamePasswordLoginModule.login(UsernamePasswordLoginModule.java:270) [picketbox-4.0.7.Final.jar:4.0.7.Final]
                              at org.jboss.security.auth.spi.UsersRolesLoginModule.login(UsersRolesLoginModule.java:155) [picketbox-4.0.7.Final.jar:4.0.7.Final]
                              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_29]
                              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_29]
                              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_29]
                              at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_29]
                              at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769) [rt.jar:1.6.0_29]
                              at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186) [rt.jar:1.6.0_29]
                              at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683) [rt.jar:1.6.0_29]
                              at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_29]
                              at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) [rt.jar:1.6.0_29]
                              at javax.security.auth.login.LoginContext.login(LoginContext.java:579) [rt.jar:1.6.0_29]
                              at org.jboss.security.authentication.JBossCachedAuthenticationManager.defaultLogin(JBossCachedAuthenticationManager.java:449) [picketbox-infinispan-4.0.7.Final.jar:4.0.7.Final]
                              at org.jboss.security.authentication.JBossCachedAuthenticationManager.proceedWithJaasLogin(JBossCachedAuthenticationManager.java:383) [picketbox-infinispan-4.0.7.Final.jar:4.0.7.Final]
                              at org.jboss.security.authentication.JBossCachedAuthenticationManager.authenticate(JBossCachedAuthenticationManager.java:371) [picketbox-infinispan-4.0.7.Final.jar:4.0.7.Final]
                              at org.jboss.security.authentication.JBossCachedAuthenticationManager.isValid(JBossCachedAuthenticationManager.java:160) [picketbox-infinispan-4.0.7.Final.jar:4.0.7.Final]
                              at org.jboss.as.messaging.HornetQSecurityManagerAS7.validateUser(HornetQSecurityManagerAS7.java:39) [jboss-as-messaging-7.1.1.Final.jar:7.1.1.Final]
                              at org.hornetq.core.protocol.stomp.StompProtocolManager.onConnect(StompProtocolManager.java:620) [hornetq-core-2.2.13.Final.jar:]
                              at org.hornetq.core.protocol.stomp.StompProtocolManager.handleBuffer(StompProtocolManager.java:192) [hornetq-core-2.2.13.Final.jar:]
                              at org.hornetq.core.protocol.stomp.StompConnection.bufferReceived(StompConnection.java:284) [hornetq-core-2.2.13.Final.jar:]
                              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:533) [hornetq-core-2.2.13.Final.jar:]
                              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73) [hornetq-core-2.2.13.Final.jar:]
                              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100) [netty-3.2.6.Final.jar:]
                              at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372) [netty-3.2.6.Final.jar:]
                              at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:367) [netty-3.2.6.Final.jar:]
                              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) [netty-3.2.6.Final.jar:]
                              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) [netty-3.2.6.Final.jar:]
                              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:100) [netty-3.2.6.Final.jar:]
                              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.2.6.Final.jar:]
                              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) [netty-3.2.6.Final.jar:]
                              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181) [netty-3.2.6.Final.jar:]
                              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_29]
                              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_29]
                              at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29]
                    

                     

                    After which the StompClient stops receiving messages.  This exception is odd considering that you have <security-enabled>false</security-enabled> in your standalone.xml.  This looks like a bug in either the Stomp implementation code or in the AS7 messaging subsystem code.  In any event, I can't reproduce the error.

                    • 7. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                      jbertram

                      FYI - I fixed the FailedLoginException issue in my branch and managed to reproduce the IndexOutOfBoundsException.  I'm investigating further at this point.

                      • 8. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                        ericrrobinson

                        Thanks for looking into it Justin.

                        • 9. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                          jbertram

                          Just FYI, I've got a test-case in the HornetQ test-suite in my branch now that I believe reproduces this problem (and perhaps more). 

                          • 10. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                            jbertram

                            One more thing...I'm using your fix from this thread, but there still appears to be a race condition somewhere that I'm seeing in my test that you probably aren't hitting (yet).

                            • 11. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                              clebert.suconic

                              yeah.. we have been pair programming (me and Justin) and we found the real issue.. Justin should be pushing the real fix soon.

                               

                              @Eric: your fix cured a symptom but not the real issue. We were supposed to make a buffer copy without affecting the core buffer on stomp.

                              • 12. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                                jbertram

                                FYI - HORNETQ-1007 was opened for this issue and has been resolved.  You can see all the git pull requests on the JIRA as the issue has been fixed on all 3 major HornetQ branches.

                                • 13. Re: java.lang.IndexOutOfBoundsException when using a stomp client
                                  ericrrobinson

                                  Thanks for looking into this Clebert and Justin.