12 Replies Latest reply on Aug 19, 2011 7:14 AM by fabrice.dossin

    Failed to decode

    fabrice.dossin

      Hi,

       

      I am really stuck with a problem and I need some help to give me advices where to look...

      I have some jboss-6 instances connected to one jboss-6 instance acting as the central server.

      The link between the clients jboss et the server jboss is done by core-bridges.

      I use JMS APIs with EJBs (MDB listener on topics)

       

      One the client, a process post message on a topic forwarded to the server.

      There is also a listeners on this topic on the client instance.

       

      I receive a lot of the following error on client side:

       

      {noformat}

      2011-06-27 09:40:39,360 ERROR [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (Thread-20 (group:HornetQ-client-global-threads-24355971)) Failed to decode: java.lang.IllegalArgumentException: Invalid type: -106

                at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:491) [:6.0.0.Final]

                at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:379) [:6.0.0.Final]

                at org.hornetq.core.client.impl.FailoverManagerImpl$DelegatingBufferHandler.bufferReceived(FailoverManagerImpl.java:1134) [:6.0.0.Final]

                at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:135) [:6.0.0.Final]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.0.0.Final]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_24]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_24]

                at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]

      {noformat}

       

       

       

       

      The invalid type is sometimes 0, -106, -104, 82, -28 etc... This does not seem to follow a specific rules.

       

      I have also some

       

      {noformat}

      2011-06-24 13:40:09,229 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-15 (group:HornetQ-client-global-threads-2083463303)) Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

         at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleStringInternal(ChannelBufferWrapper.java:83) [:6.0.0.Final]

         at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleString(ChannelBufferWrapper.java:77) [:6.0.0.Final]

         at org.hornetq.core.protocol.core.impl.wireformat.SessionProducerCreditsMessage.decodeRest(SessionProducerCreditsMessage.java:75) [:6.0.0.Final]

         at org.hornetq.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:235) [:6.0.0.Final]

         at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:495) [:6.0.0.Final]

         at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:379) [:6.0.0.Final]

         at org.hornetq.core.client.impl.FailoverManagerImpl$DelegatingBufferHandler.bufferReceived(FailoverManagerImpl.java:1134) [:6.0.0.Final]

         at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:135) [:6.0.0.Final]

         at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.0.0.Final]

         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_24]

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_24]

         at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]

      {noformat}

       

       

       

      At this time the JVM heap space is not full (170/480Mo)

       

      I know it is not very much information. Maybe someone could tell me where I could look to go further ?

       

      Kind regards,

      Fabrice

        • 1. Re: Failed to decode
          clebert.suconic

          You have a version mismatch.

           

          We are only guaranteeing client-server compatibility post 2.2.2.

           

           

          Upgrade your client libraries and this should go away.

          • 2. Re: Failed to decode
            fabrice.dossin

            Hi,

             

            As a temporary fix I have removed all my mdb listeners from my client jboss instance. Once this is done, there is no more exception into the jboss client instance. The bridge works as expected.

             

            Today I have updated the two jboss to a jboss-6.1.0-SNAPSHOT with HornetQ 2.2.5 and replugged my mdb listener in the client.

             

            Messages are forwarded trough the bridge and my mdb receives messages but I still have this exception

             

             

            {noformat}

            2011-07-28 15:41:39,030 ERROR [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (Thread-10 (group:HornetQ-client-global-threads-10140152)) Failed to decode: java.lang.IllegalArgumentException: Invalid type: 0

              at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:529) [:]

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:424) [:]

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1302) [:]

              at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:137) [:]

              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_26]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_26]

              at java.lang.Thread.run(Thread.java:662) [:1.6.0_26]

            {noformat}

             

            The type is now always 0 and I did not see the second stack until now.

             

            The message flow is:

            Client: Message producer -> Client: Message consumer

                                                   -> Client: Bridge -> Server: bridge -> Server: Message consumer

             

            There is not any hornetq libraries into my ear and the two servers are exactly the same.

             

            Does someone has an other clue. I should miss something...

             

            Kind regards,

            Fabrice

            • 3. Re: Failed to decode
              fabrice.dossin

              Hi,

               

              I did gather more information. Maybe this will help someone to identify the problem.

               

              The problem occurs only when all the following condition are true:

              - The messages are sent from a JavaEE bean with an inVM connector, either with @Resource(mappedName = "java:/ConnectionFactory) or with

              @Resource(mappedName = "java:/JmsXA").

              - There is at least one JavaEE MDB listener on this queue

              - The message is sent on a queue that is bridged to another server

               

              If I do not have an mdb on the bridged queue, no problem. I use this as temporary solution but it is ugly:

              I send my message on a local queue and on a bridged queue and put my mdb listeners on the local queue.

               

              Another solution is to use a NettyConnectionFactory. If I use the resource @Resource(mappedName = "/ConnectionFactory) but my sending is then not transactional, worst...

               

              I tried to use NettyConnectionFactory into JmsXA by configuring it inside jms-ra.rar/META-INF/ra.xml but then I have very fastly some

              14:52:12,378 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@49d60362[local= /127.0.0.1:57842, remote=localhost/127.0.0.1:5446] [code=3]

               

              This last round of tests where done with jboss-6.0.0 (hornetq 2.1.2) but it should be the same with 2.2.5 since I also have the failed to decode error.

               

              I hope someone could look into it.

              If I can help let me know what I can do.

               

              Kind regards,

              Fabrice

              • 4. Re: Failed to decode
                clebert.suconic

                You have some version mess at your system.

                 

                It should be fairly simple. Just update your jars to a single version on server and client.

                • 5. Re: Failed to decode
                  fabrice.dossin

                  Hi,

                   

                  I do not. I reproduce this either with two jboss-6.0.0-Final server either with two jboss-6.1.0-SNAPSHOT (#82), same server version on both side.

                  Moreover, this problem occurs inside the server and there is no jars into my ear so it could have only one version of hornetq and netty.

                  Version mismatch is impossible.

                   

                  Fabrice

                  • 6. Re: Failed to decode
                    clebert.suconic

                    Look inside the Resource Adapter. Maybe you didn't update it?

                     

                     

                    This is quite a simple use case. if this was broken.. it would be broken for everybody.

                     

                     

                    You would need to provide some way to replicate this, if you want to prove that this is not an environment mismatch for you.

                    • 7. Re: Failed to decode
                      fabrice.dossin

                      Hi,

                       

                      It took me lots of time... You will find the needed files to replicate this in attachment.

                      There is a README file in the archive. Do not hesitate to contact me if I can help.

                       

                      Kind regards,

                      Fabrice

                      • 8. Re: Failed to decode
                        clebert.suconic

                        Can you provide me a detailed step, robot proof? You assumed in your instructions that I know your topology (or that I need to understand it before I run it). Include on where did you download these versions from.

                         

                         

                        The fact that you are running different versions (6.0 versus 6.1) tells me you have version mismatch between them. as I told you before HornetQ only has version compatibility after 2.2+, and server always need to be > client.

                        • 9. Re: Failed to decode
                          fabrice.dossin

                          Hi,

                           

                          Sorry for the lack of information.

                          The topoligy is:

                           

                          Jboss instance client (machine 1)

                          Topic

                          - topic/testQueue

                          MDB

                          - TestListener on topic/testQueue

                          Core Bridge to Jboss instance server on machine 2

                          - topic/testQueue

                           

                          Jboss instance server (machine 2)

                          Topic

                          - topic/testQueue

                          MDB

                          - TestListener on topic/testQueue

                           

                          I have done my test with jboss-6.1.0-SNAPSHOT build #80 on jenkins.

                          I propose you to download the last build, it should lead to same results.

                          https://hudson.jboss.org/jenkins/view/JBoss%20AS/job/JBoss-AS-6.1.x/lastSuccessfulBuild/artifact/JBossAS_6_1/build/target/jboss-6.1.x.zip

                           

                          Install JBoss on machine 1.

                          Copy hornetq config from testcase hornetq/client/config/jboss_6_1/hornetq to jboss-6.1.0-SNAPSHOT/server/default/deploy/hornetq

                          Edit the hornetq-configuration.xml file to set the proper machine 2 IP.

                          Copy war file hornetq/client/target/hornetq_client.war in instance deploy folder.

                           

                          Install JBoss on machine 2.

                          Copy hornetq config from testcase hornetq/server/config/jboss_6_1/hornetq to jboss-6.1.0-SNAPSHOT/server/default/deploy/hornetq

                          Copy war file hornetq/client/target/hornetq_server.war in instance deploy folder.

                           

                          Start JBoss on machine 2.

                          Start JBoss on machine 1.

                           

                          Open browser on machine 1 to launch the servlet test :

                          http://localhost:8080/hornetq_client/test?type=5&queue=1

                           

                          Check the bridge is ok by looking at log of machine 2. It should show received messages.

                          Look at the log of machine 1 for the stack and relaunch the test if no stack at the first try.

                           

                          You should sometimes see on client side the following stack:

                           

                          {noformat}

                          13:17:05,301 GRAVE [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Failed to decode: java.lang.IllegalArgumentException: Invalid type: 88

                                    at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:529) [:]

                                    at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:424) [:]

                                    at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1302) [:]

                                    at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:137) [:]

                                    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]

                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_26]

                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_26]

                                    at java.lang.Thread.run(Thread.java:680) [:1.6.0_26]

                           

                          {noformat}

                           

                          or sometimes

                           

                           

                          {noformat}

                          13:16:59,175 GRAVE [org.hornetq.utils.OrderedExecutorFactory] Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

                                    at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleStringInternal(ChannelBufferWrapper.java:83) [:]

                                    at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleString(ChannelBufferWrapper.java:77) [:]

                                    at org.hornetq.core.protocol.core.impl.wireformat.SessionCreateConsumerMessage.decodeRest(SessionCreateConsumerMessage.java:116) [:]

                                    at org.hornetq.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:247) [:]

                                    at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:533) [:]

                                    at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:424) [:]

                                    at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1302) [:]

                                    at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:137) [:]

                                    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]

                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_26]

                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_26]

                                    at java.lang.Thread.run(Thread.java:680) [:1.6.0_26]

                          {noformat}

                           

                                   

                          When these errors occur, the message is never sent to local MDB. However, during my test the listener on machine 2 receives the messages.

                           

                          Kind regards,

                          Fabrice

                          • 10. Re: Failed to decode
                            clebert.suconic

                            So, ou have two jboss 6.1 being used here? same exact version?

                             

                            That's contraditory to what you wrote on the readme. Is that what you're doing now?

                            • 11. Re: Failed to decode
                              fabrice.dossin

                              Hi,

                               

                              Sorry if I was not clear enough.

                              What you see in the test case is the possibility to test either with two jboss-6.0 either with two jboss-6.1 but I know I have to use the same server version on both side.

                               

                              Since the hornetq provided with jboss 6.0 is old, I think the test case with two jboss-6.1 is better.

                               

                              Kind regards,

                              Fabrice

                              • 12. Re: Failed to decode
                                fabrice.dossin

                                Hi,

                                 

                                Did you manage to reproduce it ?

                                If you need someting else, ask me.

                                 

                                Kind regards,

                                Fabrice