14 Replies Latest reply on Jan 17, 2012 4:49 AM by hushen.savani

    Min Large Message Size causing Invalid Stream Header

    jimmy.hui

      Hi,

       

      Using HornetQ2.1.1Final with JBoss 5AS.

       

      min-large-message-size is set to 1mb on the connection factories.

       

      Server 1 sends a message that is 250kb in size to Server 2. This is delivered via a Core Bridge between the 2 JMS queues

       

      Server 2 reports failure to process the message with the following exception:

       

      javax.ejb.EJBException: Error Processing Message: HornetQMessage[ID]:PERSISTENT
           ...
           at $Proxy213.onMessage()
           at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:257)
           at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:823)
           ...
      Caused by: javax.jms.JMSException: invalid stream header: 47000000
           at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:783)
           at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
           at org.hornetq.utils.ObjectInputStreamWithClassLoader.<init>(ObjectInputStreamWithClassLoader.java:40)
           at org.hornetq.jms.client.HornetQObjectMessage.getObject(HornetQObjectMessage.java:157)
           at MyMessageBean.onMessage(MyMessageBean.java:108)
           ... 54 more
      


      Once the exception is seen, you will see a message inside the largemessage folder on server 2 even though this message is not greater than 1mb (the new min-large-message-size)

       

      I can send a message that is less than 100kb (default min-large-message-size) and Server 2 will successfully consume it.

      I can send a message that is 80mb to Server 2 and it will successfully consume it. But it appears that sending a ObjectMessage that is between 100kb - 1mb i get this Invalid Stream Header exception.

       

      The journal size is left untouched (default 10mb) but I've set the journal buffer size to 2mb and I'm using NIO.

       

      Any ideas what i could be configuring wrong?

        • 1. Re: Min Large Message Size causing Invalid Stream Header
          jimmy.hui

          I updated the min-large-message-size to be 300kb and i can send a message that is 430kb in size but can not send one that is 143kb in size. Sending the 143kb message produces the same invalid stream header exception. 

          • 2. Re: Min Large Message Size causing Invalid Stream Header
            clebert.suconic

            Can you provide a testcase?

            • 3. Re: Min Large Message Size causing Invalid Stream Header
              jimmy.hui

              I'll have to get back to you for the testcase, unfortunately we don't have internet access where the dev happens..

               

              But the test simply involved updating the min-large-message-size and then sending a message that was bigger than 100kb but less than the new min-large-message-size in a deployed EJB app on JBoss

              • 4. Re: Min Large Message Size causing Invalid Stream Header
                clebert.suconic

                Before doing a test, can you validate if this will work with 2.1.2 (or 2.1.3 that should be out on the net few days).

                 

                 

                I have for instance a testcase on our dev tree testing that scenario and all works fine. Or this is something that will work on the latest version, or there is some nuance on your test that would cause a bug.

                 

                Also: Are you using MDBs? The setting for minLargeMEssageSize on that case would be on the inBoundConnection.

                 

                 

                Cheers,

                • 5. Re: Min Large Message Size causing Invalid Stream Header
                  jimmy.hui

                  I shall give 2.1.2 a try and see how that goes.

                   

                  And yes we are using MDBs, I have set the minLargeMessageSize on all connection factories at both Servers.

                  • 6. Re: Min Large Message Size causing Invalid Stream Header
                    jimmy.hui

                    Running on 2.1.2 did not solve the problem unforunately. I'm not too sure whats unique about our setup, mostly running default hornetq configurations, sends an ObjectMessage down to the queue, picked up by the core bridge, forwards to destination queue. Just don't quite understand why it works for small messages and large messages but not those in between the old and new minlargemessagesize setting.. shall try debug some more

                    • 7. Re: Min Large Message Size causing Invalid Stream Header
                      jimmy.hui

                      I created a new EJB application and deployed it, thinking that it fixed the problem. The MDB receives the message fine but the object inside the message is corrupt. So when i was only printing the message, it worked fine. But once i tried to retrieve the object inside the message, it failed.

                      • 8. Re: Min Large Message Size causing Invalid Stream Header
                        jimmy.hui

                        Hi Clebert,

                         

                        I've been debugging and looking into the code and here's what I've been finding.

                         

                        When the Core Bridge starts up it calls a method called createObjects() which creates the ClientSessionFactory object (org.hornetq.core.server.cluster.impl.BridgeImpl line 657). This calls HornetQClient.createClientSessionFactory(connectionPair.a, connectionPair.b) which creates a ClientSessionFactory using the default constructor. This sets the minLargeMessageSize to the default HornetQ value.

                         

                        Have I missed a configuration somewhere that will set this core bridge to create a clientSessionFactory with the user-specified minLargeMessageSize somewhere?

                         

                        I can see the value get set to the clientSessionFactory for the queue itself in Server1 when it creates the ObjectMessage and sends it so I know the value is set on that connectionFactory, it's just not getting set for the bridge.

                         

                        Cheers

                        • 9. Re: Min Large Message Size causing Invalid Stream Header
                          jimmy.hui

                          Has anyone else come across this issue? Modifying the BridgeImpl class to set the minLargeMessageSize of the ClientSessionFactory of 1mb prevents this issue in my test but i wouldn't want to have to keep updating this class to get around it.

                          • 10. Re: Min Large Message Size causing Invalid Stream Header
                            clebert.suconic

                            If you had provided a testcase as I asked on Sep 7h I would have taken a look already.

                            • 11. Re: Min Large Message Size causing Invalid Stream Header
                              sv_srinivaas

                              Hi Jimmy, I too have the very same issue, would like to know if you were able to find the cause of this issue? I'm using Hornetq 2.1.2 and Jboss 5.1.0

                              • 12. Re: Min Large Message Size causing Invalid Stream Header
                                jimmy.hui

                                I ended up debugging the code and found the miss-match in the minLargeMessageSize attribute when the core bridge session is created (http://community.jboss.org/message/561242#561242). At the moment we haven't worked around it and we're still using the default size. Although hard-coding the size in the BridgeImpl class (recompile hornetq and deploy with updated jars) seemed to bypass it if you really need to. We raised the issue with redhat support and were notified that the large message size mechanism is due to change though.

                                • 13. Re: Min Large Message Size causing Invalid Stream Header
                                  clebert.suconic

                                  There were a couple of fixed on clustering for large messages. Maybe you should try the tip of the branch?

                                  • 14. Re: Min Large Message Size causing Invalid Stream Header
                                    hushen.savani

                                    Hi Clebert,

                                     

                                         I've got the same problem as well. I am using JBossAS-6.1.1 and Hornetq-2.2.5.Final with Hornetq Clustering (2 nodes). Large Message Size in my application is around 3 MB. So, with the default min large message size, it was used to throw following exception:

                                    19:28:33,251 WARN  [org.hornetq.core.server.impl.QueueImpl] removing consumer which did not handle a message, consumer=org.hornetq.core.server.cluster.impl.ClusterConnectionBridge@66b2cd4f, message=Reference[3679]:RELIABLE:ServerMessage[messageID=3679,priority=4,expiration=[Tue Dec 27 19:28:43 IST 2011], durable=true, address=jms.queue.mocmMasterQueue,properties=TypedProperties[{_HQ_LARGE_SIZE=271064, _HQ_ROUTE_TOsf.mocm-cluster.afdcd7f8-3092-11e1-8e63-002128bc0745=[B@2abc60d6}]]: java.lang.NullPointerException
                                              at org.hornetq.core.buffers.impl.ResetLimitWrappedHornetQBuffer.<init>(ResetLimitWrappedHornetQBuffer.java:39) [:]
                                              at org.hornetq.core.message.impl.MessageImpl.getBodyBuffer(MessageImpl.java:267) [:]
                                              at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:223) [:]
                                              at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142) [:]
                                              at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:407) [:]
                                              at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2017) [:]
                                              at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1587) [:]
                                              at org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1472) [:]
                                              at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:72) [:]
                                              at org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2299) [:]
                                              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]
                                              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_12]
                                              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_12]
                                              at java.lang.Thread.run(Thread.java:619) [:1.6.0_12]

                                        

                                         So, I changed to min large message size to 5 MB. And it did the trick. Pl. refer to thread https://community.jboss.org/message/644232 for this.

                                     

                                         But, now following exception is thrown aribitarily even if min lage message size is greater than 3 MB:    

                                    14:49:06,223 ERROR [STDERR] javax.jms.JMSException: invalid stream header: 47000000

                                    14:49:06,223 ERROR [STDERR]           at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:783)

                                    14:49:06,223 ERROR [STDERR]           at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)

                                    14:49:06,223 ERROR [STDERR]           at org.hornetq.utils.ObjectInputStreamWithClassLoader.<init>(ObjectInputStreamWithClassLoader.java:40)

                                    14:49:06,224 ERROR [STDERR]           at org.hornetq.jms.client.HornetQObjectMessage.getObject(HornetQObjectMessage.java:157)

                                    14:49:06,224 ERROR [STDERR]           at com.elitecore.messageservice.ejb.messageserviceengine.messageservicelistener.mdb.MasterEntitiesQueueListenerMDB.onMessage(MasterEntitiesQueueListenerMDB.java:92)

                                    14:49:06,224 ERROR [STDERR]           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                                    14:49:06,224 ERROR [STDERR]           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                                    14:49:06,224 ERROR [STDERR]           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                                    14:49:06,225 ERROR [STDERR]           at java.lang.reflect.Method.invoke(Method.java:597)

                                    14:49:06,225 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)

                                    14:49:06,225 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)

                                    14:49:06,225 ERROR [STDERR]           at org.jboss.ejb3.interceptors.container.ContainerMethodInvocationWrapper.invokeNext(ContainerMethodInvocationWrapper.java:72)

                                    14:49:06,225 ERROR [STDERR]           at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76)

                                    14:49:06,226 ERROR [STDERR]           at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62)

                                    14:49:06,226 ERROR [STDERR]           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                                    14:49:06,226 ERROR [STDERR]           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                                    14:49:06,226 ERROR [STDERR]           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                                    14:49:06,226 ERROR [STDERR]           at java.lang.reflect.Method.invoke(Method.java:597)

                                    14:49:06,226 ERROR [STDERR]           at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)

                                    14:49:06,227 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,227 ERROR [STDERR]           at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:74)

                                    14:49:06,227 ERROR [STDERR]           at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_1861733324.invoke(InvocationContextInterceptor_z_fillMethod_1861733324.java)

                                    14:49:06,227 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,227 ERROR [STDERR]           at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:90)

                                    14:49:06,228 ERROR [STDERR]           at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_1861733324.invoke(InvocationContextInterceptor_z_setup_1861733324.java)

                                    14:49:06,228 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,228 ERROR [STDERR]           at org.jboss.ejb3.async.impl.interceptor.AsynchronousServerInterceptor.invoke(AsynchronousServerInterceptor.java:110)

                                    14:49:06,228 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,228 ERROR [STDERR]           at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)

                                    14:49:06,228 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,228 ERROR [STDERR]           at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.ejb3.core.context.InvocationContextAdapter.proceed(InvocationContextAdapter.java:70)

                                    14:49:06,229 ERROR [STDERR]           at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:234)

                                    14:49:06,230 ERROR [STDERR]           at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:329)

                                    14:49:06,230 ERROR [STDERR]           at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.invoke(CMTTxInterceptor.java:207)

                                    14:49:06,230 ERROR [STDERR]           at org.jboss.ejb3.tx2.aop.CMTTxInterceptorWrapper.invoke(CMTTxInterceptorWrapper.java:52)

                                    14:49:06,230 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,230 ERROR [STDERR]           at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)

                                    14:49:06,230 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,230 ERROR [STDERR]           at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:79)

                                    14:49:06,230 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.ejb3.core.context.CurrentInvocationContextInterceptor.invoke(CurrentInvocationContextInterceptor.java:47)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.ejb3.interceptor.EJB3TCCLInterceptor.invoke(EJB3TCCLInterceptor.java:86)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

                                    14:49:06,231 ERROR [STDERR]           at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:306)

                                    14:49:06,232 ERROR [STDERR]           at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:299)

                                    14:49:06,232 ERROR [STDERR]           at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:152)

                                    14:49:06,232 ERROR [STDERR]           at $Proxy150.onMessage(Unknown Source)

                                    14:49:06,232 ERROR [STDERR]           at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)

                                    14:49:06,232 ERROR [STDERR]           at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:866)

                                    14:49:06,232 ERROR [STDERR]           at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:44)

                                    14:49:06,232 ERROR [STDERR]           at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:983)

                                    14:49:06,232 ERROR [STDERR]           at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

                                    14:49:06,233 ERROR [STDERR]           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                                    14:49:06,233 ERROR [STDERR]           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                                    14:49:06,233 ERROR [STDERR]           at java.lang.Thread.run(Thread.java:619)

                                     

                                         However, message size in my application is constant everytime, i.e. around 3 MB. But, above exception comes aribitarily, and my system crashes.

                                     

                                         Can there be any work around for this as of now? Pl. suggest.

                                     

                                         Thanks.