1 2 Previous Next 20 Replies Latest reply on Aug 22, 2005 4:42 AM by Bela Ban

    message or message buffer is null followed by other exceptio

    Abby Pad Newbie

      Hi all -

      I have a cluster with JBoss Cache. Currently I am only running in the single server mode. Everytime my load gets higher I get the following messages continuouly in my logs. It always starts with the "message or message buffer is null" message for a few seconds, and then goes on to throw the other ones..which I think may be related.

      Apart from these messages, there are no other useful messages anywhere in the logs. There are no application related exceptions. What could be causing this problem? We are in production, and I need to fix this ASAP. We use TreeCache 1.2.3 (no AOP) and Jboss 3.2.6.

      2005-08-12 10:13:33,415 465540 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
      2005-08-12 10:13:33,418 465543 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
      2005-08-12 10:13:33,419 465544 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
      2005-08-12 10:13:33,419 465544 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
      2005-08-12 10:13:33,420 465545 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
      2005-08-12 10:13:33,423 465548 ERROR [org.jgroups.protocols.COMPRESS] (UpHandler (COMPRESS):) exception on uncompression: java.util.zip.DataForma
      ption: unknown compression method
      at java.util.zip.Inflater.inflateBytes(Native Method)
      at java.util.zip.Inflater.inflate(Inflater.java:219)
      at java.util.zip.Inflater.inflate(Inflater.java:236)
      at org.jgroups.protocols.COMPRESS.up(COMPRESS.java:95)
      at org.jgroups.stack.UpHandler.run(Protocol.java:59)

      2005-08-12 10:13:33,424 465549 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) exception=java.lang.IllegalArgumentException: java.io.Stream
      ptedException: invalid stream header
      2005-08-12 10:13:33,424 465549 ERROR [org.jgroups.protocols.COMPRESS] (UpHandler (COMPRESS):) exception on uncompression: java.util.zip.DataForma
      ption: unknown compression method
      at java.util.zip.Inflater.inflateBytes(Native Method)
      at java.util.zip.Inflater.inflate(Inflater.java:219)
      at java.util.zip.Inflater.inflate(Inflater.java:236)
      at org.jgroups.protocols.COMPRESS.up(COMPRESS.java:95)
      at org.jgroups.stack.UpHandler.run(Protocol.java:59)

      Thanks!

        • 1. Re: message or message buffer is null followed by other exce
          Abby Pad Newbie

          I'm sorry - I misspoke. I get the above exception when there are more than 1 jboss servers running in the cluster. When I shutdown all but one server. I no longer get the other exceptions. The only exception I get occasionally is:

          2005-08-12 10:47:05,891 ERROR [org.apache.commons.modeler.Registry] Error registering jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest95
          javax.management.InstanceAlreadyExistsException: jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest95 already registered.
          at org.jboss.mx.server.registry.BasicMBeanRegistry.add(BasicMBeanRegistry.java:704)
          at org.jboss.mx.server.registry.BasicMBeanRegistry.registerMBean(BasicMBeanRegistry.java:212)
          at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:324)
          at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
          at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:1018)
          at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:326)
          at org.apache.commons.modeler.Registry.registerComponent(Registry.java:871)
          at org.apache.jk.common.ChannelSocket.registerRequest(ChannelSocket.java:436)
          at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:443)
          at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:352)
          at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743)
          at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675)
          at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
          at java.lang.Thread.run(Thread.java:534)
          2005-08-12 10:47:05,894 WARN [org.apache.jk.common.ChannelSocket] Error registering request
          2005-08-12 10:48:04,392 ERROR [org.apache.commons.modeler.Registry] Error registering jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest179

          Thanks.

          • 2. Re: message or message buffer is null followed by other exce
            Abby Pad Newbie

            But I still need info on how to stop the jgroups related expceptions as we need to run the cluster with multiple servers.

            Please help!

            • 3. Re: message or message buffer is null followed by other exce
              Bela Ban Master

              #1 Set cache mode to LOCAL (rather than REPL_SYNC or REPL_ASYNC)
              #2 Remove COMPRESS from the protocol stack in cluster-service.xml
              #3 Upgrade to JGroups 2.2.8

              • 4. Re: message or message buffer is null followed by other exce
                Abby Pad Newbie

                Thanks for the response.

                I cant set cache mode to LOCAL as I need the cache to replicate. I dont get this problem with single server, I get in only in the cluster. So, I need to replicate my cache.

                I use JBossCache 1.2.3 and whatever JGroups that comes with that. That was what was recommended (the jboss-cache.jar and jgroups.jar should be the same). Are you suggesting I individually just change the JGroups?

                I am looking at my cluster-service.xml and I dont have COMPRESS. This is what I have:

                <!-- The JGroups protocol configuration -->


                <!-- UDP: if you have a multihomed machine,
                set the bind_addr attribute to the appropriate NIC IP address -->
                <!-- UDP: On Windows machines, because of the media sense feature
                being broken with multicast (even after disabling media sense)
                set the loopback attribute to true -->
                <!--UDP mcast_addr="228.1.2.3" mcast_port="45566"-->
                <UDP mcast_addr="229.1.2.3" mcast_port="45566"
                ip_ttl="32" ip_mcast="true"
                mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
                ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
                loopback="false" />
                <PING timeout="2000" num_initial_members="3"
                up_thread="true" down_thread="true" />
                <MERGE2 min_interval="10000" max_interval="20000" />
                <FD shun="true" up_thread="true" down_thread="true"
                timeout="2500" max_tries="5" />
                <VERIFY_SUSPECT timeout="3000" num_msgs="3"
                up_thread="true" down_thread="true" />
                <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
                max_xmit_size="8192"
                up_thread="true" down_thread="true" />
                <UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
                down_thread="true" />
                <pbcast.STABLE desired_avg_gossip="20000"
                up_thread="true" down_thread="true" />
                <FRAG frag_size="8192"
                down_thread="true" up_thread="true" />
                <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
                shun="true" print_local_addr="true" />
                <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />



                Thanks, Abitha.

                • 5. Re: message or message buffer is null followed by other exce
                  Bela Ban Master

                  I don't believe you :-):


                  2005-08-12 10:13:33,423 465548 ERROR [org.jgroups.protocols.COMPRESS] (UpHandler (COMPRESS):) exception on


                  This clearly shows you have COMPRESS. Check all of your config files, maybe you have a standalone JGroups app running somewhere ?

                  • 6. Re: message or message buffer is null followed by other exce
                    Abby Pad Newbie

                    Alright - you were right. I had COMPRESS in my jboss cache config file, and I removed it. It worked. Thank you very much.

                    I still get this exception, every few minutes in the log, although they dont seem to be doing anything (please see below).

                    Another problem we have in production, which is very bothersome is that: we have mod jk with jboss. When the load is high, after an hour or so (sometimes the time interval varies), mod jk gets locked up (ie, does not take in any more connections), and one or more of the appserver instances die with java defunct processes. I have to forcekill the app server instance to make the setup work again. This never happens in development, and always in production. Any insight on this will be very greatly appreciated.

                    Thanks!

                    -----

                    2005-08-16 10:47:52,415 WARN [org.apache.jk.common.ChannelSocket] Error registering request
                    2005-08-16 10:49:44,986 ERROR [org.apache.commons.modeler.Registry] Error registering jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest111

                    javax.management.InstanceAlreadyExistsException: jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest111 already registered.
                    at org.jboss.mx.server.registry.BasicMBeanRegistry.add(BasicMBeanRegistry.java:704)
                    at org.jboss.mx.server.registry.BasicMBeanRegistry.registerMBean(BasicMBeanRegistry.java:212)
                    at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                    at java.lang.reflect.Method.invoke(Method.java:324)
                    at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
                    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
                    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
                    at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
                    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
                    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
                    at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:1018)
                    at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:326)
                    at org.apache.commons.modeler.Registry.registerComponent(Registry.java:871)
                    at org.apache.jk.common.ChannelSocket.registerRequest(ChannelSocket.java:436)
                    at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:443)
                    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:352)
                    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743)
                    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675)
                    at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
                    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
                    at java.lang.Thread.run(Thread.java:534)
                    2005-08-16 10:49:44,987 WARN [org.apache.jk.common.ChannelSocket] Error registering request

                    • 7. Re: message or message buffer is null followed by other exce
                      Abby Pad Newbie

                      May I also ask why COMPRESS made a difference, and what it was supposed to do?

                      Thanks.

                      • 8. Re: message or message buffer is null followed by other exce
                        Bela Ban Master

                        Regarding the 2nd problem, please post on the Apache/Tomcat forum.

                        Regarding COMPRESS: this compresses packets and uncompresses them on the receiving side. If you have enough CPU power (e.g. a dual-CPU box), this might give you a huge performance gain *if* your data is easily compressable.
                        COMPRESS had a bug (if I recall in 2.2.7), which was fixed in 2.2.8. So simply upgrading should have done the job as well.

                        • 9. Re: message or message buffer is null followed by other exce
                          Abby Pad Newbie

                          Thank you Bela. I have posted in the other forums. One thing we noted yesterday was when we have the jboss die with defunct java process, the logs have a lot of the following error messages. Why and when do we get these messages? The jk log just says that "app02 is in error state" where app02 is one of the jboss servers. Can you help with this?

                          Thanks!

                          2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
                          52 not found in sent_msgs ! sent_msgs=[71274 - 71506]
                          2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
                          53 not found in sent_msgs ! sent_msgs=[71274 - 71506]
                          2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
                          54 not found in sent_msgs ! sent_msgs=[71274 - 71506]
                          2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
                          55 not found in sent_msgs ! sent_msgs=[71274 - 71506]
                          2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
                          56 not found in sent_msgs ! sent_msgs=[71274 - 71506]
                          2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
                          57 not found in sent_msgs ! sent_msgs=[71274 - 71506]
                          2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
                          58 not found in sent_msgs ! sent_msgs=[71274 - 71506]

                          • 10. Re: message or message buffer is null followed by other exce
                            Abby Pad Newbie

                            Also, interspersed with those NAKACK messages, I see the following (not necessarily at the start of the NAKACK MESSAGES).

                            java.io.IOException: Message too long
                            at java.net.PlainDatagramSocketImpl.send(Native Method)
                            at java.net.DatagramSocket.send(DatagramSocket.java:611)
                            at org.jgroups.protocols.UDP.doSend(UDP.java:858)
                            at org.jgroups.protocols.UDP.send(UDP.java:846)
                            at org.jgroups.protocols.UDP$OutgoingPacketHandler.handleMessage(UDP.java:1663)
                            at org.jgroups.protocols.UDP$OutgoingPacketHandler.run(UDP.java:1649)
                            at java.lang.Thread.run(Thread.java:534)

                            • 11. Re: message or message buffer is null followed by other exce
                              Bela Ban Master

                              This is strange because with the props you posted earlier, this should not occur as your frag size is 8192. Unless you are not using those props ?

                              • 12. Re: message or message buffer is null followed by other exce
                                Abby Pad Newbie

                                We are using those props in cluster-service.xml. The jboss config uses frag size of 60000. It also uses the max_xmit_size of 60000. Should this be changed?

                                Abitha.

                                • 13. Re: message or message buffer is null followed by other exce
                                  Abby Pad Newbie

                                  Here's our jboss config props (replSync-service.xml):


                                  <!-- JGroups protocol stack properties. Can also be a URL,
                                  e.g. file:/home/bela/default.xml

                                  -->



                                  <!-- UDP: if you have a multihomed machine,
                                  set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
                                  -->
                                  <!-- UDP: On Windows machines, because of the media sense feature
                                  being broken with multicast (even after disabling media sense)
                                  set the loopback attribute to true -->
                                  <!--UDP mcast_addr="228.1.2.3" mcast_port="48866"-->
                                  <UDP mcast_addr="229.1.2.3" mcast_port="45566"
                                  ip_ttl="32" ip_mcast="true"
                                  mcast_send_buf_size="10000000" mcast_recv_buf_size="10000000"
                                  ucast_send_buf_size="10000000" ucast_recv_buf_size="10000000"
                                  max_bundle_size="64000"
                                  max_bundle_timeout="30"
                                  use_incoming_packet_handler="false"
                                  use_outgoing_packet_handler="true"
                                  loopback="false"
                                  />
                                  <PING timeout="2000" down_thread="false" num_initial_members="3"/>
                                  <MERGE2 max_interval="10000" down_thread="false" min_interval="5000"/>
                                  <FD_SOCK down_thread="false"/>
                                  <VERIFY_SUSPECT timeout="1500" down_thread="false"/>
                                  <pbcast.NAKACK max_xmit_size="60000" down_thread="false" use_mcast_xmit="true" gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"/>
                                  <UNICAST timeout="300,600,1200,2400,3600" down_thread="false"/>
                                  <pbcast.STABLE stability_delay="1000" desired_avg_gossip="5000" down_thread="false" max_bytes="250000"/>
                                  <pbcast.GMS print_local_addr="true" join_timeout="3000" down_thread="false" join_retry_timeout="2000" shun="true"/>
                                  <FC max_credits="1000000" down_thread="false" min_threshold="0.10"/>
                                  <FRAG frag_size="60000" down_thread="false" up_thread="true"/>
                                  <pbcast.STATE_TRANSFER down_thread="false" up_thread="false"/>


                                  • 14. Re: message or message buffer is null followed by other exce
                                    Bela Ban Master

                                    Can you try with JGroups 2.2.9 ? Here's the error message you should be getting:

                                    9093 [ERROR] [TP.BundlingOutgoingPacketHandler thread] TP$BundlingOutgoingPacket
                                    Handler.run(): - failure in bundling
                                    java.lang.Exception: message size (100196) is greater than max bundling size (64
                                    000). Set the fragmentation/bundle size in FRAG and TP correctly
                                     at org.jgroups.protocols.TP$BundlingOutgoingPacketHandler.checkLength(TP
                                    .java:1264)
                                     at org.jgroups.protocols.TP$BundlingOutgoingPacketHandler.run(TP.java:1226)
                                     at java.lang.Thread.run(Thread.java:595)


                                    2.2.9 is currently CVS head.

                                    1 2 Previous Next