1 2 Previous Next 20 Replies Latest reply on Aug 22, 2005 4:42 AM by belaban

message or message buffer is null followed by other exceptio

abitha Aug 12, 2005 1:25 PM

Hi all -

I have a cluster with JBoss Cache. Currently I am only running in the single server mode. Everytime my load gets higher I get the following messages continuouly in my logs. It always starts with the "message or message buffer is null" message for a few seconds, and then goes on to throw the other ones..which I think may be related.

Apart from these messages, there are no other useful messages anywhere in the logs. There are no application related exceptions. What could be causing this problem? We are in production, and I need to fix this ASAP. We use TreeCache 1.2.3 (no AOP) and Jboss 3.2.6.

2005-08-12 10:13:33,415 465540 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
2005-08-12 10:13:33,418 465543 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
2005-08-12 10:13:33,419 465544 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
2005-08-12 10:13:33,419 465544 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
2005-08-12 10:13:33,420 465545 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) message or message buffer is null
2005-08-12 10:13:33,423 465548 ERROR [org.jgroups.protocols.COMPRESS] (UpHandler (COMPRESS):) exception on uncompression: java.util.zip.DataForma
ption: unknown compression method
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:219)
at java.util.zip.Inflater.inflate(Inflater.java:236)
at org.jgroups.protocols.COMPRESS.up(COMPRESS.java:95)
at org.jgroups.stack.UpHandler.run(Protocol.java:59)

2005-08-12 10:13:33,424 465549 ERROR [org.jgroups.blocks.RpcDispatcher] (Thread-44:) exception=java.lang.IllegalArgumentException: java.io.Stream
ptedException: invalid stream header
2005-08-12 10:13:33,424 465549 ERROR [org.jgroups.protocols.COMPRESS] (UpHandler (COMPRESS):) exception on uncompression: java.util.zip.DataForma
ption: unknown compression method
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:219)
at java.util.zip.Inflater.inflate(Inflater.java:236)
at org.jgroups.protocols.COMPRESS.up(COMPRESS.java:95)
at org.jgroups.stack.UpHandler.run(Protocol.java:59)

Thanks!

1. Re: message or message buffer is null followed by other exce

abitha Aug 12, 2005 1:53 PM (in response to abitha)

I'm sorry - I misspoke. I get the above exception when there are more than 1 jboss servers running in the cluster. When I shutdown all but one server. I no longer get the other exceptions. The only exception I get occasionally is:

2005-08-12 10:47:05,891 ERROR [org.apache.commons.modeler.Registry] Error registering jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest95
javax.management.InstanceAlreadyExistsException: jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest95 already registered.
at org.jboss.mx.server.registry.BasicMBeanRegistry.add(BasicMBeanRegistry.java:704)
at org.jboss.mx.server.registry.BasicMBeanRegistry.registerMBean(BasicMBeanRegistry.java:212)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:1018)
at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:326)
at org.apache.commons.modeler.Registry.registerComponent(Registry.java:871)
at org.apache.jk.common.ChannelSocket.registerRequest(ChannelSocket.java:436)
at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:443)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:352)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534)
2005-08-12 10:47:05,894 WARN [org.apache.jk.common.ChannelSocket] Error registering request
2005-08-12 10:48:04,392 ERROR [org.apache.commons.modeler.Registry] Error registering jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest179

Thanks.
Actions
2. Re: message or message buffer is null followed by other exce

abitha Aug 12, 2005 1:54 PM (in response to abitha)

But I still need info on how to stop the jgroups related expceptions as we need to run the cluster with multiple servers.

Please help!
Actions
3. Re: message or message buffer is null followed by other exce

belaban Aug 15, 2005 3:45 AM (in response to abitha)

#1 Set cache mode to LOCAL (rather than REPL_SYNC or REPL_ASYNC)
#2 Remove COMPRESS from the protocol stack in cluster-service.xml
#3 Upgrade to JGroups 2.2.8
Actions
4. Re: message or message buffer is null followed by other exce

abitha Aug 15, 2005 1:02 PM (in response to abitha)

Thanks for the response.

I cant set cache mode to LOCAL as I need the cache to replicate. I dont get this problem with single server, I get in only in the cluster. So, I need to replicate my cache.

I use JBossCache 1.2.3 and whatever JGroups that comes with that. That was what was recommended (the jboss-cache.jar and jgroups.jar should be the same). Are you suggesting I individually just change the JGroups?

I am looking at my cluster-service.xml and I dont have COMPRESS. This is what I have:






<UDP mcast_addr="229.1.2.3" mcast_port="45566"
ip_ttl="32" ip_mcast="true"
mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
loopback="false" />
<PING timeout="2000" num_initial_members="3"
up_thread="true" down_thread="true" />
<MERGE2 min_interval="10000" max_interval="20000" />
<FD shun="true" up_thread="true" down_thread="true"
timeout="2500" max_tries="5" />
<VERIFY_SUSPECT timeout="3000" num_msgs="3"
up_thread="true" down_thread="true" />
<pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
max_xmit_size="8192"
up_thread="true" down_thread="true" />
<UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
down_thread="true" />
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="true" down_thread="true" />
<FRAG frag_size="8192"
down_thread="true" up_thread="true" />
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
shun="true" print_local_addr="true" />
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />

Thanks, Abitha.
Actions
5. Re: message or message buffer is null followed by other exce

belaban Aug 16, 2005 3:13 AM (in response to abitha)

I don't believe you :-):

2005-08-12 10:13:33,423 465548 ERROR [org.jgroups.protocols.COMPRESS] (UpHandler (COMPRESS):) exception on

This clearly shows you have COMPRESS. Check all of your config files, maybe you have a standalone JGroups app running somewhere ?
Actions
6. Re: message or message buffer is null followed by other exce

abitha Aug 16, 2005 1:44 PM (in response to abitha)

Alright - you were right. I had COMPRESS in my jboss cache config file, and I removed it. It worked. Thank you very much.

I still get this exception, every few minutes in the log, although they dont seem to be doing anything (please see below).

Another problem we have in production, which is very bothersome is that: we have mod jk with jboss. When the load is high, after an hour or so (sometimes the time interval varies), mod jk gets locked up (ie, does not take in any more connections), and one or more of the appserver instances die with java defunct processes. I have to forcekill the app server instance to make the setup work again. This never happens in development, and always in production. Any insight on this will be very greatly appreciated.

Thanks!

-----

2005-08-16 10:47:52,415 WARN [org.apache.jk.common.ChannelSocket] Error registering request
2005-08-16 10:49:44,986 ERROR [org.apache.commons.modeler.Registry] Error registering jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest111

javax.management.InstanceAlreadyExistsException: jboss.web:type=RequestProcessor,worker=jk-8009,name=JkRequest111 already registered.
at org.jboss.mx.server.registry.BasicMBeanRegistry.add(BasicMBeanRegistry.java:704)
at org.jboss.mx.server.registry.BasicMBeanRegistry.registerMBean(BasicMBeanRegistry.java:212)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:1018)
at org.jboss.mx.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:326)
at org.apache.commons.modeler.Registry.registerComponent(Registry.java:871)
at org.apache.jk.common.ChannelSocket.registerRequest(ChannelSocket.java:436)
at org.apache.jk.common.HandlerRequest.decodeRequest(HandlerRequest.java:443)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:352)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534)
2005-08-16 10:49:44,987 WARN [org.apache.jk.common.ChannelSocket] Error registering request
Actions
7. Re: message or message buffer is null followed by other exce

abitha Aug 16, 2005 1:49 PM (in response to abitha)

May I also ask why COMPRESS made a difference, and what it was supposed to do?

Thanks.
Actions
8. Re: message or message buffer is null followed by other exce

belaban Aug 16, 2005 2:46 PM (in response to abitha)

Regarding the 2nd problem, please post on the Apache/Tomcat forum.

Regarding COMPRESS: this compresses packets and uncompresses them on the receiving side. If you have enough CPU power (e.g. a dual-CPU box), this might give you a huge performance gain *if* your data is easily compressable.
COMPRESS had a bug (if I recall in 2.2.7), which was fixed in 2.2.8. So simply upgrading should have done the job as well.
Actions
9. Re: message or message buffer is null followed by other exce

abitha Aug 17, 2005 1:01 PM (in response to abitha)

Thank you Bela. I have posted in the other forums. One thing we noted yesterday was when we have the jboss die with defunct java process, the logs have a lot of the following error messages. Why and when do we get these messages? The jk log just says that "app02 is in error state" where app02 is one of the jboss servers. Can you help with this?

Thanks!

2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
52 not found in sent_msgs ! sent_msgs=[71274 - 71506]
2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
53 not found in sent_msgs ! sent_msgs=[71274 - 71506]
2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
54 not found in sent_msgs ! sent_msgs=[71274 - 71506]
2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
55 not found in sent_msgs ! sent_msgs=[71274 - 71506]
2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
56 not found in sent_msgs ! sent_msgs=[71274 - 71506]
2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
57 not found in sent_msgs ! sent_msgs=[71274 - 71506]
2005-08-16 20:43:28,255 ERROR [org.jgroups.protocols.pbcast.NAKACK] (requester=10.100.1.50:39841, local_addr=10.100.1.52:40517) message with seqno=70
58 not found in sent_msgs ! sent_msgs=[71274 - 71506]
Actions
10. Re: message or message buffer is null followed by other exce

abitha Aug 17, 2005 1:32 PM (in response to abitha)

Also, interspersed with those NAKACK messages, I see the following (not necessarily at the start of the NAKACK MESSAGES).

java.io.IOException: Message too long
at java.net.PlainDatagramSocketImpl.send(Native Method)
at java.net.DatagramSocket.send(DatagramSocket.java:611)
at org.jgroups.protocols.UDP.doSend(UDP.java:858)
at org.jgroups.protocols.UDP.send(UDP.java:846)
at org.jgroups.protocols.UDP$OutgoingPacketHandler.handleMessage(UDP.java:1663)
at org.jgroups.protocols.UDP$OutgoingPacketHandler.run(UDP.java:1649)
at java.lang.Thread.run(Thread.java:534)
Actions
11. Re: message or message buffer is null followed by other exce

belaban Aug 17, 2005 5:18 PM (in response to abitha)

This is strange because with the props you posted earlier, this should not occur as your frag size is 8192. Unless you are not using those props ?
Actions
12. Re: message or message buffer is null followed by other exce

abitha Aug 17, 2005 6:12 PM (in response to abitha)

We are using those props in cluster-service.xml. The jboss config uses frag size of 60000. It also uses the max_xmit_size of 60000. Should this be changed?

Abitha.
Actions
13. Re: message or message buffer is null followed by other exce

abitha Aug 17, 2005 6:14 PM (in response to abitha)

Here's our jboss config props (replSync-service.xml):






<UDP mcast_addr="229.1.2.3" mcast_port="45566"
ip_ttl="32" ip_mcast="true"
mcast_send_buf_size="10000000" mcast_recv_buf_size="10000000"
ucast_send_buf_size="10000000" ucast_recv_buf_size="10000000"
max_bundle_size="64000"
max_bundle_timeout="30"
use_incoming_packet_handler="false"
use_outgoing_packet_handler="true"
loopback="false"
/>
<PING timeout="2000" down_thread="false" num_initial_members="3"/>
<MERGE2 max_interval="10000" down_thread="false" min_interval="5000"/>
<FD_SOCK down_thread="false"/>
<VERIFY_SUSPECT timeout="1500" down_thread="false"/>
<pbcast.NAKACK max_xmit_size="60000" down_thread="false" use_mcast_xmit="true" gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"/>
<UNICAST timeout="300,600,1200,2400,3600" down_thread="false"/>
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="5000" down_thread="false" max_bytes="250000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000" down_thread="false" join_retry_timeout="2000" shun="true"/>
<FC max_credits="1000000" down_thread="false" min_threshold="0.10"/>
<FRAG frag_size="60000" down_thread="false" up_thread="true"/>
<pbcast.STATE_TRANSFER down_thread="false" up_thread="false"/>
Actions

14. Re: message or message buffer is null followed by other exce

belaban Aug 18, 2005 2:55 AM (in response to abitha)

Can you try with JGroups 2.2.9 ? Here's the error message you should be getting:

9093 [ERROR] [TP.BundlingOutgoingPacketHandler thread] TP$BundlingOutgoingPacket
Handler.run(): - failure in bundling
java.lang.Exception: message size (100196) is greater than max bundling size (64
000). Set the fragmentation/bundle size in FRAG and TP correctly
 at org.jgroups.protocols.TP$BundlingOutgoingPacketHandler.checkLength(TP
.java:1264)
 at org.jgroups.protocols.TP$BundlingOutgoingPacketHandler.run(TP.java:1226)
 at java.lang.Thread.run(Thread.java:595)

2.2.9 is currently CVS head.

1 2 Previous Next

Go to original post