4 Replies Latest reply on Jan 22, 2008 4:04 AM by jeremystone

    Occasional NPE when use JGroups multiplexor

    jeremystone

      When we start up our JBoss application we sometimes get the exception shown below. It occurs on roughly 1/4 of startups. We are running JBossAS 4.2.0.GA with JBoss Messaging 1.4.0.SP3 (also saw problem with SP2). We are running with clustered destinations (but with just one cluster node at present). We use JGroups 2.5.1 with the multiplexor using the udp-sync channel config (shown below) for the control channel.

      Debugging, the following seems to happen at application server startup:

      - Main thread calls GroupMember.start() which connects to the controlChannel
      - Multiplexer thread calls back to the ControlMembershipListener viewAccepted() which creates the latch object
      - Main thread continues to call latch.countDown(). There is clearly a race condition here which causes latch to be null sometimes. Bang!

      Note:
      - problem does not occur if dont use multiplexor in JBoss messaging config (i.e. use configuration bundled with JBM): i.e. callback to viewAccepted() is on the main thread.

      - but it does occur if use multiplexor with the udp-sync channel config replaced by that in the configuration bundled with JBM. This differs from that in udp-sync (below) in that in the former use_flush is set for STATE_TRANSFER and GMS, and in udp-sync use_concurrent_stack=true with thread_pool and oob.thread_pool.enabled both =true. So these differences do not account for the problem.

      Setting the system property jgroups.mux.enabled=false makes it work again (with the multiplexor-style configuration) and is our solution for the time being.

      Any thoughts?

      2008-01-17 19:02:33,118 37694 WARN [org.jboss.system.ServiceController]
      (main:) Problem starting service jboss.messaging:service=PostOffice
      java.lang.NullPointerException
       at
      org.jboss.messaging.core.impl.postoffice.GroupMember.start(GroupMember.java:160)
       at
      org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.start(MessagingPostOffice.java:347)
       at
      org.jboss.messaging.core.jmx.MessagingPostOfficeService.startService(MessagingPostOfficeService.java:427)
       at
      org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
       at
      org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at
      org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at
      org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at
      org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at
      org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at
      org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:417)
       at org.jboss.system.ServiceController.start(ServiceController.java:435)
       at org.jboss.system.ServiceController.start(ServiceController.java:435)
       at org.jboss.system.ServiceController.start(ServiceController.java:435)
       at org.jboss.system.ServiceController.start(ServiceController.java:435)
       at org.jboss.system.ServiceController.start(ServiceController.java:435)
       at org.jboss.system.ServiceController.start(ServiceController.java:435)
       at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
       at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at
      org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at
      org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy4.start(Unknown Source)
       at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at
      org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at
      org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at
      org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at
      org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy43.start(Unknown Source)
       at org.jboss.deployment.XSLSubDeployer.start(XSLSubDeployer.java:197)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
       at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
       at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at
      org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at
      org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at
      org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at
      org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy9.deploy(Unknown Source)
       at
      org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
       at
      org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
       at
      org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
       at
      org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
       at
      org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
       at
      org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
       at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
       at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at
      org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at
      org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at
      org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:417)
       at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
       at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at
      org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at
      org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy4.start(Unknown Source)
       at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at
      org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at
      org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at
      org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at
      org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy5.deploy(Unknown Source)
       at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
       at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
       at org.jboss.Main.boot(Main.java:200)
       at org.jboss.Main$1.run(Main.java:508)
       at java.lang.Thread.run(Thread.java:595)


      JGroups stack used for controll channel:
      <stack name="udp-sync"
       description="IP multicast based stack, without flow control and without message bundling. This should be used
       instead of udp if (1) synchronous calls are used and (2) the message volume (rate and size)
       is not that large. Don't use this configuration if you send messages at a high sustained rate, or you might
       run out of memory">
       <config>
       <UDP
       mcast_addr="${jboss.partition.udpGroup:228.1.2.3}"
       mcast_port="${jgroups.udp.mcast_port:45599}"
       tos="8"
       ucast_recv_buf_size="20000000"
       ucast_send_buf_size="640000"
       mcast_recv_buf_size="25000000"
       mcast_send_buf_size="640000"
       loopback="false"
       discard_incompatible_packets="true"
       max_bundle_size="64000"
       max_bundle_timeout="30"
       use_incoming_packet_handler="true"
       ip_ttl="${jgroups.udp.ip_ttl:2}"
       enable_bundling="true"
      
       use_concurrent_stack="true"
      
       thread_pool.enabled="true"
       thread_pool.min_threads="1"
       thread_pool.max_threads="25"
       thread_pool.keep_alive_time="5000"
       thread_pool.queue_enabled="false"
       thread_pool.queue_max_size="100"
       thread_pool.rejection_policy="Run"
      
       oob_thread_pool.enabled="true"
       oob_thread_pool.min_threads="1"
       oob_thread_pool.max_threads="8"
       oob_thread_pool.keep_alive_time="5000"
       oob_thread_pool.queue_enabled="false"
       oob_thread_pool.queue_max_size="100"
       oob_thread_pool.rejection_policy="Run"/>
       <PING timeout="2000"
       num_initial_members="3"/>
       <MERGE2 max_interval="100000"
       min_interval="20000"/>
       <FD_SOCK/>
       <FD timeout="10000" max_tries="5" shun="true"/>
       <VERIFY_SUSPECT timeout="1500" />
       <pbcast.NAKACK max_xmit_size="60000"
       use_mcast_xmit="false" gc_lag="0"
       retransmit_timeout="300,600,1200,2400,4800"
       discard_delivered_msgs="false"/>
       <UNICAST timeout="300,600,1200,2400,3600"/>
       <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
       max_bytes="400000"/>
       <pbcast.GMS print_local_addr="true" join_timeout="3000"
       join_retry_timeout="2000" shun="false"
       view_bundling="true"/>
       <FRAG2 frag_size="60000" />
       <!--
       <pbcast.STREAMING_STATE_TRANSFER/>
       -->
       <pbcast.STATE_TRANSFER/>
       <pbcast.FLUSH timeout="0"/>
       </config>
       </stack>