5 Replies Latest reply on Sep 20, 2007 8:23 AM by timfox

    Can't configure a single, shared JGroups channel

    brian.stansberry

      The way MessagingPostOffice and MultiplexerJChannelFactory work together preclude using a single, multiplexed JGroups channel for both the ControlChannel and the DataChannel, i.e.:

      <attribute name="ChannelFactoryName">jboss.jgroups:service=MultiplexerChannelFactory</attribute>
       <attribute name="ControlChannelName">udp</attribute>
       <attribute name="DataChannelName">udp</attribute>
       <attribute name="ChannelPartitionName">${jboss.partition.name:DefaultPartition}-JMS</attribute>
      


      When MultiplexerJChannelFactory tries to get the channels from JGroups, in both cases it passes the value of "ChannelPartitionName" as the "id" param to createMultiplexerChannel(). This results in the error below.

      Being able to use a single channel for all channels in an AS 5 instance is an important goal for the release, as it significantly eases the configuration pain involved in keeping clusters isolated. So, I'd like to see this work.

      Here's the error you get:

      2007-09-10 18:12:51,627 ERROR [org.jboss.messaging.util.ExceptionUtil] org.jboss.messaging.core.jmx.MessagingPostOfficeService@d2821b startService
      javax.management.MBeanException
       at org.jboss.mx.interceptor.ReflectedDispatcher.handleInvocationExceptions(ReflectedDispatcher.java:184)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:165)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
       at org.jboss.messaging.core.impl.jchannelfactory.MultiplexerJChannelFactory.createDataChannel(MultiplexerJChannelFactory.java:131)
       at org.jboss.messaging.core.impl.postoffice.GroupMember.start(GroupMember.java:126)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.start(MessagingPostOffice.java:330)
       at org.jboss.messaging.core.jmx.MessagingPostOfficeService.startService(MessagingPostOfficeService.java:413)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
       at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
       at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:184)
       at $Proxy4.start(Unknown Source)
       at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
       at org.jboss.system.microcontainer.ServiceControllerContextAction.install(ServiceControllerContextAction.java:46)
       at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
       at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327)
       at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:238)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309)
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
       at org.jboss.system.ServiceController.doChange(ServiceController.java:659)
       at org.jboss.system.ServiceController.start(ServiceController.java:431)
       at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:146)
       at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:107)
       at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
       at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.deploy(AbstractSimpleRealDeployer.java:65)
       at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:169)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:728)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:749)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:669)
       at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:327)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1309)
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:734)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:862)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:784)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:622)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:411)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:495)
       at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:354)
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:245)
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:131)
       at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:382)
       at org.jboss.Main.boot(Main.java:210)
       at org.jboss.Main$1.run(Main.java:529)
       at java.lang.Thread.run(Thread.java:595)
      Caused by: java.lang.Exception: service ID "DefaultPartition-JMS" is already registered, cannot register duplicate ID
       at org.jgroups.mux.Multiplexer.createMuxChannel(Multiplexer.java:410)
       at org.jgroups.JChannelFactory.createMultiplexerChannel(JChannelFactory.java:294)
       at org.jboss.ha.framework.server.JChannelFactory.createMultiplexerChannel(JChannelFactory.java:68)
       at org.jgroups.jmx.JChannelFactory.createMultiplexerChannel(JChannelFactory.java:60)
       at AOPContainerProxy$22.createMultiplexerChannel(AOPContainerProxy$22.java)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
       ... 60 more


        • 1. Re: Can't configure a single, shared JGroups channel
          timfox

          Won't having a single channel introduce performance issues?

          JBM uses the the control channel for sending control messages (adding/removing a queue etc), and a data channel for fast multicasting of messages to topics - also this is used for primary-secondary communication for failover.

          JBM needs virtual synchrony for the control channel, but is much looser on the data channel. Does a "one size fits all" channel really work?

          Wouldn't it make more sense for AS5 to ship with a selection of different stacks that can be used for different purposes?

          E.g. you could have a totall ordered channel, a virtual synchony channel, another tuned for fast UDP etc.

          I can see you want to avoid problems with startup times for many channels, but would it be hard to add a feature to the JGroups so it "lazy loads" the channel confis - i.e. it doesn't load it until it is requested?

          • 2. Re: Can't configure a single, shared JGroups channel
            brian.stansberry

            The AS will ship with a selection of stacks that can be used for different purposes. And channels will not be created from those stacks until requested by services.

            Maybe a shared channel's not the optimal config for a high performance messaging scenario. Maybe it shouldn't even be the default setup. For AS 5 we need to have quite a bit of testing to see what's optimal in different usage scenarios. But, I don't think JBM should preclude use of a single channel.

            If JBM can't handle a shared channel, people who use JBM in a low-volume way are forced to deal with a bunch of configuration hassles in order to properly isolate production, QA and dev clusters. See http://wiki.jboss.org/wiki/Wiki.jsp?page=PromiscuousTraffic and http://wiki.jboss.org/wiki/Wiki.jsp?page=TwoClustersSameNetwork for all the painful hoops people deal with now.

            • 3. Re: Can't configure a single, shared JGroups channel
              timfox

              The problem with using the same channel for control messages and "data" messages in JBM is that we use the JGroups MessageDispatcher on the channel to handle to the control messages, but we use a standard receiver on the other channel for the "data" messages.

              IIRC, once you use the MessageDispatcher on a channel, this precludes receiving any messages that were sent not using the MessageDispatcher - i.e. we wouldn't receive any data messages.

              Any ideas how we get around this?

              • 4. Re: Can't configure a single, shared JGroups channel
                brian.stansberry

                This shouldn't be an issue. From the JBM point of view, you have two separate channels. Code written to use your control channel will not see messages destined for your data channel, and vice versa. It's just that the JGroups multiplexes them on top of the same underlying JChannel.

                When you call JChannelFactory.createMultiplexerChannel("udp", "DefaultPartition-JMS", true, "DefaultPartition-JMS") you're asking the channel factory to give you a channel based on the "udp" config. If there isn't already a multiplexed channel based on "udp" the factory will create a regular JChannel based on the "udp" config. It will then attach an instance of Multiplexer to that JChannel. It will then create an instance of MuxChannel and register it with the Multiplexer under the id "DefaultPartition-JMS". That MuxChannel is what gets returned to you. When you send messages using the MuxChannel, the Multiplexer adds "DefaultPartition-JMS" as a header to the message. On the other side, the Multiplexer reads messages off the underlying JChannel, reads out the "DefaultPartition-JMS" header, and passes them on to the correct MuxChannel.

                Note that the factory will only create one underlying JChannel based on "udp"; anyone who passes "udp" as an arg will get a MuxChannel that's using that same underlying JChannel. So, if other AS services also use "udp", they'll be sharing the same underlying channel with JBM.

                The problem I'm describing occurs from this usage in MultiplexerJChannelFactory:

                 public JChannel createControlChannel() throws Exception
                 {
                 return (JChannel) server.invoke(this.channelFactory, MUX_OPERATION,
                 new Object[]{syncStack, uniqueID, Boolean.TRUE, uniqueID}, MUX_SIGNATURE);
                 }
                
                 public JChannel createDataChannel() throws Exception
                 {
                 return (JChannel) server.invoke(this.channelFactory, MUX_OPERATION,
                 new Object[]{asyncStack, uniqueID, Boolean.TRUE, uniqueID}, MUX_SIGNATURE);
                 }
                


                If syncStack and asyncStack are the same, JGroups will barf when you ask for the 2nd channel. This fixes the problem in a brute way:

                 public JChannel createControlChannel() throws Exception
                 {
                 String controlId = uniqueId + "-CTRL";
                 return (JChannel) server.invoke(this.channelFactory, MUX_OPERATION,
                 new Object[]{syncStack, controlId, Boolean.TRUE, controlID}, MUX_SIGNATURE);
                 }
                
                 public JChannel createDataChannel() throws Exception
                 {
                 String dataId = uniqueId + "-DATA";
                 return (JChannel) server.invoke(this.channelFactory, MUX_OPERATION,
                 new Object[]{asyncStack, dataID, Boolean.TRUE, dataID}, MUX_SIGNATURE);
                 }
                



                BTW, I see in MultiplexerJChannelFactory you're casting the return from createMultiplexerChannel() to the concrete class JChannel. The API for the method returns interface Channel. The cast works because MuxChannel subclasses JChannel. But that's really an implementation detail you *shouldn't* count on. I doubt Bela will change it any time soon, but...

                • 5. Re: Can't configure a single, shared JGroups channel
                  timfox

                  Ok, I see.

                  I have made the necessary changes in TRUNK.

                  http://jira.jboss.com/jira/browse/JBMESSAGING-1077