1 2 Previous Next 22 Replies Latest reply on Dec 17, 2009 4:23 PM by brian.stansberry

    JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot

    jaikiran

      The recent upgrade of JBossTS to  4.9.0.GA in AS trunk is showing up an issue in EJB3 Dev build environment where multiple AS instances can no longer be booted on the same system. Here's the exception being thrown while booting a second instance (while the first is running) on the same system:

       

      18:53:50,805 DEBUG [loggerI18N] 
      [com.arjuna.ats.jbossatx.jta.RecoveryManagerService.create] JBossTS 
      Recovery Service (tag: JBOSSTS_4_9_0_GA) - JBoss Inc.
      
      18:53:50,853 DEBUG [arjLoggerI18N] 
      [com.arjuna.ats.internal.arjuna.recovery.RecActivatorLoader_6] - Start 
      RecoveryActivators
      
      18:53:50,910 DEBUG [arjLoggerI18N] 
      [com.arjuna.ats.arjuna.recovery.TransactionStatusManager_1] - Starting 
      service com.arjuna.ats.arjuna.recovery.ActionStatusService on port 4713
      
      18:53:50,923 ERROR [AbstractKernelController] Error installing to 
      Create: name=RecoveryManager state=Configured: 
      com.arjuna.ats.arjuna.exceptions.FatalError: 
      [com.arjuna.ats.internal.arjuna.utils.SocketProcessId_2] - 
      SocketProcessId.getpid could not get unique port.
      
      at 
      com.arjuna.ats.internal.arjuna.utils.SocketProcessId.<init>(SocketProcessId.java:85)
      
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      
      at 
      sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      
      at 
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      
      at java.lang.Class.newInstance0(Class.java:355)
      
      at java.lang.Class.newInstance(Class.java:308)
      
      at com.arjuna.ats.arjuna.utils.Utility.initDefaultProcess(Utility.java:366)
      
      at com.arjuna.ats.arjuna.utils.Utility.getProcess(Utility.java:380)
      
      at com.arjuna.ats.arjuna.utils.Utility.getpid(Utility.java:312)
      
      at com.arjuna.ats.arjuna.common.Uid.<init>(Uid.java:118)
      
      at com.arjuna.ats.arjuna.utils.Utility.initProcessUid(Utility.java:333)
      
      at com.arjuna.ats.arjuna.utils.Utility.getProcessUid(Utility.java:324)
      
      at 
      com.arjuna.ats.internal.arjuna.recovery.TransactionStatusManagerItem.<init>(TransactionStatusManagerItem.java:365)
      
      at
      

       

       

      To rule out any environment issues, i tried to reproduce this locally and am seeing the same issue on my local setup with the current AS trunk. Here are the steps to reproduce this:

       

      1) Checkout and build the current AS trunk

      2) Start the first instance (binds to 127.0.0.1 and starts up the "default" config):

       

      ./run.sh
      

       

      3) Afer the first one has started, start another instance with a different bind address (-b 127.0.0.2) and a different config ("all"):

       

      ./run.sh -b 127.0.0.2 -c all
      

       

      You'll see the above exception when the second on is starting and the boot fails.

       

      Although this exception itself is not something new, and has various JIRAs to address this (like https://jira.jboss.org/jira/browse/JBTM-412 and https://jira.jboss.org/jira/browse/JBTM-348), multiple instances on same system was working fine in 5.1.0, 6.0.0.M1 and even the AS trunk up until the JBossTS upgrade. So looks like something changed in this release.

       

      Any ideas?

        • 1. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
          jaikiran

          jaikiran wrote:

           

          multiple instances on same system was working fine in 5.1.0, 6.0.0.M1


          6.0.0.M1 uses

           

          <version.jboss.jbossts>4.6.1.GA</version.jboss.jbossts>
          

           

          And 6.0.0.M1 boots up fine with multiple instances.

          • 2. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
            jhalliday

            As it happens I've just been corresponding with Brian on the same topic, as this affects the clustering tests too.

             

            "The port for socketProcessId is configured via  ServiceBindingManager. As long as the two servers you start  use different port configs it's not a problem. As far as I  can tell, that's not done with the clustering tests for some  reason. That means it will need manual configuration to  override the socket setting e.g. munge the  transaction-jboss-beans.xml to hardcode a port rather than  using the SBM. Or use a different Process Id impl, but that  will likewise require munging the xml so it's no better really."

             

            Jonathan.

            • 3. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
              jaikiran

              jhalliday wrote:

               

               

               

              "The port for socketProcessId is configured via  ServiceBindingManager. As long as the two servers you start  use different port configs it's not a problem. As far as I  can tell, that's not done with the clustering tests for some  reason. That means it will need manual configuration to  override the socket setting e.g. munge the  transaction-jboss-beans.xml to hardcode a port rather than  using the SBM. Or use a different Process Id impl, but that  will likewise require munging the xml so it's no better really."

               

              Jonathan.

               

              Yeah, it's the same issue here. Although up until now, we used to allow users to start multiple instances either by specifying a different set of ports or by specifying a different bind address (-b option). The latter is what the EJB3 tests (and even clustering tests i guess) are using. Looks like this approach will no longer work?

              • 4. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                wolfc

                I'll again reiterate that SocketProcessId should not communicate with SBM, because it's the actual binding is an implementation detail around a hack. It's not a true service.

                 

                TransactionManager and AS' ServiceBindingManager

                • 5. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                  jhalliday

                  > Looks like this approach will no longer work?

                   

                  Correct - the default JBossTS config no longer supports that. If the AS community feels they want that functionality out of hte box, either keep the socketProcessId but change the default config so it can hunt for randon free ports (which will incur the wrath of sysadmins who like fixed port sets) or swap the socketProcessId for some other Process id implementation.

                  • 6. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                    wolfc

                    Now where is that rolling eyes smiley.

                    <!-- JBossTS SocketProcessId.  The address part is ignored, it will always use localhost/127.0.0.1. -->
                    <bean class="org.jboss.services.binding.ServiceBindingMetadata">
                       <property name="serviceName">TransactionManager</property>
                       <property name="bindingName">socketProcessId</property>
                       <property name="port">4714</property>
                       <property name="description">Socket used to provide unique
                    process id for JBossTS. The address configuration is ignored; will
                    always use localhost/127.0.0.1</property>
                    </bean>
                    

                     

                    • 7. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                      jhalliday

                      > I'll again reiterate that SocketProcessId should not communicate with SBM

                       

                      And I'll reiterate again: that's nonsense. The SMB exists specifically for cases like this. It's there to allow different servers to define different port sets so they don't interfere with each other, which is exactly what's needed here. Ingoring it just forces admins to hack a separate config file for each server instance, which will not make you popular.

                      • 8. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                        jhalliday

                        > Now where is that rolling eyes smiley.

                         

                        SocketProcessId exists specifically to guarantee a port number that is uniq to all instances on the host, regardless of how many network interfaces that host has. It's functioning as designed. If you don't like, don't use it - the Process id implementation is pluggable.

                        • 9. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                          jaikiran

                          A related question - I was reading the other referenced thread and noticed these 2 comments which sort of look contradictary:

                          The transaction manager makes extensive use of Uids. One component of
                          uniqueness in the Uid is a process id. Java provides no standard way to
                          get the process id of the JVM from the O/S, so we fudge this by binding
                          to a particular port and using that port number as a pseudo process id
                          

                           

                          So going by the above comment, if there are 2 JVMs running on the same system, then you would want 2 different IDs.

                           

                          Then the other comment:

                          The socketProcessId listener binds to localhost/127.0.0.1 regardless of
                          the address used by the server. That's intentional - all instances on
                          the same machine must use the same address to guarantee that the port
                          number is uniq, which is what the service is trying to achieve.
                          

                           

                          This comment seems to imply that if there are multiple JVM instances running (multiple processes) on the same machine then you would want the ID to be the same?

                          • 10. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                            adinn

                            The second comment is  not inconsistent. It may, perhaps, be unclear. It says that the host address

                            needs to be the same on all nodes but does not say anything about the portbeing the same. The point

                            it is trying to make relates to the use of an algorithm which scans upwards from a specified port

                            value looking for a free port. If JVMs used different bind addresses then some of them might possibly

                            bind to the same port number. If they all use the same bind address then no two JVMs can bind to the

                            same port. This is misleading because the configuration mechanism for the clustering requires assigning

                            specific ports via the SMB i.e. the docs talk about implementation details not visible to the user.

                            Add it to the list of TX doc defects (not to mention the AS ones).

                            • 11. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                              jhalliday
                              The bind is on the same interface but different port, specifically to ensure the port number is different - we don't use the value of the address, only the port number. If we let SocketProcessId bind different interfaces, then we could wind up with a non-uniq port number in cases where we have e.g. 1.2.3.4/4714 and 1.2.3.5/4714.
                              • 12. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                                jaikiran

                                Thanks Andrew and Jonathan for that explanation. That helped.

                                 

                                The point it is trying to make relates to the use of an algorithm which scans upwards from a specified port value looking for a free port.

                                 

                                I read in the other thread too that the algorithm is capable of checking the next available port if the current one is not available (or in-use), by a simple increment operation. But the reason why we sort of don't configure it that way currently is because:

                                 

                                Use the retry mechanism that tells socketProcessId to try consecutive
                                ports counting up from the specified one until it finds a free one.
                                This is disabled by default as I don't particularly like it. IMO if
                                something is using the specified port that's an error that should be
                                brought to the user's attention, not silently worked around. Upside: a
                                relatively minor config change. Downside: the AS now uses an
                                unpredictable set of ports, which some users won't like.
                                

                                 

                                How about configuring this port metadata in SBM to use a known/fixed "range" of ports. So instead of setting it to:

                                 

                                <property name="port">4714</property>
                                

                                 

                                maybe we configure it to:

                                 

                                <property name="portRange">4714,4718</property>
                                

                                (Ignore the property name and the syntax used in the value - it's just a pseudo code).

                                 

                                That would mean that the algorithm would start from 4714 and if port is not available, then keep incrementing till it reaches 4718. This way, even though the port is not fixed, the user/admin will still bind to a set of "known" ports. 

                                 

                                The current SBM may not have this kind of port "range" feature but does this kind of configuration look OK? After all the algorithm does provide a mechanism to move to the next port, but we are just not using it. Moreover, the ability to boot multiple instances by just using a different bind address has always been an easy way for users.

                                • 13. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                                  jhalliday
                                  You can extend SBM to support that if you like, but the end result won't be much different from using a fixed port (the bottom of the range) in SMB and a retry counter (the number of ports up from there to try in turn) in the transaction beans file, which is already possible with 0 dev effort. In my view it's not worth the dev effort to move it into SBM unless other things need it to.  Besides, I stick to the opinion that using such functionality by default will (re-)introduce a degree of randomness that many users won't like.
                                  • 14. Re: JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot
                                    brian.stansberry

                                    Binding multiple AS instances to different IPs isn't just a simple convenience for unsophisticated users.

                                     

                                    The recommendation for how to run multiple AS instances on a single  server has never been to configure separate port sets; that's always  been Plan B. The recommendation has always been to multihome the machine  and bind the different instances to different IP addresses.[1]

                                    The multi-server tests in the testsuite all run with our recommended  configuration (except for those specifically testing using separate port  sets).

                                    Requiring use of separate port sets or munging of configuration is a massive regression. We have a policy that commits that cause significant testsuite regressions are reverted. That is going to happen with this change if we don't come up with a solution by the end of the day

                                    [1] http://community.jboss.org/docs/DOC-9384

                                    1 2 Previous Next