JBossTS 4.9.0.GA in AS trunk - Multiple instances on same system fail to boot

[jaikiran]

The recent upgrade of JBossTS to  4.9.0.GA in AS trunk is showing up an issue in EJB3 Dev build environment where multiple AS instances can no longer be booted on the same system. Here's the exception being thrown while booting a second instance (while the first is running) on the same system:

18:53:50,805 DEBUG [loggerI18N] 
[com.arjuna.ats.jbossatx.jta.RecoveryManagerService.create] JBossTS 
Recovery Service (tag: JBOSSTS_4_9_0_GA) - JBoss Inc.

18:53:50,853 DEBUG [arjLoggerI18N] 
[com.arjuna.ats.internal.arjuna.recovery.RecActivatorLoader_6] - Start 
RecoveryActivators

18:53:50,910 DEBUG [arjLoggerI18N] 
[com.arjuna.ats.arjuna.recovery.TransactionStatusManager_1] - Starting 
service com.arjuna.ats.arjuna.recovery.ActionStatusService on port 4713

18:53:50,923 ERROR [AbstractKernelController] Error installing to 
Create: name=RecoveryManager state=Configured: 
com.arjuna.ats.arjuna.exceptions.FatalError: 
[com.arjuna.ats.internal.arjuna.utils.SocketProcessId_2] - 
SocketProcessId.getpid could not get unique port.

at 
com.arjuna.ats.internal.arjuna.utils.SocketProcessId.<init>(SocketProcessId.java:85)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

at java.lang.Class.newInstance0(Class.java:355)

at java.lang.Class.newInstance(Class.java:308)

at com.arjuna.ats.arjuna.utils.Utility.initDefaultProcess(Utility.java:366)

at com.arjuna.ats.arjuna.utils.Utility.getProcess(Utility.java:380)

at com.arjuna.ats.arjuna.utils.Utility.getpid(Utility.java:312)

at com.arjuna.ats.arjuna.common.Uid.<init>(Uid.java:118)

at com.arjuna.ats.arjuna.utils.Utility.initProcessUid(Utility.java:333)

at com.arjuna.ats.arjuna.utils.Utility.getProcessUid(Utility.java:324)

at 
com.arjuna.ats.internal.arjuna.recovery.TransactionStatusManagerItem.<init>(TransactionStatusManagerItem.java:365)

at

To rule out any environment issues, i tried to reproduce this locally and am seeing the same issue on my local setup with the current AS trunk. Here are the steps to reproduce this:

1) Checkout and build the current AS trunk

2) Start the first instance (binds to 127.0.0.1 and starts up the "default" config):

./run.sh

3) Afer the first one has started, start another instance with a different bind address (-b 127.0.0.2) and a different config ("all"):

./run.sh -b 127.0.0.2 -c all

You'll see the above exception when the second on is starting and the boot fails.

Although this exception itself is not something new, and has various JIRAs to address this (like https://jira.jboss.org/jira/browse/JBTM-412 and https://jira.jboss.org/jira/browse/JBTM-348), multiple instances on same system was working fine in 5.1.0, 6.0.0.M1 and even the AS trunk up until the JBossTS upgrade. So looks like something changed in this release.

Any ideas?

[jaikiran]

jaikiran wrote:

multiple instances on same system was working fine in 5.1.0, 6.0.0.M1

6.0.0.M1 uses

<version.jboss.jbossts>4.6.1.GA</version.jboss.jbossts>

And 6.0.0.M1 boots up fine with multiple instances.

[jhalliday]

As it happens I've just been corresponding with Brian on the same topic, as this affects the clustering tests too.

"The port for socketProcessId is configured via  ServiceBindingManager. As long as the two servers you start  use different port configs it's not a problem. As far as I  can tell, that's not done with the clustering tests for some  reason. That means it will need manual configuration to  override the socket setting e.g. munge the  transaction-jboss-beans.xml to hardcode a port rather than  using the SBM. Or use a different Process Id impl, but that  will likewise require munging the xml so it's no better really."

Jonathan.

[jaikiran]

jhalliday wrote:

"The port for socketProcessId is configured via  ServiceBindingManager. As long as the two servers you start  use different port configs it's not a problem. As far as I  can tell, that's not done with the clustering tests for some  reason. That means it will need manual configuration to  override the socket setting e.g. munge the  transaction-jboss-beans.xml to hardcode a port rather than  using the SBM. Or use a different Process Id impl, but that  will likewise require munging the xml so it's no better really."

Jonathan.

Yeah, it's the same issue here. Although up until now, we used to allow users to start multiple instances either by specifying a different set of ports or by specifying a different bind address (-b option). The latter is what the EJB3 tests (and even clustering tests i guess) are using. Looks like this approach will no longer work?

[wolfc]

I'll again reiterate that SocketProcessId should not communicate with SBM, because it's the actual binding is an implementation detail around a hack. It's not a true service.

TransactionManager and AS' ServiceBindingManager

[jhalliday]

> Looks like this approach will no longer work?

Correct - the default JBossTS config no longer supports that. If the AS community feels they want that functionality out of hte box, either keep the socketProcessId but change the default config so it can hunt for randon free ports (which will incur the wrath of sysadmins who like fixed port sets) or swap the socketProcessId for some other Process id implementation.

[wolfc]

Now where is that rolling eyes smiley.

<!-- JBossTS SocketProcessId.  The address part is ignored, it will always use localhost/127.0.0.1. -->
<bean class="org.jboss.services.binding.ServiceBindingMetadata">
   <property name="serviceName">TransactionManager</property>
   <property name="bindingName">socketProcessId</property>
   <property name="port">4714</property>
   <property name="description">Socket used to provide unique
process id for JBossTS. The address configuration is ignored; will
always use localhost/127.0.0.1</property>
</bean>

[jhalliday]

> I'll again reiterate that SocketProcessId should not communicate with SBM

And I'll reiterate again: that's nonsense. The SMB exists specifically for cases like this. It's there to allow different servers to define different port sets so they don't interfere with each other, which is exactly what's needed here. Ingoring it just forces admins to hack a separate config file for each server instance, which will not make you popular.

[jhalliday]

> Now where is that rolling eyes smiley.

SocketProcessId exists specifically to guarantee a port number that is uniq to all instances on the host, regardless of how many network interfaces that host has. It's functioning as designed. If you don't like, don't use it - the Process id implementation is pluggable.

[jaikiran]

A related question - I was reading the other referenced thread and noticed these 2 comments which sort of look contradictary:

The transaction manager makes extensive use of Uids. One component of
uniqueness in the Uid is a process id. Java provides no standard way to
get the process id of the JVM from the O/S, so we fudge this by binding
to a particular port and using that port number as a pseudo process id

So going by the above comment, if there are 2 JVMs running on the same system, then you would want 2 different IDs.

Then the other comment:

The socketProcessId listener binds to localhost/127.0.0.1 regardless of
the address used by the server. That's intentional - all instances on
the same machine must use the same address to guarantee that the port
number is uniq, which is what the service is trying to achieve.

This comment seems to imply that if there are multiple JVM instances running (multiple processes) on the same machine then you would want the ID to be the same?

[adinn]

The second comment is  not inconsistent. It may, perhaps, be unclear. It says that the host address
needs to be the same on all nodes but does not say anything about the portbeing the same. The point
it is trying to make relates to the use of an algorithm which scans upwards from a specified port
value looking for a free port. If JVMs used different bind addresses then some of them might possibly
bind to the same port number. If they all use the same bind address then no two JVMs can bind to the
same port. This is misleading because the configuration mechanism for the clustering requires assigning
specific ports via the SMB i.e. the docs talk about implementation details not visible to the user.
Add it to the list of TX doc defects (not to mention the AS ones).

[jhalliday]

The bind is on the same interface but different port, specifically to ensure the port number is different - we don't use the value of the address, only the port number. If we let SocketProcessId bind different interfaces, then we could wind up with a non-uniq port number in cases where we have e.g. 1.2.3.4/4714 and 1.2.3.5/4714.

[jaikiran]

Thanks Andrew and Jonathan for that explanation. That helped.

The point it is trying to make relates to the use of an algorithm which scans upwards from a specified port value looking for a free port.

I read in the other thread too that the algorithm is capable of checking the next available port if the current one is not available (or in-use), by a simple increment operation. But the reason why we sort of don't configure it that way currently is because:

Use the retry mechanism that tells socketProcessId to try consecutive
ports counting up from the specified one until it finds a free one.
This is disabled by default as I don't particularly like it. IMO if
something is using the specified port that's an error that should be
brought to the user's attention, not silently worked around. Upside: a
relatively minor config change. Downside: the AS now uses an
unpredictable set of ports, which some users won't like.

How about configuring this port metadata in SBM to use a known/fixed "range" of ports. So instead of setting it to:

<property name="port">4714</property>

maybe we configure it to:

<property name="portRange">4714,4718</property>

(Ignore the property name and the syntax used in the value - it's just a pseudo code).

That would mean that the algorithm would start from 4714 and if port is not available, then keep incrementing till it reaches 4718. This way, even though the port is not fixed, the user/admin will still bind to a set of "known" ports. 

The current SBM may not have this kind of port "range" feature but does this kind of configuration look OK? After all the algorithm does provide a mechanism to move to the next port, but we are just not using it. Moreover, the ability to boot multiple instances by just using a different bind address has always been an easy way for users.

[jhalliday]

You can extend SBM to support that if you like, but the end result won't be much different from using a fixed port (the bottom of the range) in SMB and a retry counter (the number of ports up from there to try in turn) in the transaction beans file, which is already possible with 0 dev effort. In my view it's not worth the dev effort to move it into SBM unless other things need it to.  Besides, I stick to the opinion that using such functionality by default will (re-)introduce a degree of randomness that many users won't like.

[brian.stansberry]

Binding multiple AS instances to different IPs isn't just a simple convenience for unsophisticated users.

The recommendation for how to run multiple AS instances on a single  server has never been to configure separate port sets; that's always  been Plan B. The recommendation has always been to multihome the machine  and bind the different instances to different IP addresses.[1]

The multi-server tests in the testsuite all run with our recommended  configuration (except for those specifically testing using separate port  sets).

Requiring use of separate port sets or munging of configuration is a massive regression. We have a policy that commits that cause significant testsuite regressions are reverted. That is going to happen with this change if we don't come up with a solution by the end of the day

[1] http://community.jboss.org/docs/DOC-9384