TransactionManager and AS' ServiceBindingManager

[brian.stansberry]

Follow-up on a brief discussion that came up on the jboss-as list a few weeks ago.

ServiceBindingManager is an AS service that provides support for preventing port conflicts if 2 AS instances are started on the same machine and the ASs aren't bound to different interfaces. I've been having to redo the SBM to allow it to deal with pojo services, since before it could only work with mbeans. If you're interested in details on that, see http://wiki.jboss.org/wiki/AS5ServiceBindingManager and the forum thread linked at the top of that wiki.

I can now get SBM working with all services that deploy in the AS, except the TransactionManager -- specifically its recovery manager, which is configured via conf/jbossjta-properties.xml. It appears the TM internally parses that file, so the existing SBM integration hooks won't work.

Some possibilities:

1) Expose some property on the "TransactionManager" bean that trumps the value that comes out of jbossjta-properties.xml. The microcontainer can then use the SBM as the source of the correct value.

2) Have whatever parses jbossjta-properties.xml support system property substitution, e.g.

<property name="com.arjuna.ats.arjuna.recovery.recoveryPort" value="${jbossts.recovery.port:4712}"/>

We'd then deploy a bean along with the SBM that sets the proper system property. Hacky, and leaves the problem of ensuring the SBM deploys before the TransactionManager.

3) If the location where this jbossjta-properties.xml comes from can be made configurable on the TransactionManager bean, we can use a hack like we use on the Tomcat server.xml file. The SBM is given the location of the base file, does an XSL Transform on it to put in the correct values, saves the result as a temp file, and the service is told to use the temp file for its configuration. A long-standing hack whose usage I for sure would hate to see expanded.

Any thoughts?

[jhalliday]

I've been waiting for ServiceBindingManager to stabilize in AS trunk before updating the transaction integration code to play nice with it. Of course there is an element of chicken and egg here - SBM's not done until it works with the transaction manager :-)

The MC bean lifecycle methods in the TransactionManagerService already override some of the property file configuration, for example logging and IP address binding. In the latter case we fish the address out of the ServerConfig.SERVER_BIND_ADDRESS system property, largely though ignorance of any more elegant solution. Is there an MC style way to get this value from the server meta data?

For the port binding, I was planning on changing transaction-beans.xml to contain a port, probably something like:

property name="port" value="some ref to SBM here"

and then use that value to override the one from the jta properties file. MC would then automatically ensure SBM deployed before the transaction manager. As I understand it this is essentially your proposal 1). Since I've not actually started work on it yet I'm happy to change the approach if you feel one of the others is preferable.

As it happens there is already a JBTM JIRA for adding system property substitution support, but a) I'm not sure it will make the cut for the next release and b) I don't think it's as elegant an approach for this particular problem.

---

There is an additional related problem that need some attention and may as well be tacked on to this discussion:

The transaction manager makes extensive use of Uids. One component of uniqueness in the Uid is a process id. Java provides no standard way to get the process id of the JVM from the O/S, so we fudge this by binding to a particular port and using that port number as a pseudo process id. We never actually process traffic sent to the listening port, we are essentially just reserving it to guarantee uniqueness for the process lifetime.

There are two issues here: all the transaction using JVMs have to agree which interface to bind the port on. They can't use different real addresses, since the port is uniq only in the scope of the address. In the absence of any override we use localhost. This works from a technical standpoint but confuses users who don't understand why the server is listening on that address even if they have specified something else with -b.

Secondly, each process must obviously have a different port number. Right now we start at a given number and walk the range until we find an unoccupied one. The drawback of this is we may wind up on a port that something else which has not yet started also needs.

If we wish to configure this though SBM too we clearly need some more properties...

[brian.stansberry]

"jhalliday" wrote:
I've been waiting for ServiceBindingManager to stabilize in AS trunk before updating the transaction integration code to play nice with it. Of course there is an element of chicken and egg here - SBM's not done until it works with the transaction manager :-)

True enough. Luckily the SBM that's in place now doesn't work anymore, so that resolves our dilemna. I can just check in what I have to replace it and we don't end up any more broken.

The MC bean lifecycle methods in the TransactionManagerService already override some of the property file configuration, for example logging and IP address binding. In the latter case we fish the address out of the ServerConfig.SERVER_BIND_ADDRESS system property, largely though ignorance of any more elegant solution. Is there an MC style way to get this value from the server meta data?

Simplest is to dependency inject it, with the system property as the value. End result is the same as you have now, but it can be changed by the user.

<property name="bindAddress">${jboss.bind.address}</property>

I think better is to get the value from the SBM; that way the SBM becomes the source for all binding related metadata. See example below.

For the port binding, I was planning on changing transaction-beans.xml to contain a port, probably something like:

property name="port" value="some ref to SBM here"

and then use that value to override the one from the jta properties file. MC would then automatically ensure SBM deployed before the transaction manager.

Yep. The syntax would look like this:

<bean name="TransactionManager" class="com.arjuna.ats.jbossatx.jta.TransactionManagerService">
 <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss:service=TransactionManager", exposedInterface=com.arjuna.ats.jbossatx.jta.TransactionManagerServiceMBean.class, registerDirectly=true)</annotation>

 <property name="transactionTimeout">300</property>
 <property name="objectStoreDir">${jboss.server.data.dir}/tx-object-store</property>
 <property name="mbeanServer"><inject bean="JMXKernel" property="mbeanServer"/></property>

 <property name="recoveryBindAddress">
 <value-factory bean="ServiceBindingManager"
 method="getInetAddressBinding"
 parameter="TransactionManager"/>
 </property>

 <property name="recoveryPort">
 <value-factory bean="ServiceBindingManager"
 method="getIntBinding"
 parameter="TransactionManager"/>
 </property>
 </bean>

It can get more involved than that if there are a bunch of bindings (see wiki), but that handles a single pair of properties for an address and port.

If the address property was of type String rather than InetAddress, the "method" attribute in its value-factory element would have a value of "getStringBinding".

The presence of the bean="ServiceBindingManager" in the value-factory elements triggers the proper dependency relationship in the MC.

As I understand it this is essentially your proposal 1). Since I've not actually started work on it yet I'm happy to change the approach if you feel one of the others is preferable.

No, I think proposal 1) is best; just threw out the others to spark discussion in case that wasn't workable.

As it happens there is already a JBTM JIRA for adding system property substitution support, but a) I'm not sure it will make the cut for the next release and b) I don't think it's as elegant an approach for this particular problem.

Cool. I think system property substitution is a nice thing to have in all our config files, but I agree it's not as elegant for this usage.

There are two issues here: all the transaction using JVMs have to agree which interface to bind the port on. They can't use different real addresses, since the port is uniq only in the scope of the address. In the absence of any override we use localhost. This works from a technical standpoint but confuses users who don't understand why the server is listening on that address even if they have specified something else with -b.

Secondly, each process must obviously have a different port number. Right now we start at a given number and walk the range until we find an unoccupied one. The drawback of this is we may wind up on a port that something else which has not yet started also needs.

If we wish to configure this though SBM too we clearly need some more properties...

Multiple properties are not a problem. The examples above use an overloaded method on the SBM that only takes a single parameter to identify the binding. There are other variants that take a second param to identify a particular binding.

As for the localhost vs -b issue, that's a configuration choice. The SBM can certainly handle using a different address value for this binding vs. the others, allowing you to keep "localhost" if you want. Here's how it could all be set up:

In bindings.xml:

 <bean class="org.jboss.bindings.ServiceBinding">
 <constructor>
 <parameter>TransactionManager</parameter>
 <parameter>RecoveryManager</parameter>
 <parameter>${jboss.bind.address}</parameter>
 <parameter>4712</parameter>
 </constructor>
 </bean>

 <bean class="org.jboss.bindings.ServiceBinding">
 <constructor>
 <parameter>TransactionManager</parameter>
 <parameter>UUIDFactor</parameter>
 <parameter>localhost</parameter>
 <parameter>1234</parameter>
 </constructor>
 </bean>

The TransactionManager bean then becomes:

<bean name="TransactionManager" class="com.arjuna.ats.jbossatx.jta.TransactionManagerService">
 <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss:service=TransactionManager", exposedInterface=com.arjuna.ats.jbossatx.jta.TransactionManagerServiceMBean.class, registerDirectly=true)</annotation>

 <property name="transactionTimeout">300</property>
 <property name="objectStoreDir">${jboss.server.data.dir}/tx-object-store</property>
 <property name="mbeanServer"><inject bean="JMXKernel" property="mbeanServer"/></property>

 <property name="recoveryBindAddress">
 <value-factory bean="ServiceBindingManager"
 method="getInetAddressBinding">
 <parameter>TransactionManager</parameter>
 <parameter>RecoveryManager</parameter>
 <value-factory>
 </property>

 <property name="recoveryPort">
 <value-factory bean="ServiceBindingManager"
 method="getIntBinding" >
 <parameter>TransactionManager</parameter>
 <parameter>RecoveryManager</parameter>
 <value-factory>
 </property>

 <property name="uuidBindAddress">
 <value-factory bean="ServiceBindingManager"
 method="getInetAddressBinding">
 <parameter>TransactionManager</parameter>
 <parameter>UUIDFactor</parameter>
 <value-factory>
 </property>

 <property name="uuidPort">
 <value-factory bean="ServiceBindingManager"
 method="getIntBinding" >
 <parameter>TransactionManager</parameter>
 <parameter>UUIDFactor</parameter>
 <value-factory>
 </property>
 </bean>

[brian.stansberry]

The pojo version of SBM is now being used in AS trunk. Let me know what bindings you'll want and I can add them to the bindings.xml file the AS uses (or, if you want to do it yourself, it's at https://svn.jboss.org/repos/jbossas/trunk/server/src/etc/conf/default/bindings.xml.)

Currently it has this related to the transaction manager:

 <!-- ********************* deploy/transaction-jboss-beans.xml ********************** -->

 <!-- JBossTS Recovery Manager -->
 <bean class="org.jboss.services.binding.ServiceBinding">
 <constructor>
 <parameter>TransactionManager</parameter>
 <parameter>${jboss.bind.address}</parameter>
 <parameter>4712</parameter>
 </constructor>
 </bean>

But that is unused, really more of a temporary placeholder, so it can be changed as you wish.

[jhalliday]

A quick question: is there a way to construct a service binding just for the port number, rather than a host/port pair?

 <constructor>
 <parameter>TransactionManager</parameter>
 <parameter>socketProcessId</parameter>
 <parameter>10001</parameter>
 </constructor>

does not do what I want.

Thanks.

[brian.stansberry]

Yes and no. There is an overloaded constructor with 4 params; the third is the host name and accepts null:

<constructor>
 <parameter>TransactionManager</parameter>
 <parameter>socketProcessId</parameter>
 <parameter><null/></parameter>
 <parameter>10001</parameter>
</constructor>

ServiceBinding actually exposes two getters related to the hostname passed to the constructor:

String getHostName() -- simple getter
InetAddress getBindAddress() -- derived from hostname

The first is a simple getter of what's passed in; the second is derived from hostname:

this.bindAddress = InetAddress.getByName(hostName);

InetAddress.getByName() will return the loopback address if passed null; so in your case calling getBindAddress() will return the loopback address, not null.

That could probably be changed to return null; I didn't think about it much when I rewrote SBM; just left in place the existing behavior.

Of course, what getBindAddress() returns only matters if someone invokes on SBM asking for it, which I imagine JBossTS won't be doing. :-)

[jhalliday]

Right, thanks. I'm basically looking for some way to make clear in the bindings file that the IP will be ignored. Right now I use the localhost address but have a comment warning it's irrelevant. Using null plus the comment seems even better, I think I'll go with that. Users will only get a surprise if they change null to something else AND fail to read the comment. But at least that means it's an issue only for the really top notch idiots out there :-)


One other problem: I have two ports that don't need particular numbers. Right now we use a default of '0', which means pick the first free port you come across. If I put '0' in the bindings file though, it only works for the default port set. For others it gets offset e.g. 0->100, which results in different semantics. I guess the best option is to pick fixed ports to use in the bindings file. Any preferences?

[brian.stansberry]

I don't have any opinions on what ports to use. :)

It is possible to override for a set of bindings the default behavior of incrementing the port by 100 in ports-01, 200 in ports-01 etc. I can show you how to do that if needed, although I think having a fixed ports sounds better. Admins seem to like that.

[marklittle]

I vote for fixed ports.

[jhalliday]

Ok, probably 4713 and 4714 then, to make them consecutive with the existing 4712 usage.

[jhalliday]

In this week's exciting new episode: Jonathan breaks the AS cluster tests...

The clustering test mechanism does not use SBM, but rather uses the same default port set but with each server bound to a different IP using the '-b' option. This is fine for the JBossTS ports that bind to the same IP as the server.

The socketProcessId listener binds to localhost/127.0.0.1 regardless of the address used by the server. That's intentional - all instances on the same machine must use the same address to guarantee that the port number is uniq, which is what the service is trying to achieve.

When using the SBM with a fixed port value (4714) for the socketProccessId, the SBM offsets the value (4814, 4914, etc) for additional instances and you don't get a port conflict. When using the '-b' instead the same port number is used and things break.

Options:

- Use a value of '0', which tells the socketProcessId to use a random free port. Upside: probably the easiest fix, since it's a one line change in bindings.xml Downside: odd semantic change when 0 is offset (100, 200, etc) by the SBM - it's now fixed no longer random. The offset would also need to be greater that 1024 or you wind up trying to bind low numbered ports.

- Use the retry mechanism that tells socketProcessId to try consecutive ports counting up from the specified one until it finds a free one. This is disabled by default as I don't particularly like it. IMO if something is using the specified port that's an error that should be brought to the user's attention, not silently worked around. Upside: a relatively minor config change. Downside: the AS now uses an unpredictable set of ports, which some users won't like.

- Use an alternative implementation to get a uniq number. We have a file lock based one too for example. Getting the process ID for the JVM would also work, if there was some clean way to do it without resorting to native code.

- Change clustering tests to use the SBM. Of course this ignores the fact that customers may also want to run multiple servers using '-'b'. How common is that and will they tolerate having to make a config file change to do it?

[adrian.brock]

"jhalliday" wrote:

- Use an alternative implementation to get a uniq number. We have a file lock based one too for example. Getting the process ID for the JVM would also work, if there was some clean way to do it without resorting to native code.

Why can't you use a GUID as discussed on a different thread.
Its a waste of resources (a file handle) to open a port or lock a file.

- Change clustering tests to use the SBM. Of course this ignores the fact that customers may also want to run multiple servers using '-'b'. How common is that and will they tolerate having to make a config file change to do it?

-b is the recommended approach since it is easier to configure
(assuming you can assign an extra ip address to the machine).
It also makes it a lot easier to control routing.

[jhalliday]

> Why can't you use a GUID as discussed on a different thread.

( http://www.jboss.com/index.html?module=bb&op=viewtopic&t=141496 )

Changing the way arjuna's UIDs are structured is a big deal and a) will take more time than we have before component freeze for AS GA and b) is not worth the hassle in light of the number of alternatives available.

> Its a waste of resources (a file handle) to open a port or lock a file.

Feel free to provide any other implementation of process uniqueness you happen to like the look of, since the two we provide don't seem to meet with your approval. As long as it returns a value 0-65535 that's uniq in the scope of the O/S it's running on it should be possible to plug it in without too much hassle.

[brian.stansberry]

"jhalliday" wrote:

- Use a value of '0', which tells the socketProcessId to use a random free port. Upside: probably the easiest fix, since it's a one line change in bindings.xml Downside: odd semantic change when 0 is offset (100, 200, etc) by the SBM - it's now fixed no longer random. The offset would also need to be greater that 1024 or you wind up trying to bind low numbered ports.

The SBM supports configuring certain beans to ignore the "increase ports-default by 100" approach:

 <!-- The ports-01 bindings are obtained by taking ports-default and adding 100 to each port value -->
 <bean name="Ports01Bindings" class="org.jboss.bindings.impl.PortOffsetServiceBindingSet">

 <constructor>
 <parameter><inject bean="PortsDefaultBindings"/></parameter>
 <parameter>100</parameter>
 <!-- Bindings to which the "offset by 100" approach can't be applied -->
 <parameter>
 <set>
 <!-- Can't apply an offset to an ephemeral port value, so we redeclare this one. -->
 <bean class="org.jboss.bindings.ServiceBinding">
 <constructor>
 <parameter>jboss:service=invoker,type=pooled</parameter>
 <parameter>${jboss.bind.address}</parameter>
 <parameter>0</parameter>
 </constructor>
 </bean>
 </set>
 </parameter>
 <!-- Default host name -->
 <parameter>${jboss.bind.address}</parameter>
 </constructor>
 </bean>

[wolfc]

How can we plug-in another implementation? It appears to me that the usage of SocketProcessId is hard-coded.