How to programatically manage 1,000 to 6,000 outbound ejb connections in JBoss7?

[jmacdonald]

My use case is that I have a central Jboss 7 server, or possibly a cluster that acts as a central location.  I also have up to 6,000 remote locations.  Each of these locations has an ejb with a remote interface exported in JNDI.  Let's call it ejb:/RemoteAPP/remote-api// ... some EJB etc.

The central server needs to connect to all these remote ejb's (one after another) and read information unique to each remote location.  Say statistics for example.

In Jboss 4,5,6 this was easy, we would use an InitialContext, and pass it a URL with a different IP:PORT.  This has worked sufficiently for years.  But we want to move to Jboss 7.

So it seems in Jboss 7, you are suppose to... set up 6,000 outbound connections using the jboss-cli interface? manually adding them to your standalone.xml?  Seems untennable to me.  There must be a betteer way.  I realize there might be a clustering answer in here, but my use case is specific, please keep reading.

At any time, remote sites can come online, or offline.  We don't want to have to restart our central server to become aware of these new outbound connections.  Currently, we manage all this connection information in an SQL database in an automated fashion.  With previous versions of Jboss we simply read it out, create our InitialContext with our URL that has a PORT and IP and everyone is happy (regardless of wether this is actually the correct way to do it).

So in Jboss 7 I am trying to figure out how to manage this.  I am currenly using 7.1.2 tag that I compiled.  I have also worked on this scenario in Jboss 7.1.1.

I tried setting up a few Jboss7 server to server outbound connections in standalone.xml.  What I observed is that when I lookup up my ejb:/RemoteAPP/remote-api// app, the Receiver would get stuck on the first one it found.  I could not switch between remote servers 1, 2 or 3.  So ok, I thought there must be away around this.  I read about setting the selector and creating jboss-ejb-client.properties programtically.  Oh but you can't change the selector when running inside the container because it throws a security exception.  The selector is locked.

I think the we are going to look at changing the application name in the application.xml or, set the distinctName globally for each remote node to deal with this issue.  However I would still have two remaning issues.

• Firstly, How how can I programatically manage thousands of outbound-socket-binding's and remote-outbound-connection's, sometimes coming online new, or being removed, without restarting the central server(s)?  I don't think putting 1,000 or more outbound-socket-binding's and remote-outbound-connection's in the standalone.xml is tennable.  I have to use the database as a source to manage this information.
• Secondly, even if I can find a way to manage outbound-socket-binding's and remote-outbound-connectio'sn programatically, how can I getaway without making thousands of entries in the jboss-ejb-client.xml of my application ear that lives on the centra server(s).  Perhaps I can interface withs omething that lets me publish these remote connections dynamically in the global jndi namespace as they come and go?

So what are my options?  Some kind of Management APi to create outbound-socket-binding's and remote-outbound-connection's dynaimically?  Does one exist?  A plugin that can boot strap the outbound connection information from an implementation I could write and then reads from our database?  Some kind of clustering configuration where I have fine grain control on each lookup or invocation which target remote server will be used to execute an EJB call?

I would appreciate any feedback on how to solve this use case in Jboss7.

-James

[wdfink]

Hi James,

you are right and not alone in this point, there are several issues around migrating 'EJB-calls'.

the EJB is identified by it's name 'ejb:<app>/<module>/distinct-name>/<Bean>!<View>' and the selector will try one of the available references.

It is not possible to select a special server/bean with the current implementation.

With the management API you are able to create remote connections, but AFAIK you need to restart the server (which is also possible via CLI if you use doamain mode). But from my perspective this looks also not feasible

There are several changes in progress but they are still not finished.

I think there will be an enhancement that the application can have more control about the EJB-selection without such configuration and direct access to the JBoss specific API classes.

You might watch the following JIRA's EJBCLIENT-34 AS7-5047

Also I'm working on some quickstarts in this area, you might watch this documentation (keep in mind that the link might change for 7.2 and further releases).

If there are enhancements I hope that I (or someone else) add this to the existing or some new quickstarts to the "JBoss Developer Framework" in time.

[jmacdonald]

Thanks you sir, for confirming my suspicions.

We have been able to work around this for the time being by switching to a model where remote only contacts central.  Most of our functionality over EJB is remote looking up to central so with the exception of one corner case (which we are fixing with a rest service), we are able to proceed.

-James

[jason.greene]

Until we have something like EJBCLIENT-34, you can programatically manage your own connections and invocation on them like in the post here:

https://community.jboss.org/message/724714#724714

[jason.greene]

Although I should warn you if you are running this in-container, you will have to use reflection to disable the selector lock, and then you will need to take care and restore it in a try/finally for your invocations

[wdfink]

The given link from  Jason Greene  will show a soltution with the remoting-project which is marked as deprecated.

I recommend to not use this approach except as workaround.

I hope that the EJBCLIENT-34 will solve most of the trouble with the new ejb remote calls

[sujes]

What about invoking the EJB's through RMI-IIOP? Is this a good idea?

I've been trying to set up something similar to use JTS albeit with less instances. As a reference, I was looking at the JTS quickstart. Although, in this quickstart it sets up the reference using injection, I was able to lookup the exported EJB through the JNDI. In my initial tests it works and I don't need to statically configure all the references to my instances in jboss config; I just pass a different host and port.

[jason.greene]

Wolf-Dieter Fink wrote:

The given link from  Jason Greene  will show a soltution with the remoting-project which is marked as deprecated.

I recommend to not use this approach except as workaround.

I hope that the EJBCLIENT-34 will solve most of the trouble with the new ejb remote calls

What? What exactly is deprecated?

[sujes]

I don't think that's deprecated. Personally I didn't use that approach because I had lots of difficulties using EJBClientContext.setSelector(this.selector) because I was trying to invoke EJB's in one container instance from inside another instance. Firstly the setSelector has permissions locked from the security manager and secondly the JTS couldn't commit the transaction. The transaction manager commit runs in another thread with another classloader and I had no control over the client context it tries to use.

[jason.greene]

Simon Mc Glynn wrote:

I don't think that's deprecated. Personally I didn't use that approach because I had lots of difficulties using EJBClientContext.setSelector(this.selector) because I was trying to invoke EJB's in one container instance from inside another instance. Firstly the setSelector has permissions locked from the security manager and secondly the JTS couldn't commit the transaction. The transaction manager commit runs in another thread with another classloader and I had no control over the client context it tries to use.

Right you would have to use reflection to access our selector:

https://github.com/jbossas/jboss-as/blob/master/ejb3/src/main/java/org/jboss/as/ejb3/remote/TCCLEJBClientContextSelectorService.java

And then do some kind of alter connect change and restore. Unfortunately due to a bug, you would have to control the transaction manually.

If you are using JTS though, and you want to participate in container transactions on the client side (the one making the remote call) you could look at using IIOP, it allows you to dynamically register a connection.

I have bookmarked your usecase with the work we are doing in EJBCLIENT-34. I hope to make sure it is covered without any hacking.

[henk53]

Just out of curiousity, what was really wrong with providing a URL to initial context?

I remember initial posts about AS 7 saying that the main reason for dropping it was that is was not in the Java EE and EJB specs, and JBoss wanted to explore alternative ways. Maybe I've missed a post, but I've never really seen an explanation that detailed what was really wrong with the URL approach.

[jaikiran]

Simon Mc Glynn wrote:

The transaction manager commit runs in another thread with another classloader and I had no control over the client context it tries to use.

This was a bug which we just fixed and a pull request has been sent. I think we should be able to merge this to AS7 upstream by tomorrow. I'll let you know once it's available in a nightly build.

[sujes]

Thanks. I am using IIOP. From what tests I have done it appears to work well. Actually, I would suggest that using IIOP could be the solution to the original problem. Aside from offering JTS support it allows a hostname:port style API which is much more "JBoss 4,5,6" as James put it.