ClientCallback Multiplex

[aslak]

I'm having a bit of an issue here..

We have setup 3 servers:
DMZ = Jboss Messaging 1.4.0 GA server in a DMZ zone.
CORE = Jboss Messaging 1.4.0 GA server in a internal zone.
APP = Jboss AS 4.2.0 server in same zone as CORE.

APP is setup with 2 remote JMS providers, one for CORE and one for DMZ.
Connection flow:
DMZ <- APP -> CORE

Between the DMZ and the APP server there is a firewall. In the DMZ server log we can see constant
ERROR messages of type "received new control socket for unrecognized listenerId: xxx". I suspect the reason for this
is that the Client opens up random callback ports but the server is not allowed through the firewall.

Normally one would set up the client to use a specific callback port by supplying the VM arg jboss.messaging.callback.bind.port=.

In this case the APP server acts as a clinet for two servers and specifying the jboss.messaging.callback.bind.port on the VM level results in one of the providers failing
with a "The invoker for locator (....) is already in use by another Connector" exception.

The Invoker subsystem seems to be configurable through the remoting-*-service.xml on the Server side, but on the Client side it
is hardcoded in JMSRemotingConnection.start to use the static final variable ServerPeer.REMOTING_JMS_SUBSYSTEM.

The use of the VM variable jboss.messaging.callback.bind.port is also hardcoded in the JMSRemotingConnection and there seem to be no way of adding
your own metadata to a Client instance.

Maybe one should be able to supply this kinda information by setting properties on the JMSProviderAdaptor or something similar.

-aslak-

[aslak]

The actually issue is not the combination of two providers on the same server, but rather using the jboss.messaging.callback.bind.port variable at all in the application server. I see the same behavior on a single local provider.

When deploying a MDB it first creates a connection to the DLQ, then failes to create a connection to the actual destination.

12:41:47,891 ERROR [JmsActivation] Unable to reconnect org.jboss.resource.adapter.jms.inflow.JmsActivationSpec@10f8ed(ra=org.jboss.resource.adapter.jms.JmsResourceAdapter@4dd5a1 destination=/queue/jmsxa.in isTopic=false tx=true durable=false reconnect=10 provider=java:/DefaultJMSProvider user=jmsxatest pass=<not shown> maxMessages=1 minSession=1 maxSession=15 keepAlive=60000 useDLQ=true DLQHandler=org.jboss.resource.adapter.jms.inflow.dlq.GenericDLQHandler DLQJndiName=queue/DLQ DLQUser=null DLQMaxResent=5)
org.jboss.jms.exception.MessagingJMSException: Failed to invoke
 at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:269)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:187)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
 at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
 at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
 at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
 at org.jboss.jms.client.JBossConnectionFactory.createXAQueueConnection(JBossConnectionFactory.java:142)
 at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupQueueConnection(JmsActivation.java:448)
 at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupConnection(JmsActivation.java:422)
 at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:303)
 at org.jboss.resource.adapter.jms.inflow.JmsActivation.handleFailure(JmsActivation.java:249)
 at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:589)
 at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
 at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
 at java.lang.Thread.run(Thread.java:619)
Caused by: org.jboss.remoting.InvalidConfigurationException: The invoker for locator (InvokerLocator [bisocket://10.111.111.197:4459/callback?callbackServerHost=10.111.111.197&callbackServerPort=4459&callbackServerProtocol=bisocket&clientMaxPoolSize=200&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&datatype=jms&isCallbackServer=true&onewayThreadPool=org.jboss.jms.server.remoting.DirectThreadPool&serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper]) is already in use by another Connector. Either change the locator or add new handlers to existing Connector.
 at org.jboss.remoting.InvokerRegistry.createServerInvoker(InvokerRegistry.java:519)
 at org.jboss.remoting.transport.Connector.init(Connector.java:396)
 at org.jboss.remoting.transport.Connector.create(Connector.java:782)
 at org.jboss.remoting.transport.Connector.start(Connector.java:301)
 at org.jboss.remoting.Client.addListener(Client.java:918)
 at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:237)
 at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:316)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
 ... 16 more

[clebert.suconic]

>> jboss.messaging.callback.bind.port

This variable is not used any more. We have fixed the documentation removing this old option from doc. On 1.3.0 we started using something called Bisocket where the client will create sockets and the server will use them to callback on the client.

Can you try the new 1.4.0.SP1 release? We have done a bunch of fixed on remoting and configuration improvements. If you are behind a firewal you will also see some extra configuration options on the Remoting-bisocket.

[aslak]

Sorry for the slow response on this, but I've finally gotten the time to look more into the issue.
It's the same issue mentioned in this post:
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=119302


A solution to the problem is to add Persistence to the XMBean desc and generate a serverPeerID if not already set.
This would remove the need for manual serverPeerId configuration, but still keep the option to set it if needed. In the case where a node is moved to a new server and one would want to keep the messages, one could either set the generated serverPeerId in the mbean xml or copy over the xmbean persistence data.

Something like this should work (not widely tested):

Index: src/etc/xmdesc/ServerPeer-xmbean.xml
===================================================================
--- src/etc/xmdesc/ServerPeer-xmbean.xml (revision 3286)
+++ src/etc/xmdesc/ServerPeer-xmbean.xml (working copy)
@@ -4,6 +4,11 @@
 "http://www.jboss.org/j2ee/dtd/jboss_xmbean_1_2.dtd">
 <mbean>
 <description>JBoss Messaging Server Peer</description>
+ <descriptors>
+ <persistence persistPolicy="OnUpdate"/>
+ <persistence-manager value="org.jboss.mx.persistence.DelegatingPersistenceManager"/>
+ </descriptors>
+
 <class>org.jboss.jms.server.ServerPeer</class>

 <!-- Managed constructors -->
Index: src/main/org/jboss/jms/server/ServerPeer.java
===================================================================
--- src/main/org/jboss/jms/server/ServerPeer.java (revision 3286)
+++ src/main/org/jboss/jms/server/ServerPeer.java (working copy)
@@ -35,6 +35,7 @@
 import java.util.Map;
 import java.util.Set;
 import java.util.StringTokenizer;
+import java.util.UUID;

 import javax.management.Attribute;
 import javax.management.InstanceNotFoundException;
@@ -218,9 +219,15 @@
 return;
 }

+ // if no serverPeerID is set manually, generate one.
 if (serverPeerID < 0)
 {
- throw new IllegalStateException("ServerPeerID not set");
+ //throw new IllegalStateException("ServerPeerID not set");
+ serverPeerID = generateServerPeerID();
+ // set the ServerPeerID via the MBeanServer to trigger a Persistence if configured
+ this.server.setAttribute(
+ this.serviceName,
+ new Attribute("ServerPeerID", new Integer(serverPeerID)));
 }

 log.debug(this + " starting");
@@ -1393,6 +1400,15 @@
 }
 }

+ // Generate a ServerPeerId, do not return < 0
+ private int generateServerPeerID() {
+ int serverId = UUID.randomUUID().hashCode();
+ if(serverId < 0) {
+ serverId = generateServerPeerID();
+ }
+ return serverId;
+ }
+
 private String deployDestinationDefault(boolean isQueue, String name, String jndiName)
 throws Exception
 {

MessageXAResource uses the ServerPeerID to identify the uniqueness of the resource manager in the isSameRM method. This could be changed to use a more unique variable, ip/vmid/serverpeer hashcode? This is likely to fix the cluster 1 to cluster 2 type communication, as long as they do not use the same database backend.

[timfox]

aslak - I think you posted on the wrong thread?

This thread is about configuring bisocket transport ports.

[aslak]

Ohh, not at all! :o)

When I started debugging my issue that's where I thought the problem was, but a bit more research on the firewall etc I found that not to be true.

The whole issue was two different jms clusters using the same serverPeerIds, resulting in XA transactions failing for a bridge like application.

The Bisocket config works like a charm as far as this issue goes..

-aslak-

[timfox]

Ok sorry, you're right :)

Yeah, the unique server id issue is a known one - as you know, it's been discussed on other threads.

If we could use a string, then it would be trivial - we could just generate a GUID and save back into the xml like you say.

(However I don't think hashCodes are a good idea, since hashCodes are not supposed to be unique!)

One thing we have to consider with JBM 1.x is that this value is used in the PK in the database for transactions, so this might have performance implications.

For JBM 2 this would be a non issue.