JBoss Cluster using Gossip Router

[amardeep21]

Hi,

I need some info on connection to JBoss in Cluster mode. I'll brief what we are trying to achieve.

We are clustering jboss using Gossip Router because one of our jboss
is on vpn network and other on static public facing IP (Firewalled)

Issue : Cluster works fine, the problem when we establish the connection from client side which is firewall on public IP but can access via vpn

Setup: Gossip Router running on Static and VPN IP
JBOSS NODE 1 : Running/Binding on VPN IP
JBOSS NODE 2 : Running/Binding on STATIC IP

Now when we try to connect to jboss using jnlp it show these lines in java console.

network: Connecting socket://vpnip:1199 with proxy=DIRECT
network: Connecting socket://vpnip:1198 with proxy=DIRECT
network: Connecting socket://vpnip:5446 with proxy=DIRECT
network: Connecting socket://public:5446 with proxy=DIRECT
network: Connecting socket://public:5446 with proxy=DIRECT

What i don't understand here is, connecting to jboss on VPNIP goes fine upto
port 5446 but then it tries to connect to public IP as well on port 5446
which is firewalled.

Then it keep retrying for 20 to 30 min and then establish the connection to jboss.

Can anyone suggest how can we connect to just VPNIP if we make connection to
vpn only.

Thanks in advance.

Amardeep

[brian.stansberry]

The connections to 1199 and 1198 are for a JNDI lookup. The 5446 connection is most likely made by an EJB proxy that was returned by the JNDI lookup.

Have a look at the config of the "jboss.remoting:service=Connector,transport=socket" mbean in conf/jboss-service.xml. By default the "serverBindAddress" attribute is set to ${jboss.bind.address} which will resolve to whatever address you pass to -b. If you want the server to listen on vpnip, you'd start JBoss with -b vpnip.

You can also have a look at the "jboss:service=Naming" which has "BindAddress" and "RmiBindAddress" attributes. Compare those configs to what you see in the "jboss.remoting:service=Connector,transport=socket"; it will probably give you a clue as to why JNDI is behaving differently from the remoting connector in your config.

[amardeep21]

Hi,

We are starting jboss using -b option.

./run.sh -c vpn -b vpnip
./run.sh -c static -b staticip

So, bind address should be of VPN node and connection should go to VPN IP :5446
instead if keep trying on STATIC IP:5446

One thing to notice was the jndi lookup goes to VPN IP but why it slips to STATIC IP
to create ejb connection where it should go to VPN IP.

I have compared the config and can't locate what could be the problem.

Extract from the jboss-service.xml

<mbean code="org.jboss.naming.NamingService"
name="jboss:service=Naming"
xmbean-dd="resource:xmdesc/NamingService-xmbean.xml">
<!-- The call by value mode. true if all lookups are unmarshalled using
 the caller's TCL, false if in VM lookups return the value by reference. -->
 <attribute name="CallByValue">false</attribute>
 <!-- The listening port for the bootstrap JNP service. Set this to -1
 to run the NamingService without the JNP invoker listening port. -->
 <attribute name="Port">1099</attribute>
<attribute name="BindAddress">${jboss.bind.address}</attribute>
<!-- The port of the RMI naming service, 0 == anonymous -->
 <attribute name="RmiPort">1098</attribute>
 <attribute name="RmiBindAddress">${jboss.bind.address}</attribute>
 <!-- The thread pool service used to control the bootstrap lookups -->
 <depends optional-attribute-name="LookupPool"
 proxy-type="attribute">jboss.system:service=ThreadPool</depends>
 <!-- An example of using the unifed invoker as the transport.
<depends optional-attribute-name="InvokerProxyFactory"
proxy-type="attribute">jboss:service=proxyFactory,type=unified,target=Naming
</depends>-->
<depends optional-attribute-name="Naming"
 proxy-type="attribute">jboss:service=NamingBeanImpl</depends>
</mbean>

<!-- The Connector is the core component of the remoting server service. -->
 <!-- It binds the remoting invoker (transport protocol, callback configuration, -->
 <!-- data marshalling, etc.) with the invocation handlers. -->
 <mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.remoting:service=Connector,transport=socket"
 display-name="Socket transport Connector">

 <attribute name="Configuration">
 <config>
 <!-- Other than transport type and handler, none of these configurations are required (will just use defaults). -->
 <invoker transport="socket">
 <attribute name="dataType" isParam="true">invocation</attribute>
 <attribute name="marshaller" isParam="true">org.jboss.invocation.unified.marshall.InvocationMarshaller</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.invocation.unified.marshall.InvocationUnMarshaller</attribute>
 <attribute name="clientMaxPoolSize" isParam="true">2000</attribute>
 <attribute name="socketTimeout" isParam="true">600000</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">4446</attribute>
 <!-- <attribute name="clientConnectAddress">216.23.33.2</attribute> -->
 <!-- <attribute name="clientConnectPort">7777</attribute> -->
 <attribute name="enableTcpNoDelay" isParam="true">true</attribute>
 <!-- <attribute name="backlog">200</attribute>-->

Thanks,
Amardeep

[brian.stansberry]

From the ports you've described, I assume you are using the ServiceBindingManager. For this remoting Connector mbean, it's doing an XSL transform; perhaps there is a problem there?

[amardeep21]

For this remoting Connector mbean, it's doing an XSL transform; perhaps there is a problem there?

We using default configuration so not sure about ServiceBindingManager.

I didn't get this XSL transform can you please elaborate more..

Thanks,
Amardeep

[brian.stansberry]

Your config says:

<attribute name="serverBindPort">4446</attribute>

but the service is listening on 5446. That tells me you are using the ServiceBindingManager. For more on that see
http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Server_Configuration_Guide/beta422/html/Additional_Services-Services_Binding_Management.html

You can look in docs/examples/binding-manager/sample-bindings, which is most likely where the 5446 is coming from. The AS 4.x SBM is fairly complex; in the the case of this Remoting Connect mbean it is using an XSL transform to alter the config.

I can try to make some more detailed suggestions in a few days if you ping me. But not today or tomorrow, sorry. That's why I'm pointing you to the details. Only suggestion I can make now is look in that sample-bindings.xml for

<service-config name="jboss.messaging:service=Connector,transport=bisocket"

That configures a different remoting connector, and doesn't use an XSL transform to do it. The same approach could probably be applied to the "jboss.remoting:service=Connector,transport=socket" config as well.

If your JBoss version doesn't have the "jboss.messaging:service=Connector,transport=bisocket" config in sample-bindings, you can see what I'm talking about at:

http://anonsvn.jboss.org/repos/jbossas/branches/Branch_4_2/varia/src/resources/services/binding/sample-bindings.xml

[amardeep21]

Thanks for the explanation really appreciate that.

Now, I understand your point its using port 5446 because i am using ports-01 which is then using the sample-bindings and XSL transform to rewrite the ports.

But, if use normal default ports config which i tried as well it gives same result.

network: Connecting socket://vpnip:1099 with proxy=DIRECT
network: Connecting socket://vpnip:1098 with proxy=DIRECT
network: Connecting socket://staticip:1099 with proxy=DIRECT
network: Connecting socket://staticip:1099 with proxy=DIRECT
network: Connecting socket://staticip:1099 with proxy=DIRECT
network: Connecting socket://vpnip:1099 with proxy=DIRECT
network: Connecting socket://staticip:1099 with proxy=DIRECT
network: Connecting socket://staticip:4446 with proxy=DIRECT
network: Connecting socket://staticip:1099 with proxy=DIRECT
network: Connecting socket://staticip:4446 with proxy=DIRECT
network: Connecting socket://staticip:1099 with proxy=DIRECT
network: Connecting socket://staticip:4446 with proxy=DIRECT
network: Connecting socket://staticip:4446 with proxy=DIRECT

So, is it a problem with sample-bindings?

Thanks

[brian.stansberry]

If it is due to the ServiceBindingManager, you'd have the same problem with ports-default.

You can confirm it's due to the SBM by commenting out the following in conf/jboss-service.xml. If the problem goes away, it's the SBM.

<mbean code="org.jboss.services.binding.ServiceBindingManager"
 name="jboss.system:service=ServiceBindingManager">
 <attribute name="ServerName">ports-01</attribute>
 <attribute name="StoreURL">${jboss.home.url}/docs/examples/binding-manager/sample-bindings.xml</attribute>
 <attribute name="StoreFactoryClassName">
 org.jboss.services.binding.XMLServicesStoreFactory
 </attribute>
 </mbean>

BTW, what version of JBoss AS are we talking about here?

[amardeep21]

I'll try that now.

BTW, what version of JBoss AS are we talking about here?

jboss-4.2.2.GA

Thanks

[amardeep21]

I have commented ports-01 but it makes no difference.

Now it tries to connect 4446 on staticip.

Any other suggestions please?

Thanks,
Amardeep

[brian.stansberry]

"amardeep21" wrote:
I have commented ports-01 but it makes no difference.

Just to be clear: you commented the entire mbean, right?

[amardeep21]

Just to be clear: you commented the entire mbean, right?

Yes right the entire mbean and now jboss is running on 1099,1098, 4446.

Thanks,
Amardeep

[brian.stansberry]

Sorry for the wild goose chase with the SBM. I suspect there is something obvious we're missing, as I'd think if that connector wasn't respecting the -b value I'd have heard about it long before. I've pinged the Remoting folks (http://www.jboss.org/index.html?module=bb&op=viewtopic&t=155089) to see if they have any input.

[amardeep21]

Thanks really appreciate that.

I am still not convinced that is something to do with jboss bind address
for its is unpredictable behavior on port 5446.

As it does the correct lookup on vpnip but then instead of creating connection to ejb on vpnip:5446 it tries connection to other cluster node!!

[brian.stansberry]

Oh, I completely misunderstood the problem! I thought the calls going to staticip:5446 were hitting the same AS instance as the calls to vpnip:109x, but were coming in on the wrong interface.

What is being looked up with the JNDI calls? I expect it's an EJB session bean. Stateless or Stateful?

Either way, the JNDI lookup is downloading a cluster-aware EJB proxy. That proxy is free to invoke on any node in the cluster, otherwise the bean isn't HA. It doesn't know what node it was downloaded from and thus can't guarantee it will give preference to invoking on that node.

If clients can't reliably connect to both nodes in the cluster, I recommend you don't make the beans clustered. Failover won't work anyway, so there is no benefit to having them be clustered. A non-clustered EJB proxy will only know how to invoke on the node it came from, solving your problem.