6 Replies Latest reply on Jul 16, 2015 8:56 AM by rhusar

Wildfly Standalone servers in cluster mode with same bind address -b 0.0.0.0 throwing address already in use exception

gnsunayana Jul 15, 2015 9:50 AM

We are using the link http://middlewaremagic.com/jboss/?p=1952 to create two standalone servers in cluster mode.We are starting the servers as follows

./standalone.sh -c standalone-ha.xml -b 0.0.0.0 -u 230.0.0.4 -Djboss.bind.address.management=10.91.154.68 -Djboss.management.http.port=55011 -Djboss.server.config.dir=$fwkInstancesDir/server1/configuration -Djboss.node.name=server1 -Djboss.ajp.port=8010 -Djboss.server.base.dir=$fwkInstancesDir/server1 -Djboss.http.port=55000

./standalone.sh -c standalone-ha.xml -b 0.0.0.0 -u 230.0.0.4 -Djboss.bind.address.management=10.91.154.68 -Djboss.management.http.port=55031 -Djboss.server.config.dir=$fwkInstancesDir/server2/configuration -Djboss.node.name=server2 -Djboss.ajp.port=8020 -Djboss.server.base.dir=$fwkInstancesDir/server2 -Djboss.http.port=55020

We are keeping the Bind address same for both the standalone servers. Both the servers get started and running, but the second server log shows error as below. When we try to connect to server through jboss-cli and get the JNDI view the second server is not shown.

[31m02:49:32,371 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 54) MSC000001: Failed to start service jboss.infinispan.web.global-component-registry: org.jboss.msc.service.StartException in service jboss.infinispan.web.global-component-registry: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport

at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [wildfly-clustering-common-8.2.0.Final.jar:8.2.0.Final]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]

at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]

Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport

at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:241)

at org.jboss.as.clustering.infinispan.subsystem.GlobalComponentRegistryService.start(GlobalComponentRegistryService.java:33)

at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [wildfly-clustering-common-8.2.0.Final.jar:8.2.0.Final]

... 4 more

Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type ChannelTransport

at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)

at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:219)

... 6 more

Caused by: org.infinispan.commons.CacheException: Unable to start JGroups Channel

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:198)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:187)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]

at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]

at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)

... 11 more

Caused by: java.lang.Exception: connecting to channel "web" failed

at org.jgroups.JChannel._connect(JChannel.java:544)

at org.jgroups.JChannel.connect(JChannel.java:290)

at org.jgroups.JChannel.connect(JChannel.java:275)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:196)

... 17 more

Caused by: java.lang.IllegalArgumentException: failed to start server socket

at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:321)

at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:218)

at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:84)

at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:533)

at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:576)

at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:347)

at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1045)

at org.jgroups.protocols.FlowControl.down(FlowControl.java:340)

at org.jgroups.protocols.FRAG2.down(FRAG2.java:136)

at org.jgroups.protocols.RSVP.down(RSVP.java:142)

at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1024)

at org.jgroups.JChannel.down(JChannel.java:760)

at org.jgroups.JChannel._connect(JChannel.java:538)

... 20 more

Caused by: java.net.BindException: bind_addr /0.0.0.0 is not a valid interface: java.net.BindException: Address already in use

at org.jgroups.util.Util.createServerSocket(Util.java:3567)

at org.jgroups.protocols.FD_SOCK.startServerSocket(FD_SOCK.java:584)

at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:318)

... 33 more

In addition we have tried specifying different interface for Jgroups as suggested in the link 7.7.7. Binding JGroups Channels to a particular interface , but still we are facing the same error as above. Can anyone please suggest any solutions for the above issue.

1. Re: Wildfly Standalone servers in cluster mode with same bind address -b 0.0.0.0 throwing address already in use exception

teacurran Jul 15, 2015 1:46 PM (in response to gnsunayana)

It looks like you changed your ports for HTTP and AJP, but JGroups needs it's port changed as well.

Did you try using a port offset to automatically increment all ports?
-Djboss.socket.binding.port-offset=100
Actions
2. Re: Wildfly Standalone servers in cluster mode with same bind address -b 0.0.0.0 throwing address already in use exception

gnsunayana Jul 16, 2015 2:29 AM (in response to teacurran)

Iam specifying ports for HTTP and AJP separately in the command only as you can see below

./standalone.sh -c standalone-ha.xml -b 0.0.0.0 -u 230.0.0.4 -Djboss.bind.address.management=10.91.154.68 -Djboss.management.http.port=55011 -Djboss.server.config.dir=$fwkInstancesDir/server1/configuration -Djboss.node.name=server1 -Djboss.ajp.port=8010 -Djboss.server.base.dir=$fwkInstancesDir/server1 -Djboss.http.port=55000

Iam not using -Djboss.socket.binding.port-offset=100 . Is there any way to set ports for jgroups ?
Actions
3. Re: Wildfly Standalone servers in cluster mode with same bind address -b 0.0.0.0 throwing address already in use exception

wdfink Jul 16, 2015 2:40 AM (in response to gnsunayana)

JGroups can not use the 0.0.0.0 address to bind to. You need to run it with a real IP address.

One possibility is to create an additional interface (copy public and set a different name for jboss.bind.address or use the IP)
and add the attribute "interface" with the reference to this for each jgroups socket binding.

This should work
Actions
4. Re: Wildfly Standalone servers in cluster mode with same bind address -b 0.0.0.0 throwing address already in use exception

gnsunayana Jul 16, 2015 4:49 AM (in response to wdfink)

Hi wolf-Dieter ,

Could you please provide any links suggesting how to proceed further with your suggestions.Thanks.
Actions
5. Re: Wildfly Standalone servers in cluster mode with same bind address -b 0.0.0.0 throwing address already in use exception

gnsunayana Jul 16, 2015 8:24 AM (in response to wdfink)

As you suggested IP is working for me and i am able to run both the server instances in the standalone/clustered mode successfully.

Now the problem is, while using the jboss-cli for finding the cluster view under /subsystem/jgroups/channel/web , I get only the view of that particular instance view and not the clustered instances view. eg. node1/web (1),,, where as if i am running 2 instances it should be [node1/web|1] [node1/web, node2/web].
Actions
6. Re: Wildfly Standalone servers in cluster mode with same bind address -b 0.0.0.0 throwing address already in use exception

rhusar Jul 16, 2015 8:56 AM (in response to gnsunayana)

Caused by: java.net.BindException: bind_addr /0.0.0.0 is not a valid interface: java.net.BindException: Address already in use

at org.jgroups.util.Util.createServerSocket(Util.java:3567)

at org.jgroups.protocols.FD_SOCK.startServerSocket(FD_SOCK.java:584)

You would also have to change the FD_SOCK ports and other ports that are used (FD_SOCK).

There is a property that handles all port offset to run on the same IP, use -Djboss.socket.binding.port-offset=100.

Nevertheless, binding to all adresses (0.0.0.0) should not be used.
Actions

Go to original post