bridge related: not working between source server and target server from a Vitual machine

[ohmygod]

The bridge is working well between two servers on same machine, but not working when I use a target server which is a Vitual machine running in source server.

The error message from target server:

2013-12-22 22:06:12,687;[MSC service thread 1-1];ERROR;org.hornetq.core.server;HQ224002: Failure in initialisation: org.jboss.netty.channel.ChannelException: Failed to bind to: /192.168.1.2:5447
  at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
  at org.hornetq.core.remoting.impl.netty.NettyAcceptor.startServerChannels(NettyAcceptor.java:505)
  at org.hornetq.core.remoting.impl.netty.NettyAcceptor.start(NettyAcceptor.java:458)
  at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.start(RemotingServiceImpl.java:256)
  at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1542)
  at org.hornetq.core.server.impl.HornetQServerImpl.access$1300(HornetQServerImpl.java:164)
  at org.hornetq.core.server.impl.HornetQServerImpl$SharedNothingLiveActivation.run(HornetQServerImpl.java:2498)
  at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:416)
  at org.hornetq.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:464)
  at org.jboss.as.messaging.jms.JMSService.start(JMSService.java:74)
  at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)
  at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.BindException: Cannot assign requested address: JVM_Bind
  at java.net.PlainSocketImpl.socketBind(Native Method)
  at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
  at java.net.ServerSocket.bind(ServerSocket.java:328)
  at org.jboss.netty.channel.socket.oio.OioServerSocketPipelineSink.bind(OioServerSocketPipelineSink.java:128)
  at org.jboss.netty.channel.socket.oio.OioServerSocketPipelineSink.handleServerSocket(OioServerSocketPipelineSink.java:79)
  at org.jboss.netty.channel.socket.oio.OioServerSocketPipelineSink.eventSunk(OioServerSocketPipelineSink.java:53)
  at org.jboss.netty.channel.Channels.bind(Channels.java:561)
  at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
  at org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:382)
  at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
  at org.jboss.netty.channel.socket.oio.OioServerSocketChannel.<init>(OioServerSocketChannel.java:78)
  at org.jboss.netty.channel.socket.oio.OioServerSocketChannelFactory.newChannel(OioServerSocketChannelFactory.java:127)
  at org.jboss.netty.channel.socket.oio.OioServerSocketChannelFactory.newChannel(OioServerSocketChannelFactory.java:87)
  at org.jboss.netty.bootstrap.ServerBootstrap.bindAsync(ServerBootstrap.java:329)
  at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:266)
  ... 14 more

Another error message in source server:

2013-12-23 14:07:16,941;[Old I/O client worker ([id: 0xa1f1a0d2, /192.168.1.2:59223 => /192.168.1.4:5447])];ERROR;org.hornetq.core.client;HQ214036: Failed to decode packet: java.lang.IllegalArgumentException: HQ119074: Invalid type: 0
  at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:427)
  at org.hornetq.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:64)
  at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:485)
  at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1658)
  at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72)
  at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
  at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281)
  at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
  at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
  at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71)
  at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73)
  at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51)
  at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  at java.lang.Thread.run(Thread.java:619)

Settings on source server:

<connectors>
     <netty-connector name="netty" socket-binding="messaging"/>
      <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
           <param key="batch-delay" value="50"/>
      </netty-connector>
      <in-vm-connector name="in-vm" server-id="0"/>
           <connector name="remote-connector">
                <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                <param key="host" value="192.168.1.4"/>
                <param key="port" value="5447"/>
           </connector>
</connectors>

192.168.1.4 is the ip address of the VM server.

settings on target server from the VM:

<acceptors>
    <netty-acceptor name="netty" socket-binding="messaging"/>
    <n
etty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
         <param key="batch-delay" value="50"/>
         <param key="direct-deliver" value="false"/>
    </netty-acceptor>
    <in-vm-acceptor name="in-vm" server-id="0"/>
    <acceptor name="remote-connector">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host" value="192.168.1.2"/>    
         <param key="port" value="5447"/>
    </acceptor>
</acceptors>

192.168.1.2 is the ip address of the source server.

The bridge works fine on the same machine (two servers) if the "<param key="host" value="192.168.1.x"/>" is not configured.

[jbertram]

You don't need to configure a special acceptor on the "target" server, and even if you did it wouldn't use the IP address of the "source" server.  An acceptor can only bind to a local address - an address it can use to accept connections.

The connector on the source server simply needs to point to the target server on the normal HornetQ port (i.e. 5445).

Furthermore, it's better to use a netty-connector with a socket-binding rather than a generic connector.  For example:

<netty-connector name="remote-connector" socket-binding="remote-socket-binding"/> 
...
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
  ...
    <outbound-socket-binding name="remote-socket-binding">
        <remote-destination host="192.168.1.4" port="5445"/>
    </outbound-socket-binding>
</socket-binding-group>

[ohmygod]

Thanks. Do you mean an accepter named like "remote-connector" is not needed on "target" server? I have removed the accepter named "remote-connector" on "target" server and added the netty-connector and outbound-socket-binding as you mentioned on "source" server, but no good result.

Here is the connect time out error on "source" server. I have started both "source" and "target" server.

2013-12-25 11:21:02,550;[Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=a3f8e6e8-6b84-11e3-a261-0b6a9beedf18-1203129835)];ERROR;org.hornetq.core.client;HQ214045: Failed to create netty connection: java.net.SocketTimeoutException: connect timed out
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
  at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
  at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
  at java.net.Socket.connect(Socket.java:519)
  at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:105)
  at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:65)
  at org.jboss.netty.channel.Channels.connect(Channels.java:634)
  at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:207)
  at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229)
  at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182)
  at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:505)
  at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1225)
  at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1071)
  at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:246)
  at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:826)
  at org.hornetq.core.server.cluster.impl.BridgeImpl.createSessionFactory(BridgeImpl.java:766)
  at org.hornetq.core.server.cluster.impl.BridgeImpl.connect(BridgeImpl.java:795)
  at org.hornetq.core.server.cluster.impl.BridgeImpl$ConnectRunnable.run(BridgeImpl.java:1101)
  at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  at java.lang.Thread.run(Thread.java:619)

Anything still missed to make it run?

BTW, I have not found any schema definition for "outbound-socket-binding" in jboss-as-messaging_1_3.xsd.

[ohmygod]

I have also tried with the "target" server running on the same host machine with "source" server and also not working if no acceptor is defined on "target" server. The only difference is there is no "connect time out" error.

[jbertram]

Do you mean an accepter named like "remote-connector" is not needed on "target" server?

That's correct.

Anything still missed to make it run?

Since it isn't working, I would say yes, there is still something missing. 

When you start the "target" server are you binding it to the interface to which the "source" server is trying to connect (i.e. 192.168.1.4)?  Is the "target" server using any kind of port binding offset?

The whole connector/acceptor concept is pretty simple.  A connector just needs to point to a host:port where there is an acceptor listening for connections.  It's as simple as that.

BTW, I have not found any schema definition for "outbound-socket-binding" in jboss-as-messaging_1_3.xsd.

That's because it isn't in there.  The <socket-binding-group> is not part of the messaging subsystem.  Look in, e.g. jboss-as-config_1_3.xsd.

[ohmygod]

When you start the "target" server are you binding it to the interface to which the "source" server is trying to connect (i.e. 192.168.1.4)?

Not sure how to do this? Can you adive a little bit more? I do not think I have done anything related to the binding for now.

Is the "target" server using any kind of port binding offset?

I do not think so.

[jbertram]

Not sure how to do this? Can you adive a little bit more? I do not think I have done anything related to the binding for now.

When just about any piece of software that is going to accept network connections starts it binds to one (or more) of the machines network interfaces so it can actually receive connections.  Most machines on a network have several interfaces - at least one "loopback" interface (e.g. 127.0.0.1) that can only be used by local clients and at least one remote interface that can be used by remote clients connecting to the machine.  By default JBossAS/Wildfly will bind to 127.0.0.1 which means that no remote clients can connect to it.  This default behavior can be overridden by using a command line switch (e.g. "-b" or "-Djboss.bind.address=...") or by modifying the "public" interface in the <interfaces> subsystem in standalone*.xml or domain.xml.  If you're not doing this on your "target" server then nobody will be able to connect to it since it will be bound by default to a loopback interface.

[ohmygod]

Understood. Thanks for the explanation. And I just tried and confirmed it works fine now. Thanks so much.

But I am wondering if there is a way to use a new interface with the address (192.168.1.4) bound and then specify this interface to the remote-socket-binding the "source" server defined instead of modifying the "public" interface in "target" server standalone*.xml, like

<socket-binding name="remote-socket-binding" interface="for-remote-connect" port="${xxxxxxxx:xxxx}"/>

Can this be done?

Or is there any other method to configure this beside setting the outbound-socket-binding?

[jbertram]

What you want to do should be possible.  Just look through the jboss-as-config_1_3.xsd schema for the configuration options for <socket-binding> and <interface>.

[ohmygod]

Hi Justin,

I tried following settings in "target" server's standalone.xml and it seems working fine. One question is why the port bridge uses has to be 5445 (the same number with the "target" server's messaging socket-binding port)? I used other port number but did not work. Can you explain it a little bit?

<interface name="bridge">
       <inet-address value="${jboss.bind.address.unsecure:192.168.1.4}"/>
</interface>
 
<socket-binding name="bridge" interface="bridge" port="${jboss.bridge.port:5445}"/>
 

The outbound-socket-binding in "source" server is

<outbound-socket-binding name="remote-socket-binding">
     <remote-destination host="192.168.1.4" port="5445"/>
</outbound-socket-binding>
 

As I see there are two socket-bindings related to the netty connector: one is "messaging", the other is called "messaging-throughput". What is the difference between them and how are they used? I tried both "5445" and "5455" port numbers in above setting and they both work fine.

<netty-connector name="netty" socket-binding="messaging"/>
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
     <param key="batch-delay" value="50"/>
</netty-connector>
 
<socket-binding name="messaging" port="5445"/>
<socket-binding name="messaging-throughput" port="5455"/>

 

If above method I tried has no problem, then does this grammar look perfect enough or is there any other better setting to satisfy the same target?

<socket-binding name="bridge" interface="bridge" port="${jboss.bridge.port:5445}"/>
 

[jbertram]

One question is why the port bridge uses has to be 5445 (the same number with the "target" server's messaging socket-binding port)? I used other port number but did not work. Can you explain it a little bit?

The bridge has to connect to a port where the server is actually listening (e.g. 5445 or 5455 by default).  I'm not sure I can explain it any more simply than that.  I believe we've discussed this already on this thread and it's certainly covered in the documentation (e.g. here and here).

As I see there are two socket-bindings related to the netty connector: one is "messaging", the other is called "messaging-throughput". What is the difference between them and how are they used? I tried both "5445" and "5455" port numbers in above setting and they both work fine.

Each acceptor has a matching connector so that, for example, a JMS connection factory can be configured with a specific connector to connect to a specific acceptor.  The "messaging" connector/acceptor pair has the most basic configuration possible.  Every configuration value uses the default.  The "messaging-throughput" connector/acceptor pair is slightly more advanced.  It has a few settings to enable greater overall throughput (hence the name).  You can read more about all the configuration options (including those already set for "messaging-throughput") in the documentation.

In general, I recommend you thoroughly review the HornetQ User Guide as I think it would help clarify many areas of confusion for you.

[ohmygod]

Thanks for all your help Justin. So far it is running so good. I could continue bother you if I meet with more problems..