AS 7 Messaging ( HORNETQ ) Client access remote:// Problem with Server listening on 0.0.0.0 Interface

[rodakr]

Hi

I see interessting Problem, when AS7 Server is configured on IP 0.0.0.0 ( means all IP's on this server AS7 runs on).

Simple Stand Alone Client Accessing JMS Connection Factory over Jboss Naming Remote :

             final Properties env = new Properties();
            env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");
            env.put(Context.PROVIDER_URL, "remote://linuxServer:4447");
            env.put(Context.SECURITY_PRINCIPAL, "admin");
            env.put(Context.SECURITY_CREDENTIALS, "administrator");    
            InitialContext context = new InitialContext(env);
            return context;

            String connectionFactoryString = System.getProperty("connection.factory", "jms/RemoteConnectionFactory");
            connectionFactory = (ConnectionFactory) context.lookup(connectionFactoryString); 
           destination = (Destination) context.lookup("jms/queue/test");
           connection = connectionFactory.createConnection(System.getProperty("username", admin), System.getProperty("password",administrator));
           session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

My remote://  standalone Client can connect to AS7 server, pass authentication, can lookup JMS Connection Factory and when try to create  JMS Session gets this Exception:

log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
15.06.2012 07:53:16 com.test.client.QuickJmsTest testRemoteJMS
INFO: Attempting to acquire connection factory "jms/RemoteConnectionFactory"
15.06.2012 07:53:16 com.test.client.QuickJmsTest testRemoteJMS
INFO: Found connection factory "jms/RemoteConnectionFactory" in JNDI
15.06.2012 07:53:16 com.test.client.QuickJmsTest testRemoteJMS
INFO: Attempting to acquire destination "jms/queue/test"
15.06.2012 07:53:16 com.test.client.QuickJmsTest testRemoteJMS
INFO: Found destination "jms/queue/test" in JNDI
15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate trace
AM FEINSTEN: getConnectionWithRetry::1 with retryInterval = 2000 multiplier = 1.0
java.lang.Exception: trace
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:974)
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:225)
    at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:752)
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
    at com.test.client.QuickJmsTest.testRemoteJMS(QuickJmsTest.java:140)
    at com.test.client.QuickJmsTest.main(QuickJmsTest.java:75)
15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate debug
FEIN: Trying reconnection attempt 0/1
15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate debug
FEIN: Trying to connect with connector = org.hornetq.core.remoting.impl.netty.NettyConnectorFactory@a4488, parameters = {port=5445, host=0.0.0.0} connector = NettyConnector [host=0.0.0.0, port=5445, httpEnabled=false, useServlet=false, servletPath=/messaging/HornetQServlet, sslEnabled=false, useNio=false]
15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate debug
FEIN: Started Netty Connector version 3.2.5.Final-a96d88c
15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate debug
FEIN: Trying to connect at the main server using connector :org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=0-0-0-0
15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate debug
FEIN: Remote destination: /0.0.0.0:5445
15.06.2012 07:53:17 org.hornetq.core.logging.impl.JULLogDelegate debug
FEIN: Main server is not up. Hopefully there's a backup configured now!
15.06.2012 07:53:17 org.hornetq.core.logging.impl.JULLogDelegate trace
AM FEINSTEN: No Backup configured!
java.lang.Exception: trace
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1224)
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:994)
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:225)
    at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:752)
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
    at com.test.client.QuickJmsTest.testRemoteJMS(QuickJmsTest.java:140)
    at com.test.client.QuickJmsTest.main(QuickJmsTest.java:75)
15.06.2012 07:53:17 org.hornetq.core.logging.impl.JULLogDelegate trace
AM FEINSTEN: returning connection = null as tc == null
15.06.2012 07:53:17 org.hornetq.core.logging.impl.JULLogDelegate debug
FEIN: Trying to connect towards ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=0-0-0-0], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=0-0-0-0, backupConfig=null]
15.06.2012 07:53:17 com.test.client.QuickJmsTest testRemoteJMS
SCHWERWIEGEND: Failed to create session factory
Exception in thread "main" javax.jms.JMSException: Failed to create session factory
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
    at com.test.client.QuickJmsTest.testRemoteJMS(QuickJmsTest.java:140)
    at com.test.client.QuickJmsTest.main(QuickJmsTest.java:75)
Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
    at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:774)
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
    ... 3 more

The Problem is that  org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry  use informations aquired from remote side from org.hornetq.core.remoting.impl.netty.NettyConnectorFactory to connect to hornetq JMS Server in AS7... and not the information client passt to initial context for lookup...

I see here some possible coding solutions:

- Server Side resolve 0.0.0.0 to IP Addresses  and set this ins o rg.hornetq.core.remoting.impl.netty.NettyConnectorFactory

- Client side api, if address 0.0.0.0, use ip from JNDI lookup....

Or ist there allready another solution?

[jbertram]

The first point to make here (which has already been explained on the forum) is that the JNDI lookup is 100% independent of HornetQ.  You can retrieve JMS resources from JNDI but there are completely unrelated aside from that.  Therefore, it would be unsuitable to link the two functionally so as to have HornetQ use data from the JNDI lookup.

I just ran a quick test with a freshly unzipped version of JBoss AS 7.1.1.Final.  I started it like so:

./standalone.sh -c standalone-full.xml -b 0.0.0.0

Then I executed a simple client to perform a JNDI lookup on 127.0.0.1 to get a JMS connection factory and destination and then send a message via the JMS API.  It worked without issue.  Can you attach your standalone*.xml?

[rodakr]

You can't reproduce the error if client and Server runs on same machine!

Because on same machine 0.0.0.0 is also 127.0.0.1 ....

You have to test it on two different Servers. On one you have your client, on second you have your server. Then you will see same exception.

I'm using standalon.xml without any modification. If I start AS7.1 on dedicated Interface ( -b 192.168.0.10 )  and Test on Client 192.168.0.11  then every think works of course so quick solution it just not to use 0.0.0.0... but it sucks if you have clusters...

Other solution could be to check in Server Side API if configured Interface is 0.0.0.0, and if this is the case, then set the Interface to result from InetAddress.getLocalHost().getHostAddress();

This way 0.0.0.0 on server side will not break the clients on different machines... ( found this here http://stackoverflow.com/questions/4044306/why-is-inet-0-0-0-0-interpreted-as-inet-00000000-by-com-sun-net-httpser )

[jbertram]

Why does binding to a specific interface suck if you have clusters?

Also, while using InetAddress.getLocalHost().getHostAddress() would be better than the current behavior it would not be a fool-proof solution because InetAddress.getLocalHost().getHostAddress() may not actually be visible to the remote client.  The code has no knowledge of which interface is actually visible to the remote client so that means manual configuration of the "connector" would be required in certain cases (which is already supported).

I'll need to track this down to see if the fix should go into AS7 or HornetQ itself.  In the mean-time you can just bind to a specific interface or manually configure the host address used by the connector.

[rodakr]

Hi Justin

Sorry I used wrong word. It doesn't sucks but is one step more I have to do, each cluster Member needs to know binding address, where with 0.0.0.0 I don't need to distinct between cluster members.

Also, while using InetAddress.getLocalHost().getHostAddress() would be better than the current behavior it would not be a fool-proof solution because InetAddress.getLocalHost().getHostAddress() may not actually be visible to the remote client.  The code has no knowledge of which interface is actually visible to the remote client so that means manual configuration of the "connector" would be required in certain cases (which is already supported).

True. But I don't want to change existing behaviour. Just add this special case with 0.0.0.0 so I can configure it  ... pseudo code  :

if ( getManualConfiguredIP() == "0.0.0.0" ) {

    org-hornetq-core-remoting-impl-netty-NettyConnectorFactory.setHost(getHostIpWhichIsNotLocalhost());

}

With this if I put manually "-b 0.0.0.0"  my remote client can connect to Hornetq.

Without I have 0% chance to connect...

Thanks for taking care!

[jbertram]

I think you misunderstood my previous comment.  I wasn't talking about using InetAddress.getLocalHost().getHostAddress() for the general case, but only for when the client specifies 0.0.0.0 for the jboss.bind.address.  My point was that using InetAddress.getLocalHost().getHostAddress() when the client specifies 0.0.0.0 doesn't solve every problem because the interface selected by InetAddress.getLocalHost().getHostAddress() still might not be visible to the remote client.  In that situation manual configuration would still be required.

As it stands now you can use "-b 0.0.0.0" and manually configure the connector like so:

<connector name="netty">
   <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
   <param key="host" value="visibleHostName"/>
   <param key="port" value="5445"/>
</connector>

[rodakr]

...you mean I hard code visibleHostName in standalone.xml on each Clsuter Member , and have to change it each time cluster moves to other server.

And then I start it with "-b 0.0.0.0" which has no effects on  it... :-)

I can live with just passing "-b `uname -n`" :-)

[jbertram]

...you mean I hard code visibleHostName in standalone.xml on each Clsuter Member , and have to change it each time cluster moves to other server.

Based on your comments, it appeared to me like you wanted to bind to 0.0.0.0 yet still have remote JMS clients work.  You mistakenly thought there was "0%" chance of this working.  I provided you with a configuration that would allow you to bind to 0.0.0.0 and still enable remote JMS clients as a work-around until this can be addressed in the code.  Apparently that doesn't work with your particular use-case, but then again you've not really explained your underlying use-case.

And then I start it with "-b 0.0.0.0" which has no effects on  it... :-)

I'm not sure what you mean here.  When you use "-b 0.0.0.0" then network services will be listening on all available network interfaces.  How the "netty" connector is configured doesn't change this.

I can live with just passing "-b `uname -n`" :-)

I have to wonder why you didn't say this in the first place.  I take it that the whole point of using 0.0.0.0 wasn't actually to bind to all available network interfaces (which is usually the goal when using 0.0.0.0), but simply to avoid having to hard-configure the IP address.

[rodakr]

I like to use 0.0.0.0. Mabe  we can meet on some jboss public events , then I can spend you a beer and try to explain what I mean :-)

[jaikiran]

I think I understand what the usecase is:

1) Remote client is on a network interface A

2) Server is bound to all available interfaces (A, B, C) via -b 0.0.0.0

3) Client looks up the JMS connection factory and invokes createConnection(). Internally HornetQ API on the client side ends up using a wrong network interface - an interface which probably isn't usable from that remote client.

Radek, is that right?

In AS 7.1.1 to solve issues like these, each socket-binding (in standalone.xml) can optionally specify a set of "client-mappings" (@see the definition of socket-binding-client-mappingType in jboss-as-config_1_2.xsd).

<xs:complexType name="socket-bindingType">
        <xs:annotation>
            <xs:documentation>Configuration information for a socket.</xs:documentation>
        </xs:annotation>
        <xs:choice>
            <xs:element name="client-mapping" type="socket-binding-client-mappingType"
                        minOccurs="0" maxOccurs="unbounded">
                <xs:annotation>
                    <xs:documentation>
                        Specifies zero or more client mappings for this socket binding.
                        A client connecting to this socket should use the destination address
                        specified in the mapping that matches its desired outbound interface.
                        This allows for advanced network topologies that use either network
                        address translation, or have bindings on multiple network interfaces
                        to function.

                        Each mapping should be evaluated in declared order, with the first successful
                        match used to determine the destination.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>

Each such client-mapping looks like:

<xs:complexType name="socket-binding-client-mappingType">
        <xs:annotation>
                <xs:documentation>
                    Type definition for a client mapping on a socket binding. A client
                    mapping specifies how external clients should connect to this
                    socket's port, provided that the client's outbound interface
                    match the specified source network value.
                </xs:documentation>
            </xs:annotation>
        <xs:attribute name="source-network" type="xs:string" use="optional">
            <xs:annotation>
                <xs:documentation>
                    Source network the client connection binds on. This value is in
                    the form of ip/netmask. A client should match this value against
                    the desired client host network interface, and if matched the
                    client should connect to the corresponding destination values.

                    If omitted this mapping should match any interface.
                </xs:documentation>
            </xs:annotation>
        </xs:attribute>
        <xs:attribute name="destination-address" type="xs:string" use="required">
            <xs:annotation>
                <xs:documentation>
                    The destination address that a client should connect to if the
                    source-network matches. This value can either be a hostname or
                    an ip address.
                </xs:documentation>
            </xs:annotation>
        </xs:attribute>
        <xs:attribute name="destination-port" type="xs:string" use="optional">
            <xs:annotation>
                <xs:documentation>
                    The destination port that a client should connect to if the
                    source-network matches.

                    If omitted this mapping will reuse the effective socket binding
                    port.
                </xs:documentation>
            </xs:annotation>
        </xs:attribute>
    </xs:complexType>

An example:

<client-mapping source-network="192.168.1.1/24" destination-address="hostname-or-ip-address-to-use-by-the-remote-client"/>

So effectively, on the server side we now have a configuration which can let the remote clients know what IPs/hostname to use for a specific network interface on which the client is running. Of course, for this to work, the client side APIs which interact with the server should be aware of these client-mapping(s) and what they mean. In AS7, the JBoss EJB client API, is capable of dealing with these mappings on the remote client and using the correct destination address to connect to the server. The HornetQ client API, IMO, will have to start using this too - perhaps make a generic client mapping construct in HornetQ and for AS7 provide an integration which makes use of the client-mapping(s) in standalone/domain.xml.

[rodakr]

Hi jaikiran

Not exactly.

1) and 2) are correct.

3)  From stack trace:

FEIN: Trying to connect with connector = org.hornetq.core.remoting.impl.netty.NettyConnectorFactory@a4488, parameters = {port=5445, host=0.0.0.0} connector = NettyConnector AS 7 Messaging ( HORNETQ ) Client access remote:// Problem with Server listening on 0.0.0.0 Interface

15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate debug

FEIN: Started Netty Connector version 3.2.5.Final-a96d88c

15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate debug

FEIN: Trying to connect at the main server using connector :org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=0-0-0-0

15.06.2012 07:53:16 org.hornetq.core.logging.impl.JULLogDelegate debug

FEIN: Remote destination: /0.0.0.0:5445

Client side get as information 0.0.0.0 ( what is configured ) but not a List of A,B,C...  when started with "-b 0.0.0.0"

This code would gives you A,B,C if A,B,C ist configured on given server :

import java.net.*;

import java.util.*;

public class Test {

public static void main(String args[]) throws Exception {

    List<InetAddress> addrList = new ArrayList<InetAddress>();

    Enumeration<NetworkInterface> interfaces = null;

    try {

        interfaces = NetworkInterface.getNetworkInterfaces();

    } catch (SocketException e) {

        e.printStackTrace();

    }

    InetAddress localhost = null;

    try {

        localhost = InetAddress.getByName("127.0.0.1");

    } catch (UnknownHostException e) {

        e.printStackTrace();

    }

    while (interfaces.hasMoreElements()) {

        NetworkInterface ifc = interfaces.nextElement();

        Enumeration<InetAddress> addressesOfAnInterface = ifc.getInetAddresses();

        while (addressesOfAnInterface.hasMoreElements()) {

            InetAddress address = addressesOfAnInterface.nextElement();

            if (!address.equals(localhost) && !address.toString().contains(":")) {

                addrList.add(address);

                System.out.println("FOUND ADDRESS ON NIC: " + address.getHostAddress());

            }

        }

    }

}

}

Credits are going to person which post this code on http://stackoverflow.com/questions/494465/how-to-enumerate-ip-addresses-of-all-enabled-nic-cards-from-java

I quickly testet it on my OSX box.

// just A

$ java Test

FOUND ADDRESS ON NIC: 192.168.0.100

// adding B

$ sudo ifconfig en0 alias 193.168.0.200 netmask 255.255.255.0

$ java Test

FOUND ADDRESS ON NIC: 193.168.0.200

FOUND ADDRESS ON NIC: 192.168.0.100

// adding C

$ sudo ifconfig en0 alias 194.168.0.220 netmask 255.255.255.0

java Test

FOUND ADDRESS ON NIC: 194.168.0.220

FOUND ADDRESS ON NIC: 193.168.0.200

FOUND ADDRESS ON NIC: 192.168.0.100

$ ping 194.168.0.220

PING 194.168.0.220 (194.168.0.220): 56 data bytes

64 bytes from 194.168.0.220: icmp_seq=0 ttl=64 time=0.094 ms

64 bytes from 194.168.0.220: icmp_seq=1 ttl=64 time=0.069 ms

// remove B and C

$ sudo ifconfig en0 -alias 194.168.0.220

$ sudo ifconfig en0 -alias 193.168.0.200

$ java Test

FOUND ADDRESS ON NIC: 192.168.0.100

[haukem]

Hi,

I have a similar problem. The JBoss is bind to 0.0.0.0 by default and remote jndi/jms connections do not work when bind to 0.0.0.0, when the server is only bind to the external address it works.

One JBoss 7.1.2 instance on one server wants to connect to an other JBoss 7.1.2 instance on an other server.

On the server I added / modified in the standalone.xml the following way:

<socket-binding name="remoting" port="4447">

<client-mapping destination-address="10.100.1.231"/>

</socket-binding>

<socket-binding name="messaging" port="5445">

<client-mapping destination-address="10.100.1.231"/>

</socket-binding>

The client JBoss still wants to connect to 0.0.0.0 which he interprets as local host, which fails.

10.100.1.231 is the external IP address of the server. I left out the source-network option just for tests, it should be added later.

Message was edited by: Hauke Mehrtens

[jbertram]

I can't speak for JNDI, but as far as HornetQ is concerned this issue was resolved via HORNETQ-952.  You can use a AS7 nightly build to get this fix.

[haukem]

Yes, solved this by using  a more recent HornetQ version, Currently we are using 2.2.21.Final in Jboss 7.1.2, Jboss 7.1.3 already includes this version.

[will.tatam]

For reference, under 7.1.3 it looks like you must do the following as the suggestion of adding client-mapping did not work for me on 7.1.3.Final

Replace

<netty-connector name="netty" socket-binding="messaging"/>

with

<connector name="netty"><factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class><param key="host" value="my-public-ip-hostname"/><param key="port" value="5445"/></connector>

inside <hornetq-server><connectors> inside your standalone-full.xml