WildFly 8 and 9. Connecting to topic using http-remoting fails when server is behind NAT firewall

[geturner]

I was hoping version 9 would fix this problem, but alas, not.

Caused by: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers

    at org.hornetq.core.clinet.impl.ServerLocatorImpl.createSessionFactory(ServletLocatorImpl.java:905)

We are using standard http-remoting connection code.  We can lookup and call remote EJB methods fine.  We have used a workaround in the code that involves adding hornet to the plugin manifest and adding this line in the code:

     final ConnectionFactory cf = (ConnectionFactory) ctx.lookup("jms/RemoteConnectionFactory");

    if (HornetQJMSConnectionFactory.class.isAssignableFrom(cf.getClass())) {

        ((HornetQJMSConnectionFactory)cf).getServletLocator().getStaticTransportConfigurations()[0].getParams().put("host", ip);

    }

Nothing else seems to work, any help would be great.

[jaikiran]

Could you please add more context to this, including complete exception stacktraces and more details on what you are doing? It's hard to understand what this is about without that context? Was there any bug report that was filed against 8.x and was expected to be fixed in 9.x?

[geturner]

This is actually pretty straight forward and I am shocked that this fails, as it has been a standard capability in JMS for years, but I suspect that the http-remoting has something to do with a mis-connect with HornetQ.

Simple client call:

Properties topicProperties = new Properties();

topicProperties.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");

topicProperties.put(Context.PROVIDER_URL, "http-remoting://" + host + ":" + port);

InitialContext ctx = new InitialContext(topicProperties);

ConnectionFactory tmp = (ConnectionFactory) topicCtx.lookup("jms/RemoteConnectionFactory");

connection = tmp.createConnection();

Results in:

Jun 11, 2015 8:26:07 AM org.xnio.Xnio <clinit>

INFO: XNIO version 3.3.1.Final

Jun 11, 2015 8:26:07 AM org.xnio.nio.NioXnio <clinit>

INFO: XNIO NIO Implementation Version 3.3.1.Final

Jun 11, 2015 8:26:07 AM org.jboss.remoting3.EndpointImpl <clinit>

INFO: JBoss Remoting version 4.0.9.Final

javax.jms.JMSException: Failed to create session factory

  at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:673)

  at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:112)

  at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:107)

  at com.lmco.spacefence.netcentric.test.client.TestMessageConsumer.<init>(TestMessageConsumer.java:36)

  at com.lmco.spacefence.netcentric.test.client.TestMessageConsumer.main(TestMessageConsumer.java:24)

Caused by: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.]

  at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:905)

  at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:669)

  ... 4 more

Disconnected from the target VM, address: '127.0.0.1:54275', transport: 'socket'

The server is behind a NAT firewall

[geturner]

By the way, the ConnectionFactory instance returned shows the following in a debugger.  The params host value is of the IP of the server itself behind the NAT firewall instead of the IP address that we connect to.

initialConnectors = {org.hornetq.api.core.TransportConfiguration[1]@2183}

0 = {org.hornetq.api.core.TransportConfiguration@2196} "TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=8080&host=10-10-20-77&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor"

  name = {java.lang.String@2198} "http-connector"

  factoryClassName = {java.lang.String@2199} "org.hornetq.core.remoting.impl.netty.NettyConnectorFactory"

  params = {java.util.HashMap@2200}  size = 4

   0 = {java.util.HashMap$Node@2203} "port" -> "8080"

   1 = {java.util.HashMap$Node@2204} "host" -> "10.10.20.77"

   2 = {java.util.HashMap$Node@2205} "http-upgrade-enabled" -> "true"

   3 = {java.util.HashMap$Node@2206} "http-upgrade-endpoint" -> "http-acceptor"

[jaikiran]

This has been reported a few times and I thought this was solved but it looks like not. Here's one such thread discussing this AS 7 Messaging ( HORNETQ ) Client access remote:// Problem with Server listening on 0.0.0.0 Interface (it starts off on a slightly different note), but the underlying problem is the same. Read through that thread.

The workaround appears to be Re: AS 7 Messaging ( HORNETQ ) Client access remote:// Problem with Server listening on 0.0.0.0 Interface (it was being discussed in the context of AS7, so you'll have to check if that works for WildFly 8/9). Here's an explanation of that workaround https://developer.jboss.org/message/810113#810113

[geturner]

Would you please provide a concrete solution that works for WildFly 8/9?  The references that you have provided show multiple solutions, going back to version 7, and the configuration is different.  I cannot determine if adding the netty connector (where I have no "messaging" connector) is the solution or by using the client-mapping, which I believe is probably what I need to do, but again, I have no socket-binding or messaging or remoting, as they were removed.

[geturner]

I tried to use the client mapping, no success

[geturner]

BTW, I would like to enter this as a current bug, can you help with that?

[jaikiran]

You can file bug reports (JIRA) here WildFly - JBoss Issue Tracker. Post relevant details including the sample code/application if any and point to this thread.

[jukka.sirvio]

Did you find workaround to this? I have quite exactly similar problem!

Normal JMS messaging have (of course) worked fine for years...

[jukka.sirvio]

Ok, it was quite easy to solve in JMS client, below is related code how to do it:

......

     final Properties env = new Properties();

  env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");

  env.put(Context.PROVIDER_URL, "http-remoting://xxx.xxx.xxx.xxx:8080"); //<-- put here the natted ip address or dns name & port

  env.put(Context.SECURITY_PRINCIPAL, "user");

  env.put(Context.SECURITY_CREDENTIALS, "pass");

     Context context;

  try {

  context = new InitialContext(env);

            //<-- begin of essential part

  Map<String, Object> connectionParams = new HashMap<String, Object>();

            connectionParams.put(org.hornetq.core.remoting.impl.netty.TransportConstants.HOST_PROP_NAME, "xxx.xxx.xxx.xxx"); // <-- put here the natted ip address!

            connectionParams.put(org.hornetq.core.remoting.impl.netty.TransportConstants.PORT_PROP_NAME, "8080");

            connectionParams.put(org.hornetq.core.remoting.impl.netty.TransportConstants.HTTP_UPGRADE_ENABLED_PROP_NAME, "true");

            connectionParams.put(org.hornetq.core.remoting.impl.netty.TransportConstants.HTTP_UPGRADE_ENDPOINT_PROP_NAME, "http-acceptor");

            TransportConfiguration transportConfiguration =

                new TransportConfiguration(

                "org.hornetq.core.remoting.impl.netty.NettyConnectorFactory",

                connectionParams);

            //<-- end of essential part

            connectionFactory = HornetQJMSClient.createConnectionFactoryWithoutHA(JMSFactoryType.TOPIC_CF, transportConfiguration);

  /// continue as usual, destination, connection, session, producer/consumer etc...

......

[jukka.sirvio]

Dear George Turner, this is response to your inquiry on WFLY-4769 Jira, here is the code for wILDFLY 10.0 ArtemisMQ, Please mark this as answer if you find this useful !!

String serverName = "xxxyyyzzz";

env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");

env.put(Context.PROVIDER_URL, serverName);

env.put(Context.SECURITY_PRINCIPAL, "user");

env.put(Context.SECURITY_CREDENTIALS, "pass");

Context context;

try {

context = new InitialContext(env);

Map<String, Object> connectionParams = new HashMap<String, Object>();

connectionParams.put(org.apache.activemq.artemis.core.remoting.impl.netty.TransportConstants.HOST_PROP_NAME, "127.0.0.1"); // <-- PUT THE NATTED IP HERE!

connectionParams.put(org.apache.activemq.artemis.core.remoting.impl.netty.TransportConstants.PORT_PROP_NAME, "80");

connectionParams.put(org.apache.activemq.artemis.core.remoting.impl.netty.TransportConstants.HTTP_UPGRADE_ENABLED_PROP_NAME, "true");

connectionParams.put(org.apache.activemq.artemis.core.remoting.impl.netty.TransportConstants.HTTP_UPGRADE_ENDPOINT_PROP_NAME, "http-acceptor");

TransportConfiguration transportConfiguration =

   new org.apache.activemq.artemis.api.core.TransportConfiguration(

     "org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory",

     connectionParams);

connectionFactory = ActiveMQJMSClient.createConnectionFactoryWithoutHA(JMSFactoryType.TOPIC_CF, transportConfiguration);

[buddhikasgj]

Dear jukka sirviö,

Does this require special configuration in jboss. Im using above JMS client code with below standalong-full.xml. But still ended up with below messages from log

[Remoting "config-based-naming-client-endpoint" task-2] DEBUG org.jboss.remoting.remote.client - Client authentication failed for mechanism JBOSS-LOCAL-USER: javax.security.sasl.SaslException: Failed to read server challenge [Caused by java.io.FileNotFoundException: \home\janaka\jboss-eap-7.0\standalone\tmp\auth\local4810015721206975876.challenge (The system cannot find the path specified)]

.......

.......

ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ119007: Cannot connect to server(s). Tried with all available servers.

Im using  Jboss EAP 7.0 which is based on wildfly 10.0.

Please note that my JMS queus can be accessed without any issue from same server or any other server from the same network but cant be accessed from out side through NAT gateway.  Any kind of clue help would be highly appreciated.

Standalone.xml

  <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
  <server name="default">
  <security-setting name="#">
  <role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
  </security-setting>
  <address-setting name="#" message-counter-history-day-limit="10" page-size-bytes="2097152" max-size-bytes="10485760" expiry-address="jms.queue.ExpiryQueue" dead-letter-address="jms.queue.DLQ"/>
  <http-connector name="http-connector" endpoint="http-acceptor" socket-binding="http"/>
  <http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http">
  <param name="batch-delay" value="50"/>
  </http-connector>
  <in-vm-connector name="in-vm" server-id="0"/>
  <http-acceptor name="http-acceptor" http-listener="default"/>
  <http-acceptor name="http-acceptor-throughput" http-listener="default">
  <param name="batch-delay" value="50"/>
  <param name="direct-deliver" value="false"/>
  </http-acceptor>
  <in-vm-acceptor name="in-vm" server-id="0"/>
  <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
  <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
  <jms-queue name="fromNGSQueue" entries="java:/jboss/exported/jms/queue/fromNGS"/>
  <jms-queue name="toNGSQueue" entries="java:/jboss/exported/jms/queue/toNGS"/>
  <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
  <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
  <pooled-connection-factory name="activemq-ra" transaction="xa" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm"/>
  </server>
  </subsystem>

Console output.

14:13:55.301 [main] DEBUG org.jboss.logging - Logging Provider: org.jboss.logging.Slf4jLoggerProvider

14:13:55.326 [main] DEBUG org.jboss.naming.remote.client.InitialContextFactory - Looking for jboss-naming-client.properties using classloader sun.misc.Launcher$AppClassLoader@4e0e2f2a

14:13:55.327 [main] DEBUG org.jboss.naming.remote.client.InitialContextFactory - jboss.naming.client.endpoint.create.options. has the following options {}

14:13:55.328 [main] DEBUG org.jboss.naming.remote.client.InitialContextFactory - jboss.naming.client.remote.connectionprovider.create.options. has the following options {}

14:13:55.349 [main] INFO org.xnio - XNIO version 3.3.6.Final

14:13:55.367 [main] INFO org.xnio.nio - XNIO NIO Implementation Version 3.3.6.Final

14:13:55.559 [Remoting "config-based-naming-client-endpoint" Accept] DEBUG org.xnio.nio - Started channel thread 'Remoting "config-based-naming-client-endpoint" Accept', selector sun.nio.ch.WindowsSelectorImpl@714ebb1c

14:13:55.559 [Remoting "config-based-naming-client-endpoint" I/O-1] DEBUG org.xnio.nio - Started channel thread 'Remoting "config-based-naming-client-endpoint" I/O-1', selector sun.nio.ch.WindowsSelectorImpl@6e6846c0

14:13:55.603 [main] INFO org.jboss.remoting - JBoss Remoting version 4.0.0.Final

14:13:55.628 [main] DEBUG org.jboss.naming.remote.client.InitialContextFactory - jboss.naming.client.connect.options. has the following options {}

14:13:55.756 [main] DEBUG org.apache.activemq.artemis.logs - using dummy address ffffffa1:46:3f:fffffff1:78:76

14:13:58.205 [Remoting "config-based-naming-client-endpoint" task-2] DEBUG org.jboss.remoting.remote.client - Client authentication failed for mechanism JBOSS-LOCAL-USER: javax.security.sasl.SaslException: Failed to read server challenge [Caused by java.io.FileNotFoundException: \home\janaka\jboss-eap-7.0\standalone\tmp\auth\local4810015721206975876.challenge (The system cannot find the path specified)]

14:13:59.878 [main] DEBUG org.apache.activemq.artemis.core.client - Trying reconnection attempt 0/1

14:13:59.878 [main] DEBUG org.apache.activemq.artemis.core.client - Trying to connect with connector = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@491666ad, parameters = {httpUpgradeEnabled=true, port=8080, httpPpgradeEndpoint=http-acceptor, host=192.168.1.154} connector = null

14:13:59.888 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework

14:13:59.892 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available

14:13:59.892 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available

14:13:59.892 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available

14:13:59.892 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true

14:13:59.893 [main] DEBUG io.netty.util.internal.PlatformDependent - Platform: Windows

14:13:59.894 [main] DEBUG io.netty.util.internal.PlatformDependent - Java version: 8

14:13:59.894 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false

14:13:59.894 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available

14:13:59.894 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noJavassist: false

14:13:59.895 [main] DEBUG io.netty.util.internal.PlatformDependent - Javassist: unavailable

14:13:59.895 [main] DEBUG io.netty.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes.  Please check the configuration for better performance.

14:13:59.896 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: C:\Users\user\AppData\Local\Temp (java.io.tmpdir)

14:13:59.896 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)

14:13:59.896 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false

14:13:59.899 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetectionLevel: simple

14:13:59.918 [main] DEBUG io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 8

14:13:59.935 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false

14:13:59.936 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512

14:13:59.961 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768

14:13:59.962 [main] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192

14:13:59.975 [main] DEBUG org.apache.activemq.artemis.core.client - Started Netty Connector version 4.0.30.Final

14:13:59.975 [main] DEBUG org.apache.activemq.artemis.core.client - Remote destination: /192.168.1.154:8080

14:13:59.982 [main] DEBUG io.netty.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0x2f1ff72a552357d5 (took 1 ms)

14:14:00.014 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: unpooled

14:14:00.014 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 65536

14:14:21.053 [main] DEBUG org.apache.activemq.artemis.core.client - Connector towards NettyConnector [host=192.168.1.154, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed

Exception in thread "main" javax.jms.JMSRuntimeException: Failed to create session factory

  at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:88)

  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:262)

  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:248)

  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:238)

  at com.vishuo.icmdb.ngs.jms.HelloWorldJMSConsumer2.main(HelloWorldJMSConsumer2.java:63)

Caused by: javax.jms.JMSException: Failed to create session factory

  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:727)

  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:255)

  ... 3 more

Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ119007: Cannot connect to server(s). Tried with all available servers.]

  at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:775)

  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:724)

  ... 4 more

14:14:21.124 [Remoting "config-based-naming-client-endpoint" task-9] DEBUG org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1 - Channel end notification received, closing channel Channel ID b6654f8f (outbound) of Remoting connection 274dfe6d to vishuo.ddns.net/203.116.141.46:1800

[jbertram]

The question related to this forum thread has already been answered.  Please start a new thread with a full explanation of your use-case and problem.