14 Replies Latest reply on Dec 20, 2016 7:25 AM by olegich

    WildFly 10 -  AMQP/QPID remote client failed to connect

    achmykh

      Hi,


      I'm trying to connect to WildFly 10 Message Queue using AMQP/QPID remote client (HelloWorld.java).

      Unfortunately it's hanging on connection start, but the same WildFly remote connector works just fine while using the ActiveMQ/Artemis client (Client.java).

      On other hand same AMQP/QPID client works fine by connecting to standalone Artemis.

      I tried WildFly 10 final as well latest nightly build.


      The client programs, wildfly config/log and  wireshark captures are attached.


      Any help would be much appreciated.


      Thanks,

      Alex

       

      java version "1.8.0_91"

      Java(TM) SE Runtime Environment (build 1.8.0_91-b14)

      Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

        • 1. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
          jbertram

          Try replacing your existing "netty-acceptor" with this:

           

          <remote-acceptor name="netty-acceptor" socket-binding="messaging-amqp">
            <param name="protocols" value="AMQP, CORE"/>
          </remote-acceptor>
          
          • 2. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
            achmykh

            Thanks for reply Justin, unfortunately it didn't help. I was playing with this parameter earlier and found that if the "protocols" parameter is missing then all available are supported:

             

              <remote-acceptor name="netty-acceptor" socket-binding="messaging-amqp"/>
            

             

            17:24:16,963 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 64) AMQ221020: Started Acceptor at 127.0.0.1:5672 for protocols [CORE,AMQP,HORNETQ,STOMP]
            

            vs

            <remote-acceptor name="netty-acceptor" socket-binding="messaging-amqp">
              <param name="protocols" value="AMQP,CORE"/>
              </remote-acceptor>
            

             

            17:31:43,419 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 64) AMQ221020: Started Acceptor at 127.0.0.1:5672 for protocols [CORE,AMQP]
            
            • 3. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
              jcrossley

              I'm having a pretty similar issue using the AMQPQueueExample from the artemis source: activemq-artemis/AMQPQueueExample.java at master · apache/activemq-artemis · GitHub

               

              I'm using the correct user creds and what I think is the correct remote URI (I've tried a few): "amqp://localhost:8080". It always hangs in the createSession call, as if there's some issue with the http upgrade, but I've also configured a remote-acceptor with its own socket-binding and saw the same hanging behavior.

               

              Any leads would be appreciated!

              Jim

              • 4. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                jbertram

                I believe I've just reproduced what you're seeing.  My guess is that there's some kind of bug in Artemis 1.1.0 with regard to AMQP.  Artemis 1.3.0 was released back in June which may have a fix for this issue.  If the client works against standalone Artemis then I'd just use that for the time being.

                • 5. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                  jcrossley

                  Glad to hear you're seeing it, too! Is it worth creating a JIRA for it? I noticed there's already this: [WFLY-6730] Upgrade Artemis 1.3.0 - JBoss Issue Tracker

                  But there's no fix version set for it. Any idea when it might make it into a release? Thanks.

                  • 6. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                    jbertram

                    I work on Artemis itself and not Wildfly so I don't know when that's going to make it into a release.  However, I can say that Artemis should release 1.4.0 soon so that JIRA will be out of date. 

                     

                    In any event, I've tracked down the problem.  It was a combination of things:

                     

                    1. The Artemis AMQP protocol implementation module (at <WFLY_HOME>/modules/system/layers/base/org/apache/activemq/artemis/protocol/amqp/main) needs a dependency on Netty in its module.xml (e.g. <module name="io.netty"/>).
                    2. Artemis requires Proton-J 0.10 and Wildfly ships with 0.8 so you can copy proton-j-0.10.jar and proton-jms-0.10.jar from Artemis' /lib directory to <WFLY_HOME>/modules/system/layers/base/org/apache/qpid/proton/main and update the module.xml accordingly.


                    Once you do that I believe AMQP should work.  Keep in mind that AMQP doesn't know anything about the HTTP upgrade stuff implemented in Wildfly so you'll need to create a new remote-acceptor for it.  I made these changes and also turned off security and enabled automatic JMS queue creation and the AMQP example shipped with Artemis ran successfully against Wildfly.

                    • 7. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                      jcrossley

                      Thanks, Justin. I appreciate the help, but I'm not quite home yet. After making your suggested changes, I see these spew when my client attempts to connect: *wf*.txt · GitHub

                       

                      So my client still hangs, but my server is much louder.

                       

                      I have the remote-acceptor configured. I didn't disable security, so I did see "ActiveMQSecurityException[errorType=SECURITY_EXCEPTION message=AMQ119031: Unable to validate user: null]" errors until I passed a valid user/pw to the CF in the "queue" example client, e.g.

                       

                          ConnectionFactory connectionFactory = new JmsConnectionFactory("testuser", "testuser1!", "amqp://localhost:5672");

                       

                      I'm not entirely sure how to enable automatic queue creation, but my "security-setting/role/create-non-durable-queue" attribute is set to "true". Other than the additional remote-connector/acceptor elements, I'm using the default standalone-full.xml from WF 10.1.0.Final.

                       

                      What might I be missing?

                      • 8. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                        jbertram

                        That very well could be a bug.  A lot of work has been done on the AMQP implementation between 1.1.0 and now.  Do you get the same error when you run against standalone 1.1.0 or 1.3.0?

                        • 9. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                          jcrossley

                          I've only tried with the standalone server generated by the examples on the master branch. I can download the distros for 1.1 and 1.3 and see. I'm curious what's different about our environments that you don't see that error, though.

                          • 10. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                            jcrossley

                            Ok, Justin, both the 1.1 and 1.3 standalone servers seem to work, except they both toss a NullPointerException in AbstractConnectionContext$LocalListener.onFlow when I run the proton-ruby receive.rb script. The send.rb works fine, as do the proton-cpp and queue projects. Keep in mind that these clients are using the 1.5-SNAPSHOT master branch, so that NPE is probably expected.

                             

                            Regarding the HTTP upgrade, it looks as if the NettyConnector does support a "httpUpgradeEnabled" option. Is that not something I can pass to the ConnectionFactory? Is that expected to work eventually?

                             

                            Thanks, again!

                            • 11. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                              jbertram

                              Regarding the HTTP upgrade, it looks as if the NettyConnector does support a "httpUpgradeEnabled" option. Is that not something I can pass to the ConnectionFactory? Is that expected to work eventually?

                              I wouldn't expect you to be able to pass anything related to the Artemis Netty connector to an AMQP JMS client implementation, and I wouldn't expect that to work eventually either, at least not without some kind of change to the AMQP specification.  Remember, AMQP specifies both a wire protocol and an API (as opposed to say JMS which specifies an API but not a wire protocol).

                              • 12. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                                jcrossley

                                Yeah, I realized that after I posted it. I might expect the artemis client to support it, at some point, just as the hornetq client did, but I certainly wouldn't expect *any* AMQP client to support it, especially a .NET one, which is what I'm working with specifically. Thanks for the reply anyway.

                                • 13. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                                  jbertram

                                  To be clear, the Artemis client does support the HTTP upgrade functionality in Wildfly (via the httpUpgradeEnabled and httpUpgradeEndpoint properties).

                                  • 14. Re: WildFly 10 -  AMQP/QPID remote client failed to connect
                                    olegich

                                    Hello,

                                     

                                    I'm experiencing problem similar to described in original post.

                                    After corrections suggested by Justin:

                                    • The Artemis AMQP protocol implementation module (at <WFLY_HOME>/modules/system/layers/base/org/apache/activemq/artemis/protocol/amqp/main) needs a dependency on Netty in its module.xml (e.g. <module name="io.netty"/>).
                                    • Artemis requires Proton-J 0.10 and Wildfly ships with 0.8 so you can copy proton-j-0.10.jar and proton-jms-0.10.jar from Artemis' /lib directory to <WFLY_HOME>/modules/system/layers/base/org/apache/qpid/proton/main and update the module.xml accordingly.

                                     

                                    The server starts throwing exception right after connecting remote client to it:

                                    13:01:00,713 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955)) java.nio.ReadOnlyBufferException

                                    13:01:00,716 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at java.nio.ByteBuffer.array(ByteBuffer.java:996)

                                    13:01:00,720 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.buffer.UnsafeByteBufUtil.setBytes(UnsafeByteBufUtil.java:368)

                                    13:01:00,730 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:205)

                                    13:01:00,734 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:877)

                                    13:01:00,745 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.proton.plug.handler.impl.ProtonHandlerImpl.outputBuffer(ProtonHandlerImpl.java:226)

                                    13:01:00,749 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.proton.plug.context.AbstractConnectionContext.flushBytes(AbstractConnectionContext.java:145)

                                    13:01:00,760 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.proton.plug.context.AbstractConnectionContext$LocalListener.onTransport(AbstractConnectionContext.java:160)

                                    13:01:00,766 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.proton.plug.handler.impl.ProtonHandlerImpl.dispatch(ProtonHandlerImpl.java:349)

                                    13:01:00,770 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.proton.plug.handler.impl.ProtonHandlerImpl.flush(ProtonHandlerImpl.java:257)

                                    13:01:00,783 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.proton.plug.handler.impl.ProtonHandlerImpl.inputBuffer(ProtonHandlerImpl.java:158)

                                    13:01:00,789 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.proton.plug.context.AbstractConnectionContext.inputBuffer(AbstractConnectionContext.java:81)

                                    13:01:00,800 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.apache.activemq.artemis.core.protocol.proton.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:127)

                                    13:01:00,807 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:619)

                                    13:01:00,814 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)

                                    13:01:00,820 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)

                                    13:01:00,832 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)

                                    13:01:00,839 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:216)

                                    13:01:00,849 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527)

                                    13:01:00,855 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521)

                                    13:01:00,866 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351)

                                    13:01:00,871 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:322)

                                    13:01:00,882 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:299)

                                    13:01:00,887 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder.decode(ProtocolHandler.java:175)

                                    13:01:00,902 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:360)

                                    13:01:00,913 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)

                                    13:01:00,920 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder.channelRead(ProtocolHandler.java:118)

                                    13:01:00,931 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)

                                    13:01:00,937 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)

                                    13:01:00,949 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)

                                    13:01:00,954 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)

                                    13:01:00,965 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

                                    13:01:00,969 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

                                    13:01:00,981 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

                                    13:01:00,986 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

                                    13:01:00,998 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)

                                    13:01:01,003 ERROR [stderr] (Thread-1 (activemq-netty-threads-1534405955))      at java.lang.Thread.run(Thread.java:745)

                                    I tried also last night build but it behaves the same except doesn't throw the exception.

                                    Could someone help with this please?