1 2 Previous Next 28 Replies Latest reply on Sep 22, 2017 11:53 PM by Preeta Kuruvilla

    Upgrade to Wildfly 10.0.0.Final: Messaging System issue

    Preeta Kuruvilla Newbie

      Widlffly 8.2 was based on HornetQ JMS message broker while Wildfly 10 uses ActiveMQ JMS message broker.

      The HornetQ code base was donated to the Apache ActiveMQ community late last year and now resides as a sub project under the ActiveMQ umbrella named 'Artemis’. http://activemq.apache.org/artemis.

      http://hornetq.blogspot.in/2015/06/hornetq-apache-donation-and-apache.html

       

      The issue we are facing is after we upgraded our application to wildfly 10.0.0.Final from Wildfly 8.2, the messages are not consistently getting into the queue. This issue is not seen for Wildfly-8.2.

       

      Below is how we configured the messaging subsystem in standalone-full.xml for Wildfly 10.0.0.Final :-


      Let me know if this is good. Also I have attached the entire standalone-full.xml with this case.

       

      <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
      <server name="default">
      <security enabled="false"/>
      <statistics enabled="true"/>
      <security-setting name="#">
      <role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
      <role name="jmsrole" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
      </security-setting>
      <address-setting name="#" redistribution-delay="1000" message-counter-history-day-limit="10" page-size-bytes="2097152" max-size-bytes="10485760" expiry-address="jms.queue.ExpiryQueue" dead-letter-address="jms.queue.DLQ"/>
      <http-connector name="http-connector" endpoint="http-acceptor" socket-binding="remote-http"/>
      <http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http"/>
      <in-vm-connector name="in-vm" server-id="0"/>
      <http-acceptor name="http-acceptor" http-listener="default"/>
      <http-acceptor name="http-acceptor-throughput" http-listener="default">
      <param name="batch-delay" value="50"/>
      <param name="direct-deliver" value="false"/>
      </http-acceptor>
      <in-vm-acceptor name="in-vm" server-id="0"/>
      <jms-queue name="testQueue" entries="java:/queue/test java:jboss/exported/jms/queue/test"/>
      <jms-queue name="ISEEOutboundQueue" entries="java:/ISEEOutboundQueue java:jboss/exported/jms/queue/ISEEOutboundQueue"/>
      <jms-queue name="ISEEInboundQueue" entries="java:/ISEEInboundQueue java:jboss/exported/jms/queue/ISEEInboundQueue"/>
      <jms-queue name="BEEEAuthorizationsQueue" entries="java:/BEEEAuthorizationsQueue java:jboss/exported/jms/queue/BEEEAuthorizationsQueue"/>
      <jms-queue name="BEEERequisitionsQueue" entries="java:/BEEERequisitionsQueue java:jboss/exported/jms/queue/BEEERequisitionsQueue"/>
      <jms-queue name="BEEEInboundQueue" entries="java:/BEEEInboundQueue java:jboss/exported/jms/queue/BEEEInboundQueue"/>
      <jms-topic name="testTopic" entries="java:/topic/test java:jboss/exported/jms/topic/test"/>
      <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
      <connection-factory name="RemoteConnectionFactory" entries="java:/RemoteConnectionFactory java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
      <pooled-connection-factory name="activemq-ra" transaction="xa" entries="/JmsXA jboss/DefaultJMSConnectionFactory" connectors="in-vm"/>
      </server>
      </subsystem>

      InitialContext is created with parameters - jmsuser, password, and
      BEEERequisitions.JndiProviderUrl=http-remoting://<ip-address>:<port>
      BEEERequisitions.JndiFactory=org.jboss.naming.remote.client.InitialContextFactory

      Issue we face is that after we upgraded to wildfly 10, the messages are not getting in the queues. This is not consistent as once in a while messages are getting processed.

       

       

       

      I have attached below standalone-full.xmls for both Wildfly-8.2 where it works properly as well as standalone-full.xml for Wildfly 10.0.0.Final where we have JMS issue for your reference :-

      1. standalone-full-WL-8.2.xml is for Wildfly 8.2 which is working fine.

      2. standalone-full-WL-10.0.Final.xml is for Wildfly 10.0.Final which has this JMS issue.

       

        • 1. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
          Justin Bertram Master

          Your configuration looks fine as far as I can tell.  Do you have a test-case which I could use to reproduce the failure you're seeing?

          • 2. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
            Jeff Mesnil Master

            I don't understand your configuration.

             

            You have defined a http-connector to connect to 10.76.80.29:${jboss.http.port:6080} that is used by your remote JMS connection factory.

            This means that your JMS client will connect to the server at this address, not on  your own server which accepts connections on its http port at 8080.

             

            Is this the intended use case?

            • 3. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
              Preeta Kuruvilla Newbie

              Hi Jeff,

               

              Thanks for responding. So we have for our application, 2 wars deployed - RC.war and SL.war. The JMS is configured on SL. The http port for RC is 8080 and http port for SL is 6080.

               

              RC connects to SL through JMS on this http port which is 6080 (by utilizing HTTP upgrade). I have attached a working standalone-full.xml from Wildfly 8.2 with the same configuration. You may perhaps refer that.

               

              Thanks,

              Preeta

              1 of 1 people found this helpful
              • 4. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                Avinash G A Newbie

                What do you mean by "the messages are not consistently getting into the queue" ? Are the messages queued successfully anytime or they are failing always? If failing whats in the log?

                • 5. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                  Preeta Kuruvilla Newbie

                  Hi Avinash,

                   

                  Messaging are occasionally getting into queue. The logs got no errors or exception.

                  • 6. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                    Avinash G A Newbie

                    A message producer will always throws an exception in the case of failure to deliver the message to a queue/topic. It helps if you can provide the unit tests to reproduce.

                    • 7. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                      Preeta Kuruvilla Newbie

                      Exceptions will only be thrown when there is a issue when sending or receiving messages and currently we don't have any exceptions. Here the only thing that has changed is the configuration in standalone-full.xml for wildfly 10.

                      • 8. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                        Miroslav Novak Master

                        Hi Preeta,

                         

                        I tried you WF10 config and could send and receive messages using RemoteConnectionFactory (I just disabled security in messaging-activemq subsystem). Could you check that when RC.war sends messages then those messages are in the queue using CLI, for example like:

                         

                        /subsystem=messaging-activemq/server=default/jms-queue=ISEEInboundQueue:count-messages()

                         

                        Does SL.war contain MDB which is consuming messages from some queue? How it's configured?

                        Could you enable tracing for org.apache.activemq.artemis?

                         

                        Thanks,

                        Mirek

                        • 9. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                          Preeta Kuruvilla Newbie

                          Thanks for trying it out at your end and responding. Will try out what you suggested and get back soon.

                          • 10. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                            Preeta Kuruvilla Newbie

                            Hi Mirek,

                             

                            I enabled Trace for org.apache.activemq.artemis and then submitted a Requistion from RC.

                             

                            I see the below in the logs now:-

                             

                            2016-09-22 14:19:03,520 DEBUG [org.apache.activemq.artemis.core.client] (default task-22) Session was already closed, giving up now, this=ClientSessionImpl [name=2fe7b853-810a-11e6-9c7e-49748552169f, username=jmsuser, closed=true, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@77a24f35, metaData=(jms-session=,)]@1b5c6377

                            2016-09-22 14:19:03,521 TRACE [org.apache.activemq.artemis.core.client] (Thread-6 (ActiveMQ-client-global-threads-1058954121)) AMQ214026: Failure captured on connectionID=-94,657,003, performing failover or reconnection now: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ119006: Channel disconnected]

                                at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:342)

                                at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:911)

                                at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100)

                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                                at java.lang.Thread.run(Thread.java:745)

                             

                            So we have to configure the JMS Failover ?

                             

                            Thanks,

                            Preeta

                            • 11. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                              Miroslav Novak Master

                              JMS Failover/HA is not necessary. It seems that for some reason JMS connection created by RC.war was disconnected. There might be more information on server with SL.war. Could you try to enable tracing for Artemis on server with SL.war and check happens in time when RC.war was disconnected?

                               

                              Also I think that you could try to configure infinite reconnect-attemts on RemoteConnectionFactory, following configuration should do that:

                              <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>

                              • 12. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                                Preeta Kuruvilla Newbie

                                Also after the above error, we have the below in logs that say -there was nothing in BEEERequsitionQueue.

                                 

                                2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Thread-1 (ActiveMQ-client-netty-threads-51974265)) handling packet PACKET(SessionReceiveMessage)[type=75, channelID=14, packetObject=SessionReceiveMessage,message=ClientMessage[messageID=8602897902, durable=false, address=jms.queue.BEEERequisitionsQueue,userID=null,properties=TypedProperties[_hornetq.forced.delivery.seq=125]]]

                                2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #9 - JmsConsumer[BEEERequisitionsQueue]) There was nothing on the queue, leaving it now:: returning null

                                2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #8 - JmsConsumer[BEEERequisitionsQueue]) Returning null

                                2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #8 - JmsConsumer[BEEERequisitionsQueue]) Forcing delivery

                                2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #8 - JmsConsumer[BEEERequisitionsQueue]) Sending packet nonblocking PACKET(SessionForceConsumerDelivery)[type=78, channelID=12, packetObject=SessionForceConsumerDelivery, consumerID=0, sequence=125] on channeID=12

                                2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #8 - JmsConsumer[BEEERequisitionsQueue]) Writing buffer for channelID=12

                                2016-09-22 14:19:03,602 TRACE [org.apache.activemq.artemis.core.client] (Thread-1 (ActiveMQ-client-netty-threads-51974265)) handling packet PACKET(SessionReceiveMessage)[type=75, channelID=12, packetObject=SessionReceiveMessage,message=ClientMessage[messageID=8602897903, durable=false, address=jms.queue.BEEERequisitionsQueue,userID=null,properties=TypedProperties[_hornetq.forced.delivery.seq=125]]]

                                 

                                Found these link:[JBEAP-1527] Clients wait for connection timeout during failback - JBoss Issue Tracker

                                activemq-artemis/ClientSessionFactoryImpl.java at master · apache/activemq-artemis · GitHub (look for "ActiveMQNotConnectedException")

                                • 13. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                                  Preeta Kuruvilla Newbie

                                  In the SL log I see the below:-

                                   

                                  15:11:17,053 DEBUG [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) Cache container web will use an externalizer for org.wildfly.clustering.web.infinispan.session.SessionAccessMetaDataKey

                                  15:11:17,051 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) artemis-native-64 -> error loading the native library: java.lang.UnsatisfiedLinkError: no artemis-native-64 in java.library.path

                                      at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) [rt.jar:1.8.0_77]

                                      at java.lang.Runtime.loadLibrary0(Runtime.java:870) [rt.jar:1.8.0_77]

                                      at java.lang.System.loadLibrary(System.java:1122) [rt.jar:1.8.0_77]

                                      at org.apache.activemq.artemis.jlibaio.LibaioContext.loadLibrary(LibaioContext.java:68) [artemis-native-1.1.0.wildfly-011.jar:]

                                      at org.apache.activemq.artemis.jlibaio.LibaioContext.<clinit>(LibaioContext.java:88) [artemis-native-1.1.0.wildfly-011.jar:]

                                      at org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory.isSupported(AIOSequentialFileFactory.java:112) [artemis-journal-1.1.0.wildfly-011.jar:]

                                      at org.wildfly.extension.messaging.activemq.ActiveMQServerService.start(ActiveMQServerService.java:155) [wildfly-messaging-activemq-10.0.0.Final.jar:10.0.0.Final]

                                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

                                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_77]

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_77]

                                      at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_77]

                                   

                                  15:11:17,056 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) Library artemis-native-64 not found!

                                  15:11:17,057 DEBUG [org.jboss.modules] (MSC service thread 1-7) Module org.joda.time:main defined by local module loader @3930015a (finder: local module finder @629f0666 (roots: C:\newcluster-pc-all-10\wildfly-10.0.0.Final\modules,C:\newcluster-pc-all-10\wildfly-10.0.0.Final\modules\system\layers\base))

                                  15:11:17,057 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) artemis-native-32 -> error loading the native library: java.lang.UnsatisfiedLinkError: no artemis-native-32 in java.library.path

                                      at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) [rt.jar:1.8.0_77]

                                      at java.lang.Runtime.loadLibrary0(Runtime.java:870) [rt.jar:1.8.0_77]

                                      at java.lang.System.loadLibrary(System.java:1122) [rt.jar:1.8.0_77]

                                      at org.apache.activemq.artemis.jlibaio.LibaioContext.loadLibrary(LibaioContext.java:68) [artemis-native-1.1.0.wildfly-011.jar:]

                                      at org.apache.activemq.artemis.jlibaio.LibaioContext.<clinit>(LibaioContext.java:88) [artemis-native-1.1.0.wildfly-011.jar:]

                                      at org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory.isSupported(AIOSequentialFileFactory.java:112) [artemis-journal-1.1.0.wildfly-011.jar:]

                                      at org.wildfly.extension.messaging.activemq.ActiveMQServerService.start(ActiveMQServerService.java:155) [wildfly-messaging-activemq-10.0.0.Final.jar:10.0.0.Final]

                                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

                                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_77]

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_77]

                                      at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_77]

                                   

                                  15:11:17,058 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) Library artemis-native-32 not found!

                                  15:11:17,058 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) Couldn't locate LibAIO Wrapper

                                  1 2 Previous Next