Upgrade to Wildfly 10.0.0.Final: Messaging System issue

[kpreeta12]

Widlffly 8.2 was based on HornetQ JMS message broker while Wildfly 10 uses ActiveMQ JMS message broker.

The HornetQ code base was donated to the Apache ActiveMQ community late last year and now resides as a sub project under the ActiveMQ umbrella named 'Artemis’. http://activemq.apache.org/artemis.

http://hornetq.blogspot.in/2015/06/hornetq-apache-donation-and-apache.html

The issue we are facing is after we upgraded our application to wildfly 10.0.0.Final from Wildfly 8.2, the messages are not consistently getting into the queue. This issue is not seen for Wildfly-8.2.

Below is how we configured the messaging subsystem in standalone-full.xml for Wildfly 10.0.0.Final :-

Let me know if this is good. Also I have attached the entire standalone-full.xml with this case.

<subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
<server name="default">
<security enabled="false"/>
<statistics enabled="true"/>
<security-setting name="#">
<role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
<role name="jmsrole" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
</security-setting>
<address-setting name="#" redistribution-delay="1000" message-counter-history-day-limit="10" page-size-bytes="2097152" max-size-bytes="10485760" expiry-address="jms.queue.ExpiryQueue" dead-letter-address="jms.queue.DLQ"/>
<http-connector name="http-connector" endpoint="http-acceptor" socket-binding="remote-http"/>
<http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http"/>
<in-vm-connector name="in-vm" server-id="0"/>
<http-acceptor name="http-acceptor" http-listener="default"/>
<http-acceptor name="http-acceptor-throughput" http-listener="default">
<param name="batch-delay" value="50"/>
<param name="direct-deliver" value="false"/>
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
<jms-queue name="testQueue" entries="java:/queue/test java:jboss/exported/jms/queue/test"/>
<jms-queue name="ISEEOutboundQueue" entries="java:/ISEEOutboundQueue java:jboss/exported/jms/queue/ISEEOutboundQueue"/>
<jms-queue name="ISEEInboundQueue" entries="java:/ISEEInboundQueue java:jboss/exported/jms/queue/ISEEInboundQueue"/>
<jms-queue name="BEEEAuthorizationsQueue" entries="java:/BEEEAuthorizationsQueue java:jboss/exported/jms/queue/BEEEAuthorizationsQueue"/>
<jms-queue name="BEEERequisitionsQueue" entries="java:/BEEERequisitionsQueue java:jboss/exported/jms/queue/BEEERequisitionsQueue"/>
<jms-queue name="BEEEInboundQueue" entries="java:/BEEEInboundQueue java:jboss/exported/jms/queue/BEEEInboundQueue"/>
<jms-topic name="testTopic" entries="java:/topic/test java:jboss/exported/jms/topic/test"/>
<connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
<connection-factory name="RemoteConnectionFactory" entries="java:/RemoteConnectionFactory java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
<pooled-connection-factory name="activemq-ra" transaction="xa" entries="/JmsXA jboss/DefaultJMSConnectionFactory" connectors="in-vm"/>
</server>
</subsystem>

InitialContext is created with parameters - jmsuser, password, and
BEEERequisitions.JndiProviderUrl=http-remoting://<ip-address>:<port>
BEEERequisitions.JndiFactory=org.jboss.naming.remote.client.InitialContextFactory

Issue we face is that after we upgraded to wildfly 10, the messages are not getting in the queues. This is not consistent as once in a while messages are getting processed.

I have attached below standalone-full.xmls for both Wildfly-8.2 where it works properly as well as standalone-full.xml for Wildfly 10.0.0.Final where we have JMS issue for your reference :-

1. standalone-full-WL-8.2.xml is for Wildfly 8.2 which is working fine.

2. standalone-full-WL-10.0.Final.xml is for Wildfly 10.0.Final which has this JMS issue.

[jbertram]

Your configuration looks fine as far as I can tell.  Do you have a test-case which I could use to reproduce the failure you're seeing?

[jmesnil]

I don't understand your configuration.

You have defined a http-connector to connect to 10.76.80.29:${jboss.http.port:6080} that is used by your remote JMS connection factory.

This means that your JMS client will connect to the server at this address, not on  your own server which accepts connections on its http port at 8080.

Is this the intended use case?

[kpreeta12]

Hi Jeff,

Thanks for responding. So we have for our application, 2 wars deployed - RC.war and SL.war. The JMS is configured on SL. The http port for RC is 8080 and http port for SL is 6080.

RC connects to SL through JMS on this http port which is 6080 (by utilizing HTTP upgrade). I have attached a working standalone-full.xml from Wildfly 8.2 with the same configuration. You may perhaps refer that.

Thanks,

Preeta

[avinashga23]

What do you mean by "the messages are not consistently getting into the queue" ? Are the messages queued successfully anytime or they are failing always? If failing whats in the log?

[kpreeta12]

Hi Avinash,

Messaging are occasionally getting into queue. The logs got no errors or exception.

[avinashga23]

A message producer will always throws an exception in the case of failure to deliver the message to a queue/topic. It helps if you can provide the unit tests to reproduce.

[kpreeta12]

Exceptions will only be thrown when there is a issue when sending or receiving messages and currently we don't have any exceptions. Here the only thing that has changed is the configuration in standalone-full.xml for wildfly 10.

[mnovak]

Hi Preeta,

I tried you WF10 config and could send and receive messages using RemoteConnectionFactory (I just disabled security in messaging-activemq subsystem). Could you check that when RC.war sends messages then those messages are in the queue using CLI, for example like:

/subsystem=messaging-activemq/server=default/jms-queue=ISEEInboundQueue:count-messages()

Does SL.war contain MDB which is consuming messages from some queue? How it's configured?

Could you enable tracing for org.apache.activemq.artemis?

Thanks,

Mirek

[kpreeta12]

Thanks for trying it out at your end and responding. Will try out what you suggested and get back soon.

[kpreeta12]

Hi Mirek,

I enabled Trace for org.apache.activemq.artemis and then submitted a Requistion from RC.

I see the below in the logs now:-

2016-09-22 14:19:03,520 DEBUG [org.apache.activemq.artemis.core.client] (default task-22) Session was already closed, giving up now, this=ClientSessionImpl [name=2fe7b853-810a-11e6-9c7e-49748552169f, username=jmsuser, closed=true, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@77a24f35, metaData=(jms-session=,)]@1b5c6377

2016-09-22 14:19:03,521 TRACE [org.apache.activemq.artemis.core.client] (Thread-6 (ActiveMQ-client-global-threads-1058954121)) AMQ214026: Failure captured on connectionID=-94,657,003, performing failover or reconnection now: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ119006: Channel disconnected]

    at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:342)

    at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:911)

    at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at java.lang.Thread.run(Thread.java:745)

So we have to configure the JMS Failover ?

Thanks,

Preeta

[mnovak]

JMS Failover/HA is not necessary. It seems that for some reason JMS connection created by RC.war was disconnected. There might be more information on server with SL.war. Could you try to enable tracing for Artemis on server with SL.war and check happens in time when RC.war was disconnected?

Also I think that you could try to configure infinite reconnect-attemts on RemoteConnectionFactory, following configuration should do that:

<connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>

[kpreeta12]

Also after the above error, we have the below in logs that say -there was nothing in BEEERequsitionQueue.

2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Thread-1 (ActiveMQ-client-netty-threads-51974265)) handling packet PACKET(SessionReceiveMessage)[type=75, channelID=14, packetObject=SessionReceiveMessage,message=ClientMessage[messageID=8602897902, durable=false, address=jms.queue.BEEERequisitionsQueue,userID=null,properties=TypedProperties[_hornetq.forced.delivery.seq=125]]]

2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #9 - JmsConsumer[BEEERequisitionsQueue]) There was nothing on the queue, leaving it now:: returning null

2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #8 - JmsConsumer[BEEERequisitionsQueue]) Returning null

2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #8 - JmsConsumer[BEEERequisitionsQueue]) Forcing delivery

2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #8 - JmsConsumer[BEEERequisitionsQueue]) Sending packet nonblocking PACKET(SessionForceConsumerDelivery)[type=78, channelID=12, packetObject=SessionForceConsumerDelivery, consumerID=0, sequence=125] on channeID=12

2016-09-22 14:19:03,598 TRACE [org.apache.activemq.artemis.core.client] (Camel (BEEEcamelAgent) thread #8 - JmsConsumer[BEEERequisitionsQueue]) Writing buffer for channelID=12

2016-09-22 14:19:03,602 TRACE [org.apache.activemq.artemis.core.client] (Thread-1 (ActiveMQ-client-netty-threads-51974265)) handling packet PACKET(SessionReceiveMessage)[type=75, channelID=12, packetObject=SessionReceiveMessage,message=ClientMessage[messageID=8602897903, durable=false, address=jms.queue.BEEERequisitionsQueue,userID=null,properties=TypedProperties[_hornetq.forced.delivery.seq=125]]]

Found these link:[JBEAP-1527] Clients wait for connection timeout during failback - JBoss Issue Tracker

activemq-artemis/ClientSessionFactoryImpl.java at master · apache/activemq-artemis · GitHub (look for "ActiveMQNotConnectedException")

[kpreeta12]

In the SL log I see the below:-

15:11:17,053 DEBUG [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) Cache container web will use an externalizer for org.wildfly.clustering.web.infinispan.session.SessionAccessMetaDataKey

15:11:17,051 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) artemis-native-64 -> error loading the native library: java.lang.UnsatisfiedLinkError: no artemis-native-64 in java.library.path

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) [rt.jar:1.8.0_77]

    at java.lang.Runtime.loadLibrary0(Runtime.java:870) [rt.jar:1.8.0_77]

    at java.lang.System.loadLibrary(System.java:1122) [rt.jar:1.8.0_77]

    at org.apache.activemq.artemis.jlibaio.LibaioContext.loadLibrary(LibaioContext.java:68) [artemis-native-1.1.0.wildfly-011.jar:]

    at org.apache.activemq.artemis.jlibaio.LibaioContext.<clinit>(LibaioContext.java:88) [artemis-native-1.1.0.wildfly-011.jar:]

    at org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory.isSupported(AIOSequentialFileFactory.java:112) [artemis-journal-1.1.0.wildfly-011.jar:]

    at org.wildfly.extension.messaging.activemq.ActiveMQServerService.start(ActiveMQServerService.java:155) [wildfly-messaging-activemq-10.0.0.Final.jar:10.0.0.Final]

    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_77]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_77]

    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_77]

15:11:17,056 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) Library artemis-native-64 not found!

15:11:17,057 DEBUG [org.jboss.modules] (MSC service thread 1-7) Module org.joda.time:main defined by local module loader @3930015a (finder: local module finder @629f0666 (roots: C:\newcluster-pc-all-10\wildfly-10.0.0.Final\modules,C:\newcluster-pc-all-10\wildfly-10.0.0.Final\modules\system\layers\base))

15:11:17,057 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) artemis-native-32 -> error loading the native library: java.lang.UnsatisfiedLinkError: no artemis-native-32 in java.library.path

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) [rt.jar:1.8.0_77]

    at java.lang.Runtime.loadLibrary0(Runtime.java:870) [rt.jar:1.8.0_77]

    at java.lang.System.loadLibrary(System.java:1122) [rt.jar:1.8.0_77]

    at org.apache.activemq.artemis.jlibaio.LibaioContext.loadLibrary(LibaioContext.java:68) [artemis-native-1.1.0.wildfly-011.jar:]

    at org.apache.activemq.artemis.jlibaio.LibaioContext.<clinit>(LibaioContext.java:88) [artemis-native-1.1.0.wildfly-011.jar:]

    at org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory.isSupported(AIOSequentialFileFactory.java:112) [artemis-journal-1.1.0.wildfly-011.jar:]

    at org.wildfly.extension.messaging.activemq.ActiveMQServerService.start(ActiveMQServerService.java:155) [wildfly-messaging-activemq-10.0.0.Final.jar:10.0.0.Final]

    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_77]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_77]

    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_77]

15:11:17,058 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) Library artemis-native-32 not found!

15:11:17,058 DEBUG [org.apache.activemq.artemis.jlibaio] (MSC service thread 1-8) Couldn't locate LibAIO Wrapper

[kpreeta12]

I see your comments related to this here [JBEAP-756] Natives for ActiveMQ Artemis are not loaded - JBoss Issue Tracker