Wildfly 10 ONCE_AND_ONLY_ONCE JMS bridge

[just4f]

Hi, I'm testing simple JMS bridge between two local standalone !Wildfly 10 instances.

<jms-bridge name="mynode-bridge" max-batch-time="100" max-batch-size="10" max-retries="30" failure-retry-interval="60000" quality-of-service="ONCE_AND_ONLY_ONCE">
        <source destination="queue/xyz.mynode.input" connection-factory="ConnectionFactory" />
        <target user="jmsuser" password="xxx" destination="jms/queues/xyz.mynode.input" connection-factory="jms/RemoteConnectionFactory">
            <target-context>
                <property name="java.naming.factory.initial" value="org.jboss.naming.remote.client.InitialContextFactory" />
                <property name="java.naming.provider.url" value="http-remoting://localhost:8180" />
            </target-context>
        </target>
</jms-bridge>

Bridge works, JMS message is delivered and processed. However source node is unable to commit resources, so it retries to send message multiple times with same error:

2016-02-23 19:21:12,812 WARN  [com.arjuna.ats.jta] (Thread-96) ARJUNA016087: TransactionImple.delistResource - unknown resource
2016-02-23 19:21:12,821 WARN  [org.apache.activemq.artemis.jms.bridge] (Thread-96) AMQ342009: JMS Bridge failed to send + acknowledge batch, closing JMS objects: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commit(TransactionImple.java:224)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatchXA(JMSBridgeImpl.java:1350)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatch(JMSBridgeImpl.java:1248)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.access$1500(JMSBridgeImpl.java:75)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl$BatchTimeChecker.run(JMSBridgeImpl.java:1794)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

2016-02-23 19:21:16,769 ERROR [org.apache.activemq.artemis.core.client] (Thread-96) AMQ214005: XA start operation failed Cannot find xid in resource manager: XidImpl (88832931 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.-64.-88.-33.1.102.60.66.-22.86.-52.84.-9.0.1.68.87.0.0.0.0.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.-64.-88.-33.1.102.60.66.-22.86.-52.84.-9.0.1.68.86.49 base64:AAAAAAAAAAAAAP__wKjfAWY8QupWzFT3AAFEVwAAAAAAAAAAAAAAAAAAAAAAAP__wKjfAWY8QupWzFT3AAFEVjEHAgIA code:-4
2016-02-23 19:21:16,783 WARN  [com.arjuna.ats.jta] (Thread-96) ARJUNA016089: TransactionImple.enlistResource - xa_start  - caught: XAException.XAER_NOTA for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a8df01:663c42ea:56cc54f7:14456, node_name=1, branch_uid=0:ffffc0a8df01:663c42ea:56cc54f7:14457, subordinatenodename=null, eis_name=unknown eis name >: javax.transaction.xa.XAException
    at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.xaStart(ActiveMQSessionContext.java:499)
    at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1361)
    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:741)
    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:423)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.enlistResources(JMSBridgeImpl.java:826)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatchXA(JMSBridgeImpl.java:1375)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatch(JMSBridgeImpl.java:1248)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.access$1500(JMSBridgeImpl.java:75)
    at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl$BatchTimeChecker.run(JMSBridgeImpl.java:1794)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

2016-02-23 19:21:16,847 ERROR [stderr] (Thread-96) javax.transaction.xa.XAException
2016-02-23 19:21:16,912 ERROR [stderr] (Thread-96)     at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.xaStart(ActiveMQSessionContext.java:499)
2016-02-23 19:21:16,923 ERROR [stderr] (Thread-96)     at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1361)
2016-02-23 19:21:16,927 ERROR [stderr] (Thread-96)     at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:741)
2016-02-23 19:21:16,941 ERROR [stderr] (Thread-96)     at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:423)
2016-02-23 19:21:16,946 ERROR [stderr] (Thread-96)     at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.enlistResources(JMSBridgeImpl.java:826)
2016-02-23 19:21:16,959 ERROR [stderr] (Thread-96)     at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatchXA(JMSBridgeImpl.java:1375)
2016-02-23 19:21:16,963 ERROR [stderr] (Thread-96)     at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.sendBatch(JMSBridgeImpl.java:1248)
2016-02-23 19:21:16,976 ERROR [stderr] (Thread-96)     at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.access$1500(JMSBridgeImpl.java:75)
2016-02-23 19:21:16,980 ERROR [stderr] (Thread-96)     at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl$BatchTimeChecker.run(JMSBridgeImpl.java:1794)
2016-02-23 19:21:16,993 ERROR [stderr] (Thread-96)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2016-02-23 19:21:16,997 ERROR [stderr] (Thread-96)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2016-02-23 19:21:17,010 ERROR [stderr] (Thread-96)     at java.lang.Thread.run(Thread.java:745)

Any suggestions? Thanks.

[jbertram]

Your source and target connection factories are the local "ConnectionFactory" and the remote "jms/RemoteConnectionFactory" respectively.  These connection factories need to support XA because ONCE_AND_ONLY_ONCE across multiple brokers demands it.  Have you configured these connection factories to support XA appropriately?

[jbertram]

BTW, to configure a connection-factory to support XA just add this attribute:

  factory-type="XA_GENERIC"

[just4f]

Connection factories were not configured to support XA, now updated both nodes, but still the same error.

<connection-factory name="InVmConnectionFactory" factory-type="XA_QUEUE" entries="java:/ConnectionFactory" connectors="in-vm"/>
<connection-factory name="RemoteConnectionFactory" factory-type="XA_QUEUE" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>

[mnovak]

Not sure what different in your config but following configuration works for me:

Server 1 with connection factory and queue OutQueue - it's taget for JMS bridge:

<jms-queue name="OutQueue" entries="jms/queue/OutQueue java:jboss/exported/jms/queue/OutQueue"/>
<connection-factory name="RemoteConnectionFactory" factory-type="XA_GENERIC" reconnect-attempts="-1" block-on-acknowledge="true" ha="true" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>

Server 2 with JMS bridge, connection-factory and queue InQueue - it's source for the bridge

<jms-queue name="InQueue" entries="jms/queue/InQueue java:jboss/exported/jms/queue/InQueue"/>
<connection-factory name="InVmConnectionFactory" factory-type="XA_GENERIC" entries="java:/ConnectionFactory" connectors="in-vm"/>
...
<jms-bridge name="myBridge" add-messageID-in-header="true" max-batch-time="100" max-batch-size="10" max-retries="-1" failure-retry-interval="1000" quality-of-service="ONCE_AND_ONLY_ONCE" module="org.apache.activemq.artemis">
                <source destination="jms/queue/InQueue" connection-factory="java:/ConnectionFactory"/>
                <target destination="jms/queue/OutQueue" connection-factory="jms/bridgeCF">
                    <target-context>
                        <property name="java.naming.factory.initial" value="org.jboss.naming.remote.client.InitialContextFactory"/>
                        <property name="java.naming.provider.url" value="http-remoting://127.0.0.1:8080"/>
                    </target-context>
                </target>
            </jms-bridge>

Could you attach your configuration of the whole messaging-activemq subsystem from both of the servers. I'll take a look.

Thanks,

Mirek

[just4f]

Hi, not sure why it did not work before. Now updated configuration also with

add-messageID-in-header="true"

and it works correctly.
Thanks

[mnovak]

I'm happy that it helped. I tried your config except the username and password for connection-factory and it worked for me. Maybe this might be the issue, I'll try it when I have some time.