Wildfly 18 artemis bridge failover issue

[corab500]

Hello everyone,

i'm currently facing an issue where my failover on the artemis bridge is correctly reconnecting but after the initial start if the bridge fails to connect, upon first connection all incoming messages are ignored and in the log i see the following warning :

20-01-21 12:32:36,913 INFO  [org.apache.activemq.artemis.jms.bridge] (Thread-117) AMQ341000: Failed to set up JMS bridge RemoteBrokerInBridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 5,000 ms

2020-01-21 12:32:42,137 INFO  [org.apache.activemq.artemis.jms.bridge] (Thread-117) AMQ341001: JMS Bridge RemoteBrokerInBridge succeeded in reconnecting to servers

2020-01-21 12:33:49,289 WARN  [org.apache.activemq.ActiveMQSessionExecutor] (ActiveMQ Transport: ssl://localhost/127.0.0.1:61616) Received a message on a connection which is not yet started. Have you forgotten to call Connection.start()? Connection: ActiveMQConnection {id=ID:virtualBox-46367-1579605931017-607:1,clientId=ID:virtualBox-46367-1579605931017-606:1,started=false} Received: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:virtualBox-46367-1579605931017-607:1:1:1, destination = queue://amq.test.in, message = ActiveMQTextMessage {commandId = 5, responseRequired = true, messageId = ID:virtualBox-43797-1579606353532-6:1:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:virtualBox-43797-1579606353532-6:1:1:1, destination = queue://amq.test.in, transactionId = null, expiration = 0, timestamp = 1579606428975, arrival = 0, brokerInTime = 1579606428983, brokerOutTime = 1579606429122, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@1b4ff930, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false, jmsXGroupFirstForConsumer = false, text = :20:INSTR-MT101

testMessage123

}, redeliveryCounter = 0}

my bridge is configured like this :

<connection-factory name="nonPooledQueueConnectionFactory" entries="java:/jms/BridgeConnectionFactory java:/jms/RemoteInQueueConnectionFactory" connectors="in-vm" factory-type="XA_QUEUE"/>

<jms-bridge name="RemoteBrokerInBridge" module="org.apache.activemqclient" quality-of-service="ONCE_AND_ONLY_ONCE" failure-retry-interval="5000" max-retries="-1" max-batch-size="50" max-batch-time="500">

                <source connection-factory="XAConnectionFactory" destination="bridgeToRemoteBrokerInQueue">

                    <source-context>

                        <property name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>

                        <property name="java.naming.provider.url" value="failover:(ssl://localhost:61616)?maxReconnectAttempts=0&amp;startupMaxReconnectAttempts=0&amp;updateURIsSupported=false&amp;warnAfterReconnectAttempts=-1"/>

                        <property name="queue.bridgeToRemoteBrokerInQueue" value="amq.test.in"/>

                    </source-context>

                </source>

                <target connection-factory="/jms/BridgeConnectionFactory" destination="/jms/RemoteBrokerInQueue">

                    <target-context/>

                </target>

            </jms-bridge>

Does anyone have any idea of what is going wrong?

if i don't use the failover in the url it works as expected. Same if i disconnect from my bridge and i reconnect messages are correctly being consumed.

[mnovak]

Hi, could you try a little bit different approach, please? Instead of trying to figure out failover url in source connection factory, you can let Wildfly to create such connection factory for you, it should be like:

<server ...>
<remote-connector name="remote-connector" socket-binding="remote-artemis"/>
<connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory java:/jms/RemoteConnectionFactory" connectors="remote-connector" ha="true" compress-large-messages="false" min-large-message-size="102400" block-on-acknowledge="true" retry-interval="2000" retry-interval-multiplier="1.0" reconnect-attempts="-1" factory-type="XA_QUEUE" />
</server>
...
<jms-bridge name="myBridge" module="org.apache.activemq.artemis" quality-of-service="ONCE_AND_ONLY_ONCE" failure-retry-interval="1000" max-retries="-1" max-batch-size="10" max-batch-time="100" add-messageID-in-header="true">
                <source connection-factory="java:/ConnectionFactory" destination="jms/queue/InQueue"/>
                <target connection-factory="jms/RemoteConnectionFactory" destination="jms/queue/OutQueue"/>                  
            </jms-bridge>
...
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
...
<outbound-socket-binding name="remote-artemis">
            <remote-destination host="127.0.0.1" port="61616"/>
        </outbound-socket-binding>

Thanks,

Mirek

[corab500]

Thanks for the quick reply.

I tried to set up as you mentioned but I'm facing an error at bootup.

this is how i've setup my bridge now :

<jms-bridge name="RemoteBrokerInBridge" module="org.apache.activemq.artemis" quality-of-service="ONCE_AND_ONLY_ONCE" failure-retry-interval="1000" max-retries="-1" max-batch-size="10" max-batch-time="500" add-messageID-in-header="true"> 

                <source connection-factory="/jms/RemoteConnectionFactory" destination="jms/queue/amq.test.in"/>

                <target connection-factory="/jms/BridgeConnectionFactory" destination="/jms/RemoteBrokerInQueue"/>             

            </jms-bridge>

where amq.test.in is the name of my queue on my remote broker, i get the following error :

2020-01-21 17:30:50,369 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "messaging-activemq"),
    ("jms-bridge" => "RemoteBrokerInBridge")
]) - failure description: {
    "WFLYCTL0412: Required services that are not installed:" => ["jboss.naming.context.java.jms.queue.\"amq.test.in\""],
    "WFLYCTL0180: Services with missing/unavailable dependencies" => ["jboss.messaging-activemq.jms-bridge.RemoteBrokerInBridge is missing [jboss.naming.context.java.jms.queue.\"amq.test.in\"]"]
}

[ehugonnet]

is the local queue 'amq.test.in' correctly created and defined ?

[corab500]

thanks for your reply,

my amq.test.in is the name of the queue on the remote broker, my bridge is use to read from the remote as source and produce on the local for the target, my RemoteBrokerInQueue is defined locally but not the other one as it's the source.

should i use the source context in such case ?

[mnovak]

Now that kind of hacky but  there is not needed to get instance of queue from remote Artemis broker, Wildfly can create it itself and just use it. Could you add into your configuration:

<jms-queue name="amq.test.in" entries="java:/jms/queue/amq.test.in"/>

As /jms/RemoteConnectionFactory is pointing to remote Artemis broker, it will try find such destination (or create dynamically) on remote broker. This has negative side effect that the same queue will be created in Artemis in Wildfly server (which is the hacky thing)

[corab500]

Thanks for your suggestion.

My artemis is correctly starting but when wildfly tries to connect looks like there is a communication issue :

2020-01-23 16:10:26,549 WARN  [org.apache.activemq.artemis.core.client] (Thread-4 (ActiveMQ-client-global-threads)) AMQ212037: Connection failure to localhost/127.0.0.1:61616 has been detected: java.lang.IllegalArgumentException: AMQ219032: Invalid type: 1 [code=GENERIC_EXCEPTION]
2020-01-23 16:10:26,741 ERROR [org.apache.activemq.artemis.core.client] (Thread-8 (ActiveMQ-client-netty-threads)) AMQ214013: Failed to decode packet: java.lang.IllegalArgumentException: AMQ219032: Invalid type: 1
at org.apache.activemq.artemis.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:475)
at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:67)
at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:42)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:377)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1198)
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


2020-01-23 16:10:26,743 ERROR [org.apache.activemq.artemis.core.client] (Thread-8 (ActiveMQ-client-netty-threads)) AMQ214031: Failed to decode buffer, disconnect immediately.: java.lang.IllegalStateException: java.lang.IllegalArgumentException: AMQ219032: Invalid type: 1
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:390)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1198)
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.lang.IllegalArgumentException: AMQ219032: Invalid type: 1
at org.apache.activemq.artemis.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:475)
at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:67)
at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:42)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:377)
... 29 more

i see the same on my remote broker :

 WARN | Transport Connection to: tcp://127.0.0.1:48856 failed: java.io.IOException: Unknown data type: 77
java.io.IOException: Unknown data type: 77
at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:358)
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:277)
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:748)
 WARN | Transport Connection to: tcp://127.0.0.1:48858 failed: java.io.IOException: Unknown data type: 77
java.io.IOException: Unknown data type: 77
at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:358)
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:277)
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:748)

[mnovak]

What is the Artemis broker version? There might be problems with used protocol, Wildfly is using CORE protocol but it looks like that acceptor on Artemis broker thinks it's AMQP. Could you try to change acceptor on Artemis broker to use just CORE protocol.

[corab500]

in my current setup i'm talking to an activemq 5 version not an artemis instance.