6 Replies Latest reply on Jul 25, 2012 4:27 PM by clebert.suconic

    Bug in HornetQ or API misuse?

    _sst_

      First of all: I am NOT sure whether it is a bug. It is quite possible that it happens to be an API misuse. I am merely asking for advices/suggestions on what might be causing it.

       

      While running jBoss 6.1.0 (community edition) with HornetQ 2.2.5 from time to time we get a very interesting exception:

       

      2012-04-04 11:18:49,553 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (Thread-2912 (group:HornetQ-client-global-threads-1107999515)) XA operation failed Cannot start, session is already doing work in a transaction null code:-6

      2012-04-04 11:18:49,556 WARN  [com.arjuna.ats.jta] (Thread-2912 (group:HornetQ-client-global-threads-1107999515)) ARJUNA-16061 TransactionImple.enlistResource - XAResource.start returned: XAException.XAER_PROTO for < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffffac1f7e01:126a:4f7b4fce:7b68, node_name=1, branch_uid=0:ffffac1f7e01:126a:4f7b4fce:7b69, eis_name=unknown eis name >: javax.transaction.xa.XAException

      at org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1552) [:6.1.0.Final]

      at org.hornetq.core.client.impl.DelegatingSession.start(DelegatingSession.java:507) [:6.1.0.Final]

      at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:629) [:6.1.0.Final]

      at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:390) [:6.1.0.Final]

      at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.startTransaction(MessageInflowLocalProxy.java:399) [:1.7.21]

      at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.before(MessageInflowLocalProxy.java:235) [:1.7.21]

      at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:135) [:1.7.21]

      at $Proxy205.beforeDelivery(Unknown Source)     at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:267) [:6.1.0.Final]

      at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:866) [:6.1.0.Final]

      at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:44) [:6.1.0.Final]

      at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:983) [:6.1.0.Final]

      at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.1.0.Final]

      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_30]

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_30]

      at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]

       

       

      ClientSessionImpl code suggests that exception happened in server-side code. Here is a relevant code fragment:

       

         public void xaStart(final Xid xid) throws Exception

         {

            if (tx != null)

            {

               final String msg = "Cannot start, session is already doing work in a transaction " + tx.getXid();

       

               throw new HornetQXAException(XAException.XAER_PROTO, msg);

            }

            else

            { ... }

         }

       

      While I was checking source code of this class I found out that this situation might be possible: constructor creates a default transaction if  (boolean xa) argument is false.

      At this point I see 2 possible solutions:

           a) 'xa' argument should be set to true to avoid creating a default transaction

           b) add check in ServerSessionImpl.xaStart() that takes this.xa in account. ( same is done for ServerSessionImpl.individualAcknowledge() and ServerSessionImpl.rollback() )

       

      I tried googling it but couldn't find anything (i'd be glad to google it again if you hint me on proper keywords)

       

      While running same code in my local instance of jBoss I stumbled upon another exception being thrown (might be related to https://community.jboss.org/message/566463#566463):

       

      2012-03-22 16:32:49,830 WARN  [org.hornetq.core.protocol.core.ServerSessionPacketHandler] (Thread-16 (group:HornetQ-remoting-threads1330942-21892593)) Sending unexpected exception to the client: java.lang.NullPointerException

          at org.hornetq.core.server.impl.ServerSessionImpl.xaEnd(ServerSessionImpl.java:682) [:6.1.0.Final]

          at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:308) [:6.1.0.Final]

          at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474) [:6.1.0.Final]

          at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496) [:6.1.0.Final]

          at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457) [:6.1.0.Final]

          at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459) [:6.1.0.Final]

          at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:137) [:6.1.0.Final]

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.1.0.Final]

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_30]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_30]

          at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]

       

      2012-03-22 16:32:49,830 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (Thread-9 (group:HornetQ-client-global-threads-11522794)) Caught jmsexecptione : HornetQException[errorCode=0 message=null]

          at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:286) [:6.1.0.Final]

          at org.hornetq.core.client.impl.ClientSessionImpl.end(ClientSessionImpl.java:1275) [:6.1.0.Final]

          at org.hornetq.core.client.impl.DelegatingSession.end(DelegatingSession.java:332) [:6.1.0.Final]

          at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:588) [:6.1.0.Final]

          at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2287) [:6.1.0.Final]

          at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1452) [:6.1.0.Final]

          at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99) [:6.1.0.Final]

          at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159) [:6.1.0.Final]

          at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1158) [:6.1.0.Final]

          at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.1.0.Final]

          at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.1.0.Final]

          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.endTransaction(MessageInflowLocalProxy.java:464) [:1.7.21]

          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.finish(MessageInflowLocalProxy.java:343) [:1.7.21]

          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.after(MessageInflowLocalProxy.java:259) [:1.7.21]

          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140) [:1.7.21]

          at $Proxy205.afterDelivery(Unknown Source)    at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:287) [:6.1.0.Final]

          at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:866) [:6.1.0.Final]

          at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:44) [:6.1.0.Final]

          at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:983) [:6.1.0.Final]

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.1.0.Final]

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_30]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_30]

          at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]

       

       

      Here is the relevant code fragment:


        public void xaEnd(final Xid xid) throws Exception

        {

            if (tx != null && tx.getXid().equals(xid))

            { ... }

            ....

        }

       

      I've added some crude logging capability that logs when an empty (xid=null) transaction is being created and it turns out that:

       

      java.lang.Exception: [server] new Transaction() - hashCode(this), thread=Thread[Thread-12 (group:HornetQ-server-threads28104970-30661940),5,HornetQ-server-threads28104970-30661940]

           at org.hornetq.core.transaction.impl.TransactionImpl.<init>(TransactionImpl.java:81)

           at org.hornetq.core.server.impl.ServerSessionImpl.commit(ServerSessionImpl.java:614)

          at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:285)

           at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)

           at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)

           at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.access$000(RemotingConnectionImpl.java:45)

           at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl$1.run(RemotingConnectionImpl.java:436)

           at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

           at java.lang.Thread.run(Thread.java:662)

       

      and right after that:

       

      java.lang.Exception: [server] xaEnd: argXid=26169233, hashCode(tx)=32530158, hashCode(tx.xid)=0, thread=Thread[Thread-18 (group:HornetQ-remoting-threads18469713-8425294),5,HornetQ-remoting-threads18469713-8425294]

           at org.hornetq.core.server.impl.ServerSessionImpl.xaEnd(ServerSessionImpl.java:690)

           at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:308)

           at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)

           at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)

           at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457)

           at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)

           at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:137)

           at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

       

       

      I have 2 guesses:

      a) race condition: session object is being shared by different threads (commit() was called by Thread-12, while xaEnd() is called by Thread-18)

      b) bug: commit() should create new transaction object only if '!xa' and set 'this.tx=null' otherwise ( same way like rollback() does )

       

      I came up with a fix for (b) and attached a patch. It removes previous exception and exposes another one that I was getting before applying a patch anyway:

       

      2012-07-16 14:26:13,423 WARN  [com.arjuna.ats.jta] (Thread-7 (group:HornetQ-client-global-threads-23489525)) ARJUNA-16039 onePhaseCommit on < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffffac1c2caa:126a:50045c1f:21, node_name=1, branch_uid=0:ffffac1c2caa:126a:50045c1f:23, eis_name=unknown eis name > (org.hornetq.core.client.impl.DelegatingSession@8dd70b) failed with exception XAException.XAER_PROTO: javax.transaction.xa.XAException

          at org.hornetq.core.client.impl.ClientSessionImpl.end(ClientSessionImpl.java:1293) [:]

          at org.hornetq.core.client.impl.DelegatingSession.end(DelegatingSession.java:332) [:]

          at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:588) [:6.1.0.Final]

          at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2287) [:6.1.0.Final]

          at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1452) [:6.1.0.Final]

          at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99) [:6.1.0.Final]

          at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159) [:6.1.0.Final]

          at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1158) [:6.1.0.Final]

          at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.1.0.Final]

          at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.1.0.Final]

          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.endTransaction(MessageInflowLocalProxy.java:464) [:1.7.21]

          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.finish(MessageInflowLocalProxy.java:343) [:1.7.21]

          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.after(MessageInflowLocalProxy.java:259) [:1.7.21]

          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140) [:1.7.21]

          at $Proxy154.afterDelivery(Unknown Source)    at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:287) [:6.1.0.Final]

          at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:866) [:]

          at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:44) [:]

          at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:983) [:]

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_27]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_27]

          at java.lang.Thread.run(Thread.java:662) [:1.6.0_27]

       

       

      I'm trying to come up with some kind of reproducible test, but have no luck so far.

      Any help and suggestions are greatly appreciated!

       

      Thanks.

        • 1. Re: Bug in HornetQ or API misuse?
          ataylor

          The XA operation failed Cannot start, session is already doing work in a transaction exception can only happen if either multiple threads are making xa calls on the xa session, without more infor or a test its hard to say. what is your application doing and do you have non standard configuration, i.e. changes to RA etc. Also what load are you running on the server?

          • 2. Re: Bug in HornetQ or API misuse?
            _sst_

            > what is your application doing?

            we have 2 distinct applications:

            - GUI application that on certain events sends several messages to 4 JMS queues on remote server...

            - several MDB beans in web-app subscribed to aforementioned queues that saves received data in database...

             

             

            > what load are you running on the server?

            none that I know of...

             

            Stock jBoss 6.1.0 Final (community edition) in default configuration that hosts a single custom web-app (along with all default apps out there) containing 4 different MDB beans subscribed to different topics...

             

            I do required steps in my app, send a bunch of messages to JMS queue and see 'XA operation failed Cannot start, session is already doing work in a transaction' in log file.

             

            Messages are sent using following pattern:

              session = connection.createSession()

              producer = session.createProducer()

             

              for (Message msg : messagesToSend)

                  message = session.createTextMessage()

                  message.setProperties()

                  producer.send(message)


              session.commit()

              session.close()

              connection.close()

             

            Note: despite the fact that it uses array in practice we always send 1 message at a time

             

             

            > do you have non standard configuration

            Doesn't seems like it. We have only 2 xml files packed in .ear file that contain kind of web-app configuration related to these errors:

            first one defines security settings and the other one defines topics:

             

            <configuration xmlns="urn:hornetq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd"

            >

              <security-settings>

                <security-setting match="#">

                  <permission type="createNonDurableQueue" roles="guest" />

                  <permission type="deleteNonDurableQueue" roles="guest" />

                  <permission type="createDurableQueue" roles="guest" />

                  <permission type="deleteDurableQueue" roles="guest" />

                  <permission type="consume" roles="guest" />

                  <permission type="send" roles="guest" />

                </security-setting>

              </security-settings>

            </configuration>

             

            <configuration xmlns="urn:hornetq"

                        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                        xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">

             

            <topic name="T1">

                <entry name="/topic/T1"/>

              </topic>

              <topic name="T2">

                <entry name="/topic/T2"/>

              </topic>

              <topic name="T3">

                <entry name="/topic/T3"/>

              </topic>

             

              <queue name="Q1">

                <entry name="/queue/Q1"/>

              </queue>

            </configuration>

             

             

            We do have some strange MDB configuration (IMO). In particular error happens only for a very specific topic that has following subscribers:

             

             

            @MessageDriven(name = "<name1>", activationConfig = {

                @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),

                @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "CLIENT_ACKNOWLEDGE"),

                @ActivationConfigProperty(propertyName = "destination", propertyValue = "/topic/T3"),

                @ActivationConfigProperty(propertyName = "messageSelector", propertyValue = "key = 'value1"),

                @ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue = "Durable"),

                @ActivationConfigProperty(propertyName = "clientId", propertyValue = "<name1>"),

                @ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "<name1>")}, messageListenerInterface = MessageListener.class)

            class <name1> extends <name2> implements MessageListener {}

             

            @MessageDriven(name = "<name2>", activationConfig = {

                @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),

                @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "CLIENT_ACKNOWLEDGE"),

                @ActivationConfigProperty(propertyName = "destination", propertyValue = "/topic/T3"),

                @ActivationConfigProperty(propertyName = "messageSelector", propertyValue = "key <> 'value1"),

                @ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue = "Durable"),

                @ActivationConfigProperty(propertyName = "clientId", propertyValue = "<name2>"),

                @ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "<name3>")}, messageListenerInterface = MessageListener.class)

            class <name2> extends BaseMDB implements MessageListener {}

             

            @MessageDriven(name="<name4>", activationConfig = {

                @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Topic"),

                @ActivationConfigProperty(propertyName="acknowledgeMode", propertyValue="CLIENT_ACKNOWLEDGE"),

                @ActivationConfigProperty(propertyName="destination", propertyValue="/topic/T3"),

                @ActivationConfigProperty(propertyName="messageSelector", propertyValue="key IN ('value1', 'value2', 'value3')"),

                @ActivationConfigProperty(propertyName="subscriptionDurability", propertyValue="Durable"),

                @ActivationConfigProperty(propertyName="clientId", propertyValue="<name5>"),

                @ActivationConfigProperty(propertyName="subscriptionName", propertyValue="<name6>"),

                @ActivationConfigProperty(propertyName="useDLQ", propertyValue="false")}, // don't use the EJB dead letter Q so the JBoss messaging DLQ configuration becomes active.

                  messageListenerInterface=MessageListener.class)

            @Interceptors(SpringBeanInterceptor.class)

            @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)

            class <name4> extends BaseSpringMDB implements MessageListener {}

             

             

            Here are things I know so far:

            - it used to work before <name1> was created...

            - all 3 beans are subscribed to same topic and all 3 define 'Durable' durability

            - <name1> extends <name2> and has inverted message selector ( == value1 vs <> value1) on same key, while <name4> has a message selector that holds value1 and two other values (that particular key have a finite number of possible values - over 20 values in total)

            - <name1> is the only bean that defines same values for name, configProperty-clientId and configProperty-subscriptionName, <name2> defines a different value for configProperty-subscriptionName,

            <name4> defines 3 different values for each of these properties.

             

            As part of my next step I'll try to check whether that particular configuration fails or what bean causes exceptions.

             

            Thanks for any advices and suggestions.

             

            PS As you can guess by now I'm not the original developer and I have no idea on what it does and why. I don't expect you to give me a complete solution. I'll be most grateful if you'll throw at me a list of things and ideas to check.

             

            PSS I have tried to create a jUnit test case but was unable to even run tests in clean checkout (STATIC=false is not being set when running JavaCC which causes most testcases to fail when creating 2nd topic). I would be grateful if any developer can share his launch configuration (I have both Eclipse and IDEA)

            • 3. Re: Bug in HornetQ or API misuse?
              _sst_

              I found what was causing a problem and wish to share my findings just in case someone will ever get in same situation.

               

              Short answer: message selectors on durable subscribers is not a good idea

               

              Long answer:

              We have 2 durable subscribers to a single topic that use message selectors.

              1st subscriber uses a filter - propertyName == someValue, while the other one uses propertyName <> someValue.

               

              The idea was that one subscriber will get a certain messages, while another would get everything else.

               

              Turns out HornetQ wasn't able to deliver certain PERSISTENT messages cause subscriber would filter them out. After a while we would get a: ARJUNA-16039 onePhaseCommit on < .... > (org.hornetq.core.client.impl.DelegatingSession@8dd70b) failed with exception XAException.XAER_PROTO: javax.transaction.xa.XAException. My guess at this point is that it is a 'delivery-failed due to timeout' exception. This was actually a root of a problem.

               

              Since we couldn't deliver a PERSISTENT message - it would get stored and redelivered later .... and fail again and again and again. As our GUI application sends messages throughout a day we get more and more 'undelivered' messages.

              These messages slowly but steadily increase a load on server ... until we get to a point where we would expose a multi-thread issues.

               

              When our application sends a message it happens to get a session used by internal 'resend undelivered messages' mechanism. I might be wrong on that one though, but removing message selector from durable subscriber (<name2> in my configuration in a previous message) solved a problem.

               

              Though I'm pretty sure that some exceptions expose lurking bugs in HornetQ itself (especially when it gets down to multi-thread restrictions) I solved my issue by tweaking an MDB configuration.

               

               

               

              PS I'd like to thank Andy Taylor for his insight on checking server load.

              • 4. Re: Bug in HornetQ or API misuse?
                ataylor

                if you think you have found an issue could you please produce a test case to fix it please, something we can easily rerun.

                • 5. Re: Bug in HornetQ or API misuse?
                  clebert.suconic

                  There were some fixes on recovery recently. you should try the latest version.

                  • 6. Re: Bug in HornetQ or API misuse?
                    clebert.suconic

                    I meant... the latest 2.2.X version. (feel free to try 2.3.Alpha thouh)