getting "Message to JBossTopic ... not processed"

[ranjix]

Hey guys, sorry for the bother.

I use JBoss Community Edition 5.1.0GA, and I do have an issue with MDBs. Basically they seem to work intermittently. I do have a client which posts a message into a topic, and a subscriber which uses an MDB to pick up the respective message from the topic and use it. Fairly straightforward. The issue is that it doesn't work *every time*, from my tests it looks like once every three times it fails (messages don't make it to the subscriber). When it fails, it shows in the system.out a message like "14:41:58,469 INFO  [STDOUT] Warning:  Message to JBossTopic[com.bluemartini.DBUtilTopic] not processed: delegator->JBossMessage[5243929060474880]:NON-PERSISTENT, deliveryId=0".

the questions:

1. any idea what's going on, why it doesn't work every time? Looks like some timing issues, but no clue how to resolve. Making the subscription durable didn't help.

2. how can I get the sources to whichever class is printing out the "Message to JBossTopic"? I downloaded sources, looked through them, couldn't figure out. It seems that the part with "delegator->JBossMessage" comes from a org.jboss.jms.message.MessageProxy toString method, but can't find where is the other part coming from (the one with Message to JBossTopic). The fact that's the second issue for which I can't find the sources starts bothering me. I'll look into how to download ALL the sources (using subversion or something).

Thanks for any idea,

ranjix

[ranjix]

The JBossTopic[com.bluemartini.DBUtilTopic] seems to come from JBossTopic class, toString method.

Left to figure out the "not processed" part, keep digging...

[gaohoward]

I guess it's your MDB configuration problem. Is your subscriber durable or not?

[ranjix]

It's not impossible that my MDB config is wrong, it's bothering that same configuration works consistently in weblogic and websphere but not in jboss. As I said, I did try to make the subscription durable, see the MDB's ejb-jar.xml underneath, and DID NOT HELP. 

I also tried to make the messages persistent, but at that point the text in the system.out became "10:10:38,736 INFO  [STDOUT] Warning:  Message to JBossTopic[com.bluemartini.DBUtilTopic] not processed: delegator->JBossMessage[5244570167312385]:PERSISTENT, deliveryId=1", and the JMS message was still delivered 2 times out of 3.

Frankly there is a certain hint of flakiness coming out of this issue, in regards to JBoss capability of working with JMS. Hope I'm wrong and indeed something in our app is interfering with the JMS (although don't really see how...)

thanks/Ranjix

P.S. Again, a hint towards where in the sources is the text "Message to JBossTopic" constructed would really help. At that point I could try and debug.

the ejb-jar.xml

<?xml version="1.0" encoding="UTF-8"?>

<ejb-jar id="ejb-jar_ID"

    version="2.1"

    xmlns="http://java.sun.com/xml/ns/j2ee"

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/ejb-jar_2_1.xsd">

  <display-name>remotedesktop-JAR</display-name>

  <enterprise-beans>

    <message-driven>

      <description> db util mdb </description>

      <display-name>DBUtilTopicMDB</display-name>

      <ejb-name>DBUtilTopicMDB</ejb-name>

      <ejb-class>com.bluemartini.jms.mdb.DBUtilTopicMDB</ejb-class>

      <transaction-type>Bean</transaction-type>

      <message-destination-type>javax.jms.Topic</message-destination-type>

      <activation-config>

        <activation-config-property>

          <activation-config-property-name>subscriptionDurability</activation-config-property-name>

          <activation-config-property-value>Durable</activation-config-property-value>

        </activation-config-property>

      </activation-config>

    </message-driven>

  </enterprise-beans>

</ejb-jar>

[ranjix]

Ok, after a week of digging, I finally think I understand what's going on and I hacked something to fix the issue.

Short version, IMO is a bug in how JBoss shares JMS messages objects between subscribers on a topic.

Long version:

I have several apps, with MDBs, which subscribe to the same topic. The apps run in the same server (JVM), but obviously each in its own app classloader (the MDBs, when called, have as classloaders the EAR app classloader). This is good. The JMS message object type which I get in the onMessage implementation is org.jboss.jms.message.BytesMessageProxy (the message was supposed to be a BytesMessage). I get different instances of BytesMessageProxy in the different MDBs. This is ok. Here comes the bad part. The MessageProxy (BytesMessageProxy extends this) contains a field message which is a JBossMessage. The bad part is that each of the different instances of proxy class contains same JBossMessage object. So, now, my MDBs start reading from the proxy, which in turn read from the JBossMessage. Since I have a quad core, the reading is pretty fast, and the "second" MDB starts reading before the first finished reading and resetting the message. This makes the second MDB make a read after the end of the payloadByteArray (which is a byte array containing the BytesMessage), which results in a MessageEOFException thrown from line 169 in method readByte in JBossBytesMessage class.

So, in conclusion, IMO sharing the same JBossMessage object between the subscribers will ALWAYS be a problem.

My hack. In my onMessage method, I can't really put a synchronization block depending on the BytesMessageProxy (since is a new instance for each MDB onMessage call). But I don't have access to the contained shared JBossMessage object, so I put a synchronization block depending on BytesMessageProxy.getJMSMessageID object, which is the string representing the message's id. Since the JBossMessage shared has only one ID, then my readings will wait for the lock on the ID. This, at least, fixed my issues. Not happy at all with the implementation, if someone on JBoss JMS team reads this, please consider it for a possible bug.

thanks/ranjix

[ranjix]

just a last FYI, JBoss created a JIRA task for the issue - https://jira.jboss.org/browse/JBMESSAGING-1818