3 Replies Latest reply on Sep 14, 2012 11:47 AM by Justin Bertram

    JMS Messages are sent successfully and disappear

    Dan Williams Newbie

      Hi,

       

      Summary

       

      We are running JBOSS AS v5.1.0.GA-v2.12-1 and a separate instance of stand-alone, non-clustered hornetq 2.2.14-FINAL on the same machine.  An application running in the JBOSS 5 AS sends messages via JMS to a queue in hornetq.  These are consumed by a stand-alone application (also JAVA and also on the same machine but in a third JVM). 

       

      All messages are sent without throwing exceptions however some never show up on the queue (messageCount does not increment for these messages at any point).  About 1 in every 10 messages disappear this way.  It only happens when there are multiple producers sending to the queue.  If there is only one producer sending, it does not happen.  If we send, for example, 10x2 message (20 in total but 10 from each producer), the problem is unlikely to happen.  Same for 40.  Anything over 50 and it still may not happen but the chances get higher and for 100 messages, we are guaranteed to lose some.

       

      Whether the consumer is running at the time of sending or not makes no difference, the messages will still be lost.

       

      The configurations are all below but if, for example, producer A sends 50 messages to QUEUE_A and producer B simultaneously sends 50 messages to QUEUE_A, ultimately, they seem to lose an even number (about 4 messages) each.  If we send an uneven number then one will lose an even number and the other an odd number).  When refreshing the MBean view in jConsole in the case of sending 2x50 (100), we only see 92 in the messageCount.  Subsequently, the sole consumer on QUEUE_A will not see any of the lost messages.  They seem to get lost in the sending.

       

      We are using JDK 1.6.0 for all three JVMs and connecting via TCP even though the JVMs are on the same host (no other way to do it anyway I guess).

       

      The configuration for the hornetq connectivity FROM JBOSS is:

       

      hornetq-ds.xml

      <?xml version="1.0" encoding="UTF-8"?>

       

      <connection-factories>

        <!--

         JMS XA Resource adapter, use this for outbound JMS connections.

         Inbound connections are defined at the @MDB activation or at the resource-adapter properties.

        -->

        <tx-connection-factory>

           <jndi-name>RemoteJmsXA</jndi-name>

           <xa-transaction/>

           <rar-name>hornetq-ra.rar</rar-name>

           <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>

           <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>

           <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>

           <config-property name="ConnectionParameters" type="java.lang.String">host=ourhost.ourdomain.com;port=5445</config-property>

           <max-pool-size>20</max-pool-size>

        </tx-connection-factory>

      </connection-factories>

       

      jms-ds.xml

      <?xml version="1.0" encoding="UTF-8"?>

       

      <connection-factories>

       

        <!-- ==================================================================== -->

        <!-- JMS Stuff                                                            -->

        <!-- ==================================================================== -->

       

         <!--

             The JMS provider loader. Currently pointing to a non-clustered ConnectionFactory. Need to

             be replaced with a clustered non-load-balanced ConnectionFactory when it becomes available.

             See http://jira.jboss.org/jira/browse/JBMESSAGING-843.

         -->

         <mbean code="org.jboss.jms.jndi.JMSProviderLoader"

                name="jboss.messaging:service=JMSProviderLoader,name=JMSProvider">

            <attribute name="ProviderName">DefaultJMSProvider</attribute>

            <attribute name="ProviderAdapterClass">org.jboss.jms.jndi.JNDIProviderAdapter</attribute>

            <attribute name="FactoryRef">java:/XAConnectionFactory</attribute>

            <attribute name="QueueFactoryRef">java:/XAConnectionFactory</attribute>

            <attribute name="TopicFactoryRef">java:/XAConnectionFactory</attribute>

         </mbean>

       

          <mbean code="org.jboss.jms.jndi.JMSProviderLoader" name="hornetq:service=JMSProviderLoader,name=RemoteJMSProvider">

              <attribute name="ProviderName">HornetQJMSProvider</attribute>

              <attribute name="ProviderAdapterClass">org.jboss.jms.jndi.JNDIProviderAdapter

              </attribute>

              <attribute name="FactoryRef">XAConnectionFactory</attribute>

              <attribute name="QueueFactoryRef">XAConnectionFactory</attribute>

              <attribute name="TopicFactoryRef">XAConnectionFactory</attribute>

              <attribute name="Properties">

                  java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory

                  java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces

                  java.naming.provider.url=ourhost.ourdomain.com:7099

              </attribute>

          </mbean>

       

         <!-- JMS XA Resource adapter, use this to get transacted JMS in beans -->

         <tx-connection-factory>

            <jndi-name>JmsXA</jndi-name>

            <xa-transaction/>

            <rar-name>jms-ra.rar</rar-name>

        <connection-definition>org.jboss.resource.adapter.jms.JmsConnectionFactory</connection-definition>
        <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>
        <config-property name="JmsProviderAdapterJNDI" type="java.lang.String">java:/DefaultJMSProvider</config-property>
        <max-pool-size>20</max-pool-size>
        <security-domain-and-application>JmsXARealm</security-domain-and-application>
        <depends>jboss.messaging:service=ServerPeer</depends>

       

        <!-- turn off txn interleaving -->
        <track-connection-by-tx/>

         </tx-connection-factory>

       

      </connection-factories>

       

      The actual relevant hornetq configuration is (queue / topic names simplified):

       

      hornetq-jms.xml


      <configuration xmlns="urn:hornetq"

                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                  xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">

       

         <connection-factory name="NettyConnectionFactory">

            <xa>true</xa>

            <connectors>

               <connector-ref connector-name="netty"/>

            </connectors>

            <entries>

               <entry name="/XAConnectionFactory"/>

            </entries>

         </connection-factory>

       

         <connection-factory name="NettyConnectionFactory">

            <xa>false</xa>

            <connectors>

               <connector-ref connector-name="netty"/>

            </connectors>

            <block-on-durable-send>true</block-on-durable-send>

            <block-on-non-durable-send>true</block-on-non-durable-send>

            <entries>

               <entry name="/ConnectionFactory"/>

            </entries>

         </connection-factory>

       

         <connection-factory name="NettyThroughputConnectionFactory">

            <xa>true</xa>

            <connectors>

               <connector-ref connector-name="netty-throughput"/>

            </connectors>

            <entries>

               <entry name="/XAThroughputConnectionFactory"/>

            </entries>

         </connection-factory>

       

         <connection-factory name="NettyThroughputConnectionFactory">

            <xa>false</xa>

            <connectors>

               <connector-ref connector-name="netty-throughput"/>

            </connectors>

            <entries>

               <entry name="/ThroughputConnectionFactory"/>

            </entries>

         </connection-factory>

       

          <queue name="DLQ">

              <entry name="/queue/DLQ"/>

          </queue>

       

          <queue name="ExpiryQueue">

              <entry name="/queue/ExpiryQueue"/>

          </queue>

       

          <queue name="QUEUE_A">

              <entry name="/queue/QUEUE_A"/>

          </queue>

       

          <topic name="TOPIC_A">

              <entry name="/topic/TOPIC_A" />

          </topic>

       

          <topic name="TOPIC_B">

              <entry name="/topic/TOPIC_B" />

          </topic>

      </configuration>

       

      hornetq-configuration.xml

       

      <configuration xmlns="urn:hornetq"

                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                     xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

       

         <paging-directory>${data.dir:../data}/paging</paging-directory>

       

         <bindings-directory>${data.dir:../data}/bindings</bindings-directory>

       

         <journal-directory>${data.dir:../data}/journal</journal-directory>

       

         <journal-min-files>10</journal-min-files>

       

         <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>

       

         <connectors>

            <connector name="netty">

               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

               <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

               <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>

            </connector>

       

            <connector name="netty-throughput">

               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

               <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

               <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>

               <param key="batch-delay" value="50"/>

            </connector>

         </connectors>

       

         <acceptors>

            <acceptor name="netty">

               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

               <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

               <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>

            </acceptor>

       

            <acceptor name="netty-throughput">

               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

               <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

               <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>

               <param key="batch-delay" value="50"/>

               <param key="direct-deliver" value="false"/>

            </acceptor>

         </acceptors>

       

         <security-settings>

            <security-setting match="#">

               <permission type="createDurableQueue" roles="guest"/>

               <permission type="createNonDurableQueue" roles="guest"/>

               <permission type="deleteNonDurableQueue" roles="guest"/>

               <permission type="deleteDurableQueue" roles="guest"/>

               <permission type="createTempQueue" roles="guest" />

               <permission type="consume" roles="guest"/>

               <permission type="send" roles="guest"/>

            </security-setting>

         </security-settings>

       

         <address-settings>

            <!--default for catch all-->

            <address-setting match="#">

               <dead-letter-address>jms.queue.DLQ</dead-letter-address>

               <expiry-address>jms.queue.ExpiryQueue</expiry-address>

               <redelivery-delay>0</redelivery-delay>

               <max-size-bytes>40485760</max-size-bytes>

               <page-size-bytes>20485760</page-size-bytes>

               <message-counter-history-day-limit>10</message-counter-history-day-limit>

               <address-full-policy>PAGE</address-full-policy>

            </address-setting>

         </address-settings>

       

      </configuration>

       

      Any ideas appreciated!

       

      Dan

        • 1. Re: JMS Messages are sent successfully and disappear
          Justin Bertram Master

          Can you paste the code sending the messages?

          • 2. Re: JMS Messages are sent successfully and disappear
            Dan Williams Newbie

            Hi Justin,

            here you go (this is not compling code but it gives you enough to see what classes are being used and modes being set setc):

             

             

            ConnectionFactory connectionFactory = null;

            try {

               

                Properties props = new Properties();

             

                props.setProperty(Context.INITIAL_CONTEXT_FACTORY, CONTEXT_FACTORY);

                props.setProperty(Context.URL_PKG_PREFIXES, URL_PKG_PREFIXES);

                props.setProperty(Context.PROVIDER_URL, "jbas5:1099");

             

                Context context = new InitialContext(props);

               

                connectionFactory = (ConnectionFactory) context.lookup("java:/RemoteJmsXA");

            } catch (NamingException e) {

                throw e;

            }

             

            Destination destination = null;

            try {

                Properties props = new Properties();

             

                props.setProperty(Context.INITIAL_CONTEXT_FACTORY, CONTEXT_FACTORY);

                props.setProperty(Context.URL_PKG_PREFIXES, URL_PKG_PREFIXES);

                props.setProperty(Context.PROVIDER_URL, "hornetq:7099");

               

                Context context = new InitialContext(props);

               

                destination = (Destination) context.lookup("QUEUE_A");

            } catch (NamingException e) {

                throw e;

            }

             

            Connection conn = null;

            Session session = null;

            try {

             

                conn = connectionFactory.createConnection("user", "password"));

                session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);

                conn.start();

             

                MessageProducer producer = session.createProducer(destination);

                Message message = createMessage(session, object);

             

                try {

                    producer.send(message);

                    jmsCorrelationId = message.getJMSMessageID();

                } finally {

                    try {

                        if (producer != null)

                            producer.close();

                    } catch (JMSException e) {

                    }

                }

             

            } catch (JMSException e) {

                throw e;

            } finally {

                try {

                    if (session != null)

                        session.close();

                } catch (JMSException e) {

                }

             

                try {

                    if (conn != null)

                        conn.close();

                } catch (JMSException e) {

                }

            }

            • 3. Re: JMS Messages are sent successfully and disappear
              Justin Bertram Master

              Quick question...

               

              When you lookup "java:/RemoteJmsXA" you are setting a Context.PROVIDER_URL of "jbas5:1099" in the InitialContext.  Isn't this look-up local (i.e. in the same VM as the application client)?  If so, why are you not using an empty InitialContext?