Number of Journal Files Keep Growing

[kernow]

Hi,

After a number of hours of running my application, I have noticed that the number of journal files in the 'data/journal' directory keeps growing.  The broker has many hundreds of 000's of messages flowing through it a day.

When I look at the queues via JMX I can see that they are what I expect - around 0 (as my consumers are keeping up with the producers).  As far as I know I'm using the default settings for clean up (i.e. 30 secs).  It seems that the journal cleanUp routines don't kick in for some reason.  A restart of the broker causes the files to be cleared down.  Also, when I turn off a remote component (which produces and consumes) it sometimes results in the journals being tidied up.

I'm running 5.3.0.3. Any clues why this is happening?

Cheers,

Edited by: paulrowe on Oct 12, 2009 3:06 PM

[stlewis]

seems like something may not be acknowledging messages and so the broker is persisting messages, causing the ever increasing number of journal files.  You mention specifically that when you some remote component that produces/consumes sometimes the journal files clear up.  What does this component do?  Have you done a test where you don't run this component to see if the behavior changes?

[kernow]

Hi SLewis,

The problem is intermitant.  The component both consumes from a queue on the broker, and also produces to another.  To clarify: the component is actually identically installed and run on 5 'processing' servers.  Every day or so, it seems that one of the 'processors' mis-behaves with its JMS handling - resulting in the journals.  What's more the problem only appears when the entire system is under heavy load.

The system relies on having at least one of these 'processors' running, so disabling them all would not give us anything.

If there is a problem with a particular processor not acknowledging the messages correctly, is there a way of finding out which messages/queues/hosts are affected via, say, JMX?

[kernow]

I've done some more digging around and am seeing a high InFlightCount on the queues my components are consuming from.

The queues have 5 consumers, each with prefetch set to 1.  However, my the InFlightCount is 283 on one of the queues - and gradually growing.  What isn't obvious is which of my consumers are responsable for the 283 InFlightMessages.

I don't know what it causing this - I am a little suspicous of my use of Spring's JmsTemplate to explicitly 'recieve' from the queue (although I am using Jencks for pooling).  My Spring config is as follows:

<amq:redeliveryPolicy id="redeliveryPolicy1" initialRedeliveryDelay="1000"
          useExponentialBackOff="false" maximumRedeliveries="5" />

<bean id="prefetchPolicy" class="org.apache.activemq.ActiveMQPrefetchPolicy">
          <property name="queuePrefetch" value="1"></property>
</bean>


<bean id="jmsControllerFactory" class="org.jencks.amqpool.PooledConnectionFactory" destroy-method="stop">
    <property name="connectionFactory">
        <bean class="org.apache.activemq.ActiveMQConnectionFactory">
          <property name="brokerURL">
               <value>${jms.controller.brokerURL}</value>
          </property>
          <property name="redeliveryPolicy" ref="redeliveryPolicy1"></property>
          <property name="prefetchPolicy" ref="prefetchPolicy"></property>
     </bean>
   </property>
</bean>


<bean id="getQueue" class="org.apache.activemq.command.ActiveMQQueue">
         <constructor-arg index="0" value="getQueue"></constructor-arg>
</bean>
    
<bean id="getCommandJmsTemplate" class="org.springframework.jms.core.JmsTemplate">
     <property name="connectionFactory" ref="jmsControllerFactory"></property>
     <property name="defaultDestination">
          <ref local="getQueue"></ref>
     </property>
     <property name="messageConverter">
             <ref local="getMessageConverter"></ref>
     </property>
     <property name="receiveTimeout" value="10000"></property>
     <property name="sessionTransacted" value="true"></property>
</bean>