problem with Paging/Drop address full policy on jms topic

[danielzietz]

We have some problems with JMS Topics provided by HornetQ 2.2.2 integrated in Jboss AS 5 on a production server:

- If address-full-policy is configured to use "PAGING", HornetQ does not delete page files even if all messages are consumed. This fills harddisk till no space is available anymore and jboss stops working.(This may be related to https://issues.jboss.org/browse/HORNETQ-529)

- if address-full-policy is configured to use "DROP" the topic stops accepting messages after some time(i.e. the value of "Messages Added" in the managment interface does not increment anymore and no more messages are delivered)

The topic receives about 500-700 messages per minute and deliveres them to an average of 10 consumers.

It is likely that the client connections are terminated by network errors quite often.

Nevertheless the only logging I found referring this is "WARN  [org.hornetq.core.server.impl.ServerSessionImpl] [T:248] Client connection failed, clearing up resources for session f1813b6e-0af1-11e1-bf3f-001f3b211d7d".

The configuration of the topic in hornetq-jms.xml looks like this:

	<topic name="events.notifications">
	<entry name="/topic/events.notifications"/>
	</topic>

And the configuration of address-full-policy in hornetq-configuration looks like this:

<address-settings>   <!--default for catch all-->   <address-setting match="#">      <dead-letter-address>jms.queue.DLQ</dead-letter-address>      <expiry-address>jms.queue.ExpiryQueue</expiry-address>      <redelivery-delay>0</redelivery-delay>      <max-size-bytes>50000000</max-size-bytes>        <message-counter-history-day-limit>10</message-counter-history-day-limit>      <page-size-bytes>100000</page-size-bytes> <address-full-policy>DROP</address-full-policy>   </address-setting>         </address-settings>

First tests showed that the paging problem might not appear with HornetQ 2.2.8, but since the problem is only randomly reproduceable on local installations this is not for sure.

Therefore I would be thankful for every tip (e.g. configuration problems) that could help to solve this problem.

[clebert.suconic]

First tests showed that the paging problem might not appear with HornetQ 2.2.8, but since the problem is only randomly reproduceable on local installations this is not for sure.

I have fixed fixed something related to that. So it is probably fixed.

[danielzietz]

Thanks for your response. Unfortunatly problem partly persists in version 2.2.8.

I have created two testcases - both used to fail with 2.2.2 and only one still fails with 2.2.8:

First I create quite a high load of messages (about 1000 per second). Afterwards i kill the connection of the topic subscriptors:

- In first testcase I kill the process consuming the topic messages

- In second testcase I disable network connection for a short time

- In a third testcase I first disabled network connection, killed the process and restarted network connection

First testcase is successfull using version 2.2.8(i.e. pagefiles are deleted after connection times out)

Second and third testcase still fail (i.e. a lot of pagefiles remain till they are deleted manually).

Do you have any idea what can cause the different behaviour?

[ataylor]

did you wait for the connection to timeout on 2 and 3?

[danielzietz]

I'm not sure what you mean with waiting:

-I did wait for connection timeout before checking if pagefiles were deleted(and till management interface told me that there was only the reconnected/no consumer left and message counter showed "0")

- I did not (always) wait for connection timeout before reenabling network connection. However on 2. the client started a second connection and on 3.  no client was left.

[ataylor]

what i mean is when you kill a client th eOS will send a close signal on the connection and the server will close the connection, you will see this in the logs. If you just pull out the network cable (or stop the interface etc) TCP will keep trying to send packets, in this case the server will not close the connection until it times out, are you waiting for this timeout, you will see the server close the connection in the server logs? if you are its a bug and you should rase a jira