Paging issue?

[noky]

Hi, we are running HornetQ 2.1.2 an encountered a problem today. First some quick background: We mainly have one topic that about 3 publishers take turns writing to and about a hundred subscribers listen on. The throughput isn't too high, there are typically about 3000 messages/minute. Most subscribers use a selector to listen to a subset of the data.

The symptom was that subscribers were getting stale data. Apparently the topic queue was backed up. Restarting the HornetQ server fixed the problem.

Investigating the log files, I saw thousands of messages like this:

[Thread-0 (group:HornetQ-server-threads33153822-24678922)] 07:31:03,850 WARNING [org.hornetq.core.paging.impl.PagingStoreImpl]  Transaction 2494730065 used during paging not found

Unfortunately, I don't have any history before these messages. All 5 of our rotated saved hornetq logs (10MB each) were filled with these messages. Each log file contained over 50K of these error messages, each covering a time span of only 2 seconds!

Can you help me understand exactly what happened? It appears something to do with the paging system, but I'm not sure what. Badly behaved client not processing messages? I would like to take steps to prevent this from happening again, but I need to understand exactly what happened first.

I know HornetQ 2.2.2 had the paging system overhauled... Do you recomend that we upgrade the HQ server? Is there a clear path to upgrade, since the client-server protocol has changed and we have roughly 100 separate clients that we'd need to upgrade?

Thanks for you help,

Mike

[clebert.suconic]

Transaction 2494730065 used during paging not found

^^ That means that either the TX was not persisted (you sent a message and didn't commit it).

Or maybe you sent the message as non-persistent.. and when depaging you won't see the TX while depaging.

- or you removed the journal and left paging?

I would recommend you to upgrade.. Maybe going beyond 2.2.2 already (Branch_2_2_EAP). I'm consdering having a release of the stable branch soon.

From 2.2+ we are keeping client-server protocol compatible. Older clients are always compatible with newer servers. The opposite may not be true though.(An older server with an newer client won't work)

[noky]

Well... we are not using JMS transactions or persistent messages. All messagses are sent AUTO_ACKNOWLEDGE. The journal on the server was never touched. Note that the thousands of log messages each list different transaction numbers. What is going on?

It is good to know that older clients are compatible with newer servers! I didn't realize that part. Is there a document which describes how to migrate from 2.1.x to 2.2.x? I have been looking through the 2.2.2 docs, but haven't found a "What's New" or "Changes" section. Is config from 2.1.x compatible with 2.2.x or do we need to add/remove/change certain parameters?

[clebert.suconic]

That is only from 2.2+.

If you upgrade from 2.1.2 to 2.2+, you will have to upgrade both client and server.

From now on the story will be different.

The fact that you have PageTransactions, tells me you were using Transactions somehow. Aren't you using the JCA connection factory? On that case the Auto_ACK is ignored.

[noky]

We are not using transactions, at least not intentionally. We are running HornetQ standalone, in a non-J2EE setting, we are not using JCA. Here is how we obtain connections:

Hashtable<String, String> props = new Hashtable<String, String>();

props.put("java.naming.provider.url","jnp://aaa.bbb.com:11099,jnp://ccc.bbb.com:11099");

props.put("java.naming.factory.initial","org.jnp.interfaces.NamingContextFactory");

props.put("java.naming.factory.url.pkgs","org.jboss.naming:org.jnp.interfaces");

InitialContext initialContext = new InitialContext(props);

Topic topic = (Topic)initialContext.lookup(topic);

ConnectionFactory cf = (ConnectionFactory)initialContext.lookup("/ConnectionFactory");

We are using a nearly stock hornetq-jms.xml file, with the addition of the topics we are using. I'm not sure how we could get a JCA connection factory with this setup. Am I missing something?

[clebert.suconic]

I'm not sure how you're using transactions. But the system is definitely using transactions somehow. Otherwise you wouldn't have page transactions.