Server side exception doesn't break transaction commit

[ysliu]

I'm using HornetQ_2_2_5_Final as JMS provider.

In my configuration paging is enabled.

In my program JMS client publish event in a JTA transaction boundary.

I ran into a case that disk use to store paging file is full, then server side got some IO exception. I expect to get some exception when try to commit the transaction in client side. But it didn't.

My question is:

Will hornetQ server handle exception and mark the transaction rollback_only?
May I make any configuration to let JMS client commit transaction fail in such a case?

Since HornetQ server logged an error of the exception so I got a call stack.

I went through the code alone the call stack and didn't find logic that handle such kind exception.

And finally this exception was discard by logging an error in ServerSessionPacketHandler.

I also go through latest code in main branch and found the logic is simillar. is there any exception handling strategy I didn't found?

The exception logged by HornetQ server is as following:

2012-03-09 05:31:08,058 ERROR [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught unexpected exception

java.io.IOException: No space left on device
       at sun.nio.ch.FileDispatcher.write0(Native Method)

       at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:44)

       at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)

       at sun.nio.ch.IOUtil.write(IOUtil.java:43)

       at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:198)

       at org.hornetq.core.journal.impl.NIOSequentialFile.doInternalWrite(NIOSequentialFile.java:343)

       at org.hornetq.core.journal.impl.NIOSequentialFile.internalWrite(NIOSequentialFile.java:302)

       at org.hornetq.core.journal.impl.NIOSequentialFile.writeDirect(NIOSequentialFile.java:265)

       at org.hornetq.core.paging.impl.PageImpl.write(PageImpl.java:195)

       at org.hornetq.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:878)

       at org.hornetq.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:320)

       at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:886)

       at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:634)

       at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:561)

       at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1365)

       at org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1107)

       at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:440)

       at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)

       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)

       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457)

       at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)

       at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:137)

       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

         at java.lang.Thread.run(Thread.java:662)

[ataylor]

you cant page to disc indefinately, obviously at some point the disc will become full, you need to either

a) use a bigger hard disc

b) use blocking instead of paging

[ysliu]

Andy, thanks for your reply.

Yes, I agree disc exhaustion is a corner case.

What I can't understand is why in case the message fail to be delivered, the transaction still can be commit successfully.

I noticed in ServerSessionImpl before handle message sending there are some logic to check the access privilige.

And if the checing fail, the transaction associated with the message will be marked as rollback_only.

Why there are not similiar logic for other exceptions when try to send the messeage?

e.g. the IO exception I met.

In client side if I get exception when I try to commit the transaction, at least I know there are somthing wrong in the server side.

And I will get a chance to stop my program instead go on sending messages and assume they are delivered successfully. 

[ataylor]

with issues like these that basically kill a server, others include OOM's etc, it may not be possible to notify the client (altho a timeout or disconnect should occur), the whole point here is that you should configure your server so this doesnt happen.

regarding thetx being commited thyat shouldnt happen, if it does its a bug, can you check and provide a test case id it is definately an issue.

[clebert.suconic]

There was a bug on 2.2.5 on cleaning up page BTW. Maybe that "helped" you to go over the disk limit?

[ysliu]

hi Andy, I got a case to reproduce the issue base on hornetq out of box example jms\xa-send.

I submitted a bug at

https://issues.jboss.org/browse/HORNETQ-972

and attached the case with the bug.

summary of the case:

since in this case what kind of exception that break the message delivering is not the key point, I configured hornetq pageing folder to an not exist path so that the pageing will fail.

in the case client program will use xa session to send batch of messages to a queue, and server side will get IO exception due to paging failure.

then client side try to commit the transaction.

the commition success and the case fail.

[ysliu]

hi Clebert, thanks for your reply.

since the case is not founding in production enviroment so I think the disk limitation is not the problem itself.

in this case the transaction comittion should fail is the key point.