One interesting sidenote here. In my application, I am using the REST interface to create subscriptions. I am not subscribing directly from clients, but rather through my own service so that I do a few things prior to establishing the subscription. One of those things is to create a dedicated JMS Queue for that subscription. That all is working fine. Another thing I do is destroy that temporary queue when the client unsubscribes. When I do an unsubscribe and remove the queue, that seems to jar something loose, and that stuck thread wakes up. The server then proceeds to try to send all the backed up messages, but of course it can't, because I destroyed the queue. Here is the TRACE level logging associated with the destroying of the queue and what looks to be the resumption of my stuck thread. It's helpful having the thread names in the log file. Nothing really jumps out at me with this logging, but I figured I'd post it in case a better trained eye could spot something.
{noformat}
2011-04-24 21:10:27,079 DEBUG [com.osi.notification.test.NotificationTestBean] (http-0.0.0.0-8080-2) Unsubscribe Request for 1
2011-04-24 21:10:27,132 DEBUG [com.osi.notification.subscription.ActorBasedSubscriptionManager] (akka:event-driven:dispatcher:global-12) Received Message: com.osi.notification.subscription.UnsubscribeMessage@54fc218f
2011-04-24 21:10:27,134 DEBUG [com.osi.notification.subscription.SubscriptionManager] (akka:event-driven:dispatcher:global-12) Removing Subscription: 1
2011-04-24 21:10:27,182 DEBUG [com.osi.notification.subscription.SubscriptionHelper] (akka:event-driven:dispatcher:global-8) Removing subscription: http://localhost:8080/hornetq-rest/queues/jms.queue.testQueue1/push-consumers/1-1303691322274
2011-04-24 21:10:27,218 DEBUG [com.osi.notification.destination.DestinationHelper] (akka:event-driven:dispatcher:global-8) Destroying Queue: testQueue1
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRAMessageProducer] (akka:event-driven:dispatcher:global-4) sent org.hornetq.ra.HornetQRAMessageProducer@17fa711f result=HornetQMessage[ID:20d61254-6ed3-11e0-b206-001c42000009]:PERSISTENT
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRASession] (akka:event-driven:dispatcher:global-4) unlock()
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (akka:event-driven:dispatcher:global-4) unlock()
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRAMessageProducer] (akka:event-driven:dispatcher:global-4) close org.hornetq.ra.HornetQRAMessageProducer@17fa711f
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRASession] (akka:event-driven:dispatcher:global-4) removeProducer(org.hornetq.ra.HornetQRAMessageProducer@17fa711f)
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRASession] (akka:event-driven:dispatcher:global-4) close()
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRASessionFactoryImpl] (akka:event-driven:dispatcher:global-4) closeSession(org.hornetq.ra.HornetQRASession@1051694a)
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRASession] (akka:event-driven:dispatcher:global-4) Closing session
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (akka:event-driven:dispatcher:global-4) stop()
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (akka:event-driven:dispatcher:global-4) removeHandle(org.hornetq.ra.HornetQRASession@1051694a)
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (akka:event-driven:dispatcher:global-4) sendEvent(javax.resource.spi.ConnectionEvent[source=org.hornetq.ra.HornetQRAManagedConnection@cee2beb])
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRASessionFactoryImpl] (akka:event-driven:dispatcher:global-4) close() org.hornetq.ra.HornetQRASessionFactoryImpl@393e0cc5
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (Transaction Reaper Worker 0) unlock()
2011-04-24 21:10:27,266 TRACE [org.hornetq.ra.HornetQRAXAResource] (Transaction Reaper Worker 0) rollback(< formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffffc0a80145:126a:4db4c024:9ebc, node_name=1, branch_uid=0:ffffc0a80145:126a:4db4c024:9ebe, eis_name=unknown eis name >)
2011-04-24 21:10:27,269 TRACE [org.hornetq.utils.UTF8Util] (Thread-2 (group:HornetQ-client-global-threads-2133005797)) Reading string with utfSize=235
2011-04-24 21:10:27,280 WARN [com.arjuna.ats.jta] (Transaction Reaper Worker 0) ARJUNA-16045 attempted rollback of < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffffc0a80145:126a:4db4c024:9ebc, node_name=1, branch_uid=0:ffffc0a80145:126a:4db4c024:9ebe, eis_name=unknown eis name > (org.hornetq.ra.HornetQRAXAResource@1a376e79) failed with exception code XAException.XAER_NOTA: javax.transaction.xa.XAException
at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1488) [:]
at org.hornetq.ra.HornetQRAXAResource.rollback(HornetQRAXAResource.java:166) [:]
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:336) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2902) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2881) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1602) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:119) [:6.0.0.Final]
at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:212) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:367) [:6.0.0.Final]
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:79) [:6.0.0.Final]
2011-04-24 21:10:27,281 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (Transaction Reaper Worker 0) cleanup()
2011-04-24 21:10:27,281 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (Transaction Reaper Worker 0) destroyHandles()
2011-04-24 21:10:27,281 WARN [com.arjuna.ats.arjuna] (akka:event-driven:dispatcher:global-4) ARJUNA-12077 Abort called on already aborted atomic action 0:ffffc0a80145:126a:4db4c024:9ebc
2011-04-24 21:10:27,289 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA-12113 TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,jboss] missed interrupt when cancelling TX 0:ffffc0a80145:126a:4db4c024:9ebc -- exiting as zombie (zombie count decremented to 0)
2011-04-24 21:10:27,284 ERROR [com.osi.notification.NotificationHelper] (akka:event-driven:dispatcher:global-4) Transaction Error sending notification: javax.transaction.RollbackException: ARJUNA-16063 The transaction is not active!
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1148) [:6.0.0.Final]
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]
at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:162) [:6.0.0.Final]
at com.osi.notification.NotificationHelper.sendNotificationWithTx(NotificationHelper.java:41) [:]
at com.osi.notification.actors.DestinationSenderActor.onReceive(DestinationSenderActor.java:30) [:]
at akka.actor.UntypedActor$$anonfun$receive$1.apply(UntypedActor.scala:70) [:]
at akka.actor.UntypedActor$$anonfun$receive$1.apply(UntypedActor.scala:69) [:]
at akka.actor.Actor$$anonfun$4.apply(Actor.scala:481) [:]
at akka.actor.Actor$$anonfun$4.apply(Actor.scala:464) [:]
at akka.actor.Actor$class.apply(Actor.scala:435) [:]
at akka.actor.UntypedActor.apply(UntypedActor.scala:63) [:]
at akka.actor.LocalActorRef.akka$actor$LocalActorRef$$dispatch(ActorRef.scala:1012) [:]
at akka.actor.LocalActorRef$$anonfun$invoke$1.apply$mcV$sp(ActorRef.scala:832) [:]
at akka.actor.LocalActorRef$$anonfun$invoke$1.apply(ActorRef.scala:828) [:]
at akka.actor.LocalActorRef$$anonfun$invoke$1.apply(ActorRef.scala:828) [:]
at akka.util.ReentrantGuard.withGuard(LockUtil.scala:19) [:]
at akka.actor.LocalActorRef.invoke(ActorRef.scala:827) [:]
at akka.dispatch.MessageInvocation.invoke(MessageHandling.scala:23) [:]
at akka.dispatch.ExecutableMailbox$class.processMailbox(ExecutorBasedEventDrivenDispatcher.scala:190) [:]
at akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$1.processMailbox(ExecutorBasedEventDrivenDispatcher.scala:109) [:]
at akka.dispatch.ExecutableMailbox$class.run(ExecutorBasedEventDrivenDispatcher.scala:166) [:]
at akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$1.run(ExecutorBasedEventDrivenDispatcher.scala:109) [:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_22]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_22]
at java.lang.Thread.run(Thread.java:680) [:1.6.0_22]
at akka.dispatch.MonitorableThread.run(ThreadPoolBuilder.scala:185) [:]
{noformat}