Scott,

Thanks for your reply. I have found the problem and it appears to be with our firewall situation. Well I found that connections initiated from server were being denied since our clients are behind firewall themselves. Opening up a client box did resolve this issue.

But this is not an option for us unless we know which ports on the client the server is going to connect to. Can you suggest how to achieve notifications from server in a situation where client is behind a firewall.

I am posting the thread dumps from the run that hangs.

I hope you can throw some light on how to resolve this issue or some workaround.


Thread dump follow:

CLIENT
--------------

sun.applet.AppletViewer at localhost:10831 (Suspended)
System Thread [Finalizer] (Suspended)
Object.wait(long) line: not available [native method]
ReferenceQueue.remove(long) line: 111
ReferenceQueue.remove() line: 127
Finalizer$FinalizerThread.run() line: 159
System Thread [Reference Handler] (Suspended)
Object.wait(long) line: not available [native method]
Reference$Lock(Object).wait() line: 426
Reference$ReferenceHandler.run() line: 113
System Thread [Signal Dispatcher] (Suspended)
Thread [AWT-Shutdown] (Suspended)
Object.wait(long) line: not available [native method]
Object.wait() line: 426
AWTAutoShutdown.run() line: 259
Thread.run() line: 536
Thread [AWT-Windows] (Suspended)
WToolkit.eventLoop() line: not available [native method]
WToolkit.run() line: 253
Thread.run() line: 536
Thread [Java2D Disposer] (Suspended)
Object.wait(long) line: not available [native method]
ReferenceQueue.remove(long) line: 111
ReferenceQueue.remove() line: 127
Disposer.run() line: 97
Thread.run() line: 536
Thread [AWT-EventQueue-0] (Suspended)
Object.wait(long) line: not available [native method]
EventQueue(Object).wait() line: 426
EventQueue.getNextEvent() line: 333
EventDispatchThread.pumpOneEventForHierarchy(int, Component) line: 161
EventDispatchThread.pumpEventsForHierarchy(int, Conditional, Component) line: 150
EventDispatchThread.pumpEvents(int, Conditional) line: 144
EventDispatchThread.pumpEvents(Conditional) line: 136
EventDispatchThread.run() line: 99
Thread [thread applet-iTree.iTree.class] (Suspended)
Object.wait(long) line: not available [native method]
Thread.join(long) line: 1008
Thread.join() line: 1061
AppletViewerPanel(AppletPanel).run() line: 328
Thread.run() line: 536
Thread [DestroyJavaVM] (Suspended)
Thread [Thread-2] (Suspended)
SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) line: not available [native method]
SocketInputStream.read(byte[], int, int) line: 129
BufferedInputStream.fill() line: 183
BufferedInputStream.read() line: 201
DataInputStream.readByte() line: 276
StreamRemoteCall.executeCall() line: 189
UnicastRef2(UnicastRef).invoke(Remote, Method, Object[], long) line: 133
JRMPInvoker_Stub.invoke(Invocation) line: not available
JRMPInvokerProxy.invoke(Invocation) line: 135
InvokerInterceptor.invoke(Invocation) line: 96
InvokerAdaptorClientInterceptor.invoke(Invocation) line: 58
SecurityInterceptor.invoke(Invocation) line: 45
ClientMethodInterceptor.invoke(Invocation) line: 55
ClientContainer.invoke(Object, Method, Object[]) line: 85
$Proxy0.createMBean(String, ObjectName, Object[], String[]) line: not available
RMIConnectorImpl.createMBean(String, ObjectName, Object[], String[]) line: 230
RMIClientNotificationListener(ClientNotificationListener).createListener(RemoteMBeanServer, String, Object[], String[]) line: 73
RMIClientNotificationListener.(ObjectName, NotificationListener, Object, NotificationFilter, RemoteMBeanServer) line: 48
RMIConnectorImpl.addNotificationListener(ObjectName, NotificationListener, NotificationFilter, Object) line: 446
MBeanUtil.registerListener(MBeanServer, ObjectName, NotificationListener) line: 237
ITreeMBeanService.registerListener(NotificationListener) line: 70
ITreeSvcProxy.registerListener(ITreeNotificationListener) line: 95
DataLoadingThread.(DataManager) line: 41
DataManager.() line: 36
trapsStorage.() line: 32
iTree.() line: 1666
NativeConstructorAccessorImpl.newInstance0(Constructor, Object[]) line: not available [native method]
NativeConstructorAccessorImpl.newInstance(Object[]) line: 39
DelegatingConstructorAccessorImpl.newInstance(Object[]) line: 27
Constructor.newInstance(Object[]) line: 274
Class.newInstance0() line: 306
Class.newInstance() line: 259
AppletViewerPanel(AppletPanel).createApplet(AppletClassLoader) line: 567
AppletViewerPanel(AppletPanel).runLoader() line: 496
AppletViewerPanel(AppletPanel).run() line: 293
Thread.run() line: 536
Thread [TimerQueue] (Suspended)
Object.wait(long) line: not available [native method]
TimerQueue.run() line: 231
Thread.run() line: 536
Thread [Thread-3] (Suspended)
Object.wait(long) line: not available [native method]
TaskQueue(Object).wait() line: 426
TimerThread.mainLoop() line: 403
TimerThread.run() line: 382
Thread [Thread-4] (Suspended)
Object.wait(long) line: not available [native method]
TaskQueue(Object).wait() line: 426
TimerThread.mainLoop() line: 403
TimerThread.run() line: 382
System Thread [GC Daemon] (Suspended)
Object.wait(long) line: not available [native method]
GC$Daemon.run() line: 100
System Thread [RMI RenewClean-[216.168.164.210:4444]] (Suspended)
Object.wait(long) line: not available [native method]
ReferenceQueue.remove(long) line: 111
DGCClient$EndpointEntry$RenewCleanThread.run() line: 500
Thread.run() line: 536
System Thread [RMI TCP Accept-0] (Suspended)
PlainSocketImpl.socketAccept(SocketImpl) line: not available [native method]
PlainSocketImpl.accept(SocketImpl) line: 353
ServerSocket.implAccept(Socket) line: 439
ServerSocket.accept() line: 410
TCPTransport.run() line: 334
Thread.run() line: 536
Thread [Thread-6] (Suspended)
Object.wait(long) line: not available [native method]
TaskQueue(Object).wait() line: 426
TimerThread.mainLoop() line: 403
TimerThread.run() line: 382
System Thread [RMI Reaper] (Suspended)
Object.wait(long) line: not available [native method]
ReferenceQueue.remove(long) line: 111
ReferenceQueue.remove() line: 127
ObjectTable$Reaper.run() line: 330
Thread.run() line: 536



____________________
SERVER

__________________

Full thread dump Java HotSpot(TM) Server VM (1.4.2_03-b02 mixed mode):

"RMI RenewClean-[192.168.5.236:3909]" daemon prio=5 tid=0x00f9ff70 nid=0x4c in Object.wait() [ec401000..ec4019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf38c55a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0xf38c55a0> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:500)
at java.lang.Thread.run(Thread.java:534)

"RMI TCP Connection(12)-216.168.161.254" daemon prio=5 tid=0x00d716b0 nid=0x4a runnable [ec5fe000..ec6019a0]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305)
- locked <0xf38c5bc8> (a java.net.PlainSocketImpl)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158)
at java.net.Socket.connect(Socket.java:452)
at java.net.Socket.connect(Socket.java:402)
at java.net.Socket.(Socket.java:309)
at java.net.Socket.(Socket.java:124)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:562)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:313)
at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
at sun.rmi.transport.DGCClient$EndpointEntry.registerRefs(DGCClient.java:275)
at sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:112)
at sun.rmi.transport.LiveRef.read(LiveRef.java:274)
at sun.rmi.server.UnicastRef.readExternal(UnicastRef.java:464)
at java.rmi.server.RemoteObject.readObject(RemoteObject.java:420)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:838)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1746)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1646)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1274)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1603)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1271)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1603)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1271)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:324)
at org.jboss.invocation.MarshalledValue.get(MarshalledValue.java:78)
at org.jboss.invocation.MarshalledInvocation.getArguments(MarshalledInvocation.java:346)
at org.jboss.jmx.connector.invoker.InvokerAdaptorService.invoke(InvokerAdaptorService.java:218)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:367)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
at sun.rmi.transport.Transport$1.run(Transport.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:534)

"JNP Server" prio=5 tid=0x00e30680 nid=0x48 runnable [ef801000..ef8019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf2a7b2a8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jnp.server.Main.run(Main.java:281)
at java.lang.Thread.run(Thread.java:534)

"RMI RenewClean-[192.168.5.236:3872]" daemon prio=5 tid=0x0083fe50 nid=0x42 in Object.wait() [ec501000..ec5019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf36b62a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0xf36b62a0> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:500)
at java.lang.Thread.run(Thread.java:534)

"Thread-29" prio=5 tid=0x00669498 nid=0x3d in Object.wait() [ec801000..ec8019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf361cb78> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:772)
- locked <0xf361cb78> (a java.lang.Object)

"StandardManager[/itree-prov]" daemon prio=5 tid=0x00eb8688 nid=0x3c waiting on condition [ec901000..ec9019a0]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
at java.lang.Thread.run(Thread.java:534)

"StandardManager[/itree-svc]" daemon prio=5 tid=0x002cb948 nid=0x3b waiting on condition [ecb01000..ecb019a0]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
at java.lang.Thread.run(Thread.java:534)

"StandardManager[/web-console]" daemon prio=5 tid=0x001e63d8 nid=0x3a waiting on condition [eca01000..eca019a0]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
at java.lang.Thread.run(Thread.java:534)

"StandardManager[/jmx-console]" daemon prio=5 tid=0x00df38d8 nid=0x38 waiting on condition [ecc01000..ecc019a0]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
at java.lang.Thread.run(Thread.java:534)

"Thread-28" daemon prio=5 tid=0x0066e650 nid=0x37 runnable [eccff000..ecd019a0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:718)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:690)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1405)
at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:889)
- locked <0xf32776b0> (a oracle.jdbc.ttc7.TTC7Protocol)
at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1681)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1870)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:363)
- locked <0xf1d17118> (a oracle.jdbc.driver.OraclePreparedStatement)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:314)
- locked <0xf1d17118> (a oracle.jdbc.driver.OraclePreparedStatement)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:304)
at net.sf.hibernate.impl.BatcherImpl.getResultSet(BatcherImpl.java:83)
at net.sf.hibernate.loader.Loader.getResultSet(Loader.java:790)
at net.sf.hibernate.loader.Loader.doQuery(Loader.java:184)
at net.sf.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:128)
at net.sf.hibernate.loader.Loader.loadCollection(Loader.java:905)
at net.sf.hibernate.loader.Loader.loadCollection(Loader.java:880)
at net.sf.hibernate.loader.OneToManyLoader.initialize(OneToManyLoader.java:79)
at net.sf.hibernate.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:286)
at net.sf.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:3128)
at net.sf.hibernate.collection.PersistentCollection.initialize(PersistentCollection.java:203)
at net.sf.hibernate.collection.PersistentCollection.read(PersistentCollection.java:69)
at net.sf.hibernate.collection.Set.iterator(Set.java:131)
at com.ibasis.aqr.domain.Alarm.getRoutingChoiceStats(Unknown Source)
at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.exolab.castor.mapping.loader.FieldHandlerImpl.getValue(FieldHandlerImpl.java:373)
at org.exolab.castor.xml.FieldValidator.validate(FieldValidator.java:190)
at org.exolab.castor.xml.util.XMLClassDescriptorImpl.validate(XMLClassDescriptorImpl.java:885)
at org.exolab.castor.xml.Validator.validate(Validator.java:122)
at org.exolab.castor.xml.FieldValidator.validate(FieldValidator.java:233)
at org.exolab.castor.xml.util.XMLClassDescriptorImpl.validate(XMLClassDescriptorImpl.java:885)
at org.exolab.castor.xml.Validator.validate(Validator.java:122)
at org.exolab.castor.xml.Marshaller.validate(Marshaller.java:1770)
at org.exolab.castor.xml.Marshaller.marshal(Marshaller.java:717)
at com.ibasis.aqr.util.DomainMarshaller.marshallToXML(Unknown Source)
at com.ibasis.aqr.itree.service.jmx.ITreeAlarmSvc.retrieveITreeData(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
at org.jboss.varia.scheduler.Scheduler$MBeanListener.handleNotification(Scheduler.java:1354)
at org.jboss.mx.server.NotificationListenerProxy.handleNotification(NotificationListenerProxy.java:69)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:95)
at javax.management.timer.Timer.sendNotifications(Timer.java:441)
at javax.management.timer.Timer.access$000(Timer.java:31)
at javax.management.timer.Timer$RegisteredNotification.doRun(Timer.java:612)
at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:164)
at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:225)

"Thread-27" prio=5 tid=0x01042ef8 nid=0x36 in Object.wait() [ece01000..ece019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf338fd58> (a org.jboss.mx.util.RunnableScheduler)
at java.lang.Object.wait(Object.java:429)
at org.jboss.mx.util.RunnableScheduler.waitOutstanding(RunnableScheduler.java:181)
- locked <0xf338fd58> (a org.jboss.mx.util.RunnableScheduler)
at org.jboss.mx.util.RunnableScheduler.run(RunnableScheduler.java:93)
at java.lang.Thread.run(Thread.java:534)

"IdleRemover" prio=5 tid=0x001910c8 nid=0x35 in Object.wait() [ecf01000..ecf019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf32440b0> (a java.util.ArrayList)
at org.jboss.resource.connectionmanager.IdleRemover$1.run(IdleRemover.java:66)
- locked <0xf32440b0> (a java.util.ArrayList)
at java.lang.Thread.run(Thread.java:534)

"UILServerILService Accept Thread" prio=5 tid=0x0103a0b0 nid=0x34 runnable [ed001000..ed0019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf31e0768> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:118)
at java.lang.Thread.run(Thread.java:534)

"RMI TCP Accept-0" daemon prio=5 tid=0x006429e0 nid=0x33 runnable [ed101000..ed1019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf31e0890> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)

"OIL2 Worker Server" prio=5 tid=0x00cb3de0 nid=0x32 runnable [ed201000..ed2019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf31e09b8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.mq.il.oil2.OIL2ServerILService.run(OIL2ServerILService.java:364)
at java.lang.Thread.run(Thread.java:534)

"OIL Worker Server" prio=5 tid=0x0020ff38 nid=0x31 runnable [ed301000..ed3019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf31a9028> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.mq.il.oil.OILServerILService.run(OILServerILService.java:472)
at java.lang.Thread.run(Thread.java:534)

"Thread-19" prio=5 tid=0x0078db88 nid=0x30 in Object.wait() [ed401000..ed4019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf31a90d8> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:429)
at org.jboss.resource.connectionmanager.PoolFiller$1.run(PoolFiller.java:75)
- locked <0xf31a90d8> (a java.util.LinkedList)
at java.lang.Thread.run(Thread.java:534)

"Thread-18" prio=5 tid=0x010878c8 nid=0x2f waiting on condition [ed501000..ed5019a0]
at java.lang.Thread.sleep(Native Method)
at org.hsqldb.Log.run(Unknown Source)
at java.lang.Thread.run(Thread.java:534)

"Thread-17" daemon prio=5 tid=0x01087728 nid=0x2e in Object.wait() [ed601000..ed6019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf31a91d8> (a org.jboss.util.timeout.TimeoutFactory)
at java.lang.Object.wait(Object.java:429)
at org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:497)
- locked <0xf31a91d8> (a org.jboss.util.timeout.TimeoutFactory)
at org.jboss.util.timeout.TimeoutFactory.access$300(TimeoutFactory.java:27)
at org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:542)

"Thread-16" daemon prio=5 tid=0x00a46970 nid=0x2d in Object.wait() [ed701000..ed7019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf31a9240> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xf31a9240> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

"JBossMQ Cache Reference Softner" daemon prio=5 tid=0x00cb4f08 nid=0x2c in Object.wait() [ed801000..ed8019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf3070eb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0xf3070eb8> (a java.lang.ref.ReferenceQueue$Lock)
at org.jboss.mq.server.MessageCache.run(MessageCache.java:225)
at java.lang.Thread.run(Thread.java:534)

"SnmpPortal--1-FastReceiver" prio=5 tid=0x009beaa0 nid=0x2b runnable [edf01000..edf019a0]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
- waiting to lock <0xf2db4360> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xf1409af8> (a java.net.DatagramPacket)
- locked <0xf2db4390> (a java.net.DatagramSocket)
at org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:305)
at java.lang.Thread.run(Thread.java:534)

"SnmpPortal--1" prio=5 tid=0x007d7898 nid=0x2a in Object.wait() [ee001000..ee0019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2db4428> (a java.util.LinkedList)
at org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:372)
- locked <0xf2db4428> (a java.util.LinkedList)
at java.lang.Thread.run(Thread.java:534)

"SnmpTimer" prio=5 tid=0x009bdeb0 nid=0x29 in Object.wait() [ee101000..ee1019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2db4498> (a java.lang.Object)
at org.opennms.protocols.snmp.SnmpTimer$Scheduler.run(SnmpTimer.java:208)
- locked <0xf2db4498> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:534)

"SnmpPortal--1-FastReceiver" prio=5 tid=0x00eef830 nid=0x28 runnable [ee201000..ee2019a0]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
- waiting to lock <0xf2d80bf0> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xf1411af8> (a java.net.DatagramPacket)
- locked <0xf2d80c20> (a java.net.DatagramSocket)
at org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:305)
at java.lang.Thread.run(Thread.java:534)

"Thread-15" prio=5 tid=0x00881570 nid=0x27 in Object.wait() [ee301000..ee3019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2d80cb8> (a org.jboss.mx.util.RunnableScheduler)
at java.lang.Object.wait(Object.java:429)
at org.jboss.mx.util.RunnableScheduler.waitOutstanding(RunnableScheduler.java:181)
- locked <0xf2d80cb8> (a org.jboss.mx.util.RunnableScheduler)
at org.jboss.mx.util.RunnableScheduler.run(RunnableScheduler.java:93)
at java.lang.Thread.run(Thread.java:534)

"SnmpPortal--1" prio=5 tid=0x009dc348 nid=0x26 in Object.wait() [ee401000..ee4019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2d80d18> (a java.util.LinkedList)
at org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:372)
- locked <0xf2d80d18> (a java.util.LinkedList)
at java.lang.Thread.run(Thread.java:534)

"StandardManager[/jbossmq-httpil]" daemon prio=5 tid=0x003b3410 nid=0x25 waiting on condition [ef601000..ef6019a0]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
at java.lang.Thread.run(Thread.java:534)

"StandardManager[/invoker]" daemon prio=5 tid=0x00a9c0e0 nid=0x24 waiting on condition [ee701000..ee7019a0]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
at java.lang.Thread.run(Thread.java:534)

"TP-Monitor" daemon prio=5 tid=0x00981338 nid=0x23 in Object.wait() [ee801000..ee8019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2bdb238> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
- locked <0xf2bdb238> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:534)

"TP-Processor4" daemon prio=5 tid=0x00980f48 nid=0x22 runnable [ee901000..ee9019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf2bdb2c0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:287)
at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:581)
at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:767)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)

"TP-Processor3" daemon prio=5 tid=0x003e0610 nid=0x21 in Object.wait() [eea01000..eea019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2bdb4e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
- locked <0xf2bdb4e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)

"TP-Processor2" daemon prio=5 tid=0x003e0470 nid=0x20 in Object.wait() [eeb01000..eeb019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2bdb560> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
- locked <0xf2bdb560> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)

"TP-Processor1" daemon prio=5 tid=0x00981f98 nid=0x1f in Object.wait() [eec01000..eec019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2bdb5e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
- locked <0xf2bdb5e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)

"TP-Monitor" daemon prio=5 tid=0x003e3550 nid=0x1e in Object.wait() [eed01000..eed019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2b82368> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
- locked <0xf2b82368> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:534)

"TP-Processor4" daemon prio=5 tid=0x00d41b48 nid=0x1d runnable [eee01000..eee019a0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:767)
at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:428)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:743)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:605)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)

"TP-Processor3" daemon prio=5 tid=0x00d419a8 nid=0x1c runnable [eef01000..eef019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf2b823f0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:107)
at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:387)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:569)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)

"TP-Processor2" daemon prio=5 tid=0x00367100 nid=0x1b in Object.wait() [ef001000..ef0019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2b825d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
- locked <0xf2b825d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)

"TP-Processor1" daemon prio=5 tid=0x00769608 nid=0x1a runnable [ef101000..ef1019a0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:767)
at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:428)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:743)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:605)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)

"Thread-4" prio=5 tid=0x007d42f8 nid=0x19 runnable [ef401000..ef4019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf2a8c218> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:211)
at java.lang.Thread.run(Thread.java:534)

"RMI LeaseChecker" daemon prio=5 tid=0x0033b880 nid=0x18 waiting on condition [ef501000..ef5019a0]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
at java.lang.Thread.run(Thread.java:534)

"RMI TCP Accept-4444" daemon prio=5 tid=0x00905048 nid=0x16 runnable [ef701000..ef7019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf2a8c378> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)

"GC Daemon" daemon prio=2 tid=0x006d6b78 nid=0x14 in Object.wait() [ef901000..ef9019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2a7b358> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0xf2a7b358> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=5 tid=0x006d68f8 nid=0x13 in Object.wait() [efa01000..efa019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2a7b3a8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0xf2a7b3a8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
at java.lang.Thread.run(Thread.java:534)

"Thread-3" daemon prio=5 tid=0x006e1e98 nid=0x12 in Object.wait() [efb01000..efb019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2a7b420> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:429)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0xf2a7b420> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

"RMI TCP Accept-1098" daemon prio=5 tid=0x006e1cf8 nid=0x11 runnable [efc01000..efc019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf2a7b488> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)

"Thread-2" daemon prio=5 tid=0x006d5ee8 nid=0x10 runnable [efd01000..efd019a0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xf2a7b580> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.web.WebServer.run(WebServer.java:258)
at org.jboss.web.ThreadPool$Worker.run(ThreadPool.java:163)

"ScannerThread" daemon prio=5 tid=0x0068c1d0 nid=0xf waiting on condition [efe01000..efe019a0]
at java.lang.Thread.sleep(Native Method)
at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:218)
at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:191)

"Thread-0" daemon prio=5 tid=0x00264f00 nid=0xe in Object.wait() [eff01000..eff019a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2a04d38> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:429)
- locked <0xf2a04d38> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)

"DestroyJavaVM" prio=5 tid=0x0002d0f8 nid=0x1 waiting on condition [0..ffbeed88]

"Signal Dispatcher" daemon prio=10 tid=0x000e6d10 nid=0x8 waiting on condition [0..0]

"Finalizer" daemon prio=8 tid=0x000e3778 nid=0x6 in Object.wait() [fc281000..fc2819a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2964f50> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0xf2964f50> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x000e2618 nid=0x5 in Object.wait() [fc381000..fc3819a0]
at java.lang.Object.wait(Native Method)
- waiting on <0xf2964c18> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:429)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
- locked <0xf2964c18> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=5 tid=0x000e1f80 nid=0x4 runnable

"VM Periodic Task Thread" prio=10 tid=0x000eac90 nid=0xc waiting on condition
"Suspend Checker Thread" prio=10 tid=0x000e48e8 nid=0x7 runnable

incidentally changing the jboss-service.xml to pooled invoker in in jmx-invoker-adapter-server.sar seems to have resolved my hang problem. But notifications are not coming back. Don't see any exception either.

There's probably more to than just changing it to pooled invoker. I kept the client code still the same. Do I need to change something else also. Any pointers would be greatkly appreciated. Also if someone can tell me what level of trace or debug to turn on to see why notifications are not coming back up that would be great. Just setting org.jboss to debug didn't help any.

Here's what I did. Lines in red denotesmy changes.

<!-- The JRMP invoker proxy configuration for the InvokerAdaptorService -->
 <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
 name="jboss.jmx:type=adaptor,name=Invoker,protocol=jrmp,service=proxyFactory">
 <!-- Use the standard JRMPInvoker from conf/jboss-service.xxml -->
 <attribute name="InvokerName">jboss:service=invoker,type=pooled</attribute>
 <!-- The target MBean is the InvokerAdaptorService configured below -->
 <attribute name="TargetName">jboss.jmx:type=adaptor,name=Invoker</attribute>
 <!-- Where to bind the RMIAdaptor proxy -->
 <attribute name="JndiName">jmx/invoker/RMIAdaptor</attribute>
 <!-- The RMI compabitle MBeanServer interface -->
 <attribute name="ExportedInterface">org.jboss.jmx.adaptor.rmi.RMIAdaptor</attribute>
 <attribute name="ClientInterceptors">
 <interceptors>
 <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
 <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
 <interceptor>org.jboss.jmx.connector.invoker.client.InvokerAdaptorClientInterceptor</interceptor>
 <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
 </interceptors>
 </attribute>
 <depends>jboss:service=invoker,type=pooled</depends>
</mbean>