3 Replies Latest reply on May 30, 2010 8:55 AM by venkat.g99

    QueueClosedException in jboss cluster

    venkat.g99
    venkat.g99 Feb 22, 2010 9:32 AM

      We have  a jboss cluster in production, we have two servers in each server we have two jboss nodes running, Once in a while we are getting the below excetpion in one of the node after that node is not serving the requests.

       

      Kindly help me to resolve this issue.

       

       

       

      2010-02-21 22:37:57,761 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:37:57,762 INFO  [STDOUT] 924370863 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:37:57,921 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:37:57,922 INFO  [STDOUT] 924371023 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:00,478 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:00,478 INFO  [STDOUT] 924373580 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:01,078 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:01,078 INFO  [STDOUT] 924374180 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:02,360 INFO  [STDOUT] 924375461 [ContainerBackgroundProcessor[StandardEngine[jboss.web]]] DEBUG org.jboss.web.tomcat.service.session.JBossCacheManager  - Looking for sessions that have expired ...
      2010-02-21 22:38:05,487 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:05,488 INFO  [STDOUT] 924378589 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:05,887 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:05,888 INFO  [STDOUT] 924378989 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:06,665 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:06,666 INFO  [STDOUT] 924379767 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:07,292 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:07,293 INFO  [STDOUT] 924380394 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:07,781 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:07,782 INFO  [STDOUT] 924380883 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:10,317 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:10,318 INFO  [STDOUT] 924383419 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:10,419 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:10,419 INFO  [STDOUT] 924383521 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:10,497 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:10,498 INFO  [STDOUT] 924383599 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:10,697 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:10,698 INFO  [STDOUT] 924383799 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:12,205 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:12,206 INFO  [STDOUT] 924385307 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:12,370 INFO  [STDOUT] 924385472 [ContainerBackgroundProcessor[StandardEngine[jboss.web]]] DEBUG org.jboss.web.tomcat.service.session.JBossCacheManager  - Looking for sessions that have expired ...
      2010-02-21 22:38:12,489 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:12,490 INFO  [STDOUT] 924385591 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:13,592 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:13,593 INFO  [STDOUT] 924386694 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:14,141 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:14,142 INFO  [STDOUT] 924387243 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:14,199 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:14,199 INFO  [STDOUT] 924387301 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:15,507 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:15,508 INFO  [STDOUT] 924388609 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:15,508 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:15,509 INFO  [STDOUT] 924388610 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:15,928 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:15,929 INFO  [STDOUT] 924389030 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:15,988 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:15,989 INFO  [STDOUT] 924389090 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:17,802 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:17,802 INFO  [STDOUT] 924390904 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:18,816 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:18,817 INFO  [STDOUT] 924391918 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:19,269 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:19,270 INFO  [STDOUT] 924392371 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:20,317 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:20,318 INFO  [STDOUT] 924393419 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:20,517 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:20,518 INFO  [STDOUT] 924393619 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:21,699 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:21,700 INFO  [STDOUT] 924394801 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:22,380 INFO  [STDOUT] 924395482 [ContainerBackgroundProcessor[StandardEngine[jboss.web]]] DEBUG org.jboss.web.tomcat.service.session.JBossCacheManager  - Looking for sessions that have expired ...
      2010-02-21 22:38:22,612 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:22,613 INFO  [STDOUT] 924395714 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:25,127 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:25,128 INFO  [STDOUT] 924398229 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:25,527 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:25,528 INFO  [STDOUT] 924398629 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:26,513 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:26,514 INFO  [STDOUT] 924399615 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:27,469 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:27,470 INFO  [STDOUT] 924400571 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:27,822 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:27,822 INFO  [STDOUT] 924400924 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:28,033 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:28,033 INFO  [STDOUT] 924401135 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:29,937 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:29,938 INFO  [STDOUT] 924403039 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:30,537 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:30,538 INFO  [STDOUT] 924403639 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:30,597 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:30,598 INFO  [STDOUT] 924403699 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:32,390 INFO  [STDOUT] 924405492 [ContainerBackgroundProcessor[StandardEngine[jboss.web]]] DEBUG org.jboss.web.tomcat.service.session.JBossCacheManager  - Looking for sessions that have expired ...
      2010-02-21 22:38:34,747 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:34,748 INFO  [STDOUT] 924407849 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:35,547 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:35,548 INFO  [STDOUT] 924408649 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:37,842 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:37,843 INFO  [STDOUT] 924410944 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:38,382 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:38,383 INFO  [STDOUT] 924411484 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:38,408 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:38,408 INFO  [STDOUT] 924411510 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:38,698 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:38,699 INFO  [STDOUT] 924411800 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:38,949 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:38,949 INFO  [STDOUT] 924412051 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,096 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,096 INFO  [STDOUT] 924412198 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,536 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,536 INFO  [STDOUT] 924412638 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,557 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,558 INFO  [STDOUT] 924412659 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,812 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,813 INFO  [STDOUT] 924412914 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,941 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:39,942 INFO  [STDOUT] 924413043 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,183 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,184 INFO  [STDOUT] 924413285 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,557 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,558 INFO  [STDOUT] 924413659 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,769 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,770 INFO  [STDOUT] 924413871 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,809 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,810 INFO  [STDOUT] 924413911 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,987 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:40,988 INFO  [STDOUT] 924414089 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:41,249 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:41,250 INFO  [STDOUT] 924414351 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:41,360 INFO  [STDOUT] 924414462 [IdleRemover] DEBUG org.jboss.resource.connectionmanager.IdleRemover  - run: IdleRemover notifying pools, interval: 450000
      2010-02-21 22:38:42,181 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:42,182 INFO  [STDOUT] 924415283 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:42,400 INFO  [STDOUT] 924415502 [ContainerBackgroundProcessor[StandardEngine[jboss.web]]] DEBUG org.jboss.web.tomcat.service.session.JBossCacheManager  - Looking for sessions that have expired ...
      2010-02-21 22:38:42,401 INFO  [STDOUT] 924415503 [ContainerBackgroundProcessor[StandardEngine[jboss.web]]] DEBUG org.jboss.web.tomcat.service.session.JBossCacheService  - Remove session from my own distributed store only. Fqn: /JSESSION/localhost/atheebweb/pIIwScKT3AcsD1nUPxDt8g**
      2010-02-21 22:38:42,412 ERROR [org.jgroups.protocols.TCP] failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)
      2010-02-21 22:38:42,413 INFO  [STDOUT] 924415514 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException

      • 5068 Views
      • Tags:
        • 1. Re: QueueClosedException in jboss cluster
          pbhanupratap
          pbhanupratap Mar 9, 2010 9:31 AM (in response to venkat.g99)

          We have a Jboss cluster of 8nodes, 4 nodes on Server1 and 4nodes on Server2...

          There is no firewall between the 2 servers..

          If we start all the 8nodes, we are getting the below exceptions in the 4 nodes of Server2 and not on server1

           

          13:15:17,922 ERROR [TCP] failed handling data from 10.4.21.31:10800QueueClosedException        at org.jgroups.util.Queue.add(Queue.java:139)        at org.jgroups.protocols.TP.receive(TP.java:777)        at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:226)        at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)        at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)        at java.lang.Thread.run(Thread.java:595)

           

          Also, getting the below errors on some of the nodes of Server1 and server2:

           

           

          14:44:19,303 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:7800 - 10.4.21.31:45537],5,ConnectionTableGroup]14:44:19,305 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,305 ERROR [ConnectionTable] 2nd attempt to send data failed too14:44:19,306 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:44857 - 10.4.21.30:8800],5,ConnectionTableGroup]14:44:19,307 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,307 ERROR [ConnectionTable] 2nd attempt to send data failed too14:44:19,304 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:44858 - 10.4.21.30:9800],5,ConnectionTableGroup]14:44:19,309 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,306 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:44859 - 10.4.21.30:10800],5,ConnectionTableGroup]14:44:19,312 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,303 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:7800 - 10.4.21.30:46137],5,ConnectionTableGroup]14:44:19,313 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,321 ERROR [ConnectionTable] 2nd attempt to send data failed too14:44:19,322 ERROR [ConnectionTable] 2nd attempt to send data failed too14:44:19,325 ERROR [ConnectionTable] 2nd attempt to send data failed too
            • Actions
          • 2. Re: QueueClosedException in jboss cluster
            pbhanupratap
            pbhanupratap Mar 11, 2010 9:22 AM (in response to pbhanupratap)

            This error looks like a Hardware configuration Issue of the server.

            When we increased the RAM from 8GB to 16GB and added 1 more CPU on each of the 2 servers, this error is gone.

              • Actions
            • 3. Re: QueueClosedException in jboss cluster
              venkat.g99
              venkat.g99 May 30, 2010 8:55 AM (in response to pbhanupratap)

              Hi Bhanu,

               

              You are right, we also upgraded RAM from 8GB to16GB, from last two months it is working fine.

                • Actions