1 2 Previous Next 25 Replies Latest reply on Mar 4, 2009 4:21 PM by dschneider

    JBREM-877: New Socket Connection is being Created for Every

    ron_sigal
    ron_sigal Dec 20, 2007 5:35 PM

      Statement of problem:


      whenever EJB3 client requests the Server new socket is being created. at the client side the socket connection pool is not being used. Hope there is some issue with the socket connection after getting response from the server,becoz of which the server side the connection is being closed. this increases the CPU utilization,

      The same code is running fine in jboss 4.0.4 with jboss remoting 1.4.0

      new remoting 2.2.2 SP4 only handling 1/4 of the load with high CPU utilization.


      "ron" wrote:

      Kumarselvan, what is your evidence for this behavior? Do you have some log file(s)?


      "Kumarselvan" wrote:

      i am getting the following log:
      2007-12-17 20:12:33,138 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
      2007-12-17 20:12:33,143 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#1[131.10.20.210:40824] closing socketWrapper: ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40824,localport=3873].229885]
      2007-12-17 20:12:33,143 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] wrote CLOSING
      2007-12-17 20:12:33,143 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40824,localport=3873].229885] closing
      2007-12-17 20:12:33,231 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
      2007-12-17 20:12:33,236 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#0[131.10.20.210:40825] closing socketWrapper: ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40825,localport=3873].aea1a2]
      2007-12-17 20:12:33,236 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] wrote CLOSING
      2007-12-17 20:12:33,236 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40825,localport=3873].aea1a2] closing
      2007-12-17 20:12:33,247 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
      2007-12-17 20:12:33,252 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#1[131.10.20.210:40826] closing socketWrapper: ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40826,localport=3873].5cf1a4]
      2007-12-17 20:12:33,252 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] wrote CLOSING
      2007-12-17 20:12:33,252 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40826,localport=3873].5cf1a4] closing
      2007-12-17 20:12:33,259 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
      2007-12-17 20:12:33,265 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#0[131.10.20.210:40827] closing socketWrapper: ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40827,localport=3873].150a19a]
      2007-12-17 20:12:33,265 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] wrote CLOSING
      2007-12-17 20:12:33,265 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40827,localport=3873].150a19a] closing
      2007-12-17 20:12:33,331 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
      2007-12-17 20:12:33,336 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#1[131.10.20.210:40828] closing socketWrapper: ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40828,localport=3873].1c3f810]
      2007-12-17 20:12:33,336 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] wrote CLOSING
      2007-12-17 20:12:33,336 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40828,localport=3873].1c3f810] closing
      2007-12-17 20:12:33,372 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
      2007-12-17 20:12:33,376 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#0[131.10.20.210:40829] closing socketWrapper: ServerSocketWrapper[Socket[addr=/131.10.20.210,port=40829,localport=3873].1be1b4]
      "se


      This is keep on happening.....


      also i have seen in the netstat -nac | grep 3873

      in this i am able to see the client side port is changing and it is increasing for each connections made


      I am using default configuration of ejb3 remoting conf

      <mbean code="org.jboss.remoting.transport.Connector"
      name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
      <depends>jboss.aop:service=AspectDeployer</depends>
      <attribute name="InvokerLocator">socket://${jboss.bind.address}:3873</attribute>
      <attribute name="Configuration">
      <handlers>
      <handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
      </handlers>
      </attribute>
      </mbean>

      Is there is any other configration change i need to make it with the current release so that the problem i am having will get resolved...

      One thing i am not able to understand is y the connection getting closed for each call at the server side, this closing needs to happen only after the timeout.
      i Hope there is some exception which causes the connection to close forcely...


      Kumarselvan, after a worker thread [org.jboss.remoting.transport.socket.ServerThread] processes an invocation, it should just go to the top of a loop and wait for the next invocation. If the load is very high, the server will try to close a ServerThread's socket and reuse it with another socket, but I wouldn't expect the regular behavior you're seeing. Another possibility is that some exception is being thrown in the course of handling an invocation, which would cause the socket to be closed.

      Could you turn on trace logging so we can get more information? Add 

       <category name="org.jboss.remoting">
 <priority value="TRACE"/>
 </category>


      to jboss-log4j.xml in the conf directory. See http://wiki.jboss.org/wiki/Wiki.jsp?page=ConfigureLogging for a discussion of logging in the Application Server. Thanks.



      • 50308 Views
      • Tags:
        • 1. Re: JBREM-877: New Socket Connection is being Created for Ev
          kumarselvan
          kumarselvan Dec 21, 2007 10:41 AM (in response to ron_sigal)

          Will check this by enabling the trace Log and will send you the further information by tomorrow

            • Actions
          • 2. Re: JBREM-877: New Socket Connection is being Created for Ev
            kumarselvan
            kumarselvan Dec 24, 2007 5:58 AM (in response to ron_sigal)

            2007-12-24 16:13:57,968 185859 TRACE [org.jboss.remoting.ServerInvoker] (WorkerThread#0[131.10.20.40:2508]:) SocketServerInvoker[131.10.20.40:3873] successfully dispatched invocation, returning org.jboss.aop.joinpoint.InvocationResponse@d5a86c from subsystem 'AOP' to client 3nak14-awcxm4-fakvt4xg-1-fakvt4xg-2
            2007-12-24 16:13:57,968 185859 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) SocketServerInvoker[131.10.20.40:3873].invoke() returned org.jboss.aop.joinpoint.InvocationResponse@d5a86c
            2007-12-24 16:13:57,968 185859 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) creating response instance
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] (AcceptorThread#0:3873:) SocketServerInvoker[131.10.20.40:3873] accepted Socket[addr=/131.10.20.40,port=2513,localport=3873]
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] (AcceptorThread#0:3873:) SocketServerInvoker[131.10.20.40:3873] trying to get a worker thread from threadpool for processing
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] (AcceptorThread#0:3873:) SocketServerInvoker[131.10.20.40:3873] has an empty threadpool
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] (AcceptorThread#0:3873:) SocketServerInvoker[131.10.20.40:3873] creating new worker thread
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] (AcceptorThread#0:3873:) SocketServerInvoker[131.10.20.40:3873] created WorkerThread#1[131.10.20.40:2513]
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] (AcceptorThread#0:3873:) SocketServerInvoker[131.10.20.40:3873] starting WorkerThread#1[131.10.20.40:2513]
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] (AcceptorThread#0:3873:) passed through ServerSocketRefresh.release()
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] (AcceptorThread#0:3873:) SocketServerInvoker[131.10.20.40:3873] is going to wait on serverSocket.accept()
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) wrote response to the output stream
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) preparing to process next invocation invocation
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) blocking to read version from input stream
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) read version -1 from input stream
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) EOFException received. This is likely due to client finishing communication.
            java.io.EOFException
            at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:530)
            at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
            at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)            2007-12-24 16:13:58,000 185891 DEBUG [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) WorkerThread#0[131.10.20.40:2508] closing socketWrapper: ServerSocketWrapper[Socket[addr=/131.10.20.40,port=2508,localport=3873].8f337b]
            2007-12-24 16:13:58,000 185891 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] (WorkerThread#0[131.10.20.40:2508]:) wrote CLOSING
            2007-12-24 16:13:58,000 185891 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] (WorkerThread#0[131.10.20.40:2508]:) ServerSocketWrapper[Socket[addr=/131.10.20.40,port=2508,localport=3873].8f337b] closing
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) WorkerThread#0[131.10.20.40:2508] removing itself from clientpool and going to threadpool
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#0[131.10.20.40:2508]:) WorkerThread#0[131.10.20.40:2508] begins to wait
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) beginning dorun()
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) creating the socket wrapper
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketWrapper] (WorkerThread#1[131.10.20.40:2513]:) constructing org.jboss.remoting.transport.socket.ServerSocketWrapper instance for Socket[addr=/131.10.20.40,port=2513,localport=3873], using timeout 60000
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.transport.socket.SocketWrapper] (WorkerThread#1[131.10.20.40:2513]:) ServerSocketWrapper[Socket[addr=/131.10.20.40,port=2513,localport=3873].db06de] setting timeout to 60000
            2007-12-24 16:13:58,000 185891 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] (WorkerThread#1[131.10.20.40:2513]:) ServerSocketWrapper[Socket[addr=/131.10.20.40,port=2513,localport=3873].db06de] getting output stream from Socket[addr=/131.10.20.40,port=2513,localport=3873], org.jboss.remoting.marshal.serializable.SerializableMarshaller@19febb9
            2007-12-24 16:13:58,000 185891 TRACE [org.jboss.remoting.serialization.impl.java.JavaSerializationManager] (WorkerThread#1[131.10.20.40:2513]:) Creating ObjectOutputStream
            2007-12-24 16:13:58,015 185906 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] (WorkerThread#1[131.10.20.40:2513]:) ServerSocketWrapper[Socket[addr=/131.10.20.40,port=2513,localport=3873].db06de] getting input stream from Socket[addr=/131.10.20.40,port=2513,localport=3873], org.jboss.remoting.marshal.serializable.SerializableUnMarshaller@2cd301
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.serialization.impl.java.JavaSerializationManager] (WorkerThread#1[131.10.20.40:2513]:) Creating ObjectInputStreamWithClassLoader
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.transport.socket.SocketWrapper] (WorkerThread#1[131.10.20.40:2513]:) ServerSocketWrapper[Socket[addr=/131.10.20.40,port=2513,localport=3873].db06de] setting timeout to 60000
            2007-12-24 16:13:58,015 185906 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] (WorkerThread#1[131.10.20.40:2513]:) reset timeout: 60000
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) processing first invocation without acknowledging
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) preparing to process next invocation invocation
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) blocking to read version from input stream
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) read version 22 from input stream
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) blocking to read invocation from unmarshaller
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) read InvocationRequest[d1223d, AOP, [advisedMethod=null, unadvisedMethod=null, metadata=[metaData={IS_LOCAL={GUID=[type=AS_ISvalue=jboss.j2ee:ear=voip.ear,jar=voip-ejb.jar,name=GatekeeperSFBean,service=EJB3,VMID=7fd3c9b85af1f0e9:-6d9356ce:1170bbdc8cf:-7fff]}, REMOTING={INVOKER_LOCATOR=[type=AS_ISvalue=InvokerLocator [socket://131.10.20.40:3873/]], SUBSYSTEM=[type=AS_ISvalue=AOP]}, DISPATCHER={OID=[type=AS_ISvalue=jboss.j2ee:ear=voip.ear,jar=voip-ejb.jar,name=GatekeeperSFBean,service=EJB3]}}], targetObject=null, arguments=null]] from unmarshaller
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) about to call SocketServerInvoker[131.10.20.40:3873].invoke()
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.ServerInvoker] (WorkerThread#1[131.10.20.40:2513]:) SocketServerInvoker[131.10.20.40:3873] received [advisedMethod=null, unadvisedMethod=null, metadata=[metaData={IS_LOCAL={GUID=[type=AS_ISvalue=jboss.j2ee:ear=voip.ear,jar=voip-ejb.jar,name=GatekeeperSFBean,service=EJB3,VMID=7fd3c9b85af1f0e9:-6d9356ce:1170bbdc8cf:-7fff]}, REMOTING={INVOKER_LOCATOR=[type=AS_ISvalue=InvokerLocator [socket://131.10.20.40:3873/]], SUBSYSTEM=[type=AS_ISvalue=AOP]}, DISPATCHER={OID=[type=AS_ISvalue=jboss.j2ee:ear=voip.ear,jar=voip-ejb.jar,name=GatekeeperSFBean,service=EJB3]}}], targetObject=null, arguments=null]
            2007-12-24 16:13:58,015 185906 TRACE [org.jboss.remoting.ServerInvoker] (WorkerThread#1[131.10.20.40:2513]:) SocketServerInvoker[131.10.20.40:3873] dispatching InvocationRequest[d1223d, AOP, [advisedMethod=null, unadvisedMethod=null, metadata=[metaData={IS_LOCAL={GUID=[type=AS_ISvalue=jboss.j2ee:ear=voip.ear,jar=voip-ejb.jar,name=GatekeeperSFBean,service=EJB3,VMID=7fd3c9b85af1f0e9:-6d9356ce:1170bbdc8cf:-7fff]}, REMOTING={INVOKER_LOCATOR=[type=AS_ISvalue=InvokerLocator [socket://131.10.20.40:3873/]], SUBSYSTEM=[type=AS_ISvalue=AOP]}, DISPATCHER={OID=[type=AS_ISvalue=jboss.j2ee:ear=voip.ear,jar=voip-ejb.jar,name=GatekeeperSFBean,service=EJB3]}}], targetObject=null, arguments=null]] from client 3nak14-awcxm4-fakvt4xg-1-fakvt6lc-4 to subsystem 'AOP'
            2007-12-24 16:13:58,125 186016 TRACE [org.jboss.remoting.ServerInvoker] (WorkerThread#1[131.10.20.40:2513]:) SocketServerInvoker[131.10.20.40:3873] successfully dispatched invocation, returning org.jboss.aop.joinpoint.InvocationResponse@158bc22 from subsystem 'AOP' to client 3nak14-awcxm4-fakvt4xg-1-fakvt6lc-4
            2007-12-24 16:13:58,125 186016 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) SocketServerInvoker[131.10.20.40:3873].invoke() returned org.jboss.aop.joinpoint.InvocationResponse@158bc22
            2007-12-24 16:13:58,125 186016 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) creating response instance
            2007-12-24 16:13:58,468 186359 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) wrote response to the output stream
            2007-12-24 16:13:58,468 186359 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) preparing to process next invocation invocation
            2007-12-24 16:13:58,468 186359 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) blocking to read version from input stream
            2007-12-24 16:13:58,468 186359 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) read version -1 from input stream
            2007-12-24 16:13:58,468 186359 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) EOFException received. This is likely due to client finishing communication.
            java.io.EOFException
            at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:530)
            at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
            at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)            2007-12-24 16:13:58,468 186359 DEBUG [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) WorkerThread#1[131.10.20.40:2513] closing socketWrapper: ServerSocketWrapper[Socket[addr=/131.10.20.40,port=2513,localport=3873].db06de]
            2007-12-24 16:13:58,468 186359 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] (WorkerThread#1[131.10.20.40:2513]:) wrote CLOSING
            2007-12-24 16:13:58,468 186359 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] (WorkerThread#1[131.10.20.40:2513]:) ServerSocketWrapper[Socket[addr=/131.10.20.40,port=2513,localport=3873].db06de] closing
            2007-12-24 16:13:58,468 186359 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) WorkerThread#1[131.10.20.40:2513] removing itself from clientpool and going to threadpool
            2007-12-24 16:13:58,468 186359 TRACE [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#1[131.10.20.40:2513]:) WorkerThread#1[131.10.20.40:2513] begins to wait

              • Actions
            • 3. Re: JBREM-877: New Socket Connection is being Created for Ev
              ron_sigal
              ron_sigal Dec 29, 2007 4:49 PM (in response to ron_sigal)

              Hi Kumarselvan,

              So far I see ServerThread reacting appropriately in response to an end of file condition - that is, it returns itself to the pool. The question is, why are sockets being closed on the client side?

              Can we see a trace log from the client side? Thanks.

                • Actions
              • 4. Re: JBREM-877: New Socket Connection is being Created for Ev
                ron_sigal
                ron_sigal Jan 26, 2008 3:19 AM (in response to ron_sigal)

                Hi Kumarselvan,

                Are you still experiencing this problem?

                Thanks,
                Ron

                  • Actions
                • 5. Re: JBREM-877: New Socket Connection is being Created for Ev
                  kumarselvan
                  kumarselvan Jan 31, 2008 12:46 PM (in response to ron_sigal)

                  Will check and let you know... current days i am looking in to other things
                  will update you soon

                    • Actions
                  • 6. Re: JBREM-877: New Socket Connection is being Created for Ev
                    darrengjones
                    darrengjones Feb 6, 2008 11:33 AM (in response to ron_sigal)

                    I am finding the same behaviour - every call to an EJB3 stateless session bean appears to be creating a new client socket. I have been running through the source in the debugger and it seems to be due to this code (snipped in places):

                    public class InvokeRemoteInterceptor ...
                    {
                    public Object invoke(org.jboss.aop.joinpoint.Invocation invocation) throws Throwable
                    {

                    Client client = new Client(locator, subsystem);

                    try
                    {
                    client.connect();
                    org.jboss.aop.joinpoint.InvocationResponse response = (org.jboss.aop.joinpoint.InvocationResponse)client.invoke(invocation, null);
                    invocation.setResponseContextInfo(response.getContextInfo());
                    return response.getResponse();
                    }
                    catch (Exception e)
                    {

                    }
                    finally
                    {
                    client.disconnect();
                    }
                    }

                    So, from the bold lines it seems to create a new Client Object for every remote call. This in turn creates a new SocketClientInvoker which then creates a new Socket pool rather than reusing an existing Socket pool.

                    Client.connect() does store the client invoker in the InvokerRegistry but the disconnect() then causes the Invoker to be removed from the InvokerRegistry (via destroyClientInvoker). This may explain why the Invoker (and its pool) is not being reused.

                    I am using JBoss 4.2.1GA (JBoss Remoting 2.2.1). FYI - The source above is from 2.2.0 since that is the latest downloadable version I could find.

                    Any help on a potential fix appreciated.

                    Thanks,
                    Darren.

                      • Actions
                    • 7. Re: JBREM-877: New Socket Connection is being Created for Ev
                      ron_sigal
                      ron_sigal Feb 7, 2008 7:42 PM (in response to ron_sigal)

                      Hi Darren,

                      Thanks for the big hint, which I had forgotten. I believe the problem is that a stateless EJB has no natural place for disconnecting a Remoting client, so they create a new one each time. I'll look into the connection pool issue.

                      -Ron

                        • Actions
                      • 8. Re: JBREM-877: New Socket Connection is being Created for Ev
                        darrengjones
                        darrengjones Feb 15, 2008 4:02 AM (in response to ron_sigal)

                        Hi Ron,

                        I've patched our application by simply commenting out the client.disconnect(). It seems to be safe for our particular application because there is no connectionValidator and no leasePinger involved - so the tidy up in client.disconnect() are essentially no-ops. And not disconnect()'ing leaves the invoker in the registry for the next time client.connect() is called.

                        I'm not suggesting this is a solid fix but it seems to work in our case. It may be useful to others until an official patch is available.

                        Thanks,

                        Darren.

                          • Actions
                        • 9. Re: JBREM-877: New Socket Connection is being Created for Ev
                          ron_sigal
                          ron_sigal Feb 19, 2008 4:38 AM (in response to ron_sigal)

                          Thanks, again, Darren.

                          I've proposed a solution on a thread in the EJB design forum - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=130264. Any feedback is welcome.

                            • Actions
                          • 10. Re: JBREM-877: New Socket Connection is being Created for Ev
                            ron_sigal
                            ron_sigal Feb 23, 2008 2:08 AM (in response to ron_sigal)

                            I've added a new feature to the Remoting 2.4 branch which allows the destruction of a client invoker to be delayed by some number of microseconds. The idea is to pick a delay long enough that the client invoker will still be around to be used by the next invocation on the same server. It could be useful in a situation where there are not a lot of concurrent invocations.

                            The delay is turned off by default, but it can be turned on by setting the parameter Client.INVOKER_DESTRUCTION_DELAY (actual value "invokerDestructionDelay") to a positive integer. The value can be set either by

                            * adding "invokerDestructionDelay=xxx" to the InvokerLocator, or
                            * adding the value with key "invokerDestructionDelay" to the configuration map passed to the Client constructor.

                            For EJB3 clients, the value should be set in the Connector MBean in the file .../server/default/deploy/ejb3.deployer/META-INF/jboss-service.xml.

                            Just to illustrate the value of this feature, I ran the following loop:

                             for (int i = 0; i < 10000; i++)
 {
 Client client = new Client(locator, config);
 client.connect();
 client.invoke("abc");
 client.disconnect();
 }


                            with and without a delay, and I got the following results: 

                             run delay=0 delay=5000
 -------------------------------
 1 46937 22657
 2 49265 24016
 3 47750 23375
---------------------------------
 avg: 47989 23349


                            The average speed up is a factor of 2.06. This was a simple in-jvm test, with nothing else going on, but it's still interesting.

                              • Actions
                            • 11. Re: JBREM-877: New Socket Connection is being Created for Ev
                              vink
                              vink Dec 19, 2008 6:06 AM (in response to ron_sigal)

                              I'm using SLSB on JBoss 4.2.2.GA with Remoting 2.4.0.SP1. On using invokerDestructionDelay in ejb3 connector configuration, my system goes slow & slow and run out of resources(native threads) in few minutes.

                              When removing this attribute, the system don't run out of resources, but many sockets are visible in TIME_WAIT state. As a result of this, my system run out of sockets & after a while I get sporadic disconnects.

                              Is there any real fix available for this critical problem? Any other version that can be used to avoid this situation.

                              Any suggestions.

                                • Actions
                              • 12. Re: JBREM-877: New Socket Connection is being Created for Ev
                                ron_sigal
                                ron_sigal Dec 25, 2008 1:26 AM (in response to ron_sigal)

                                 

                                "vink" wrote:

                                I'm using SLSB on JBoss 4.2.2.GA with Remoting 2.4.0.SP1. On using invokerDestructionDelay in ejb3 connector configuration, my system goes slow & slow and run out of resources(native threads) in few minutes.


                                Could you do a thread dump (if you haven't already) and tell me what kind of threads are getting created in such abundance?

                                  • Actions
                                • 13. Re: JBREM-877: New Socket Connection is being Created for Ev
                                  kiwi_clive
                                  kiwi_clive Dec 31, 2008 11:57 AM (in response to ron_sigal)

                                  Hi chaps,

                                  Apologies if this has all been asked before but I cannot seem to find the definitive answer I was looking for and this post seems as close as I can get.

                                  I am using jboss-4.2.2.GA, out of the box and using EJB3.

                                  I see the same issues that were originally posted by Kumarselvan (JBREM-877), namely I see several sockets being closed when a client is obtaining a proxy connection and want to keep the socket open for the next request.

                                  Having read your posts, I have edited
                                  /Applications/jboss-4.2.2.GA/server/default/deploy/ejb3.deployer/META-INF/jboss-server.xml so that the ejb3 connector now reads:


                                  <mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
 <depends>jboss.aop:service=AspectDeployer</depends>
 <attribute name="InvokerLocator">socket://${jboss.bind.address}:3873/?invokerDestructionDelay=5000</attribute>
 <attribute name="Configuration">
 <handlers>
 <handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
 </handlers>
 </attribute>
 </mbean>


                                  On restarting jboss, I see no difference in behaviour.

                                  I then downloaded jboss-remoting 2.4.0.GA and replaced the jboss-remoting.jar under server/default/lib, restarted the server but see no difference in behaviour.

                                  Am I being really stupid ? I hope so.

                                  How can I make the socket linger and be reused and cutdown on the socket creation/destruction overhead ?

                                  Any assistance will be gratefully received :-)

                                  Clive

                                    • Actions
                                  • 14. Re: JBREM-877: New Socket Connection is being Created for Ev
                                    vink
                                    vink Jan 2, 2009 4:40 AM (in response to ron_sigal)

                                    As suggested, please find below the thread dump; At that moment, my system was running with 32 clients. And, JBoss is running with default EJB3 connector settings.

                                    2009-01-02 14:09:37
                                    Full thread dump Java HotSpot(TM) Server VM (1.6.0_03-b05 mixed mode):

                                    "UIL2(SocketManager.MsgPool@be2377 client=150.158.73.73:8093)#87" daemon prio=6 tid=0x29be5400 nid=0x1334 in Object.wait() [0x2818f000..0x2818fc14]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x25f9d450> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
                                    at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.java:353)
                                    - locked <0x25f9d450> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkManager(2)-389" daemon prio=6 tid=0x28ce5c00 nid=0x16a4 in Object.wait() [0x3686f000..0x3686fb94]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0828aea0> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
                                    - locked <0x0828aea0> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkManager(2)-388" daemon prio=6 tid=0x29eb5400 nid=0xfd0 in Object.wait() [0x3672f000..0x3672fc14]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0828aea0> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
                                    - locked <0x0828aea0> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "JMSThread(3)-385" daemon prio=6 tid=0x28f52c00 nid=0x1444 in Object.wait() [0x33c0f000..0x33c0fc94]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x08a4a128> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
                                    - locked <0x08a4a128> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "JMSThread(3)-384" daemon prio=6 tid=0x29674c00 nid=0x153c in Object.wait() [0x2a93f000..0x2a93fd14]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x08a4a128> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
                                    - locked <0x08a4a128> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkManager(2)-387" daemon prio=6 tid=0x2978f400 nid=0x90c in Object.wait() [0x3681f000..0x3681fc94]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0828aea0> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
                                    - locked <0x0828aea0> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkManager(2)-386" daemon prio=6 tid=0x29cc2c00 nid=0xb84 in Object.wait() [0x3659f000..0x3659fd14]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0828aea0> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
                                    - locked <0x0828aea0> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "JMSThread(3)-383" daemon prio=6 tid=0x2bcbc400 nid=0x3c0 in Object.wait() [0x2f50f000..0x2f50fd94]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x08a4a128> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
                                    - locked <0x08a4a128> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "JMSThread(3)-382" daemon prio=6 tid=0x28d08c00 nid=0xc9c in Object.wait() [0x2b67f000..0x2b67fa14]
                                    java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x08a4a128> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
                                    - locked <0x08a4a128> (a java.lang.Object)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:736)
                                    at org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
                                    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:760)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "RMI TCP Connection(1733)-150.158.72.216" daemon prio=6 tid=0x2967a800 nid=0x14d4 runnable [0x375ef000..0x375efb94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x199e4e88> (a java.io.BufferedInputStream)
                                    at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - <0x199e5040> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

                                    "RMI TCP Connection(1732)-150.158.74.51" daemon prio=6 tid=0x28c4cc00 nid=0x16e4 runnable [0x374ff000..0x374ffc14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x199e9678> (a java.io.BufferedInputStream)
                                    at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - <0x199e9830> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

                                    "RMI TCP Connection(1731)-150.158.74.51" daemon prio=6 tid=0x2987cc00 nid=0x1220 runnable [0x372cf000..0x372cfc94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x199e9df8> (a java.io.BufferedInputStream)
                                    at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - <0x199e9fb0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

                                    "RMI TCP Connection(1729)-150.158.74.118" daemon prio=6 tid=0x28eb4400 nid=0x1624 runnable [0x3727f000..0x3727fd14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x199f2610> (a java.io.BufferedInputStream)
                                    at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - <0x199f27c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

                                    "RMI TCP Connection(1728)-150.158.74.51" daemon prio=6 tid=0x2c58e400 nid=0xbb8 runnable [0x3718f000..0x3718fd94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x199f2da0> (a java.io.BufferedInputStream)
                                    at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - <0x199f2f58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

                                    "RMI TCP Connection(1727)-150.158.73.101" daemon prio=6 tid=0x2bd75c00 nid=0xd7c runnable [0x3704f000..0x3704fa14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x199fc5d8> (a java.io.BufferedInputStream)
                                    at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - <0x199fc790> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

                                    "RMI TCP Connection(1726)-150.158.72.124" daemon prio=6 tid=0x2970a800 nid=0x1138 runnable [0x36a4f000..0x36a4fd94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x198fd150> (a java.io.BufferedInputStream)
                                    at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
                                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - <0x19851e30> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

                                    "Thread-539" daemon prio=6 tid=0x2c4e4c00 nid=0x1440 waiting for monitor entry [0x36b8f000..0x36b8fa14]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-538" daemon prio=6 tid=0x2b8d8400 nid=0xd20 runnable [0x37ecf000..0x37ecfb14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x10982fe8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-536" daemon prio=6 tid=0x2c82b800 nid=0x840 runnable [0x37e7f000..0x37e7fc14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x10985ab8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-535" daemon prio=6 tid=0x2959ac00 nid=0x1678 waiting for monitor entry [0x37d8f000..0x37d8fc94]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-533" daemon prio=6 tid=0x2952a000 nid=0xe0 runnable [0x3759f000..0x3759fd94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x1098b268> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-530" daemon prio=6 tid=0x28881c00 nid=0xd88 runnable [0x3663f000..0x3663fa94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0f7c28c8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-529" daemon prio=6 tid=0x2cce1c00 nid=0xd9c waiting for monitor entry [0x38a5f000..0x38a5fd14]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-527" daemon prio=6 tid=0x2c12bc00 nid=0x109c runnable [0x3722f000..0x3722fa14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0f50d288> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#90[150.158.73.73:3990]" prio=6 tid=0x28c98400 nid=0x1340 in Object.wait() [0x38f5f000..0x38f5fc94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0e914c78> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0e914c78> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-524" daemon prio=6 tid=0x29686800 nid=0x17e4 runnable [0x38faf000..0x38fafd14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0e9189d8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#89[150.158.73.73:4061]" prio=6 tid=0x29612800 nid=0x670 in Object.wait() [0x38e6f000..0x38e6fb94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0e8af778> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0e8af778> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#88[150.158.73.73:3968]" prio=6 tid=0x29865400 nid=0x16e8 in Object.wait() [0x38dcf000..0x38dcfc94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0e8aff10> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0e8aff10> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#87[150.158.73.73:4044]" prio=6 tid=0x2bd75800 nid=0xfdc in Object.wait() [0x3745f000..0x3745fd14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0e8b06a8> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0e8b06a8> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-520" daemon prio=6 tid=0x2c9c0400 nid=0xa24 waiting for monitor entry [0x3690f000..0x3690fc14]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#86[150.158.73.73:4087]" prio=6 tid=0x288bb400 nid=0x105c in Object.wait() [0x38d7f000..0x38d7fd94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0e8b0e40> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0e8b0e40> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-516" daemon prio=6 tid=0x28f50400 nid=0x288 runnable [0x368bf000..0x368bfa94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0e896738> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-511" daemon prio=6 tid=0x2be21800 nid=0x1748 runnable [0x3740f000..0x3740fd94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0e73fbd8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-510" daemon prio=6 tid=0x2cb99c00 nid=0xec4 runnable [0x3709f000..0x3709fa14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0e7425e8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-509" daemon prio=6 tid=0x28d8a400 nid=0x1620 runnable [0x3783f000..0x3783fb94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0e6adb80> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-508" daemon prio=6 tid=0x297d8c00 nid=0x134c waiting for monitor entry [0x3754f000..0x3754fc14]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-502" daemon prio=6 tid=0x28d08800 nid=0x13ac runnable [0x3882f000..0x3882fa94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0df577b0> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-501" daemon prio=6 tid=0x29813400 nid=0xe08 runnable [0x385ff000..0x385ffb94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0deeb5c8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-492" daemon prio=6 tid=0x2978dc00 nid=0xd54 runnable [0x2f83f000..0x2f83fd14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0dddbdf0> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-491" daemon prio=6 tid=0x2bceb400 nid=0x1458 runnable [0x3887f000..0x3887fa14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0dd65d18> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-490" daemon prio=6 tid=0x2d198c00 nid=0x37c waiting for monitor entry [0x38b4f000..0x38b4fa94]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-489" daemon prio=6 tid=0x289fcc00 nid=0x668 waiting for monitor entry [0x38a0f000..0x38a0fc94]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#85[150.158.73.73:4009]" prio=6 tid=0x281abc00 nid=0x143c in Object.wait() [0x38bef000..0x38befd14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0dc751e0> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0dc751e0> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#84[150.158.73.73:3913]" prio=6 tid=0x294a1400 nid=0x1618 in Object.wait() [0x386ef000..0x386efb94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0dc6dfc8> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0dc6dfc8> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#83[150.158.73.73:4071]" prio=6 tid=0x29607400 nid=0x1080 in Object.wait() [0x387df000..0x387dfa14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0dc724b0> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0dc724b0> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-484" daemon prio=6 tid=0x297e5800 nid=0x1180 runnable [0x38aaf000..0x38aafb14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0dc679e0> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#82[150.158.73.73:3921]" prio=6 tid=0x296e6400 nid=0xd10 in Object.wait() [0x389bf000..0x389bfb94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0dc60b80> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0dc60b80> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#81[150.158.73.73:3907]" prio=6 tid=0x29678400 nid=0x17fc in Object.wait() [0x366df000..0x366dfc94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0dc6ac68> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0dc6ac68> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-480" daemon prio=6 tid=0x28921c00 nid=0x12f8 runnable [0x388cf000..0x388cfd14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0dc32bc0> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-478" daemon prio=6 tid=0x28a2b800 nid=0x16a0 runnable [0x3864f000..0x3864fd94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0db29d60> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-477" daemon prio=6 tid=0x2c427400 nid=0x1490 waiting for monitor entry [0x3768f000..0x3768fa14]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-446" daemon prio=6 tid=0x29cb7800 nid=0x1494 runnable [0x3896f000..0x3896fc14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0d51e9a8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-427" daemon prio=6 tid=0x288bbc00 nid=0x1680 runnable [0x38cdf000..0x38cdfa14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0d21f9b8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-426" daemon prio=6 tid=0x294b2400 nid=0xf54 runnable [0x3713f000..0x3713fb14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0d1f1d10> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-423" daemon prio=6 tid=0x2b705800 nid=0x8c0 waiting for monitor entry [0x3677f000..0x3677fd94]
                                    java.lang.Thread.State: BLOCKED (on object monitor)
                                    at org.apache.log4j.Category.callAppenders(Category.java:201)
                                    - waiting to lock <0x07f967a0> (a org.apache.log4j.spi.RootLogger)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:82)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-420" daemon prio=6 tid=0x2970ac00 nid=0xccc runnable [0x3891f000..0x3891fd94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0d1be160> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-415" daemon prio=6 tid=0x28abe400 nid=0xbf8 runnable [0x3654f000..0x3654fd94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0d18c3a8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-413" daemon prio=6 tid=0x2b8d8800 nid=0x1290 runnable [0x38c3f000..0x38c3fb94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0d169f88> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#80[150.158.73.73:3914]" prio=6 tid=0x29709800 nid=0xfd4 in Object.wait() [0x38b9f000..0x38b9fd14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0d1627d0> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0d1627d0> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#79[150.158.73.73:3946]" prio=6 tid=0x2d3c8c00 nid=0x1100 in Object.wait() [0x2f9ef000..0x2f9efd94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0d162e18> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0d162e18> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#78[150.158.73.73:3927]" prio=6 tid=0x29782400 nid=0x17a4 in Object.wait() [0x36f0f000..0x36f0fc14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0d157510> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0d157510> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#77[150.158.73.73:3991]" prio=6 tid=0x294ea400 nid=0xe48 in Object.wait() [0x2ae8f000..0x2ae8fc14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0cdce670> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0cdce670> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-360" daemon prio=6 tid=0x289fc800 nid=0x1158 runnable [0x3819f000..0x3819fb94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0cdbf2d8> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#76[150.158.73.73:3903]" prio=6 tid=0x2d466c00 nid=0x16b8 in Object.wait() [0x367cf000..0x367cfa14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0cc54300> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0cc54300> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#75[150.158.73.73:3997]" prio=6 tid=0x2cf3dc00 nid=0xe28 in Object.wait() [0x2a84f000..0x2a84fa94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0cc51e00> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0cc51e00> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#74[150.158.73.73:3910]" prio=6 tid=0x2c709400 nid=0x1698 in Object.wait() [0x3736f000..0x3736fb94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c98d308> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c98d308> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#73[150.158.73.73:3964]" prio=6 tid=0x29d43400 nid=0x11d8 in Object.wait() [0x3731f000..0x3731fc14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c98d910> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c98d910> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#72[150.158.73.73:3898]" prio=6 tid=0x28f2f800 nid=0x1328 in Object.wait() [0x36bdf000..0x36bdfc94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c98df18> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c98df18> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#71[150.158.73.73:3950]" prio=6 tid=0x2c82bc00 nid=0x135c in Object.wait() [0x374af000..0x374afa94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c949790> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c949790> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#70[150.158.73.73:4068]" prio=6 tid=0x295d0c00 nid=0x14ec in Object.wait() [0x373bf000..0x373bfd14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c947c00> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c947c00> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#69[150.158.73.73:4085]" prio=6 tid=0x29614c00 nid=0xf58 in Object.wait() [0x371df000..0x371dfd94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c948358> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c948358> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#68[150.158.73.73:4027]" prio=6 tid=0x2954b400 nid=0x17c8 in Object.wait() [0x37c9f000..0x37c9fd94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c92fdd0> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c92fdd0> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#67[150.158.73.73:4067]" prio=6 tid=0x2958a400 nid=0x1204 in Object.wait() [0x37bff000..0x37bffa14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c92f318> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c92f318> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#66[150.158.73.73:4094]" prio=6 tid=0x28d36800 nid=0x6a0 in Object.wait() [0x370ef000..0x370efa94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c930488> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c930488> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-291" daemon prio=6 tid=0x2ca83400 nid=0x1770 runnable [0x364ff000..0x364ffb14]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0c923668> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#65[150.158.73.73:4038]" prio=6 tid=0x2bea7800 nid=0x1640 in Object.wait() [0x36faf000..0x36fafd14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c8eca20> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c8eca20> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#64[150.158.73.73:4034]" prio=6 tid=0x28466c00 nid=0x2a4 in Object.wait() [0x36f5f000..0x36f5fa94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c8ecfb8> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c8ecfb8> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#63[150.158.73.73:4023]" prio=6 tid=0x2d409c00 nid=0x1070 in Object.wait() [0x36ebf000..0x36ebfd94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c8ed600> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c8ed600> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#62[150.158.73.73:4074]" prio=6 tid=0x2d409400 nid=0x17cc in Object.wait() [0x36e6f000..0x36e6fb14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c8edbb8> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c8edbb8> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#61[150.158.73.73:4111]" prio=6 tid=0x2890fc00 nid=0x17ac in Object.wait() [0x36c7f000..0x36c7fa14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c8ee190> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c8ee190> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#60[150.158.73.73:4004]" prio=6 tid=0x297e6400 nid=0x10c4 in Object.wait() [0x36c2f000..0x36c2fb94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c8ee728> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c8ee728> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "File Reaper" daemon prio=10 tid=0x2bd1c800 nid=0xef0 in Object.wait() [0x365ef000..0x365efb14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c89fc50> (a java.lang.ref.ReferenceQueue$Lock)
                                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
                                    - locked <0x0c89fc50> (a java.lang.ref.ReferenceQueue$Lock)
                                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
                                    at org.apache.commons.io.FileCleaningTracker$Reaper.run(FileCleaningTracker.java:205)

                                    Locked ownable synchronizers:
                                    - None

                                    "Thread-275" daemon prio=6 tid=0x2a114800 nid=0xe1c runnable [0x2f8df000..0x2f8dfb94]
                                    java.lang.Thread.State: RUNNABLE
                                    at java.net.SocketInputStream.socketRead0(Native Method)
                                    at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                                    - locked <0x0c899cc0> (a java.io.BufferedInputStream)
                                    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2247)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2540)
                                    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2550)
                                    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
                                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                                    at org.apache.log4j.net.SocketNode.run(SocketNode.java:75)
                                    at com.barco.cms.logging.LogServer$ClientNode.run(Unknown Source)
                                    at java.lang.Thread.run(Thread.java:619)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#59[150.158.73.73:3890]" prio=6 tid=0x28f34800 nid=0x41c in Object.wait() [0x37f6f000..0x37f6fb14]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c2dcd20> (a org.jboss.remoting.transport.socket.ServerThread)
                                    at java.lang.Object.wait(Object.java:485)
                                    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)
                                    - locked <0x0c2dcd20> (a org.jboss.remoting.transport.socket.ServerThread)

                                    Locked ownable synchronizers:
                                    - None

                                    "DeviceManagement_CMS_EventProcessor" daemon prio=6 tid=0x28f24800 nid=0xe94 in Object.wait() [0x37f1f000..0x37f1fb94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c2d2f40> (a java.util.TaskQueue)
                                    at java.lang.Object.wait(Object.java:485)
                                    at java.util.TimerThread.mainLoop(Timer.java:483)
                                    - locked <0x0c2d2f40> (a java.util.TaskQueue)
                                    at java.util.TimerThread.run(Timer.java:462)

                                    Locked ownable synchronizers:
                                    - None

                                    "WorkerThread#58[150.158.73.73:3938]" prio=6 tid=0x28f00400 nid=0x16c0 in Object.wait() [0x37d3f000..0x37d3fa94]
                                    java.lang.Thread.State: WAITING (on object monitor)
                                    at java.lang.Object.wait(Native Method)
                                    - waiting on <0x0c2c13e8> (a org.jb

                                      • Actions
                                    1 2 Previous Next