4 Replies Latest reply on Nov 18, 2004 5:35 AM by Jens Schanz

    uil2.SocketManager -> exiting on unexpected error

    Jens Schanz Newbie

      HI folks,

      i'm running here a installation of jboss 4.0.0. After a while the server runs into an ouf of memory error.

      2004-11-16 11:08:45,074 DEBUG [org.jboss.mq.il.uil2.SocketManager] Begin ReadTask.run
      2004-11-16 11:08:45,074 DEBUG [org.jboss.mq.il.uil2.SocketManager] Begin WriteTask.run
      2004-11-16 11:08:45,074 DEBUG [org.jboss.mq.il.uil2.SocketManager] Created ObjectOutputStream
      2004-11-16 11:08:45,074 DEBUG [org.jboss.mq.il.uil2.SocketManager] Created ObjectInputStream
      2004-11-16 11:08:45,075 DEBUG [org.jboss.mq.il.uil2.SocketManager] Exiting on unexpected error in read task
      java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start(Native Method)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(PooledExecutor.java:520)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:880)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:296)
      at java.lang.Thread.run(Thread.java:534)
      2004-11-16 11:08:45,075 DEBUG [org.jboss.mq.il.uil2.ServerSocketManagerHandler] Exiting on unexpected error in read task
      java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start(Native Method)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(PooledExecutor.java:520)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:880)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:296)
      at java.lang.Thread.run(Thread.java:534)
      2004-11-16 11:08:45,075 DEBUG [org.jboss.mq.il.uil2.SocketManager] End ReadTask.run
      2004-11-16 11:08:45,076 DEBUG [org.jboss.mq.il.uil2.SocketManager] End WriteTask.run
      2004-11-16 11:12:57,870 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
      2004-11-16 11:12:57,870 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
      2004-11-16 11:20:27,880 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
      2004-11-16 11:20:27,880 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
      2004-11-16 11:20:27,884 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
      2004-11-16 11:20:27,887 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
      2004-11-16 11:20:27,890 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
      2004-11-16 11:20:27,893 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
      2004-11-16 11:20:27,896 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
      2004-11-16 11:20:27,899 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}


      With JBoss 3.2.5 my client application couldn't reconnect to the jms queue. With 4.0.0 it seems that it works ...

      The following parameters are set in run.conf:

      MAX_FD="maximum"
      JAVA_OPTS="-server -Xms128m -Xmx256m"

      The base system is SuSE Linux Enterprise Server.

      Is this problem a server or a client problem?

      Thanks,
      Jens

        • 1. Re: uil2.SocketManager -> exiting on unexpected error
          Kalyan Newbie

          What do you mean by reconnecting to a queue?

          Also, this is a problem with the server side JMS code, that you would have written. This could be a problem because of the proliferation of queue connections or queue sessions.

          If you are doing something like below, for every message you send back, then it's a problem:

           System.out.println("Dispatching the result....");
           Context ctx = new InitialContext();
           QueueConnectionFactory factory = (QueueConnectionFactory)ctx.lookup("ConnectionFactory");
           QueueConnection connection = factory.createQueueConnection();
           QueueSession session = connection.createQueueSession(false,Session.AUTO_ACKNOWLEDGE);
           Message msg = session.createTextMessage("Reply from server....");
          
           Queue queue = (Queue)ctx.lookup(myQueueName);
           QueueSender sender = session.createSender(queue);
          
           System.out.println("Sending the message....");
           sender.send(msg);


          Instead, once you have created the queue connection and queue session, if it's the same sender to the same queue, then cache them and re use them while sending the message to the client via the jms queue.

          HTH,

          Thanks,
          Kalyan.

          • 2. Re: uil2.SocketManager -> exiting on unexpected error
            Jens Schanz Newbie

            Hi,

            thanks for the answer. We don't use any self-written code on server side. I have only created one new durable topic:

            <?xml version="1.0" encoding="UTF-8"?>


            <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager
            <depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager










            Several clients are sending messages to this topic. With 10 or 20 clients, everything works fine. If I increase the size of clients (e.g. to 50 or 100) than I'll run into this outOfMemory error.

            Does anyone has an hint, how to configure more connections or threads?

            The system is running about 165 jboss threads when the error occurs.

            I don't know, how our client programmers have written the code. Is it possible, that they have forgotten to close a session or something else?

            Thanks,
            Jens

            • 3. Re: uil2.SocketManager -> exiting on unexpected error
              Kalyan Newbie

              To my knowledge, the SocketManager spawns a thread for every request or message that comes in (via JMS). Once it returns back the result, it must remove the corresponding resources. I'm sure the code would be doing this as well.

              How did you increase the no. of clients to 50? Is the client you are running, a test program or is it a production ready client? All I can think of here is that the client programs are proliferating the no. of sessions and connections, which is leading to the death of the server.

              To figure out how many threads are currently hanging in the server, do something equivalent to "Kill -3" on unix, on your linux box. That might be of some help.

              Thanks,
              Kalyan

              • 4. Re: uil2.SocketManager -> exiting on unexpected error
                Jens Schanz Newbie

                Hi,

                thanks for this reply.

                We're running here several clients. It's a production environment. We add every day 5-20 new clients to the server. We have to add about 2000 clients which should communicate via JMS with the server.

                I've asked our programmers. They open a connection to the server and send and receive messages.
                The client is sending a message and gets a few minutes later a acknowledge message for this. Until 50 client we hadn't a problem with any threads or out of memory. Last week have added 20 clients at one day and got the next day the trouble with out of memory and couldn't create a new native thread. I think, there is an error in the client code (something like a not closed connection/session) but i can't prove this. So it looks for all, that the server is running a wrong configuration.

                How many concurrent sessions could JBoss handle for JMS? Are there any hints, how to increase any caches or thread limiter?

                Thanks,
                Jens