uil2.SocketManager -> exiting on unexpected error

[jschanz]

HI folks,

i'm running here a installation of jboss 4.0.0. After a while the server runs into an ouf of memory error.

2004-11-16 11:08:45,074 DEBUG [org.jboss.mq.il.uil2.SocketManager] Begin ReadTask.run
2004-11-16 11:08:45,074 DEBUG [org.jboss.mq.il.uil2.SocketManager] Begin WriteTask.run
2004-11-16 11:08:45,074 DEBUG [org.jboss.mq.il.uil2.SocketManager] Created ObjectOutputStream
2004-11-16 11:08:45,074 DEBUG [org.jboss.mq.il.uil2.SocketManager] Created ObjectInputStream
2004-11-16 11:08:45,075 DEBUG [org.jboss.mq.il.uil2.SocketManager] Exiting on unexpected error in read task
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(PooledExecutor.java:520)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:880)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:296)
at java.lang.Thread.run(Thread.java:534)
2004-11-16 11:08:45,075 DEBUG [org.jboss.mq.il.uil2.ServerSocketManagerHandler] Exiting on unexpected error in read task
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.addThread(PooledExecutor.java:520)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:880)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:296)
at java.lang.Thread.run(Thread.java:534)
2004-11-16 11:08:45,075 DEBUG [org.jboss.mq.il.uil2.SocketManager] End ReadTask.run
2004-11-16 11:08:45,076 DEBUG [org.jboss.mq.il.uil2.SocketManager] End WriteTask.run
2004-11-16 11:12:57,870 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
2004-11-16 11:12:57,870 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
2004-11-16 11:20:27,880 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000
2004-11-16 11:20:27,880 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
2004-11-16 11:20:27,884 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
2004-11-16 11:20:27,887 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
2004-11-16 11:20:27,890 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
2004-11-16 11:20:27,893 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
2004-11-16 11:20:27,896 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}
2004-11-16 11:20:27,899 DEBUG [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] Using properties: {user=jboss2, password=--hidden--}


With JBoss 3.2.5 my client application couldn't reconnect to the jms queue. With 4.0.0 it seems that it works ...

The following parameters are set in run.conf:

MAX_FD="maximum"
JAVA_OPTS="-server -Xms128m -Xmx256m"

The base system is SuSE Linux Enterprise Server.

Is this problem a server or a client problem?

Thanks,
Jens

[kalyan120]

What do you mean by reconnecting to a queue?

Also, this is a problem with the server side JMS code, that you would have written. This could be a problem because of the proliferation of queue connections or queue sessions.

If you are doing something like below, for every message you send back, then it's a problem:

 System.out.println("Dispatching the result....");
 Context ctx = new InitialContext();
 QueueConnectionFactory factory = (QueueConnectionFactory)ctx.lookup("ConnectionFactory");
 QueueConnection connection = factory.createQueueConnection();
 QueueSession session = connection.createQueueSession(false,Session.AUTO_ACKNOWLEDGE);
 Message msg = session.createTextMessage("Reply from server....");

 Queue queue = (Queue)ctx.lookup(myQueueName);
 QueueSender sender = session.createSender(queue);

 System.out.println("Sending the message....");
 sender.send(msg);

Instead, once you have created the queue connection and queue session, if it's the same sender to the same queue, then cache them and re use them while sending the message to the client via the jms queue.

HTH,

Thanks,
Kalyan.

[jschanz]

Hi,

thanks for the answer. We don't use any self-written code on server side. I have only created one new durable topic:

<?xml version="1.0" encoding="UTF-8"?>


<depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager
<depends optional-attribute-name="SecurityManager">jboss.mq:service=SecurityManager










Several clients are sending messages to this topic. With 10 or 20 clients, everything works fine. If I increase the size of clients (e.g. to 50 or 100) than I'll run into this outOfMemory error.

Does anyone has an hint, how to configure more connections or threads?

The system is running about 165 jboss threads when the error occurs.

I don't know, how our client programmers have written the code. Is it possible, that they have forgotten to close a session or something else?

Thanks,
Jens

[kalyan120]

To my knowledge, the SocketManager spawns a thread for every request or message that comes in (via JMS). Once it returns back the result, it must remove the corresponding resources. I'm sure the code would be doing this as well.

How did you increase the no. of clients to 50? Is the client you are running, a test program or is it a production ready client? All I can think of here is that the client programs are proliferating the no. of sessions and connections, which is leading to the death of the server.

To figure out how many threads are currently hanging in the server, do something equivalent to "Kill -3" on unix, on your linux box. That might be of some help.

Thanks,
Kalyan

[jschanz]

Hi,

thanks for this reply.

We're running here several clients. It's a production environment. We add every day 5-20 new clients to the server. We have to add about 2000 clients which should communicate via JMS with the server.

I've asked our programmers. They open a connection to the server and send and receive messages.
The client is sending a message and gets a few minutes later a acknowledge message for this. Until 50 client we hadn't a problem with any threads or out of memory. Last week have added 20 clients at one day and got the next day the trouble with out of memory and couldn't create a new native thread. I think, there is an error in the client code (something like a not closed connection/session) but i can't prove this. So it looks for all, that the server is running a wrong configuration.

How many concurrent sessions could JBoss handle for JMS? Are there any hints, how to increase any caches or thread limiter?

Thanks,
Jens