Thanks for the information, Adrian. Much appreciated.

NIO: At the HttpClient project we have mixed experience with the latest NIO implementations under Linux. At least NIO in blocking mode it's worse than legacy IO. In non blocking mode however it is very good. Oleg has tested that extensively:
http://marc.info/?l=httpclient-commons-dev&m=112448628002581&w=2
http://marc.info/?l=httpclient-commons-dev&m=114328575030480&w=2

I was actually thinking of tweaking UIL2 to use NIO. But I thought I better ask here first if any work had been done on a similar thing already. I can imagine that the adventure with OIL2 failed: it's not an easy task. When I dare make a guess: does it deadlock? does a slow connection impact others?

Meanwhile we have been folding MDB code into less MDBs, as the UIL2 tweaking is errorprone and difficult. We are stuck with RedHat EL3 (kernel 2.4) at the moment, so many threads can really be a pain for the scheduler sometimes.

A big problem is this code in SocketManager

// TODO: Check the validity of this config
pool = new PooledExecutor(5);
pool.setMinimumPoolSize(1);
pool.setKeepAliveTime(1000 * 60);
pool.runWhenBlocked();
String id = "SocketManager.MsgPool@"+
 Integer.toHexString(System.identityHashCode(this))
 + " client=" + ipAddress;
pool.setThreadFactory(new UILThreadFactory(id));

If you want to monitor thread death, deploy this mbean and watch the log:

public class ThreadCoroner extends ServiceMBeanSupport implements ThreadCoronerMBean {
 private ThreadMXBean mbean;
 private Map<Long, ThreadInfo> last;
 private Timer timer;
 private int interval = 60;


 public int getInterval() {
 return interval;
 }
 public void setInterval(int interval) {
 this.interval = interval;
 }

 @Override
 protected void startService() throws Exception {
 mbean = ManagementFactory.getThreadMXBean();

 timer = new Timer("ThreadCoroner-Timer", true);
 TimerTask task = new TimerTask() {
 @Override
 public void run() {
 logDeadThreads();
 }
 };
 timer.schedule(task, new Date(), interval * 1000L);
 }

 @Override
 protected void stopService() throws Exception {
 timer.cancel();
 mbean = null;
 last = null;
 }

 private void logDeadThreads() {
 Map<Long, ThreadInfo> current = getAllThreads();
 if (last != null) {
 Set<ThreadInfo> dead = new HashSet<ThreadInfo>();
 for (Map.Entry<Long, ThreadInfo> entry : last.entrySet()) {
 if (!current.containsKey(entry.getKey())) {
 dead.add(entry.getValue());
 }
 }

 for (ThreadInfo ti : dead) {
 log.debug(ti.getThreadName());
 }
 }
 last = current;
 }

 private Map<Long, ThreadInfo> getAllThreads() {
 long myId = Thread.currentThread().getId();
 long[] ids = mbean.getAllThreadIds();
 ThreadInfo[] infos = mbean.getThreadInfo(ids);
 Map<Long, ThreadInfo> map = new HashMap<Long, ThreadInfo>(infos.length);
 for (ThreadInfo info : infos) {
 if (info.getThreadId() == myId) continue;
 map.put(new Long(info.getThreadId()), info);
 }
 return map;
 }
}

I have removed the pool from SocketManager. All still works fine and we see no more excess thread creation.

Ok, thanks. As the bug was fixed in 4.0.4RC1, we should be safe with 4.0.4GA.
So the pool was introduced as a fix against deadlocks. Okay, but this way the delivery order of messages is not known anymore (depends on scheduler). Couldn't that cause more trouble during bursts?

ksdeger, yes, we have commented all pool related stuff from SocketManager. It's not possible to get rid of it through configuration.