12 Replies Latest reply on Aug 3, 2007 11:23 AM by Katherine Degerberg

    excess UIL2 threads

    Novice

      Hi,

      We noticed that the UIL2 architecture (SocketManager) uses many threads:
      - 2 (read/write) per connection on the server side
      - 2 (read/write) per connection on the client side

      So if you have like 20 (EJB3) MDBs with a pool size of each 10 each, then UIL2 will create 20 * 10 * 4 = 400 threads that do mostly sleep and perform only ping/pong every minute or so.

      Threads cost memory and constitute a scheduling problem. The current design limits the amount of MDBs in one JVM (or in extreme cases on one machine) unnecessarily.

      Have the JBoss team ever thought about reducing this massive waste of resources? Maybe with NIO selectors und NIO buffers Threads could be shared among connections. Likewise the MQ destinations could provide selectors instead of blocking calls to pop a message from the stack. I think it makes sense to have one thread per MDB on each client and server side, handling all of the connections. It should be possible to do this without giving up the non-blocking behaviour.

      Ortwin

        • 1. Re: excess UIL2 threads
          Adrian Brock Master

          Yes, but JBossMQ is not currently being developed by Redhat employees.
          There is an old feature request to re-implement UIL2 as NIO, e.g. call it UIL3

          The replacement JBoss Messaging already has such a feature.

          FYI there was an old OIL2 that did NIO but it was buggy.

          Besides which at the time there bugs in the Microsoft NIO support in the JDK and
          the Linux implementation of NIO was slow.
          I don't think the latter are real issues anymore?

          So unless you are prepared to develop it yourself, it isn't likely to get done?

          • 2. Re: excess UIL2 threads
            Novice

            Thanks for the information, Adrian. Much appreciated.

            NIO: At the HttpClient project we have mixed experience with the latest NIO implementations under Linux. At least NIO in blocking mode it's worse than legacy IO. In non blocking mode however it is very good. Oleg has tested that extensively:
            http://marc.info/?l=httpclient-commons-dev&m=112448628002581&w=2
            http://marc.info/?l=httpclient-commons-dev&m=114328575030480&w=2

            I was actually thinking of tweaking UIL2 to use NIO. But I thought I better ask here first if any work had been done on a similar thing already. I can imagine that the adventure with OIL2 failed: it's not an easy task. When I dare make a guess: does it deadlock? does a slow connection impact others?

            Meanwhile we have been folding MDB code into less MDBs, as the UIL2 tweaking is errorprone and difficult. We are stuck with RedHat EL3 (kernel 2.4) at the moment, so many threads can really be a pain for the scheduler sometimes.

            • 3. Re: excess UIL2 threads
              Novice

              A big problem is this code in SocketManager

              // TODO: Check the validity of this config
              pool = new PooledExecutor(5);
              pool.setMinimumPoolSize(1);
              pool.setKeepAliveTime(1000 * 60);
              pool.runWhenBlocked();
              String id = "SocketManager.MsgPool@"+
               Integer.toHexString(System.identityHashCode(this))
               + " client=" + ipAddress;
              pool.setThreadFactory(new UILThreadFactory(id));
              


              This causes a thread-death every minute per connection and side. On our system we see around 5000 new threads every hour just because of that! And we can not change it, because it's hardcoded. This pool config makes the use of a thread pool absurd in terms of resource consumption.

              Actually having this extra pool is a bit of an overkill in the first place, I think. It should be enough to have the Read task execute the message synchronously in general. On the server side this is an inject into a queue I guess, which is fast. On the client side this is passing the message to an MDB thread (or is it just passed to an instance? - I haven't checked), which is fast as well.

              If this pooled is created, it should be fully configurable.

              • 4. Re: excess UIL2 threads
                Novice

                If you want to monitor thread death, deploy this mbean and watch the log:

                public class ThreadCoroner extends ServiceMBeanSupport implements ThreadCoronerMBean {
                 private ThreadMXBean mbean;
                 private Map<Long, ThreadInfo> last;
                 private Timer timer;
                 private int interval = 60;
                
                
                 public int getInterval() {
                 return interval;
                 }
                 public void setInterval(int interval) {
                 this.interval = interval;
                 }
                
                 @Override
                 protected void startService() throws Exception {
                 mbean = ManagementFactory.getThreadMXBean();
                
                 timer = new Timer("ThreadCoroner-Timer", true);
                 TimerTask task = new TimerTask() {
                 @Override
                 public void run() {
                 logDeadThreads();
                 }
                 };
                 timer.schedule(task, new Date(), interval * 1000L);
                 }
                
                 @Override
                 protected void stopService() throws Exception {
                 timer.cancel();
                 mbean = null;
                 last = null;
                 }
                
                 private void logDeadThreads() {
                 Map<Long, ThreadInfo> current = getAllThreads();
                 if (last != null) {
                 Set<ThreadInfo> dead = new HashSet<ThreadInfo>();
                 for (Map.Entry<Long, ThreadInfo> entry : last.entrySet()) {
                 if (!current.containsKey(entry.getKey())) {
                 dead.add(entry.getValue());
                 }
                 }
                
                 for (ThreadInfo ti : dead) {
                 log.debug(ti.getThreadName());
                 }
                 }
                 last = current;
                 }
                
                 private Map<Long, ThreadInfo> getAllThreads() {
                 long myId = Thread.currentThread().getId();
                 long[] ids = mbean.getAllThreadIds();
                 ThreadInfo[] infos = mbean.getThreadInfo(ids);
                 Map<Long, ThreadInfo> map = new HashMap<Long, ThreadInfo>(infos.length);
                 for (ThreadInfo info : infos) {
                 if (info.getThreadId() == myId) continue;
                 map.put(new Long(info.getThreadId()), info);
                 }
                 return map;
                 }
                }


                • 5. Re: excess UIL2 threads
                  Adrian Brock Master

                  We already know all this, see my earlier post.

                  • 6. Re: excess UIL2 threads
                    Novice

                    I have removed the pool from SocketManager. All still works fine and we see no more excess thread creation.

                    • 7. Re: excess UIL2 threads
                      Adrian Brock Master

                      Make sure you have this fix:
                      http://jira.jboss.com/jira/browse/JBAS-2476

                      I haven't had time to test UIL2 without the thread pool.
                      But problems like JBAS-2476 were why it was introduced.

                      • 8. Re: excess UIL2 threads
                        Novice

                        Ok, thanks. As the bug was fixed in 4.0.4RC1, we should be safe with 4.0.4GA.
                        So the pool was introduced as a fix against deadlocks. Okay, but this way the delivery order of messages is not known anymore (depends on scheduler). Couldn't that cause more trouble during bursts?

                        • 9. Re: excess UIL2 threads
                          Adrian Brock Master

                           

                          "oglueck" wrote:
                          Ok, thanks. As the bug was fixed in 4.0.4RC1, we should be safe with 4.0.4GA.
                          So the pool was introduced as a fix against deadlocks. Okay, but this way the delivery order of messages is not known anymore (depends on scheduler). Couldn't that cause more trouble during bursts?


                          No the thread pool doesn't affect the JMS semantics, it is simply there
                          for when the protocol is doing more than one request/response, the protocol is bi-directional
                          with requests being issued from either side "at any time".

                          More importantly the handling of a request by one side may to lead a further
                          request to the other side. e.g.

                          "You just gave me a message for a receiver/session that I'm in the process of closing,
                          please NACK that message"

                          server -> DELIVER -> client -> NACK -> server

                          • 10. Re: excess UIL2 threads
                            Katherine Degerberg Newbie

                            When you say you removed the pool from the SocketManager did you change the actual SocketManager class code?

                            I'm sure that this is a stupid question, and that the answer is yes, but I was hoping for a simple configuration answer. We're also seeing thousands of new threads being created, which can't be a good thing.

                            • 11. Re: excess UIL2 threads
                              Novice

                              ksdeger, yes, we have commented all pool related stuff from SocketManager. It's not possible to get rid of it through configuration.

                              • 12. Re: excess UIL2 threads
                                Katherine Degerberg Newbie

                                Rats! I guess I'll have to do the same thing.

                                Thanks for your answer.