-
1. Re: excess UIL2 threads
adrian.brock Jun 15, 2007 11:00 AM (in response to oglueck)Yes, but JBossMQ is not currently being developed by Redhat employees.
There is an old feature request to re-implement UIL2 as NIO, e.g. call it UIL3
The replacement JBoss Messaging already has such a feature.
FYI there was an old OIL2 that did NIO but it was buggy.
Besides which at the time there bugs in the Microsoft NIO support in the JDK and
the Linux implementation of NIO was slow.
I don't think the latter are real issues anymore?
So unless you are prepared to develop it yourself, it isn't likely to get done? -
2. Re: excess UIL2 threads
oglueck Jun 15, 2007 2:12 PM (in response to oglueck)Thanks for the information, Adrian. Much appreciated.
NIO: At the HttpClient project we have mixed experience with the latest NIO implementations under Linux. At least NIO in blocking mode it's worse than legacy IO. In non blocking mode however it is very good. Oleg has tested that extensively:
http://marc.info/?l=httpclient-commons-dev&m=112448628002581&w=2
http://marc.info/?l=httpclient-commons-dev&m=114328575030480&w=2
I was actually thinking of tweaking UIL2 to use NIO. But I thought I better ask here first if any work had been done on a similar thing already. I can imagine that the adventure with OIL2 failed: it's not an easy task. When I dare make a guess: does it deadlock? does a slow connection impact others?
Meanwhile we have been folding MDB code into less MDBs, as the UIL2 tweaking is errorprone and difficult. We are stuck with RedHat EL3 (kernel 2.4) at the moment, so many threads can really be a pain for the scheduler sometimes. -
3. Re: excess UIL2 threads
oglueck Jun 29, 2007 11:11 AM (in response to oglueck)A big problem is this code in SocketManager
// TODO: Check the validity of this config pool = new PooledExecutor(5); pool.setMinimumPoolSize(1); pool.setKeepAliveTime(1000 * 60); pool.runWhenBlocked(); String id = "SocketManager.MsgPool@"+ Integer.toHexString(System.identityHashCode(this)) + " client=" + ipAddress; pool.setThreadFactory(new UILThreadFactory(id));
This causes a thread-death every minute per connection and side. On our system we see around 5000 new threads every hour just because of that! And we can not change it, because it's hardcoded. This pool config makes the use of a thread pool absurd in terms of resource consumption.
Actually having this extra pool is a bit of an overkill in the first place, I think. It should be enough to have the Read task execute the message synchronously in general. On the server side this is an inject into a queue I guess, which is fast. On the client side this is passing the message to an MDB thread (or is it just passed to an instance? - I haven't checked), which is fast as well.
If this pooled is created, it should be fully configurable. -
4. Re: excess UIL2 threads
oglueck Jun 29, 2007 11:13 AM (in response to oglueck)If you want to monitor thread death, deploy this mbean and watch the log:
public class ThreadCoroner extends ServiceMBeanSupport implements ThreadCoronerMBean { private ThreadMXBean mbean; private Map<Long, ThreadInfo> last; private Timer timer; private int interval = 60; public int getInterval() { return interval; } public void setInterval(int interval) { this.interval = interval; } @Override protected void startService() throws Exception { mbean = ManagementFactory.getThreadMXBean(); timer = new Timer("ThreadCoroner-Timer", true); TimerTask task = new TimerTask() { @Override public void run() { logDeadThreads(); } }; timer.schedule(task, new Date(), interval * 1000L); } @Override protected void stopService() throws Exception { timer.cancel(); mbean = null; last = null; } private void logDeadThreads() { Map<Long, ThreadInfo> current = getAllThreads(); if (last != null) { Set<ThreadInfo> dead = new HashSet<ThreadInfo>(); for (Map.Entry<Long, ThreadInfo> entry : last.entrySet()) { if (!current.containsKey(entry.getKey())) { dead.add(entry.getValue()); } } for (ThreadInfo ti : dead) { log.debug(ti.getThreadName()); } } last = current; } private Map<Long, ThreadInfo> getAllThreads() { long myId = Thread.currentThread().getId(); long[] ids = mbean.getAllThreadIds(); ThreadInfo[] infos = mbean.getThreadInfo(ids); Map<Long, ThreadInfo> map = new HashMap<Long, ThreadInfo>(infos.length); for (ThreadInfo info : infos) { if (info.getThreadId() == myId) continue; map.put(new Long(info.getThreadId()), info); } return map; } }
-
5. Re: excess UIL2 threads
adrian.brock Jul 2, 2007 8:43 AM (in response to oglueck)We already know all this, see my earlier post.
-
6. Re: excess UIL2 threads
oglueck Jul 3, 2007 6:41 AM (in response to oglueck)I have removed the pool from SocketManager. All still works fine and we see no more excess thread creation.
-
7. Re: excess UIL2 threads
adrian.brock Jul 4, 2007 12:16 PM (in response to oglueck)Make sure you have this fix:
http://jira.jboss.com/jira/browse/JBAS-2476
I haven't had time to test UIL2 without the thread pool.
But problems like JBAS-2476 were why it was introduced. -
8. Re: excess UIL2 threads
oglueck Jul 4, 2007 12:26 PM (in response to oglueck)Ok, thanks. As the bug was fixed in 4.0.4RC1, we should be safe with 4.0.4GA.
So the pool was introduced as a fix against deadlocks. Okay, but this way the delivery order of messages is not known anymore (depends on scheduler). Couldn't that cause more trouble during bursts? -
9. Re: excess UIL2 threads
adrian.brock Jul 5, 2007 5:50 AM (in response to oglueck)"oglueck" wrote:
Ok, thanks. As the bug was fixed in 4.0.4RC1, we should be safe with 4.0.4GA.
So the pool was introduced as a fix against deadlocks. Okay, but this way the delivery order of messages is not known anymore (depends on scheduler). Couldn't that cause more trouble during bursts?
No the thread pool doesn't affect the JMS semantics, it is simply there
for when the protocol is doing more than one request/response, the protocol is bi-directional
with requests being issued from either side "at any time".
More importantly the handling of a request by one side may to lead a further
request to the other side. e.g.
"You just gave me a message for a receiver/session that I'm in the process of closing,
please NACK that message"
server -> DELIVER -> client -> NACK -> server -
10. Re: excess UIL2 threads
ksdeger Aug 2, 2007 2:34 PM (in response to oglueck)When you say you removed the pool from the SocketManager did you change the actual SocketManager class code?
I'm sure that this is a stupid question, and that the answer is yes, but I was hoping for a simple configuration answer. We're also seeing thousands of new threads being created, which can't be a good thing. -
11. Re: excess UIL2 threads
oglueck Aug 3, 2007 3:32 AM (in response to oglueck)ksdeger, yes, we have commented all pool related stuff from SocketManager. It's not possible to get rid of it through configuration.
-
12. Re: excess UIL2 threads
ksdeger Aug 3, 2007 11:23 AM (in response to oglueck)Rats! I guess I'll have to do the same thing.
Thanks for your answer.