Good afternoon!

Environment: Fuse MB 5.3.0.(4|5) with a network of brokers with four nodes, all processes running on multicore machines using java1.5.0_21 64 bit on Linux, brokers are under high usage.

(To understand the CPU usage given below: having a machine with n cores, c is a number between 1 and n).

We experience this on a regular basis: on jconsole you can see after a (short) while that (at least) one of the nodes has a CPU usage around c/n and remains there with the usual going up for coping with messages. I will never go down again.

This always affects an "ActiveMQ Transport" thread which will remain in an infinite loop and therefore take over a CPU core completely (and is therefore also not available for regular work anymore). And on the thread dump, one can always see that the thread is working on a HashMap instance:

"ActiveMQ Transport: tcp:///censored:58633" daemon prio=1 tid=0x000000004b64c770 nid=0x400 runnable \[0x00000000455ce000..0x00000000455ced10\]         at java.util.HashMap.putForCreate(HashMap.java:470)         at java.util.HashMap.putAllForCreate(HashMap.java:482)         at java.util.HashMap.<init>(HashMap.java:219)         at org.apache.activemq.command.Message.copy(Message.java:120)         at org.apache.activemq.command.ActiveMQMessage.copy(ActiveMQMessage.java:64)         at org.apache.activemq.command.ActiveMQObjectMessage.copy(ActiveMQObjectMessage.java:81)         at org.apache.activemq.command.ActiveMQObjectMessage.copy(ActiveMQObjectMessage.java:75)         at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:88)         at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)         at org.apache.activemq.broker.util.LoggingBrokerPlugin.send(LoggingBrokerPlugin.java:276)         at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)         at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)         at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)         at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)         at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)         at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)         at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)         at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)         at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)         - locked <0x00002aab0bde5910> (a org.apache.activemq.transport.InactivityMonitor$1)         at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)         at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)         at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)         at java.lang.Thread.run(Thread.java:595)

According to well-informed sources, this is an issue very often observed on code that doesn't care enough about the fact that HashMap is not thread safe and that runs well on singlecore machines for ages, but crashes soon on multicore machines (and the more cores, the sooner it crashes).

Has anyone similar experiences with this? Can someone open a JIRA ticket on this?

Thanks in advance for any input on this.

Have a good 2010!