Based on what you've presented, I don't see any evidence that JBoss Messaging is contributing to the OOME.
For future reference, community help for legacy platforms (i.e. JBoss AS 5.x and JBoss Messaging) is typically very weak since development is always focused on the leading edge of the code-base. If you anticipate that you will keep an application installed for many years on the same components then I recommend you get a support subscription from Red Hat as they provide extended (up to 7 years) support for every platform they ship (in your case it would be EAP 5.1) which includes bug fixes for many of those years.
Than what would it cause for OOM and many instances of EDU.oswego.cs.dl.util.concurrent.ConcurrentHashMap$Entry.