Could you take a look at the thread "Problem for Timer of BisocketServerInvoker" (https://community.jboss.org/thread/111400?tstart=0), in particular, the last note, by Andrea Ippolito, and see if it is relevant?
Initially some close() methods were missing but I added them and still it was the issue. (BiSockets getting collected). Then I realised for every message, InitialContext and ConnectionFactory are being instantiated. But all close() methods were called.
I changed the code to get InitialContext and ConnectionFactory once per queue. Now the BisocketServerInvokers are limited in number. But I still did get why BisocketServerInvokers are getting accumulated when I open 1 InitialContext and ConnectoryFactory per message, but still calling all close().
Have you tried profiling? I use jprofiler to find out what's going on in memory when I have a problem.