1 2 Previous Next 19 Replies Latest reply on Nov 8, 2007 5:59 AM by kconner

    Exhausted JmsConnectionPool

    haagenhasle

      I have made a ServiceA that contains an action that calls another ServiceB (synchronously) using ServiceInvoker, before doing something else. By viewing my logging statements I can see that ServiceB is called, but it doesn't look like anything is actually transfered back to the action in ServiceA. Instead, I get alot of these:

      16:29:58,340 INFO [JmsConnectionPool] The connection pool was exhausted. Waiting 1 second before trying again..
      16:29:59,340 INFO [JmsConnectionPool] The connection pool was exhausted. Waiting 1 second before trying again..


      Obviously I'm doing something wrong, but I'm not sure what.. I have done similar things before without any errors. Can anyone help me?

      Regards, Haagen

        • 1. Re: Exhausted JmsConnectionPool
          burrsutter

          So you have a custom action that is using the ServiceInvoker to call Service B via
          resultMsg = deliverySync(myMessage,2000);

          are you sure that you are not getting a timeout or an exception from this line of code?

          does the error occur on the first message or after several hundred?

          What version of the ESB are you using?

          • 2. Re: Exhausted JmsConnectionPool
            haagenhasle

            I figured it out.. When I created ServiceB I copied from another service, and just edited the necessary fields. Unfortunately, I forgot to edit the name-attribute of the jms-listener, and this obviously caused JBossESB to malfunction. I changed the name, and now it's working.


            Regards, Haagen

            • 3. Re: Exhausted JmsConnectionPool
              haagenhasle

              I was a little quick to state that I figured it out... After I changed the name of the JMS-listener, it worked once. But then I changed somehting in my code and redeployed the application, and it goes back to malfunctioning.

              What does it actually mean that the JMS pool is exhausted?

              To answer your question, I use JBoss AS 4.2.0GA, JBossESB 4.2MR3, and I think your old JMS-implementation (I have a jbossmq.jar and write my mbeans in jbmq-queue-service.xml).

              Regards, Haagen

              • 4. Re: Exhausted JmsConnectionPool
                burrsutter

                We have had problems with this in the past and it was somewhat cleared up before the 4.2.GA release was made available. You might try upgrading to that version to see if it clears up your issue.

                Also, try to trim back your various jboss-esb.xml to something that still exhibits the problem but has the fewest moving parts then post that jboss-esb.xml here so we can review it.

                Burr

                • 5. Re: Exhausted JmsConnectionPool
                  haagenhasle

                  I'll try to upgrade to a later version and see if it fixes the problem.

                  I don't know if this is related to my problem, but when I shutdown the server I see the following statements in the log:

                  11:35:39,850 WARN [Connection] Connection failure:
                  org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
                   at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:72)
                   at org.jboss.mq.Connection.asynchFailure(Connection.java:423)
                   at org.jboss.mq.il.uil2.UILClientILService.asynchFailure(UILClientILService.java:174)
                   at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:439)
                   at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:371)
                   at java.lang.Thread.run(Thread.java:595)
                  Caused by: java.io.EOFException
                   at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2666)
                   at java.io.ObjectInputStream.readByte(ObjectInputStream.java:875)
                   at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:316)
                   ... 1 more
                  11:35:39,850 WARN [Connection] Exception listener ended abnormally:
                  java.lang.NullPointerException
                   at org.jboss.internal.soa.esb.rosetta.pooling.JmsConnectionPoolContainer.removePool(JmsConnectionPoolContainer.java:127)
                   at org.jboss.internal.soa.esb.rosetta.pooling.JmsConnectionPool.removeSessionPool(JmsConnectionPool.java:255)
                   at org.jboss.internal.soa.esb.rosetta.pooling.JmsConnectionPool$1.onException(JmsConnectionPool.java:134)
                   at org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1356)
                   at java.lang.Thread.run(Thread.java:595)


                  Thank you for your answer!

                  Regards, Haagen

                  • 6. Re: Exhausted JmsConnectionPool
                    haagenhasle

                    I still seem to have som problems with this, even after upgrading to the lastest jbossesb-server-version.

                    Are there any administration-tools I can use to inspect the JMS Connection Pool and the different JMS queues?

                    Regards, Haagen

                    • 7. Re: Exhausted JmsConnectionPool
                      haagenhasle

                      What is really interesting is that I sometimes get the errormessage when I *start* the server, and sometimes I don't. I don't have to run any services. I can get the errors even if I didn't run any services the last time the server was up..

                      Any thoughts anyone?

                      Regards, Haagen

                      • 8. Re: Exhausted JmsConnectionPool
                        haagenhasle

                        I get 9 of these at startup, and the connection pool messages are starting right after the second one:
                        [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.

                        How come this is happening? I'm not even using Quartz...

                        Regards, Haagen

                        • 9. Re: Exhausted JmsConnectionPool
                          haagenhasle

                          I think I'm beginning to understand why I'm getting the problems that I do.. In jboss-esb.xml, I define a lot of JMS-listeners, one for each Service. I guess each of them needs to hold on to a connection to the JMS, so at the moment that NUMBER_OF_SERVICES > NUMBER_OF_CONNECTIONS_IN_POOL, then I will start getting the exhausted-errormessages.

                          A colleague of mine found the following code in org.jboss.internal.soa.esb.rosetta.pooling.JmsConnectionPool.java:

                          private static final int DEFAULT_POOL_SIZE = 20;
                          private static final int DEFAULT_SLEEP = 30;
                          
                          private static int CONFIGURED_POOL_SIZE = DEFAULT_POOL_SIZE;
                          private static int CONFIGURED_SLEEP = DEFAULT_SLEEP;
                          
                          /** Maximum number of Sessions that will be created in this pool */
                          private int MAX_SESSIONS = DEFAULT_POOL_SIZE; //TODO Make this manageable


                          No wonder I haven't found out where to configure the poolsize.. :-)

                          Now, I can't help thinking that there must be something I'm doing wrong here.. I can't be the only one to have more than 20 Services in my esb?!
                          Am I right in my assumption that each Service (or each Listener) is grabbing one connection and holding on to it forever? Can I work around this in any way?
                          Is the JmsConnectionPool shared for the entire ESB, or can I split my deployment into several .esb-files, so that each of them will get 20 connections in a pool of its own?

                          And - are there any plans to fix the TODO in JMSConnectionPool?

                          Regards, Haagen

                          • 10. Re: Exhausted JmsConnectionPool
                            kconner

                            Hiya Haagen.

                            We have just released 4.2.1GA, which is based on AS4.2.1. Can you upgrade to this and try again?

                            The connection pool limit you see is still hardcoded in 4.2.1, we haven't addressed that yet, but the way in which these resources are managed has been changed. Each .esb artifact is now given a separate pool of connections for its own use, so the maximum of 20 is now per deployment.

                            As for having 20 services in an esb archive, this sounds a lot. Is it possible to split them up into separate .esb artifacts?

                            The other solution to your problem is to use JMS via JCA. The gateway/listener is driven by the JCA system and therefore the limit you have found does not apply. The easiest way to do this, unfortunately, still requires upgrading to ESB 4.2.1. It is simply a matter of changing the 'jms-provider' element to a 'jms-jca-provider' element.

                            Kev

                            • 11. Re: Exhausted JmsConnectionPool
                              haagenhasle

                              Thanks for a quick reply!

                              I'm already using the latest version, I downloaded it yesterday, right after I saw your post saying that it had been released. :)

                              But I'll try the JCA-solution, and if that doesn't work then yes, I guess we can split our project into smaller units.

                              Regards, Haagen

                              • 12. Re: Exhausted JmsConnectionPool
                                kconner

                                Sorry about the delays you have been experiencing, we have all been very busy with the 4.2.1 release.

                                Please let us know your thoughts on the new release.

                                Kev

                                • 13. Re: Exhausted JmsConnectionPool
                                  haagenhasle

                                  I tried the jms-jca-provider, and that seems to do the trick!

                                  The 4.2.1GA release has solved a couple of my problems, so I'm quite happy with it so far. :-) I'll let you know how I feel after some more work and testing. (It still seems like quite a hassle to change database, but I've got a good description in another thread here, so I'll give that a shot.)

                                  Can you explain to me the different versions you're using? Right now you released 4.2.1GA. In your Jira I read about 4.2.1IR, where one of the bugs I reported was going to be fixed. Does IR come before or after GA? I guess this is explained somewhere, but I haven't found it yet.. :)

                                  Regards, Haagen

                                  • 14. Re: Exhausted JmsConnectionPool
                                    kconner

                                    Hiya Haagen.

                                    The IR1 and IR2 tags were internal checkpoints between the 4.2GA and 4.2.1GA releases. They were used by the team to divide the development process into three different cycles, resulting in the final GA, and to allow external users to use a reasonably stable codebase.

                                    All of the IR1/IR2 JIRA issues have now been moved into 4.2.1 and these internal versions are no longer in JIRA.

                                    Kev

                                    1 2 Previous Next