10 Replies Latest reply on May 15, 2014 12:47 AM by manu29585

    HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session

    stewar

      Hi ,

      i have a JMS producer and client , deployed in Jboss AS 6.1.0 final. which is using the embedded HornetQ 2.2.5 as JMS Provider.

      After some time of execution generally a couple days and thousands of messages. the following exceptions start happening.

       

      Caused by: javax.jms.JMSException: Failed to create session

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:778)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:264)

              at org.hornetq.jms.client.HornetQConnection.createSessionInternal(HornetQConnection.java:501)

              at org.hornetq.jms.client.HornetQConnection.createSession(HornetQConnection.java:161)

              at com.avaya.spirit.datatransport.enterprise.util.JMSMessageDeliveryHelper.createSession(JMSMessageDeliveryHelper.java:60)

              at com.avaya.spirit.datatransport.enterprise.util.JMSMessageDeliveryHelper.sendMessage(JMSMessageDeliveryHelper.java:81)

              ... 105 more

      Caused by: HornetQException[errorCode=0 message=Failed to create session]

              ... 111 more

      Caused by: java.lang.IllegalStateException: Connection is null

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:660)

              ... 110 more

       

       

       

       

      Any Ideas?

        • 1. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
          inglesharad

          Hi Sumeeth,

          I am getting the same error. Did you get any fix for this?

          • 2. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
            ahofi

            Hi,

             

            I got the same error and I think it is because createSession() does not synchronize on the same monitor as getConnection().

             

            The ClientSessionFactoryImpl.getConnection() method synchronizes on "connectionLock" writes to the field "connection" and returns.

             

            The createSessionInternal method synchronizes on "createSessionLock", "failoverLock" and later on "inCreateSessionGuard". Never on the "connectionLock". As the Java memory model does not guarantee that a non-volatile field is updated when synchronizing on another monitor as during the write to that field, this can cause the IllegalStateException("Connection is null")

             

            Here an excerpt of : http://www.cs.umd.edu/~pugh/java/memoryModel/jsr-133-faq.html#synchronization

             

                 "But there is more to synchronization than mutual exclusion. Synchronization ensures that memory writes by a thread before or during a synchronized block are made visible in a predictable manner to other threads which synchronize on the same monitor."

             

            So either ClientSessionFactoryImpl.connection must be volatile or always synchronize on the "connectionLock" before accessing the connection.

             

            What makes that problem worse: Everytime when the connection is set to null, a synchonization on "failoverLock" is done, but at the one single place, where it is set to a value that is not null, only on "connectionLock" a synchronized is done.

             

            There is lots of code comment about synchronization during failover, but it doesn't match the code as createSession() never attempts to create a new connection but always sees when a connection is set to null.

             

            As workaround you can just call ClientSessionFactory.getConnection() directly before calling createSession(). I don't think that this fully resolves the issue, but lowers the probability that it fails.

             

            I used HornetQ 2.3.5

            • 3. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
              jbertram

              Under what circumstance(s) does this error happen?  Do you have some code you could share to reproduce this?

              • 4. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
                ahofi

                That actually is really the problem. I just don't know.

                 

                I tried around with creating a test program that reproduces the problem, but failed. Unfortunately we use HornetQ in embedded mode and have a HUGE abstraction layer over it. So I just cannot provide a test program to give it to you.

                 

                I had no time yet to create a test program that reproduces the problem directly with HornetQ core.

                 

                That is why I just analyzed the code directly and analyzed if the synchronization makes sense.

                 

                My approach to reproduce the problem would be to simulate an unreliable network connection and create sessions concurrently. With some extra Thread.sleep() in ClientSessionFactoryImpl it could probably fail with the same problem. Our code is more vulnerable to the "create session" problem because we do create so many (session caching is not implemented yet)

                 

                What makes it even harder is that in our company, only the guys in Poland witnessed the problem, but never captured a logfile, or had HornetQ logging completely disabled. I am in Austria and have no access to their test servers.

                • 5. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
                  clebert.suconic

                  Can you please double check you used 2.3.5? just making sure you didn't confused it with 2.2.5...

                   

                  You're probably using maven right? 2.3.5 wasn't released as a public package on downloads.

                  • 6. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
                    clebert.suconic

                    connectionLock doesn't exist on the 2.2 branch, so it is definitely a 2.3 branch.

                     

                     

                    Can you post at least the Stack Trace?

                     

                     

                    I have been working on a refactoring for multi-protocol and I had an issue with the method sharing a global variable  (connection).. if I had connectoin returned by getConnection() and retry it properly you wouldn't have the NPE.

                     

                    We can't have the createSessionFactory using the same monitor otherwise it would dead lock with reconnections and failover.

                     

                     

                     

                    Can you provide the stack trace?

                    • 7. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
                      ahofi

                      Yes, we use Maven.

                       

                      Trying always to have newest version available. Actually I am really confused, why Maven Central has other versions available, as the HornetQ homepage.

                       

                      Clebert Suconic schrieb:

                       

                      connectionLock doesn't exist on the 2.2 branch, so it is definitely a 2.3 branch.

                       

                       

                      Can you post at least the Stack Trace?

                      The part of the stack trace that matters:

                          Caused by: HornetQException[errorType=INTERNAL_ERROR message=HQ119001: Failed to create session]

                              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:928) [hornetq-core-client-2.3.5.Final.jar:]

                              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:363) [hornetq-core-client-2.3.5.Final.jar:]

                              at com.cg.helix.mib.server.eventing.client.impl.EventContextFactory.createContext(EventContextFactory.java:219) [helix-mib-server-3.0.RC.128.jar:]

                              ... 58 more

                          Caused by: java.lang.IllegalStateException: Connection is null

                              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:797) [hornetq-core-client-2.3.5.Final.jar:]

                              ... 60 more

                       

                      Generally it is the very first call to any HornetQ method in the thread.

                       

                      I have been working on a refactoring for multi-protocol and I had an issue with the method sharing a global variable  (connection).. if I had connectoin returned by getConnection() and retry it properly you wouldn't have the NPE.

                       

                      We can't have the createSessionFactory using the same monitor otherwise it would dead lock with reconnections and failover.

                       

                      Yeah, I already thought, that the other synchronization monitor is because of a possible (in this case very likely) deadlock.

                       

                      Making connection volatile would also not solve the problem. I don't understand the reconnection logic well enough to provide a proper suggestion how to solve the problem. Therefore my partly-working workaround.

                      • 8. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
                        clebert.suconic

                        Can you provide us some pseudo code on what you're doing?

                         

                         

                        if you probably used one connection per thread you wouldn't have this issue?

                        • 9. Re: Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
                          ahofi

                          OK, I'll try as good as I can (ignore the probably completely wrong use and naming of HornetQ methods)

                           

                          First a few notes about our messaging architecture:

                           

                          • We use hornetq-core directly and embed it into our framework.
                          • The API our developer program against is so extremely far away from any existing messaging framework, that a huge abstraction layer is needed.
                          • It was the API of an in-house developed eventing framework, that was replaced by a new HornetQ based one, by keeping the old API.
                          • Our framework is metadata centric. Meaning that there is an "EventDefinition" that defines if a message should be sent with a transaction, what timeout it has, if it should be persisted, ...
                            When sending the message the programmer just writes something like
                            "eventService.createEvent("/some/definition/name", payload).fire();"
                          • Typically the application is deployed in several WAR modules on a JBOSS6 EAP server, but must also run outside of a servlet container or in Tomcat. The same framework is used for smaller installations and for bigger ones with thousands of concurrent users.
                          • When we got the problems the HornetQ server did run inside a JBoss6 EAP. But we don't use the one provided by JBoss, but start our own server.
                          • All clients run in the same VM as the server, but on different ClassLoaders. Therefore connecting over Netty.

                           

                           

                          class OurSessionFactory {

                           

                          /*

                          * So first on startup a connection and session factory are created.

                          */

                          void initMessagingClient() {

                              var connectionConfig = askFrameworkHowToConnectToHornetQServer();

                              this.serverLocator = HornetQClient.createServerLocatorWithoutHA(connectionConfig);

                           

                              // copied from some other forum, or StackOverflow post, because reconnection did not work at all

                              serverLocator.setRetryInterval(4000);

                              serverLocator.setReconnectAttempts(10);

                              serverLocator.setConfirmationWindowSize(1024*1024);

                              serverLocator.setCallFailoverTimeout(23232432);

                           

                              this.sessionFactory = serverLocator.createSessionFactory();

                          }

                           

                          /*

                          * When a client makes a call to a service on a Module and the business logic has to fire an event,

                          * then this is typically the very first method in a new Thread that accesses any hornetq-client code.

                          *

                          * Producers are created directly when they are needed. Transacted sessions are committed together with

                          * Hibernate transaction (using the Spring transaction manager).

                          *

                          * Consumers are created on startup and have their own sessions.

                          */

                          ClientSession fetchClientSession(EventMetadata metadata) {

                              // Already used ClientSession are held in the transaction

                              // When the transaction ends, then sessions are closed or returned to global cache.

                              if (transactionCache.containsSession(metadata)) {

                                  return transactionCache.getSession(metadata);

                              }

                           

                              //this is not implemented yet, so currently we create a new HornetQ session for every request

                              /*

                              * return sessionCache.fetchSession(metadata);

                              */

                           

                              return createClientSession(metadata)

                          }

                           

                          ClientSession createClientSession(EventMetadata metadata) {

                              // there are more flags and properties in the metadata that affect the created sessions

                              // but currently only the transacted flag really matters

                              if (metadata.isTransacted()) {

                                  this.sessionFactory.createTransactedSession();

                              } else {

                                  this.sessionFactory.createSession();

                              }

                          }

                           

                          }

                          • 10. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
                            manu29585


                            This problem could be because of missing handling of connection timeouts.

                            Based on the documentation @ Programming model of JMS, one connection can be used to create multiple sessions.

                            But, if there are no messages delivered after certain time, connection could be closed internally and when you tried to create new session, this could result in JMS exceptions with the posted stack trace.