-
1. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
inglesharad Jul 5, 2013 11:25 PM (in response to stewar)Hi Sumeeth,
I am getting the same error. Did you get any fix for this?
-
2. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
ahofi Jan 28, 2014 8:03 AM (in response to stewar)Hi,
I got the same error and I think it is because createSession() does not synchronize on the same monitor as getConnection().
The ClientSessionFactoryImpl.getConnection() method synchronizes on "connectionLock" writes to the field "connection" and returns.
The createSessionInternal method synchronizes on "createSessionLock", "failoverLock" and later on "inCreateSessionGuard". Never on the "connectionLock". As the Java memory model does not guarantee that a non-volatile field is updated when synchronizing on another monitor as during the write to that field, this can cause the IllegalStateException("Connection is null")
Here an excerpt of : http://www.cs.umd.edu/~pugh/java/memoryModel/jsr-133-faq.html#synchronization
"But there is more to synchronization than mutual exclusion. Synchronization ensures that memory writes by a thread before or during a synchronized block are made visible in a predictable manner to other threads which synchronize on the same monitor."
So either ClientSessionFactoryImpl.connection must be volatile or always synchronize on the "connectionLock" before accessing the connection.
What makes that problem worse: Everytime when the connection is set to null, a synchonization on "failoverLock" is done, but at the one single place, where it is set to a value that is not null, only on "connectionLock" a synchronized is done.
There is lots of code comment about synchronization during failover, but it doesn't match the code as createSession() never attempts to create a new connection but always sees when a connection is set to null.
As workaround you can just call ClientSessionFactory.getConnection() directly before calling createSession(). I don't think that this fully resolves the issue, but lowers the probability that it fails.
I used HornetQ 2.3.5
-
3. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
jbertram Jan 29, 2014 12:05 PM (in response to ahofi)Under what circumstance(s) does this error happen? Do you have some code you could share to reproduce this?
-
4. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
ahofi Jan 29, 2014 12:23 PM (in response to jbertram)That actually is really the problem. I just don't know.
I tried around with creating a test program that reproduces the problem, but failed. Unfortunately we use HornetQ in embedded mode and have a HUGE abstraction layer over it. So I just cannot provide a test program to give it to you.
I had no time yet to create a test program that reproduces the problem directly with HornetQ core.
That is why I just analyzed the code directly and analyzed if the synchronization makes sense.
My approach to reproduce the problem would be to simulate an unreliable network connection and create sessions concurrently. With some extra Thread.sleep() in ClientSessionFactoryImpl it could probably fail with the same problem. Our code is more vulnerable to the "create session" problem because we do create so many (session caching is not implemented yet)
What makes it even harder is that in our company, only the guys in Poland witnessed the problem, but never captured a logfile, or had HornetQ logging completely disabled. I am in Austria and have no access to their test servers.
-
5. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
clebert.suconic Jan 29, 2014 2:23 PM (in response to ahofi)Can you please double check you used 2.3.5? just making sure you didn't confused it with 2.2.5...
You're probably using maven right? 2.3.5 wasn't released as a public package on downloads.
-
6. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
clebert.suconic Jan 29, 2014 2:32 PM (in response to clebert.suconic)connectionLock doesn't exist on the 2.2 branch, so it is definitely a 2.3 branch.
Can you post at least the Stack Trace?
I have been working on a refactoring for multi-protocol and I had an issue with the method sharing a global variable (connection).. if I had connectoin returned by getConnection() and retry it properly you wouldn't have the NPE.
We can't have the createSessionFactory using the same monitor otherwise it would dead lock with reconnections and failover.
Can you provide the stack trace?
-
7. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
ahofi Jan 30, 2014 10:15 AM (in response to clebert.suconic)Yes, we use Maven.
Trying always to have newest version available. Actually I am really confused, why Maven Central has other versions available, as the HornetQ homepage.
Clebert Suconic schrieb:
connectionLock doesn't exist on the 2.2 branch, so it is definitely a 2.3 branch.
Can you post at least the Stack Trace?
The part of the stack trace that matters:
Caused by: HornetQException[errorType=INTERNAL_ERROR message=HQ119001: Failed to create session]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:928) [hornetq-core-client-2.3.5.Final.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:363) [hornetq-core-client-2.3.5.Final.jar:]
at com.cg.helix.mib.server.eventing.client.impl.EventContextFactory.createContext(EventContextFactory.java:219) [helix-mib-server-3.0.RC.128.jar:]
... 58 more
Caused by: java.lang.IllegalStateException: Connection is null
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:797) [hornetq-core-client-2.3.5.Final.jar:]
... 60 more
Generally it is the very first call to any HornetQ method in the thread.
I have been working on a refactoring for multi-protocol and I had an issue with the method sharing a global variable (connection).. if I had connectoin returned by getConnection() and retry it properly you wouldn't have the NPE.
We can't have the createSessionFactory using the same monitor otherwise it would dead lock with reconnections and failover.
Yeah, I already thought, that the other synchronization monitor is because of a possible (in this case very likely) deadlock.
Making connection volatile would also not solve the problem. I don't understand the reconnection logic well enough to provide a proper suggestion how to solve the problem. Therefore my partly-working workaround.
-
8. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
clebert.suconic Jan 30, 2014 12:23 PM (in response to ahofi)Can you provide us some pseudo code on what you're doing?
if you probably used one connection per thread you wouldn't have this issue?
-
9. Re: Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
ahofi Feb 4, 2014 11:51 AM (in response to clebert.suconic)OK, I'll try as good as I can (ignore the probably completely wrong use and naming of HornetQ methods)
First a few notes about our messaging architecture:
- We use hornetq-core directly and embed it into our framework.
- The API our developer program against is so extremely far away from any existing messaging framework, that a huge abstraction layer is needed.
- It was the API of an in-house developed eventing framework, that was replaced by a new HornetQ based one, by keeping the old API.
- Our framework is metadata centric. Meaning that there is an "EventDefinition" that defines if a message should be sent with a transaction, what timeout it has, if it should be persisted, ...
When sending the message the programmer just writes something like
"eventService.createEvent("/some/definition/name", payload).fire();" - Typically the application is deployed in several WAR modules on a JBOSS6 EAP server, but must also run outside of a servlet container or in Tomcat. The same framework is used for smaller installations and for bigger ones with thousands of concurrent users.
- When we got the problems the HornetQ server did run inside a JBoss6 EAP. But we don't use the one provided by JBoss, but start our own server.
- All clients run in the same VM as the server, but on different ClassLoaders. Therefore connecting over Netty.
class OurSessionFactory {
/*
* So first on startup a connection and session factory are created.
*/
void initMessagingClient() {
var connectionConfig = askFrameworkHowToConnectToHornetQServer();
this.serverLocator = HornetQClient.createServerLocatorWithoutHA(connectionConfig);
// copied from some other forum, or StackOverflow post, because reconnection did not work at all
serverLocator.setRetryInterval(4000);
serverLocator.setReconnectAttempts(10);
serverLocator.setConfirmationWindowSize(1024*1024);
serverLocator.setCallFailoverTimeout(23232432);
this.sessionFactory = serverLocator.createSessionFactory();
}
/*
* When a client makes a call to a service on a Module and the business logic has to fire an event,
* then this is typically the very first method in a new Thread that accesses any hornetq-client code.
*
* Producers are created directly when they are needed. Transacted sessions are committed together with
* Hibernate transaction (using the Spring transaction manager).
*
* Consumers are created on startup and have their own sessions.
*/
ClientSession fetchClientSession(EventMetadata metadata) {
// Already used ClientSession are held in the transaction
// When the transaction ends, then sessions are closed or returned to global cache.
if (transactionCache.containsSession(metadata)) {
return transactionCache.getSession(metadata);
}
//this is not implemented yet, so currently we create a new HornetQ session for every request
/*
* return sessionCache.fetchSession(metadata);
*/
return createClientSession(metadata)
}
ClientSession createClientSession(EventMetadata metadata) {
// there are more flags and properties in the metadata that affect the created sessions
// but currently only the transacted flag really matters
if (metadata.isTransacted()) {
this.sessionFactory.createTransactedSession();
} else {
this.sessionFactory.createSession();
}
}
}
-
10. Re: HornetQ issue : Caused by: javax.jms.JMSException: Failed to create session
manu29585 May 15, 2014 12:47 AM (in response to stewar)
This problem could be because of missing handling of connection timeouts.Based on the documentation @ Programming model of JMS, one connection can be used to create multiple sessions.
But, if there are no messages delivered after certain time, connection could be closed internally and when you tried to create new session, this could result in JMS exceptions with the posted stack trace.