JBoss Nested Throwable error

[swati20]

Hi,

I am getting the attached error in Production system

WARN  [org.jboss.util.NestedThrowable] Duplicate throwable nesting of same base type: class org.jboss.mq.SpyJMSException is assignable from: class org.jboss.mq.SpyJMSException
2017-01-18 14:25:42,107 ERROR [org.jboss.jms.asf.StdServerSession] failed to commit/rollback
org.jboss.mq.SpyXAException: Resource manager error during commit; - nested throwable: (org.jboss.mq.SpyJMSException: Could not rollback tx: 80292; - nested throwable: (java.sql.SQLException: Invalid state, the Connection object is closed.))
at org.jboss.mq.SpyXAException.getAsXAException(SpyXAException.java:72)
at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:92)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:317)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:905)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
at org.jboss.mq.SpySession.run(SpySession.java:323)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.jboss.mq.SpyJMSException: Could not rollback tx: 80292; - nested throwable: (java.sql.SQLException: Invalid state, the Connection object is closed.)
at org.jboss.mq.pm.jdbc2.PersistenceManager.rollbackPersistentTx(PersistenceManager.java:988)
at org.jboss.mq.pm.Tx.rollback(Tx.java:258)
at org.jboss.mq.pm.TxManager.rollbackTx(TxManager.java:180)
at org.jboss.mq.server.JMSDestinationManager.transact(JMSDestinationManager.java:449)
at org.jboss.mq.server.JMSServerInterceptorSupport.transact(JMSServerInterceptorSupport.java:126)
at org.jboss.mq.security.ServerSecurityInterceptor.transact(ServerSecurityInterceptor.java:197)
at org.jboss.mq.server.TracingInterceptor.transact(TracingInterceptor.java:352)
at org.jboss.mq.server.JMSServerInvoker.transact(JMSServerInvoker.java:132)
at org.jboss.mq.il.jvm.JVMServerIL.transact(JVMServerIL.java:175)
at org.jboss.mq.Connection.send(Connection.java:1110)
at org.jboss.mq.SpyXAResourceManager.commit(SpyXAResourceManager.java:166)
at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:88)
... 7 more
Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1634)
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.prepareStatement(ConnectionJDBC2.java:2366)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.doPrepareStatement(BaseWrapperManagedConnection.java:349)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.prepareStatement(BaseWrapperManagedConnection.java:344)
at org.jboss.resource.adapter.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:187)
at org.jboss.mq.pm.jdbc2.PersistenceManager.removeMarkedMessages(PersistenceManager.java:880)
at org.jboss.mq.pm.jdbc2.PersistenceManager.rollbackPersistentTx(PersistenceManager.java:972)
... 18 more

After restarting the services, Issue is resolved.

Can anyone advise what is the problem and what can be done to fix it ?

[andey]

Hi,

-  Since the  "java.sql.SQLException: Invalid state, the Connection object is closed." errors  raised by the JTDS driver, they indicate that some event outside of the JBoss JVM has closed the connection.

This could be a database server initiated timeout or some error condition on the database server. But the source for this error will likely be found outside of JBoss.

It seems the connection is most likely closed by you by calling "close()" on it. There's possibility that there's a communication error between the driver and SQL Server (either network or TDS protocol issue) and that's causing the connection to close, JBoss won't close the connection. JBoss only reports the exception.

- Try without the pool and if it works, then you know where the problem lies.

- Try to retry the query, or re-connect your session.

To solve this, you would need to enable connection validation mechanism if you have disabled, or not configured incase.

Connection validation is recommended, and should fix the issue of connections failing on a database restart. Any connections in the pool when the database goes offline for any reason will become invalid. When an application tries to use one of these connections, you'll see the error that you posted. Enabling validation will help JBoss detect these invalid connections and create new ones.

~~~

<validation>

  <validate-on-match>true</validate-on-match>

  <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.mssql.MSSQLValidConnectionChecker" />

  <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mssql.MSSQLExceptionSorter" />

</validation>

~~~

- The connection is most likely closed by you by  calling "close()" on it.There's a possibility that there's a communication error between the driver and SQL Server (either network or TDS protocol issue) and that's causing the connection to close.

[swati20]

Hi Anup, We are using the below code for creating and close the session public synchronized void addReceived(String from, String to, String type,     String filename, boolean msgAck) throws Exception {   log.info("*****INSIDE addReceived :1 **********");   MessageState msgState = new MessageState();   msgState.setFilename(filename);   msgState.setDateCreated(null);   Session s = hbFactory.openSession();   Transaction tx = null;   try {     tx = s.beginTransaction();   List list = s.createCriteria(MessageState.class).add(       Example.create(msgState).enableLike(MatchMode.ANYWHERE)         .ignoreCase()).list();   log.debug("Found " + list.size() + " matching Message");     if (list.size() > 0) {     msgState = (MessageState) list.get(0);     if (msgAck) {       log.info("updating message state " + msgState.getFilename()         + "to a state " + MessageState.STATE_ACKED);       msgState.setState(MessageState.STATE_ACKED);       s.update(msgState);     }     } else {     log.info("saving message " + msgState.getFilename()       + "to a state " + MessageState.STATE_RECEIVED);     if (msgAck) {       msgState.setFrom(to);       msgState.setTo(from);       msgState.setType(type);     } else {       msgState.setFrom(from);       msgState.setTo(to);       msgState.setType(type);     }         msgState.setState(MessageState.STATE_RECEIVED);     msgState.setDateCreated(new Date());     log.debug("saving current date as " + new Date().toString());     log.debug("the system timezone is " + TimeZone.getDefault());     s.save(msgState);     }     tx.commit();   } catch (Exception e) {     e.printStackTrace();     log.info(e.getMessage());     if (tx != null)     tx.rollback();     throw e;   } finally {     s.close();   }   } Also as you have said that there can be another process that closing this session ,so I just want to tell one thing that this issue happen every Wednesday. On Wednesday there is one job runs on the server and it throws this error Eror :- "Access to the root\MSCluster namespace was denied because the namespace is marked with RequiresEncryption but the script or application attempted to connect to this namespace with an authentication level below Pkt_Privacy. Change the authentication level to Pkt_Privacy and run the script or application again" Can this be a problem for this issue? Thanks Swati

[andey]

Hi Swati,

It may be possible.  If it weren't for the fact that this worked without this problem previously with another JDBC package, I'd say it was clearly a problem with the socket being closed by the server or network infrastructure. However, that's still something to check. It's pretty common for network infrastructure to disallow open, but inactive sockets (with the actual timeout varying based on settings). You might also check the keep-alive settings on your server and client IP stacks--if they're not exchanging some messages periodically, the socket is going to close anyway.If your transactions with the database are that infrequent, you might consider restructuring the program so that it didn't attempt to keep the connection open (as that will always be somewhat unreliable--a hiccup on the network will cause problems from time-to-time even if it's not a consistent problem, I'd guess).

At any rate, check your network settings; that might be the source of the changed behavior (if your IT folks slipped in some extra protections without announcing it, for instance).

It means that the socket connection is reset first (because of a timeout or network outage or something else out of jTDS' or Java's control). This triggers the exception thrown by jTDS and causes jTDS to forcefully close the JDBC Connection. After that all method calls on the JDBC Connection return the "Invalid state, the Connection object is closed" message.

Nothing unusual here. Use a pool with a validation query and you should be all set.

One part of the configuration that may be important in this regard is verifying that the database server side timeout value is *larger* than the timeout period in JBoss  (allowing JBoss to gracefully timeout connections rather than setting up the circumstance where connections in the pool may be externally invalidated).

The validate-on-match configuration is recommended for production systems. However, I would also (in addition to validation) recommend making sure the database timeout period is larger than the JBoss timeout. This will reduce the cost of validation (when connections have been timed out) which will improve performance of connection requests.

The connection is most likely closed by you (read "your pool implementation") calling close() on it.

There's a very-very slight possibility that there's a communication error between the driver and SQL Server (either network or TDS protocol issue) and that's causing the connection to close, but I wouldn't bet on that. Try without the pool and if it works, then you know where the problem lies.

[swati20]

Thank you Anup. One information can there be some code problem or this error is usually thrown when there is socket or network issue?

[andey]

I can guarantee the issue has been occurred due to code problem or network issue. It might be possibility that there's a communication error between the driver and SQL Server (either network or TDS protocol issue) and that's causing the connection to close.

it is likely that the database closed the connection because it was idle too long, JBoss gave it to the application, and then it failed because it was closed an not validated.

You can also check your database connectivity is properly established between your application and database

Test the network with the usual means like ping, telnet, ssh... connection tests.

The fact that restarting JBoss might fix the connections

To solve this, you would need to enable one of the two forms of connection validation if you have disabled, or set an idle timeout lower than the database's, so that the connections are not closed by the database.

The first recommendation  is on the datasource configuration i.e Enabling validation as suggested before

Set <idle-timeout-minutes>30</idle-timeout-minutes> in your data source

[swati20]

Ok thanks for your help..

[andey]

Hi,

Welcome swati, If my recommendation is helpful to you pls make it helpful.