7 Replies Latest reply on Jan 20, 2017 4:30 AM by Anup Dey

    JBoss Nested Throwable error

    Swati Jain Newbie

      Hi,

       

      I am getting the attached error in Production system

       

      WARN  [org.jboss.util.NestedThrowable] Duplicate throwable nesting of same base type: class org.jboss.mq.SpyJMSException is assignable from: class org.jboss.mq.SpyJMSException
      2017-01-18 14:25:42,107 ERROR [org.jboss.jms.asf.StdServerSession] failed to commit/rollback
      org.jboss.mq.SpyXAException: Resource manager error during commit; - nested throwable: (org.jboss.mq.SpyJMSException: Could not rollback tx: 80292; - nested throwable: (java.sql.SQLException: Invalid state, the Connection object is closed.))
      at org.jboss.mq.SpyXAException.getAsXAException(SpyXAException.java:72)
      at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:92)
      at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:317)
      at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:905)
      at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
      at org.jboss.mq.SpySession.run(SpySession.java:323)
      at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: org.jboss.mq.SpyJMSException: Could not rollback tx: 80292; - nested throwable: (java.sql.SQLException: Invalid state, the Connection object is closed.)
      at org.jboss.mq.pm.jdbc2.PersistenceManager.rollbackPersistentTx(PersistenceManager.java:988)
      at org.jboss.mq.pm.Tx.rollback(Tx.java:258)
      at org.jboss.mq.pm.TxManager.rollbackTx(TxManager.java:180)
      at org.jboss.mq.server.JMSDestinationManager.transact(JMSDestinationManager.java:449)
      at org.jboss.mq.server.JMSServerInterceptorSupport.transact(JMSServerInterceptorSupport.java:126)
      at org.jboss.mq.security.ServerSecurityInterceptor.transact(ServerSecurityInterceptor.java:197)
      at org.jboss.mq.server.TracingInterceptor.transact(TracingInterceptor.java:352)
      at org.jboss.mq.server.JMSServerInvoker.transact(JMSServerInvoker.java:132)
      at org.jboss.mq.il.jvm.JVMServerIL.transact(JVMServerIL.java:175)
      at org.jboss.mq.Connection.send(Connection.java:1110)
      at org.jboss.mq.SpyXAResourceManager.commit(SpyXAResourceManager.java:166)
      at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:88)
      ... 7 more
      Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
      at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1634)
      at net.sourceforge.jtds.jdbc.ConnectionJDBC2.prepareStatement(ConnectionJDBC2.java:2366)
      at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.doPrepareStatement(BaseWrapperManagedConnection.java:349)
      at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.prepareStatement(BaseWrapperManagedConnection.java:344)
      at org.jboss.resource.adapter.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:187)
      at org.jboss.mq.pm.jdbc2.PersistenceManager.removeMarkedMessages(PersistenceManager.java:880)
      at org.jboss.mq.pm.jdbc2.PersistenceManager.rollbackPersistentTx(PersistenceManager.java:972)
      ... 18 more

       

       

       

      After restarting the services, Issue is resolved.

       

      Can anyone advise what is the problem and what can be done to fix it ?

       

        • 1. Re: JBoss Nested Throwable error
          Anup Dey Expert

          Hi,

           

          -  Since the  "java.sql.SQLException: Invalid state, the Connection object is closed." errors  raised by the JTDS driver, they indicate that some event outside of the JBoss JVM has closed the connection.

          This could be a database server initiated timeout or some error condition on the database server. But the source for this error will likely be found outside of JBoss.

           

          It seems the connection is most likely closed by you by calling "close()" on it. There's possibility that there's a communication error between the driver and SQL Server (either network or TDS protocol issue) and that's causing the connection to close, JBoss won't close the connection. JBoss only reports the exception.

           

          - Try without the pool and if it works, then you know where the problem lies.

          - Try to retry the query, or re-connect your session.

           

          To solve this, you would need to enable connection validation mechanism if you have disabled, or not configured incase.

          Connection validation is recommended, and should fix the issue of connections failing on a database restart. Any connections in the pool when the database goes offline for any reason will become invalid. When an application tries to use one of these connections, you'll see the error that you posted. Enabling validation will help JBoss detect these invalid connections and create new ones.

           

          ~~~

          <validation>

            <validate-on-match>true</validate-on-match>

            <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.mssql.MSSQLValidConnectionChecker" />

            <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.mssql.MSSQLExceptionSorter" />

          </validation>

          ~~~

           

          - The connection is most likely closed by you by  calling "close()" on it.There's a possibility that there's a communication error between the driver and SQL Server (either network or TDS protocol issue) and that's causing the connection to close.

          • 2. Re: JBoss Nested Throwable error
            Swati Jain Newbie

            Hi Anup, We are using the below code for creating and close the session public synchronized void addReceived(String from, String to, String type,     String filename, boolean msgAck) throws Exception {   log.info("*****INSIDE addReceived :1 **********");   MessageState msgState = new MessageState();   msgState.setFilename(filename);   msgState.setDateCreated(null);   Session s = hbFactory.openSession();   Transaction tx = null;   try {     tx = s.beginTransaction();   List list = s.createCriteria(MessageState.class).add(       Example.create(msgState).enableLike(MatchMode.ANYWHERE)         .ignoreCase()).list();   log.debug("Found " + list.size() + " matching Message");     if (list.size() > 0) {     msgState = (MessageState) list.get(0);     if (msgAck) {       log.info("updating message state " + msgState.getFilename()         + "to a state " + MessageState.STATE_ACKED);       msgState.setState(MessageState.STATE_ACKED);       s.update(msgState);     }     } else {     log.info("saving message " + msgState.getFilename()       + "to a state " + MessageState.STATE_RECEIVED);     if (msgAck) {       msgState.setFrom(to);       msgState.setTo(from);       msgState.setType(type);     } else {       msgState.setFrom(from);       msgState.setTo(to);       msgState.setType(type);     }         msgState.setState(MessageState.STATE_RECEIVED);     msgState.setDateCreated(new Date());     log.debug("saving current date as " + new Date().toString());     log.debug("the system timezone is " + TimeZone.getDefault());     s.save(msgState);     }     tx.commit();   } catch (Exception e) {     e.printStackTrace();     log.info(e.getMessage());     if (tx != null)     tx.rollback();     throw e;   } finally {     s.close();   }   } Also as you have said that there can be another process that closing this session ,so I just want to tell one thing that this issue happen every Wednesday. On Wednesday there is one job runs on the server and it throws this error Eror :- "Access to the root\MSCluster namespace was denied because the namespace is marked with RequiresEncryption but the script or application attempted to connect to this namespace with an authentication level below Pkt_Privacy. Change the authentication level to Pkt_Privacy and run the script or application again" Can this be a problem for this issue? Thanks Swati

            • 3. Re: JBoss Nested Throwable error
              Anup Dey Expert

              Hi Swati,

               

              It may be possible.  If it weren't for the fact that this worked without this problem previously with another JDBC package, I'd say it was clearly a problem with the socket being closed by the server or network infrastructure. However, that's still something to check. It's pretty common for network infrastructure to disallow open, but inactive sockets (with the actual timeout varying based on settings). You might also check the keep-alive settings on your server and client IP stacks--if they're not exchanging some messages periodically, the socket is going to close anyway.If your transactions with the database are that infrequent, you might consider restructuring the program so that it didn't attempt to keep the connection open (as that will always be somewhat unreliable--a hiccup on the network will cause problems from time-to-time even if it's not a consistent problem, I'd guess).

               

              At any rate, check your network settings; that might be the source of the changed behavior (if your IT folks slipped in some extra protections without announcing it, for instance).

               

              It means that the socket connection is reset first (because of a timeout or network outage or something else out of jTDS' or Java's control). This triggers the exception thrown by jTDS and causes jTDS to forcefully close the JDBC Connection. After that all method calls on the JDBC Connection return the "Invalid state, the Connection object is closed" message.

               

              Nothing unusual here. Use a pool with a validation query and you should be all set.

               

              One part of the configuration that may be important in this regard is verifying that the database server side timeout value is *larger* than the timeout period in JBoss  (allowing JBoss to gracefully timeout connections rather than setting up the circumstance where connections in the pool may be externally invalidated).

               

              The validate-on-match configuration is recommended for production systems. However, I would also (in addition to validation) recommend making sure the database timeout period is larger than the JBoss timeout. This will reduce the cost of validation (when connections have been timed out) which will improve performance of connection requests.

               

              The connection is most likely closed by you (read "your pool implementation") calling close() on it.

               

              There's a very-very slight possibility that there's a communication error between the driver and SQL Server (either network or TDS protocol issue) and that's causing the connection to close, but I wouldn't bet on that. Try without the pool and if it works, then you know where the problem lies.

               

               

               

              • 4. Re: JBoss Nested Throwable error
                Swati Jain Newbie

                Thank you Anup. One information can there be some code problem or this error is usually thrown when there is socket or network issue?

                • 5. Re: JBoss Nested Throwable error
                  Anup Dey Expert

                  I can guarantee the issue has been occurred due to code problem or network issue. It might be possibility that there's a communication error between the driver and SQL Server (either network or TDS protocol issue) and that's causing the connection to close.

                   

                  it is likely that the database closed the connection because it was idle too long, JBoss gave it to the application, and then it failed because it was closed an not validated.

                   

                  You can also check your database connectivity is properly established between your application and database

                  Test the network with the usual means like ping, telnet, ssh... connection tests.

                  The fact that restarting JBoss might fix the connections

                   

                  To solve this, you would need to enable one of the two forms of connection validation if you have disabled, or set an idle timeout lower than the database's, so that the connections are not closed by the database.

                   

                  The first recommendation  is on the datasource configuration i.e Enabling validation as suggested before

                  Set <idle-timeout-minutes>30</idle-timeout-minutes> in your data source

                  • 6. Re: JBoss Nested Throwable error
                    Swati Jain Newbie

                    Ok thanks for your help..

                    • 7. Re: JBoss Nested Throwable error
                      Anup Dey Expert

                      Hi,

                       

                      Welcome swati, If my recommendation is helpful to you pls make it helpful.