10 Replies Latest reply on Mar 5, 2010 9:40 PM by Dean Hiller

    db going down and up and seam doesn't recover?

    Dean Hiller Expert

      Not sure what layer this is at or if seam is giving the connection pool to hibernate or what here.  In Jboss AND tomcat, once the database is shutdown and a seam web page is hit, then I boot up the database, seam/hibernate never recover and pretty much are dead.  Not sure this is jboss connectoin pool, hibernate, or does seam deal with connections?


      thanks,
      Dean

        • 1. Re: db going down and up and seam doesn't recover?
          Arbi Sookazian Master

          What does dead mean?  Show stack trace if any.  I would guess this is a Hibernate/JBoss connection pool or related issue...

          • 2. Re: db going down and up and seam doesn't recover?
            Dean Hiller Expert

            got sidetracked with other production issues had to go work on....back on the database issue again.  Well, I did some more debugging and here is some more info....


            It correctly goes into SeamPhaseListener line 202 and redirects correctly to /web/error.xhtml but then it keeps hitting line 224 over and over with infinite redirect switching between restore view phase and render response phase.  I am still debugging this further though.


            I am assuming that entityMgr.getTransaction().begin and commit never hit the database if there was nothing done in between them?????  though I am seeing database errors duruing the infinite redirect even though I basically took ALL my code out of the error.xhtml page and just made it plain text.  ie. it is almost like seam is hitting the db for no reason here?


            I will post a bit more info when I find out more here.


            I tried redirecting view-id="/web/error.html" but seam changed that to error.xhtml so then I switched to url attribute on redirect but seam didn't seem to like that and redirected back to my original page when I used


            #{facesContext.externalContext.request.scheme}://#{facesContext.externalContext.request.serverName}:#{facesContext.externalContext.request.serverPort}#{facesContext.externalContext.requestContextPath}/web/errorY.html


            hmmm, I will see if seam likes relative urls next.

            • 3. Re: db going down and up and seam doesn't recover?
              Dean Hiller Expert

              shoooot....seam is calling hibernate begin transaction which then is calling setAutocommit(true) which results in a database connection.


              Is there a work around for this?  Can I turn transactions off in seam for error.xhtml which clearly should never be using the database and only should display Hey, there is a bug or db is down, or etc. etc from facesMessages


              Also, I know this is the wrong place to ask, but does hibernate really try to start a transaction on begin instead of waiting for some real activity????  That seems like a bad practice if I have ALOT of content where the db is not needed.  Also, why setAutocommit(true) instead of false....that seems odd.


              thanks for any insight as to how to work around and display a valid error page when our connection to the db is down!!!
              thanks,
              Dean

              • 4. Re: db going down and up and seam doesn't recover?
                Dean Hiller Expert

                Well, this is a crappy hack to the problem but it works for now.
                I basically modified web.xml to this....




                   <error-page>
                        <error-code>511</error-code>
                        <location>/web/errorY.html</location>
                    </error-page>
                



                and then inside the pages.xml file, I add an exception handler
                just does this...




                     <exception class="javax.persistence.PersistenceException">
                          <end-conversation/>
                          <http-error error-code="511" />
                    </exception>
                



                It works but it really sucks because errorY.html can't use the template.xhtml file that I have so it has all the same styling as all the other pages :(....anyone have any other ideas?


                I am thinking about making a jdbc proxy which does not actually start a transaction until it really starts making jdbc calls that affect data or that do queries...could just copy log4jdbc and erase all the logging code(or just keep it I guess).


                Dean



                • 5. Re: db going down and up and seam doesn't recover?
                  Nikolay Elenkov Master

                  JBoss data sources have the check-valid-connection-sql element, maybe you need to set this?
                  For Oracle something like this works:


                  <check-valid-connection-sql>select 0 from dual</check-valid-connection-sql>
                  


                  • 6. Re: db going down and up and seam doesn't recover?
                    Dean Hiller Expert

                    found the answer!!!


                    we already had check-valid-connection...that doesn't solve it.


                    I did find the answer though.  With oracle, you HAVE to set the hibernate autocommit property to false.  With postgres, you do not need this property as it is supposed to be default of false to begin with in hibernate.  After that, it almost works...it is refreshing the login page with transaction failed now and I think I just might have my pages.xml file exception handling wrong for this one now and need to change that and then it will all work fine.

                    • 7. Re: db going down and up and seam doesn't recover?
                      Dean Hiller Expert

                      dangit!!!!  I thought I had this working last night.  This morning, hibernate is no longer calling setAutocommit(true) on the oracle adapter which hit the database yesterday causing problems.  today, the getTransaction.commit() is causing some kind of call to the database even though getTransaction.begin does not and there is no data changed at all. 


                      this all works when I switch over to postgres database and take postgres down and hit a seam static page.  Switch back to oracle and it breaks on the commit like so.....


                      man, I am dying on this one.....


                      java.sql.SQLRecoverableException: Io exception: Socket read timed out
                           at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
                           at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
                           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
                           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)
                           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:521)
                           at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:595)
                           at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3432)
                           at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3438)
                           at net.sf.log4jdbc.ConnectionSpy.commit(ConnectionSpy.java:855)
                           at org.hibernate.transaction.JDBCTransaction.commitAndResetAutoCommit(JDBCTransaction.java:170)
                           at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:146)
                           at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:54)
                           at org.jboss.seam.transaction.EntityTransaction.commit(EntityTransaction.java:110)
                           at org.jboss.seam.jsf.SeamPhaseListener.commitOrRollback(SeamPhaseListener.java:614)
                           at org.jboss.seam.jsf.SeamPhaseListener.commitOrRollback(SeamPhaseListener.java:605)
                           at org.jboss.seam.jsf.SeamPhaseListener.handleTransactionsAfterPhase(SeamPhaseListener.java:343)
                           at org.jboss.seam.jsf.SeamPhaseListener.afterServletPhase(SeamPhaseListener.java:243)
                           at org.jboss.seam.jsf.SeamPhaseListener.afterPhase(SeamPhaseListener.java:194)
                           at com.sun.faces.lifecycle.Phase.handleAfterPhase(Phase.java:175)
                           at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:114)
                           at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139)
                           at javax.faces.webapp.FacesServlet.service(FacesServlet.java:266)
                           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
                           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
                           at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:177)
                           at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:267)
                           at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:380)
                           at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:507)
                           at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.jboss.seam.web.HotDeployFilter.doFilter(HotDeployFilter.java:53)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
                           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
                           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                           at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
                           at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
                           at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
                           at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                           at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                           at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
                           at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
                           at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
                           at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
                           at java.lang.Thread.run(Thread.java:619)
                      Caused by: oracle.net.ns.NetException: Socket read timed out
                           at oracle.net.ns.Packet.receive(Packet.java:249)
                           at oracle.net.ns.DataPacket.receive(DataPacket.java:92)
                           at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:172)
                           at oracle.net.ns.NetInputStream.read(NetInputStream.java:117)
                           at oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
                           at oracle.net.ns.NetInputStream.read(NetInputStream.java:77)
                           at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1034)
                           at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1010)
                           at oracle.jdbc.driver.T4C7Ocommoncall.receive(T4C7Ocommoncall.java:97)
                           at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:539)
                           ... 50 more
                      



                      • 8. Re: db going down and up and seam doesn't recover?
                        Dean Hiller Expert

                        ok, so it is a hibernate bug, not a seam bug.  This simple test case passes when using postgres and fails when using oracle 11...




                        EntityManagerFactory sf = Persistence.createEntityManagerFactory(
                                  "xcoreNonJta", props);
                        
                        //put a breakpoint here and 
                        //pull the internet connection after hitting the breakpoint
                        //then continue the test!!! postgres passes, oracle fails!!
                        
                        EntityManager mgr = sf.createEntityManager();
                        mgr.getTransaction().begin();
                        
                        //since we do nothing, neither commit nor begin will interact
                        //with the database...isn't hibernate kick ass except when
                        //used with oracle :(
                        
                        mgr.getTransaction().commit();
                        mgr.close();
                        sf.close();
                        



                        • 9. Re: db going down and up and seam doesn't recover?
                          Dean Hiller Expert

                          well, this was very annoying but we coded up a very simple JDBC proxy that in nearly every method of Connection, we set dirty=true and in the commit method we have


                          try {
                             if(dirty=true)
                                realConnection.commit();
                          } finally {
                             dirty = false;
                          }



                          NOTE: beware other Drivers registering before your driver.  JDBC uses the first registered driver and not the last one which tied me up for 30 minutes to figure out why someone else was registering before I was.


                          later and good luck,
                          Dean

                          • 10. Re: db going down and up and seam doesn't recover?
                            Dean Hiller Expert

                            Well, I am able now to reproduce this in full detail with various settings.  The test case that fails for EVERYBODY using seam/jboss no matter what the database is thus....





                            1. jboss / database is up
                            2. bring database down(kill -9 does the trick)
                            3. go to a page with forms(NOT login page, it behaves weird) (you should be able to get to this page if half the bug is fixed)
                            4. now submit the form
                            RESULT: no infinite redirect and goes to error.xthml properly (FAILs with 
                               a. oracle OR  
                               b. postgres AND check valid connection set
                            5. now bring the database back up
                            6. now go back to the form and submit it
                            RESULT: fails if check-valid-connection is not set up of course





                            The full bug reports are here
                            https://jira.jboss.org/jira/browse/JBAS-7788
                            and
                            https://jira.jboss.org/jira/browse/JBSEAM-4528


                            Please vote if you like.  I would really like to see check-valid-connection-sql to be called AFTER there is a failure on the connection(ie. why bother calling it as long as the connection keeps working).