Issue in Production Seam-Hibernate - Spring

[avinashpuri]

Hi,

We are currently facing issues with our production website.


We use Facelets, Seam, Spring and Hibernate. Weblogic is the app server and Sun One is the web server. At a moderate to heavy load, we start seeing stuck threads and exceptions in the weblogic logs. Once we have stuck threads, we have to recycle weblogic to bring the site back up.

One of the first and the major exceptions which we saw in the logs is something trying to close an already closed resultset.

Once this exception is seeing in the logs, lot of collection loads cause the same issue and ultimately server has to be recycled.

We let Seam manage the hibernate session by using the following entry in components.xml

        <persistence:managed-hibernate-session name="hibernateSession"
                auto-create="true" session-factory="#{springSessionFactory}" />

The Spring/Hibernate config is as follows :-

<bean id="sessionFactory"
    class="org.jboss.seam.ioc.spring.SeamManagedSessionFactoryBean">
    <property name="sessionName" value="hibernateSession" />
  </bean>

  <!-- Set up the WLS Transaction Manager. Note: in order for Seam integration to work,
    you must use this ID in the Seam components.xml
  -->
  <bean id="transactionManager"
    class="org.springframework.transaction.jta.WebLogicJtaTransactionManager">
    <property name="userTransactionName"
      value="java:comp/UserTransaction" />
    <property name="transactionManagerName"
      value="javax.transaction.TransactionManager" />

  </bean>


<bean id="springSessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
        <property name="dataSource" ref="xxxxxx"/>
        <property name="hibernateProperties">
            <props>
                 <prop key="hibernate.dialect">org.hibernate.dialect.Oracle10gDialect</prop>
        <prop key="hibernate.show_sql">false</prop>
        <prop key="hibernate.use_outer_join">true</prop>
        <prop key="hibernate.format_sql">true</prop>
        <prop key="hibernate.order_updates">true</prop>
        <prop key="hibernate.use_sql_comments">true</prop>
        <prop key="hibernate.jdbc.fetch_size">50</prop>
        <prop key="hibernate.jdbc.batch_size">50</prop>
        <prop key="hibernate.default_batch_fetch_size">16</prop>
        <prop key="hibernate.cache.use_second_level_cache">false</prop>
        <prop key="hibernate.cache.use_query_cache">false</prop>
        <prop key="hibernate.generate_statistics">true</prop>
        <!-- These properties are required in order to enable JTA
             for Hibernate within a Spring context -->
        <prop key="hibernate.transaction.factory_class">org.hibernate.transaction.CMTTransactionFactory</prop>
        <prop key="hibernate.current_session_context_class">jta</prop>
        <prop key="hibernate.transaction.manager_lookup_class">org.hibernate.transaction.WeblogicTransactionManagerLookup</prop>
        <prop key="hibernate.connection.release_mode">auto</prop>
        <prop key="hibernate.connection.flush_before_completion">true</prop>
        <prop key="hibernate.connection.auto_close_session">true</prop>
            </props>
        </property>

We are not able to reproduce this in sub production environments.

We were wondering if anyone has seeing similar exceptions before and any recommendation you might have to address this.

Any help is highly appreciated.

Here is the complete stack trace

org.hibernate.exception.GenericJDBCException: could not initialize a collection batch:
            at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)

            at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)

            at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)

            at org.hibernate.loader.Loader.loadCollectionBatch(Loader.java:2039)

            at org.hibernate.loader.collection.BatchingCollectionInitializer.initialize(BatchingCollectionInitializer.java:47)

            at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:565)

            at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:63)

            at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1716)

            at org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:344)

            at org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:86)

            at org.hibernate.collection.PersistentBag.iterator(PersistentBag.java:249)

            at xxxx.getAttributeMap(xxx.java:232)

          at UIComponent.encodeBegin(UIComponent.java:74)

            at javax.faces.component.UIComponent.encodeAll(UIComponent.java:934)

            at javax.faces.component.UIComponent.encodeAll(UIComponent.java:942)

            at com.sun.facelets.FaceletViewHandler.renderView(FaceletViewHandler.java:592)

            at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:108)

            at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:189)

            at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:110)

            at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)

            at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139)

            at javax.faces.webapp.FacesServlet.service(FacesServlet.java:266)

            at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

            at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

            at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)

            at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)

            at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)

            at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:154)

            at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:260)

            at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:366)

            at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:493)

            at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)

            at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)

            at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)

            at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)

            at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:85)

            at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)

            at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)

            at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)

            at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)

            at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)

            at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)

            at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)

            at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:26)

            at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)

            at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3229)

            at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

            at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)

            at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2002)

            at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1908)

            at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1362)

            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

            at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

Caused by: java.sql.SQLException: Result set already closed

            at weblogic.jdbc.wrapper.ResultSet.checkResultSet(ResultSet.java:102)

            at weblogic.jdbc.wrapper.ResultSet.preInvocationHandler(ResultSet.java:58)

            at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.wasNull(Unknown Source)

            at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:164)

            at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:154)

            at org.hibernate.type.AbstractType.hydrate(AbstractType.java:81)

            at org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:2101)

            at org.hibernate.loader.Loader.loadFromResultSet(Loader.java:1380)

            at org.hibernate.loader.Loader.instanceNotYetLoaded(Loader.java:1308)

            at org.hibernate.loader.Loader.getRow(Loader.java:1206)

            at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:580)

            at org.hibernate.loader.Loader.doQuery(Loader.java:701)

            at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)

            at org.hibernate.loader.Loader.loadCollectionBatch(Loader.java:2032)

[valatharv]

Hi,

We faced sort of this error (not exactly) sometime back, also note application was not using seam, it used servlets, etc. and was on Weblogic.

Note, your case might very but just incase if it gives any light...

Are you also facing any Deadlock ?

a) Error said (it started happening all of a sudden ONLY on production.. :), we were never able to simulate on testing server)
"Http Request: /app/requestconf..", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> ..."
SQL Exception.;Error (-60) caught executing stored procedure:
ORA-00060: deadlock detected while waiting for resource> ...."

b) We asked our DBA to check the connections which were stuck and Oracle logs.

c) We investigated and it was caused by a proceduer. (note not always but sometimes on heavy load).
Also to some extent proceduer was not actually the root cause.

d) Mostly it can be performance problem caused by an automatic process that inserted a lot of data or heavy load.

e) Try checking the thread dump for Weblogic and check oracle logs.

Solution which fixed the issue.
Changes to the database (dropping some bitmap indices), things seem to be running reasonable well. 
http://www.akadia.com/services/ora_bitmapped_index.html

Thanks

[avinashpuri]

Thanks Val,

We also see stuck threads.

We are involving DBA's to seem, if they see something strange on the Oracle side.

Thanks for your help.

Regards