Problems with transactions / database connection
Hi,
We have a fairly heavily loaded portal installation (>1 login / sec and pageloads to match). Under load we are getting the error as displayed in the stacktrace below.
Here the error has something to do with portal objects, but we get the same errors with cms operations and user opperations.
When you get this error page it eventually goes away (after eg. reloading the page a few times) and other portal pages may work fine in the meanwhile.
Any ideas?
Database config:
<datasources> <local-tx-datasource> <jndi-name>PortalDS</jndi-name> <connection-url>jdbc:mysql://dbserver:3306/db?autoReconnect=true</connection-url> <driver-class>com.mysql.jdbc.Driver</driver-class> <user-name>root</user-name> <password>pass</password> </local-tx-datasource> </datasources>
HTTP Status 500 - type Exception report message description The server encountered an internal error () that prevented it from fulfilling this request. exception javax.servlet.ServletException: javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: a1404ac:db35:47df9a63:137b45 status: ActionStatus.ABORT_ONLY > org.jboss.portal.server.servlet.PortalServlet.service(PortalServlet.java:406) javax.servlet.http.HttpServlet.service(HttpServlet.java:803) org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) root cause javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: a1404ac:db35:47df9a63:137b45 status: ActionStatus.ABORT_ONLY > org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.checkEnlisted(TxConnectionManager.java:744) org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:577) org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:337) org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:518) org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:399) org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842) org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88) org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:69) org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423) org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144) org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139) org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547) org.hibernate.loader.Loader.doQuery(Loader.java:673) org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236) org.hibernate.loader.Loader.loadEntity(Loader.java:1907) org.hibernate.loader.entity.CollectionElementLoader.loadElement(CollectionElementLoader.java:72) org.hibernate.persister.collection.OneToManyPersister.getElementByIndex(OneToManyPersister.java:360) org.hibernate.collection.AbstractPersistentCollection.readElementByIndex(AbstractPersistentCollection.java:158) org.hibernate.collection.PersistentMap.get(PersistentMap.java:146) org.jboss.portal.core.impl.model.portal.PortalObjectImpl.getChild(PortalObjectImpl.java:372) org.jboss.portal.core.model.portal.command.mapping.DefaultPortalObjectPathMapper$1.getChild(DefaultPortalObjectPathMapper.java:90) org.jboss.portal.server.servlet.PathParser.map(PathParser.java:86) org.jboss.portal.core.model.portal.command.mapping.DefaultPortalObjectPathMapper.getTarget(DefaultPortalObjectPathMapper.java:108) org.jboss.portal.core.model.portal.PortalObjectCommandFactory.doMapping(PortalObjectCommandFactory.java:77) org.jboss.portal.core.controller.command.mapper.CommandFactoryDelegate.doMapping(CommandFactoryDelegate.java:87) org.jboss.portal.core.controller.command.mapper.DelegatingCommandFactoryService.doMapping(DelegatingCommandFactoryService.java:142) org.jboss.portal.core.model.portal.DefaultPortalCommandFactory.doMapping(DefaultPortalCommandFactory.java:72) org.jboss.portal.core.controller.Controller.handle(Controller.java:217) org.jboss.portal.server.RequestControllerDispatcher.invoke(RequestControllerDispatcher.java:51) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:131) org.jboss.portal.core.cms.aspect.IdentityBindingInterceptor.invoke(IdentityBindingInterceptor.java:47) org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115) org.jboss.portal.server.aspects.server.ContentTypeInterceptor.invoke(ContentTypeInterceptor.java:68) org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115) org.jboss.portal.core.aspects.server.LocaleInterceptor.invoke(LocaleInterceptor.java:96) org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115) org.jboss.portal.core.aspects.server.UserInterceptor.invoke(UserInterceptor.java:245) org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115) org.jboss.portal.server.aspects.server.SessionInvalidatorInterceptor.invoke(SessionInvalidatorInterceptor.java:84) org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115) org.jboss.portal.core.impl.api.user.UserEventBridgeTriggerInterceptor.invoke(UserEventBridgeTriggerInterceptor.java:65) org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115) org.jboss.portal.core.aspects.server.TransactionInterceptor.org$jboss$portal$core$aspects$server$TransactionInterceptor$invoke$aop(TransactionInterceptor.java:49) org.jboss.portal.core.aspects.server.TransactionInterceptor$invoke_N5143606530999904530.invokeNext(TransactionInterceptor$invoke_N5143606530999904530.java) org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) org.jboss.aspects.tx.TxInterceptor$RequiresNew.invoke(TxInterceptor.java:253) org.jboss.portal.core.aspects.server.TransactionInterceptor$invoke_N5143606530999904530.invokeNext(TransactionInterceptor$invoke_N5143606530999904530.java) org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) org.jboss.aspects.tx.TxInterceptor$RequiresNew.invoke(TxInterceptor.java:262) org.jboss.portal.core.aspects.server.TransactionInterceptor$invoke_N5143606530999904530.invokeNext(TransactionInterceptor$invoke_N5143606530999904530.java) org.jboss.portal.core.aspects.server.TransactionInterceptor.invoke(TransactionInterceptor.java) org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115) org.jboss.portal.server.aspects.LockInterceptor$InternalLock.invoke(LockInterceptor.java:70) org.jboss.portal.server.aspects.LockInterceptor.invoke(LockInterceptor.java:131) org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115) org.jboss.portal.common.invocation.Invocation.invoke(Invocation.java:157) org.jboss.portal.server.servlet.PortalServlet.service(PortalServlet.java:380) javax.servlet.http.HttpServlet.service(HttpServlet.java:803) org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) note The full stack trace of the root cause is available in the JBossWeb/2.0.0.GA logs. JBossWeb/2.0.0.GA
And another
11:48:24,500 WARN [loggerI18N] [com.arjuna.ats.internal.jta.transaction.arjunacore.lastResource.disallow] [com.arjuna.ats.internal.jta.transaction.arjunacore.lastResource.disallow] Adding mul
tiple last resources is disallowed. Current resource is org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@26835ed2
11:48:24,510 WARN [JDBCExceptionReporter] SQL Error: 0, SQLState: null
11:48:24,511 ERROR [JDBCExceptionReporter] Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled
to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: a1404ac:a96c:47df8dc7:1ea30 status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JB
ossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, s
ee the previous warnings. tx=TransactionImple < ac, BasicAction: a1404ac:a96c:47df8dc7:1ea30 status: ActionStatus.ABORT_ONLY >))
11:48:24,511 ERROR [JBossCachePersistenceManager] failed to read binary data: 58709461-1230-48a7-9a42-055465b7a85d/{http://www.jcp.org/jcr/1.0}data[0]
org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
at org.hibernate.loader.Loader.doQuery(Loader.java:673)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
at org.hibernate.loader.Loader.doList(Loader.java:2220)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
at org.hibernate.loader.Loader.list(Loader.java:2099)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
at org.jboss.portal.cms.hibernate.state.JBossCachePersistenceManager$DbBLOBStore.get(JBossCachePersistenceManager.java:1529)
at org.apache.jackrabbit.core.state.util.Serializer.deserialize(Serializer.java:293)
at org.jboss.portal.cms.hibernate.state.JBossCachePersistenceManager.load(JBossCachePersistenceManager.java:849)
at org.apache.jackrabbit.core.state.SharedItemStateManager.loadItemState(SharedItemStateManager.java:959)
at org.apache.jackrabbit.core.state.SharedItemStateManager.getNonVirtualItemState(SharedItemStateManager.java:884)
at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:234)
at org.apache.jackrabbit.core.version.VersionItemStateProvider.getItemState(VersionItemStateProvider.java:122)
at org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:239)
at org.apache.jackrabbit.core.state.LocalItemStateManager.getPropertyState(LocalItemStateManager.java:119)
at org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState(LocalItemStateManager.java:151)
at org.apache.jackrabbit.core.state.XAItemStateManager.getItemState(XAItemStateManager.java:235)
at org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:162)
at org.apache.jackrabbit.core.ItemManager.createItemInstance(ItemManager.java:461)
at org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:319)
at org.apache.jackrabbit.core.NodeImpl.getProperty(NodeImpl.java:2492)