3 Replies Latest reply on Jun 23, 2015 12:42 AM by nick.sree

    Wildfly 9.0-CR1 CCM warning which leads wildfly deployment failure

    nick.sree
    nick.sree Jun 15, 2015 8:09 AM

      Hi,

          When we shutdown the wildfly from eclipse, hope this kills ( uses kill -9) then we start the wildfly again, it issues the following warning and making deployment failed. If we set use-ccm=false, then no issues. Have any idea why this is happening on restart ? even though its just warning why wildfly making the deployment as failed ?

       

      16:53:18,536 WARN  [com.arjuna.ats.arjuna] (MSC service thread 1-8) ARJUNA012140: Adding multiple last resources is disallowed. Trying to add LastResourceRecord(XAOnePhaseResource(LocalXAResourceImpl@22dc9f42[connectionListener=3272dfab connectionManager=2e2506c8 warned=false currentXid=< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000101:5aef0a20:557eb586:16, node_name=1, branch_uid=0:ffff7f000101:5aef0a20:557eb586:1a, subordinatenodename=null, eis_name=java:/ezgovDatabasePool > productName=PostgreSQL productVersion=9.4.2 jndiName=java:/ezgovDatabasePool])), but already have LastResourceRecord(XAOnePhaseResource(LocalXAResourceImpl@282e7f01[connectionListener=679eb20e connectionManager=2e2506c8 warned=false currentXid=< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000101:5aef0a20:557eb586:16, node_name=1, branch_uid=0:ffff7f000101:5aef0a20:557eb586:19, subordinatenodename=null, eis_name=java:/ezgovDatabasePool > productName=PostgreSQL productVersion=9.4.2 jndiName=java:/ezgovDatabasePool]))

      16:53:18,540 WARN  [org.jboss.tm.usertx.UserTransactionRegistry] (MSC service thread 1-8) Error notifying listener org.jboss.jca.core.tx.jbossts.UserTransactionListenerImpl@511990bb of userTransactionStarted: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffff7f000101:5aef0a20:557eb586:16 status: ActionStatus.ABORT_ONLY >

        at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:883)

        at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:390)

        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.transactionStarted(TxConnectionManagerImpl.java:450)

        at org.jboss.jca.core.connectionmanager.ccm.CachedConnectionManagerImpl.userTransactionStarted(CachedConnectionManagerImpl.java:252)

        at org.jboss.jca.core.tx.jbossts.UserTransactionListenerImpl.userTransactionStarted(UserTransactionListenerImpl.java:52)

        at org.jboss.tm.usertx.UserTransactionRegistry.userTransactionStarted(UserTransactionRegistry.java:119)

        at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.begin(ServerVMClientUserTransaction.java:151)

        at org.springframework.transaction.jta.JtaTransactionManager.doJtaBegin(JtaTransactionManager.java:875)

        at org.springframework.transaction.jta.JtaTransactionManager.doBegin(JtaTransactionManager.java:832)

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)

        at org.springframework.scheduling.quartz.SchedulerAccessor.registerJobsAndTriggers(SchedulerAccessor.java:200)

        at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:512)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1625)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1562)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)

        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:743)

        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)

        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)

        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403)

        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)

        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)

        at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173)

        at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:195)

        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:86)

        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:71)

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

      • 2908 Views
      • Tags:
        • 1. Re: Wildfly 9.0-CR1 CCM warning which leads wildfly deployment failure
          jameslivingston
          jameslivingston Jun 15, 2015 8:31 PM (in response to nick.sree)

          Those messages say that it is trying to enlist two connections from java:/ezgovDatabasePool into the same transaction. It is not just a warning, there will be an exception thrown and when a context listener's contextInitiased() method (Spring's ContextLoaderListener) throws an exception, it will cause a deployment failure.

           

          What I think the CachedConnectionManager is doing here is to support the behaviour of code getting a connection first and then starting a transaction second, lazy enlistment, where it has retrieved the list of open connections for the thread and enlisting them in the transaction that is being begun. My guess would be that some other Spring bean is opening a connection outside a transaction and not closing it, or something like that.

          1 of 1 people found this helpful
            • Actions
          • 2. Re: Wildfly 9.0-CR1 CCM warning which leads wildfly deployment failure
            nick.sree
            nick.sree Jun 16, 2015 8:02 AM (in response to jameslivingston)

            This is happening while Quartz trying to start, its feed with both Tx and NonTx datasources. More exception stack trace as follows, this is after the above warning

             

            ERROR [org.springframework.web.context.ContextLoader] (MSC service thread 1-3) Context initialization failed: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pgrSchedular' defined in class path resource [config/spring/applicationQuartzContext-pgr.xml]: Invocation of init method failed; nested exception is org.quartz.JobPersistenceException: Couldn't retrieve trigger: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK [See nested exception: java.sql.SQLException: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK]

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1566)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)

              at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)

              at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

              at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)

              at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)

              at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:743)

              at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)

              at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)

              at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403)

              at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)

              at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)

              at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173)

              at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:195)

              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:86)

              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:71)

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

            Caused by: org.quartz.JobPersistenceException: Couldn't retrieve trigger: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK [See nested exception: java.sql.SQLException: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK]

              at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1533)

              at org.quartz.impl.jdbcjobstore.JobStoreSupport$12.execute(JobStoreSupport.java:1522)

              at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:245)

              at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeWithoutLock(JobStoreSupport.java:3723)

              at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1519)

              at org.quartz.core.QuartzScheduler.getTrigger(QuartzScheduler.java:1530)

              at org.quartz.impl.StdScheduler.getTrigger(StdScheduler.java:508)

              at org.springframework.scheduling.quartz.SchedulerAccessor.addTriggerToScheduler(SchedulerAccessor.java:291)

              at org.springframework.scheduling.quartz.SchedulerAccessor.registerJobsAndTriggers(SchedulerAccessor.java:235)

              at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:512)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1625)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1562)

              ... 21 more

            Caused by: java.sql.SQLException: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK

              at org.jboss.jca.adapters.jdbc.WrapperDataSource.checkTransactionActive(WrapperDataSource.java:241)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.checkTransactionActive(WrappedConnection.java:1633)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.checkStatus(WrappedConnection.java:1648)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.checkTransaction(WrappedConnection.java:1622)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:427)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.lang.reflect.Method.invoke(Method.java:483)

              at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:73)

              at com.sun.proxy.$Proxy188.prepareStatement(Unknown Source)

              at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:1758)

              at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1531)

              ... 32 more

             

             

            17:14:48,441 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./pgr: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./pgr: Failed to start service

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

            Caused by: java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pgrSchedular' defined in class path resource [config/spring/applicationQuartzContext-pgr.xml]: Invocation of init method failed; nested exception is org.quartz.JobPersistenceException: Couldn't retrieve trigger: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK [See nested exception: java.sql.SQLException: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK]

              at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:224)

              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:86)

              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:71)

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)

              ... 3 more

            Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pgrSchedular' defined in class path resource [config/spring/applicationQuartzContext-pgr.xml]: Invocation of init method failed; nested exception is org.quartz.JobPersistenceException: Couldn't retrieve trigger: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK [See nested exception: java.sql.SQLException: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK]

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1566)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)

              at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)

              at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

              at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)

              at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)

              at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:743)

              at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)

              at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)

              at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403)

              at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)

              at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)

              at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173)

              at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:195)

              ... 7 more

            Caused by: org.quartz.JobPersistenceException: Couldn't retrieve trigger: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK [See nested exception: java.sql.SQLException: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK]

              at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1533)

              at org.quartz.impl.jdbcjobstore.JobStoreSupport$12.execute(JobStoreSupport.java:1522)

              at org.quartz.impl.jdbcjobstore.JobStoreCMT.executeInLock(JobStoreCMT.java:245)

              at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeWithoutLock(JobStoreSupport.java:3723)

              at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1519)

              at org.quartz.core.QuartzScheduler.getTrigger(QuartzScheduler.java:1530)

              at org.quartz.impl.StdScheduler.getTrigger(StdScheduler.java:508)

              at org.springframework.scheduling.quartz.SchedulerAccessor.addTriggerToScheduler(SchedulerAccessor.java:291)

              at org.springframework.scheduling.quartz.SchedulerAccessor.registerJobsAndTriggers(SchedulerAccessor.java:235)

              at org.springframework.scheduling.quartz.SchedulerFactoryBean.afterPropertiesSet(SchedulerFactoryBean.java:512)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1625)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1562)

              ... 21 more

            Caused by: java.sql.SQLException: IJ031070: Transaction cannot proceed: STATUS_MARKED_ROLLBACK

              at org.jboss.jca.adapters.jdbc.WrapperDataSource.checkTransactionActive(WrapperDataSource.java:241)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.checkTransactionActive(WrappedConnection.java:1633)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.checkStatus(WrappedConnection.java:1648)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.checkTransaction(WrappedConnection.java:1622)

              at org.jboss.jca.adapters.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:427)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.lang.reflect.Method.invoke(Method.java:483)

              at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:73)

              at com.sun.proxy.$Proxy188.prepareStatement(Unknown Source)

              at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:1758)

              at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1531)

              ... 32 more

              • Actions
            • 3. Re: Wildfly 9.0-CR1 CCM warning which leads wildfly deployment failure
              nick.sree
              nick.sree Jun 23, 2015 12:42 AM (in response to nick.sree)

              The reason was we where injecting transactionmanager to spring quartzschedulerfactory, thats getting messed up in case of hard shutdown.

                • Actions