10 Replies Latest reply on Dec 22, 2008 9:43 AM by imranpariyani

    Jboss hangs after running for a whiel

    imranpariyani

      My application runs fine for a while but after that it just hangs and in the log file is see this warnings

      2008-09-16 21:03:00,116 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f42 invoked while multiple threads active within it.
      2008-09-16 21:03:00,117 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fee6:96b7:48ce8118:30f42 aborting with 1 threads active!
      2008-09-16 21:03:00,117 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f48 invoked while multiple threads active within it.
      2008-09-16 21:03:00,117 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fee6:96b7:48ce8118:30f48 aborting with 1 threads active!
      2008-09-16 21:03:00,118 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f4e invoked while multiple threads active within it.
      2008-09-16 21:03:00,118 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fee6:96b7:48ce8118:30f4e aborting with 1 threads active!
      2008-09-16 21:03:00,118 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f54 invoked while multiple threads active within it.
      2008-09-16 21:03:00,118 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fee6:96b7:48ce8118:30f54 aborting with 1 threads active!
      2008-09-16 21:05:00,121 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f5e invoked while multiple threads active within it.
      2008-09-16 21:05:00,121 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fee6:96b7:48ce8118:30f5e aborting with 1 threads active!
      2008-09-16 21:05:00,122 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f68 invoked while multiple threads active within it.
      2008-09-16 21:05:00,122 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fee6:96b7:48ce8118:30f68 aborting with 1 threads active!
      2008-09-16 21:06:54,238 DEBUG [org.hibernate.impl.SessionImpl] opened session at timestamp: 12215920142
      2008-09-16 21:06:54,238 DEBUG [org.hibernate.ejb.AbstractEntityManagerImpl] Looking for a JTA transaction to join
      2008-09-16 21:06:54,238 DEBUG [org.hibernate.jdbc.JDBCContext] successfully registered Synchronization
      2008-09-16 21:06:54,238 DEBUG [org.hibernate.ejb.AbstractEntityManagerImpl] Looking for a JTA transaction to join
      2008-09-16 21:06:54,238 DEBUG [org.hibernate.ejb.AbstractEntityManagerImpl] Transaction already joined
      2008-09-16 21:07:00,124 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f72 invoked while multiple threads active within it.
      2008-09-16 21:07:00,124 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fee6:96b7:48ce8118:30f72 aborting with 1 threads active!
      2008-09-16 21:07:00,124 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f78 invoked while multiple threads active within it.
      2008-09-16 21:07:00,125 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fee6:96b7:48ce8118:30f78 aborting with 1 threads active!
      2008-09-16 21:11:54,246 DEBUG [org.hibernate.impl.SessionImpl] opened session at timestamp: 12215923142
      2008-09-16 21:11:54,246 DEBUG [org.hibernate.ejb.AbstractEntityManagerImpl] Looking for a JTA transaction to join
      2008-09-16 21:11:54,246 DEBUG [org.hibernate.jdbc.JDBCContext] successfully registered Synchronization
      2008-09-16 21:11:54,246 DEBUG [org.hibernate.ejb.AbstractEntityManagerImpl] Looking for a JTA transaction to join
      2008-09-16 21:11:54,246 DEBUG [org.hibernate.ejb.AbstractEntityManagerImpl] Transaction already joined
      


      these warnings just keeps repeating .. is it because of some memory leak ?

      any help would be appreciated

      Thanks
      Imran Pariyani

        • 1. Re: Jboss hangs after running for a whiel
          jhalliday

          Which part of DO NOT POST USER QUESTIONS HERE is too difficult for you to understand? Please read the wiki, use the user forum, or buy a support contract.

          • 2. Re: Jboss hangs after running for a whiel
            imranpariyani

             

            "jhalliday" wrote:
            Which part of DO NOT POST USER QUESTIONS HERE is too difficult for you to understand? Please read the wiki, use the user forum, or buy a support contract.


            @jhalliday
            All of it i guess is difficult for me to understand .. was too drunk thats why it all appeared blurred ..

            @rest
            Sorry guys .. was quite stressed out and dint notice the sticky note at the top .. will repost the question to the user forum ;)

            • 3. Re: Jboss hangs after running for a whiel
              marklittle

              I have moved this posting too!

              • 4. Re: Jboss hangs after running for a whiel
                pbaker01

                Hi imranpariyani,

                Did you ever solve this? I am getting the same errors (see console log below). I have a message producer that puts messages on a queue. I also have an MDB lisener that consumes the messages. Each message can generate > 200 database inserts. The number of messages queued and to be processed is > 50,000. The max session size is 5.

                MDB Annotations:

                @MessageDriven(mappedName = "jms/SSMListener", activationConfig = {
                 @ActivationConfigProperty(propertyName="messagingType", propertyValue="javax.jms.MessageListener"),
                 @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),
                 @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
                 @ActivationConfigProperty(propertyName = "maxSession", propertyValue = "5"),
                 @ActivationConfigProperty(propertyName="destination",propertyValue="queue/SSMQueue") })
                
                public class SSMListener implements MessageListener {
                
                 @PersistenceContext(unitName="WABPU")
                 private EntityManager em;
                 SsimService ssimService;
                 SsimRsp rsp = new SsimRsp();


                Console Log:
                15:28:10,065 INFO [Server] Starting JBoss (MX MicroKernel)...
                15:28:10,066 INFO [Server] Release ID: JBoss [Trinity] 4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417)
                15:28:10,067 INFO [Server] Home Dir: C:\JBoss\jboss-4.2.3.GA
                15:28:10,067 INFO [Server] Home URL: file:/C:/JBoss/jboss-4.2.3.GA/
                15:28:10,067 INFO [Server] Patch URL: null
                15:28:10,068 INFO [Server] Server Name: messaging
                15:28:10,068 INFO [Server] Server Home Dir: C:\JBoss\jboss-4.2.3.GA\server\messaging
                15:28:10,068 INFO [Server] Server Home URL: file:/C:/JBoss/jboss-4.2.3.GA/server/messaging/
                15:28:10,068 INFO [Server] Server Log Dir: C:\JBoss\jboss-4.2.3.GA\server\messaging\log
                15:28:10,068 INFO [Server] Server Temp Dir: C:\JBoss\jboss-4.2.3.GA\server\messaging\tmp
                15:28:10,069 INFO [Server] Root Deployment Filename: jboss-service.xml
                15:28:10,292 INFO [ServerInfo] Java version: 1.6.0_10,Sun Microsystems Inc.
                15:28:10,292 INFO [ServerInfo] Java VM: Java HotSpot(TM) 64-Bit Server VM 11.0-b15,Sun Microsystems Inc.
                15:28:10,293 INFO [ServerInfo] OS-System: Windows Vista 6.0,amd64
                15:28:10,592 INFO [Server] Core system initialized
                15:28:12,127 INFO [WebService] Using RMI server codebase: http://127.0.0.1:8083/
                15:28:12,128 INFO [Log4jService$URLWatchTimerTask] Configuring from URL: resource:jboss-log4j.xml
                15:28:12,532 INFO [TransactionManagerService] JBossTS Transaction Service (JTA version) - JBoss Inc.
                15:28:12,532 INFO [TransactionManagerService] Setting up property manager MBean and JMX layer
                15:28:12,716 INFO [TransactionManagerService] Starting recovery manager
                15:28:12,784 INFO [TransactionManagerService] Recovery manager started
                15:28:12,784 INFO [TransactionManagerService] Binding TransactionManager JNDI Reference
                15:28:14,700 INFO [EJB3Deployer] Starting java:comp multiplexer
                15:28:15,452 INFO [ServerPeer] ServerPeer[0] creating server peer with ID 0
                15:28:17,325 INFO [NativeServerConfig] JBoss Web Services - Native
                15:28:17,325 INFO [NativeServerConfig] jbossws-3.0.1-native-2.0.4.GA (build=200803312044)
                15:28:18,028 INFO [Embedded] Catalina naming disabled
                15:28:18,154 INFO [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\Java\jdk1.6.0_10\bin
                15:28:18,208 INFO [Http11Protocol] Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080
                15:28:18,214 INFO [AjpProtocol] Initializing Coyote AJP/1.3 on ajp-127.0.0.1-8009
                15:28:18,214 INFO [Catalina] Initialization processed in 185 ms
                15:28:18,214 INFO [StandardService] Starting service jboss.web
                15:28:18,216 INFO [StandardEngine] Starting Servlet Engine: JBossWeb/2.0.1.GA
                15:28:18,257 INFO [Catalina] Server startup in 42 ms
                15:28:18,334 INFO [TomcatDeployer] deploy, ctxPath=/, warUrl=.../deploy/jboss-web.deployer/ROOT.war/
                15:28:18,937 INFO [TomcatDeployer] deploy, ctxPath=/invoker, warUrl=.../deploy/http-invoker.sar/invoker.war/
                15:28:19,065 INFO [TomcatDeployer] deploy, ctxPath=/jbossws, warUrl=.../deploy/jbossws.sar/jbossws-context.war/
                15:28:19,276 INFO [TomcatDeployer] deploy, ctxPath=/web-console, warUrl=.../deploy/management/console-mgr.sar/web-console.war/
                15:28:19,729 INFO [MailService] Mail Service bound to java:/Mail
                15:28:19,904 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jboss-ha-local-jdbc.rar
                15:28:19,946 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jboss-ha-xa-jdbc.rar
                15:28:19,984 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jboss-local-jdbc.rar
                15:28:20,019 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jboss-xa-jdbc.rar
                15:28:20,073 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jms-ra.rar
                15:28:20,110 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/mail-ra.rar
                15:28:20,156 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/quartz-ra.rar
                15:28:20,168 INFO [QuartzResourceAdapter] start quartz!!!
                15:28:20,257 INFO [SimpleThreadPool] Job execution threads will use class loader of thread: main
                15:28:20,295 INFO [QuartzScheduler] Quartz Scheduler v.1.5.2 created.
                15:28:20,298 INFO [RAMJobStore] RAMJobStore initialized.
                15:28:20,298 INFO [StdSchedulerFactory] Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
                15:28:20,298 INFO [StdSchedulerFactory] Quartz scheduler version: 1.5.2
                15:28:20,298 INFO [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
                15:28:20,496 INFO [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=WABDS' to JNDI name 'java:WABDS'
                15:28:20,677 INFO [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' to JNDI name 'java:JmsXA'
                15:28:20,782 INFO [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' to JNDI name 'java:DefaultDS'
                15:28:22,714 INFO [ServerPeer] JBoss Messaging 1.3.0.GA server [0] started
                15:28:23,105 INFO [ConnectionFactory] Connector bisocket://127.0.0.1:4457 has leasing enabled, lease period 10000 milliseconds
                15:28:23,106 INFO [ConnectionFactory] [/ConnectionFactory, /XAConnectionFactory, java:/ConnectionFactory, java:/XAConnectionFactory] started
                15:28:23,106 WARN [ConnectionFactoryJNDIMapper] supportsFailover attribute is true on connection factory: jboss.messaging.connectionfactory:service=ClusteredConnectionFactory but post office is non clustered. So connection factory will *not* support failover
                15:28:23,106 WARN [ConnectionFactoryJNDIMapper] supportsLoadBalancing attribute is true on connection factory: jboss.messaging.connectionfactory:service=ClusteredConnectionFactory but post office is non clustered. So connection factory will *not* support load balancing
                15:28:23,109 INFO [ConnectionFactory] Connector bisocket://127.0.0.1:4457 has leasing enabled, lease period 10000 milliseconds
                15:28:23,109 INFO [ConnectionFactory] [/ClusteredConnectionFactory, /ClusteredXAConnectionFactory, java:/ClusteredConnectionFactory, java:/ClusteredXAConnectionFactory] started
                15:28:23,189 INFO [QueueService] Queue[/queue/DLQ] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,221 INFO [QueueService] Queue[/queue/ExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,223 INFO [TopicService] Topic[/topic/testTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,224 INFO [TopicService] Topic[/topic/securedTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,225 INFO [TopicService] Topic[/topic/testDurableTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,253 INFO [QueueService] Queue[/queue/testQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,286 INFO [QueueService] Queue[/queue/A] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,304 INFO [QueueService] Queue[/queue/B] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,329 INFO [QueueService] Queue[/queue/C] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,352 INFO [QueueService] Queue[/queue/D] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,378 INFO [QueueService] Queue[/queue/ex] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,411 INFO [QueueService] Queue[/queue/PrivateDLQ] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,430 INFO [QueueService] Queue[/queue/PrivateExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,477 INFO [QueueService] Queue[/queue/QueueWithOwnDLQAndExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,478 INFO [TopicService] Topic[/topic/TopicWithOwnDLQAndExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,496 INFO [QueueService] Queue[/queue/QueueWithOwnRedeliveryDelay] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,497 INFO [TopicService] Topic[/topic/TopicWithOwnRedeliveryDelay] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,521 INFO [QueueService] Queue[/queue/testDistributedQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,522 INFO [TopicService] Topic[/topic/testDistributedTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:23,546 INFO [QueueService] Queue[/queue/SSMQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                15:28:24,169 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.entity.PersistenceUnitDeployment
                15:28:24,176 INFO [JmxKernelAbstraction] installing MBean: persistence.units:jar=WAB_EJB.jar,unitName=WABPU with dependencies:
                15:28:24,176 INFO [JmxKernelAbstraction] jboss.jca:name=WABDS,service=DataSourceBinding
                15:28:24,178 INFO [PersistenceUnitDeployment] Starting persistence unit persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                15:28:24,220 INFO [Version] Hibernate EntityManager 3.2.1.GA
                15:28:24,248 INFO [Version] Hibernate Annotations 3.2.1.GA
                15:28:24,260 INFO [Environment] Hibernate 3.2.4.sp1
                15:28:24,266 INFO [Environment] hibernate.properties not found
                15:28:24,269 INFO [Environment] Bytecode provider name : javassist
                15:28:24,281 INFO [Environment] using JDK 1.4 java.sql.Timestamp handling
                15:28:24,748 INFO [Configuration] Reading mappings from resource : META-INF/orm.xml
                15:28:24,750 INFO [Ejb3Configuration] [PersistenceUnit: WABPU] no META-INF/orm.xml found
                15:28:25,137 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineImpl -> Airline
                15:28:25,296 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineBaggagePriorityImpl -> AirlineBaggagePriority
                15:28:25,298 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineContactImpl -> AirlineContact
                15:28:25,299 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineFlightTypeImpl -> AirlineFlightType
                15:28:25,300 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineLoadInformationCodeImpl -> AirlineLoadInformationCode
                15:28:25,302 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineMessageDeliveryOptionImpl -> AirlineMessageDeliveryOption
                15:28:25,303 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineOptionsImpl -> AirlineOptions
                15:28:25,306 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineUnitsOfMeasureImpl -> AirlineUnitsOfMeasure
                15:28:25,307 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.SubscriberImpl -> Subscriber
                15:28:25,308 INFO [HbmBinder] Mapping class: com.sita.wab.entities.masterflight.MasterFlightImpl -> MasterFlight
                15:28:25,313 INFO [HbmBinder] Mapping class: com.sita.wab.entities.masterflight.MasterFlightLegImpl -> MasterFlightLeg
                15:28:25,320 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AirlineLoadInformationCodes -> AirlineLoadInformationCode
                15:28:25,321 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AirlineFlightTypes -> AirlineFlightType
                15:28:25,321 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AirlineMessageDeliveryOption -> AirlineMessageDeliveryOption
                15:28:25,321 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AirlineBaggagePriority -> AirlineBaggagePriority
                15:28:25,321 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.SubscriberImpl.Airline -> Airline
                15:28:25,321 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.SubscriberImpl.MasterFlights -> MasterFlight
                15:28:25,321 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.masterflight.MasterFlightImpl.MasterFlightLegs -> MasterFlightLeg
                15:28:25,490 INFO [ConnectionProviderFactory] Initializing connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
                15:28:25,498 INFO [InjectedDataSourceConnectionProvider] Using provided datasource
                15:28:25,511 INFO [SettingsFactory] RDBMS: MySQL, version: 5.0.67-community-nt
                15:28:25,511 INFO [SettingsFactory] JDBC driver: MySQL-AB JDBC Driver, version: mysql-connector-java-5.0.7 ( $Date: 2007-03-09 22:13:57 +0100 (Fri, 09 Mar 2007) $, $Revision: 6341 $ )
                15:28:25,545 INFO [Dialect] Using dialect: org.hibernate.dialect.MySQLDialect
                15:28:25,553 INFO [TransactionFactoryFactory] Transaction strategy: org.hibernate.ejb.transaction.JoinableCMTTransactionFactory
                15:28:25,558 INFO [TransactionManagerLookupFactory] instantiating TransactionManagerLookup: org.hibernate.transaction.JBossTransactionManagerLookup
                15:28:25,562 INFO [TransactionManagerLookupFactory] instantiated TransactionManagerLookup
                15:28:25,562 INFO [SettingsFactory] Automatic flush during beforeCompletion(): disabled
                15:28:25,562 INFO [SettingsFactory] Automatic session close at end of transaction: disabled
                15:28:25,562 INFO [SettingsFactory] JDBC batch size: 15
                15:28:25,562 INFO [SettingsFactory] JDBC batch updates for versioned data: disabled
                15:28:25,564 INFO [SettingsFactory] Scrollable result sets: enabled
                15:28:25,564 INFO [SettingsFactory] JDBC3 getGeneratedKeys(): enabled
                15:28:25,564 INFO [SettingsFactory] Connection release mode: auto
                15:28:25,566 INFO [SettingsFactory] Maximum outer join fetch depth: 2
                15:28:25,566 INFO [SettingsFactory] Default batch fetch size: 1
                15:28:25,566 INFO [SettingsFactory] Generate SQL with comments: disabled
                15:28:25,566 INFO [SettingsFactory] Order SQL updates by primary key: disabled
                15:28:25,566 INFO [SettingsFactory] Order SQL inserts for batching: disabled
                15:28:25,566 INFO [SettingsFactory] Query translator: org.hibernate.hql.ast.ASTQueryTranslatorFactory
                15:28:25,572 INFO [ASTQueryTranslatorFactory] Using ASTQueryTranslatorFactory
                15:28:25,572 INFO [SettingsFactory] Query language substitutions: {}
                15:28:25,572 INFO [SettingsFactory] JPA-QL strict compliance: enabled
                15:28:25,572 INFO [SettingsFactory] Second-level cache: enabled
                15:28:25,572 INFO [SettingsFactory] Query cache: disabled
                15:28:25,572 INFO [SettingsFactory] Cache provider: org.hibernate.cache.HashtableCacheProvider
                15:28:25,575 INFO [SettingsFactory] Optimize cache for minimal puts: disabled
                15:28:25,575 INFO [SettingsFactory] Cache region prefix: WAB_EJB_jar,WABPU
                15:28:25,575 INFO [SettingsFactory] Structured second-level cache entries: disabled
                15:28:25,588 INFO [SettingsFactory] Statistics: disabled
                15:28:25,589 INFO [SettingsFactory] Deleted entity synthetic identifier rollback: disabled
                15:28:25,589 INFO [SettingsFactory] Default entity-mode: pojo
                15:28:25,589 INFO [SettingsFactory] Named query checking : enabled
                15:28:25,644 INFO [SessionFactoryImpl] building session factory
                15:28:26,031 INFO [SessionFactoryObjectFactory] Factory name: persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                15:28:26,033 INFO [NamingHelper] JNDI InitialContext properties:{java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory, java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces}
                15:28:26,039 INFO [SessionFactoryObjectFactory] Bound factory to JNDI name: persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                15:28:26,040 WARN [SessionFactoryObjectFactory] InitialContext did not implement EventContext
                15:28:26,040 INFO [NamingHelper] JNDI InitialContext properties:{java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory, java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces}
                15:28:26,304 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.stateless.StatelessContainer
                15:28:26,312 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:jar=WAB_EJB.jar,name=AirlineEjbBean,service=EJB3 with dependencies:
                15:28:26,313 INFO [JmxKernelAbstraction] persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                15:28:26,380 INFO [EJBContainer] STARTED EJB: com.sita.wab.ejb.airline.AirlineEjbBean ejbName: AirlineEjbBean
                15:28:26,471 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.stateless.StatelessContainer
                15:28:26,471 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:jar=WAB_EJB.jar,name=SsimEjbBean,service=EJB3 with dependencies:
                15:28:26,471 INFO [JmxKernelAbstraction] persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                15:28:26,478 INFO [EJBContainer] STARTED EJB: com.sita.wab.ejb.ssim.SsimEjbBean ejbName: SsimEjbBean
                15:28:26,489 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.stateless.StatelessContainer
                15:28:26,489 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:jar=WAB_EJB.jar,name=SubscriberEjbBean,service=EJB3 with dependencies:
                15:28:26,489 INFO [JmxKernelAbstraction] persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                15:28:26,497 INFO [EJBContainer] STARTED EJB: com.sita.wab.ejb.subscriber.SubscriberEjbBean ejbName: SubscriberEjbBean
                15:28:26,507 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.mdb.MDB
                15:28:26,511 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:jar=WAB_EJB.jar,name=SSMListener,service=EJB3 with dependencies:
                15:28:26,511 INFO [JmxKernelAbstraction] persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                15:28:26,518 INFO [EJBContainer] STARTED EJB: com.sita.wab.mdb.ssim.SSMListener ejbName: SSMListener
                15:28:26,557 INFO [EJB3Deployer] Deployed: file:/C:/JBoss/jboss-4.2.3.GA/server/messaging/deploy/WAB_EJB.jar/
                15:28:26,597 INFO [TomcatDeployer] deploy, ctxPath=/WAB, warUrl=.../deploy/WAB.war/
                15:28:26,767 WARN [config] JSF1059: WARNING! The com.sun.faces.verifyObjects feature is to aid developers not using tools. It shouldn''t be enabled if using an IDE, or if this application is being deployed for production as it will impact application start times.
                15:28:30,305 INFO [CacheManager] Selected [org.ajax4jsf.cache.LRUMapCacheFactory] cache factory
                15:28:30,312 INFO [LRUMapCacheFactory] Creating LRUMap cache instance using parameters: {facelets.DEVELOPMENT=true, enable-cache=false, javax.faces.CONFIG_FILES=/WEB-INF/faces-config.xml, org.richfaces.CONTROL_SKINNING_CLASSES=enable, org.richfaces.SKIN=blueSky, com.sun.faces.injectionProvider=org.jboss.web.jsf.integration.injection.JBossInjectionProvider, com.sun.faces.validateXml=true, org.richfaces.LoadScriptStrategy=ALL, org.richfaces.LoadStyleStrategy=ALL, facelets.SKIP_COMMENTS=true, com.sun.faces.verifyObjects=true, javax.faces.DEFAULT_SUFFIX=.xhtml}
                15:28:30,313 INFO [LRUMapCacheFactory] Creating LRUMap cache instance of default capacity
                15:28:30,346 INFO [CacheManager] Selected [org.ajax4jsf.cache.LRUMapCacheFactory] cache factory
                15:28:30,346 INFO [LRUMapCacheFactory] Creating LRUMap cache instance using parameters: {facelets.DEVELOPMENT=true, enable-cache=false, javax.faces.CONFIG_FILES=/WEB-INF/faces-config.xml, org.richfaces.CONTROL_SKINNING_CLASSES=enable, org.richfaces.SKIN=blueSky, com.sun.faces.injectionProvider=org.jboss.web.jsf.integration.injection.JBossInjectionProvider, com.sun.faces.validateXml=true, org.richfaces.LoadScriptStrategy=ALL, org.richfaces.LoadStyleStrategy=ALL, facelets.SKIP_COMMENTS=true, com.sun.faces.verifyObjects=true, javax.faces.DEFAULT_SUFFIX=.xhtml}
                15:28:30,346 INFO [LRUMapCacheFactory] Creating LRUMap cache instance of default capacity
                15:28:30,384 INFO [TomcatDeployer] deploy, ctxPath=/jmx-console, warUrl=.../deploy/jmx-console.war/
                15:28:30,564 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-127.0.0.1-8080
                15:28:30,592 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-127.0.0.1-8009
                15:28:30,603 INFO [Server] JBoss (MX MicroKernel) [4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417)] Started in 20s:533ms
                15:28:49,111 INFO [STDOUT] Starting message: ID:JBM-0
                15:28:49,636 INFO [STDOUT] Starting message: ID:JBM-1
                15:28:50,441 INFO [STDOUT] Starting message: ID:JBM-2
                15:28:50,899 INFO [STDOUT] Starting message: ID:JBM-3
                15:28:51,418 INFO [STDOUT] Starting message: ID:JBM-4
                15:30:19,529 INFO [STDOUT] Completed message: ID:JBM-1
                15:30:22,566 INFO [STDOUT] Starting message: ID:JBM-5
                15:30:28,571 INFO [STDOUT] Completed message: ID:JBM-2
                15:30:31,155 INFO [STDOUT] Starting message: ID:JBM-6
                15:30:44,014 INFO [STDOUT] Completed message: ID:JBM-3
                15:30:47,128 INFO [STDOUT] Starting message: ID:JBM-7
                15:33:05,342 INFO [STDOUT] Completed message: ID:JBM-0
                15:33:10,029 INFO [STDOUT] Starting message: ID:JBM-8
                15:33:46,534 INFO [STDOUT] Completed message: ID:JBM-7
                15:33:51,218 INFO [STDOUT] Starting message: ID:JBM-9
                15:35:24,513 INFO [STDOUT] Completed message: ID:JBM-6
                15:35:29,511 INFO [STDOUT] Starting message: ID:JBM-10
                15:36:21,303 INFO [STDOUT] Completed message: ID:JBM-5
                15:36:27,915 INFO [STDOUT] Starting message: ID:JBM-11
                15:37:17,392 INFO [STDOUT] Completed message: ID:JBM-4
                15:37:25,875 INFO [STDOUT] Starting message: ID:JBM-12
                15:38:48,452 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fe99:c269:494ea6dc:71 invoked while multiple threads active within it.
                15:38:48,452 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fe99:c269:494ea6dc:71 aborting with 1 threads active!
                15:39:18,636 INFO [STDOUT] Completed message: ID:JBM-10
                15:39:23,701 INFO [STDOUT] Starting message: ID:JBM-13
                15:41:42,136 INFO [STDOUT] Completed message: ID:JBM-11
                15:41:49,472 INFO [STDOUT] Starting message: ID:JBM-14
                15:43:10,413 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fe99:c269:494ea6dc:6dc invoked while multiple threads active within it.
                15:43:10,414 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fe99:c269:494ea6dc:6dc aborting with 1 threads active!
                15:43:11,522 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[8]:PERSISTENT
                javax.ejb.EJBTransactionRolledbackException: org.hibernate.SessionException: Session is closed!
                 at org.jboss.ejb3.tx.Ejb3TxPolicy.handleInCallerTx(Ejb3TxPolicy.java:87)
                 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:130)
                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                 at $Proxy84.onMessage(Unknown Source)
                 at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                 at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                 at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                 at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                 at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                 at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                 at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                 at java.lang.Thread.run(Thread.java:619)
                Caused by: javax.persistence.PersistenceException: org.hibernate.SessionException: Session is closed!
                 at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
                 at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:99)
                 at com.sita.wab.dao.MasterFlightDAO.findMasterFlight(MasterFlightDAO.java:74)
                 at com.sita.wab.service.SsimService.buildFlight(SsimService.java:398)
                 at com.sita.wab.service.SsimService.buildFlights(SsimService.java:345)
                 at com.sita.wab.mdb.ssim.SSMListener.onMessage(SSMListener.java:51)
                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                 at java.lang.reflect.Method.invoke(Method.java:597)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                 ... 23 more
                Caused by: org.hibernate.SessionException: Session is closed!
                 at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:49)
                 at org.hibernate.impl.SessionImpl.getPersistenceContext(SessionImpl.java:1823)
                 at org.hibernate.event.def.ProxyVisitor.processEntity(ProxyVisitor.java:27)
                 at org.hibernate.event.def.AbstractVisitor.processValue(AbstractVisitor.java:104)
                 at org.hibernate.event.def.WrapVisitor.processValue(WrapVisitor.java:98)
                 at org.hibernate.event.def.AbstractVisitor.processEntityPropertyValues(AbstractVisitor.java:55)
                 at org.hibernate.event.def.DefaultFlushEntityEventListener.wrapCollections(DefaultFlushEntityEventListener.java:192)
                 at org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:125)
                 at org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:196)
                 at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:76)
                 at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:35)
                 at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:969)
                 at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1114)
                 at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
                 at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:80)
                 ... 40 more
                15:43:11,572 INFO [STDOUT] Starting message: ID:JBM-15
                15:43:21,078 INFO [STDOUT] Completed message: ID:JBM-9
                15:43:30,999 INFO [STDOUT] Starting message: ID:JBM-8
                15:47:26,072 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fe99:c269:494ea6dc:b6a invoked while multiple threads active within it.
                15:47:26,073 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fe99:c269:494ea6dc:b6a aborting with 1 threads active!
                15:47:26,122 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[12]:PERSISTENT
                javax.ejb.EJBTransactionRolledbackException: org.hibernate.SessionException: Session is closed!
                 at org.jboss.ejb3.tx.Ejb3TxPolicy.handleInCallerTx(Ejb3TxPolicy.java:87)
                 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:130)
                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                 at $Proxy84.onMessage(Unknown Source)
                 at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                 at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                 at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                 at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                 at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                 at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                 at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                 at java.lang.Thread.run(Thread.java:619)
                Caused by: javax.persistence.PersistenceException: org.hibernate.SessionException: Session is closed!
                 at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
                 at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:99)
                 at com.sita.wab.dao.MasterFlightDAO.findMasterFlight(MasterFlightDAO.java:74)
                 at com.sita.wab.service.SsimService.buildFlight(SsimService.java:398)
                 at com.sita.wab.service.SsimService.buildFlights(SsimService.java:345)
                 at com.sita.wab.mdb.ssim.SSMListener.onMessage(SSMListener.java:51)
                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                 at java.lang.reflect.Method.invoke(Method.java:597)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                 ... 23 more
                Caused by: org.hibernate.SessionException: Session is closed!
                 at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:49)
                 at org.hibernate.impl.SessionImpl.getPersistenceContext(SessionImpl.java:1823)
                 at org.hibernate.event.def.ProxyVisitor.processEntity(ProxyVisitor.java:27)
                 at org.hibernate.event.def.AbstractVisitor.processValue(AbstractVisitor.java:104)
                 at org.hibernate.event.def.WrapVisitor.processValue(WrapVisitor.java:98)
                 at org.hibernate.event.def.AbstractVisitor.processEntityPropertyValues(AbstractVisitor.java:55)
                 at org.hibernate.event.def.DefaultFlushEntityEventListener.wrapCollections(DefaultFlushEntityEventListener.java:192)
                 at org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:125)
                 at org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:196)
                 at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:76)
                 at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:35)
                 at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:969)
                 at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1114)
                 at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
                 at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:80)
                 ... 40 more
                15:47:26,128 INFO [STDOUT] Starting message: ID:JBM-16
                15:49:24,212 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fe99:c269:494ea6dc:d3b invoked while multiple threads active within it.
                15:49:24,217 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fe99:c269:494ea6dc:d3b aborting with 1 threads active!
                15:49:25,323 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[13]:PERSISTENT
                javax.ejb.EJBTransactionRolledbackException
                 at org.jboss.ejb3.tx.Ejb3TxPolicy.handleInCallerTx(Ejb3TxPolicy.java:87)
                 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:130)
                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                 at $Proxy84.onMessage(Unknown Source)
                 at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                 at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                 at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                 at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                 at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                 at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                 at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                 at java.lang.Thread.run(Thread.java:619)
                Caused by: java.lang.NullPointerException
                 at org.hibernate.connection.DatasourceConnectionProvider.closeConnection(DatasourceConnectionProvider.java:74)
                 at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:451)
                 at org.hibernate.jdbc.ConnectionManager.aggressiveRelease(ConnectionManager.java:406)
                 at org.hibernate.jdbc.ConnectionManager.afterStatement(ConnectionManager.java:281)
                 at org.hibernate.jdbc.AbstractBatcher.closePreparedStatement(AbstractBatcher.java:543)
                 at org.hibernate.jdbc.AbstractBatcher.closeStatement(AbstractBatcher.java:269)
                 at org.hibernate.jdbc.AbstractBatcher.closeQueryStatement(AbstractBatcher.java:285)
                 at org.hibernate.jdbc.AbstractBatcher.closeQueryStatement(AbstractBatcher.java:212)
                 at org.hibernate.loader.Loader.doQuery(Loader.java:726)
                 at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
                 at org.hibernate.loader.Loader.doList(Loader.java:2220)
                 at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
                 at org.hibernate.loader.Loader.list(Loader.java:2099)
                 at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)
                 at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
                 at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
                 at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
                 at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
                 at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:80)
                 at com.sita.wab.dao.MasterFlightDAO.findMasterFlight(MasterFlightDAO.java:74)
                 at com.sita.wab.service.SsimService.buildFlight(SsimService.java:398)
                 at com.sita.wab.service.SsimService.buildFlights(SsimService.java:345)
                 at com.sita.wab.mdb.ssim.SSMListener.onMessage(SSMListener.java:51)
                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                 at java.lang.reflect.Method.invoke(Method.java:597)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                 ... 23 more
                15:49:25,328 INFO [STDOUT] Starting message: ID:JBM-12
                15:51:50,224 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fe99:c269:494ea6dc:f4d invoked while multiple threads active within it.
                15:51:50,224 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fe99:c269:494ea6dc:f4d aborting with 1 threads active!
                15:51:50,280 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[14]:PERSISTENT
                javax.ejb.EJBTransactionRolledbackException: org.hibernate.SessionException: Session is closed!
                 at org.jboss.ejb3.tx.Ejb3TxPolicy.handleInCallerTx(Ejb3TxPolicy.java:87)
                 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:130)
                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                 at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                 at $Proxy84.onMessage(Unknown Source)
                 at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                 at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                 at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                 at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                 at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                 at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                 at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                 at java.lang.Thread.run(Thread.java:619)
                Caused by: javax.persistence.PersistenceException: org.hibernate.SessionException: Session is closed!
                 at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
                 at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:99)
                 at com.sita.wab.dao.MasterFlightDAO.findMasterFlight(MasterFlightDAO.java:74)
                 at com.sita.wab.service.SsimService.buildFlight(SsimService.java:398)
                 at com.sita.wab.service.SsimService.buildFlights(SsimService.java:345)
                 at com.sita.wab.mdb.ssim.SSMListener.onMessage(SSMListener.java:51)
                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                 at java.lang.reflect.Method.invoke(Method.java:597)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                 at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                 ... 23 more
                Caused by: org.hibernate.SessionException: Session is closed!
                 at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:49)
                 at org.hibernate.impl.SessionImpl.getPersistenceContext(SessionImpl.java:1823)
                 at org.hibernate.event.def.ProxyVisitor.processEntity(ProxyVisitor.java:27)
                 at org.hibernate.event.def.AbstractVisitor.processValue(AbstractVisitor.java:104)
                 at org.hibernate.event.def.WrapVisitor.processValue(WrapVisitor.java:98)
                 at org.hibernate.event.def.AbstractVisitor.processEntityPropertyValues(AbstractVisitor.java:55)
                 at org.hibernate.event.def.DefaultFlushEntityEventListener.wrapCollections(DefaultFlushEntityEventListener.java:192)
                 at org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:125)
                 at org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:196)
                 at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:76)
                 at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:35)
                 at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:969)
                 at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1114)
                 at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
                 at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:80)
                 ... 40 more
                15:51:50,283 INFO [STDOUT] Starting message: ID:JBM-13
                


                • 5. Re: Jboss hangs after running for a whiel
                  jhalliday

                  I'm not seeing any evidence of a hang there. It looks more like the large tx size (or high concurrency) is causing transactions to exceed their timeout. Maybe try to decrease the former or increase the latter.

                  • 6. Re: Jboss hangs after running for a whiel
                    pbaker01

                    Hi Jhalliday,

                    Thank you for taking the time to look at this. The system does not "hang" but it degregates to the point where, for all practical purposes, it is hung up. The console messages in the original post were the same as the ones that I was getting so I thought there may be a connection.

                    I have tried a number of combinations of sessions verse the work packet size. The work packet size represents the number of entries in each queue message. A work pack size of 1 will generate approximately 1 query and up to 30 x 2 inserts (two tables are updated on each insert).

                    The message producer and message consumer (MDB) both work at the same time.

                    JBoss: version 4.2.3
                    JBoss Messaing 1.4
                    MySQL is used as the datasource
                    JPA/Hibernate used for message consumer

                    Taking your advise, I reduced the work pack size to 5 and increased the number of sessions to 25.

                    With this configuration, the processing starts up and runs well for the first few minutes but then quickly degenerates. See below for new console output.

                    Any tuning advise will be greating appreciated. If you do find anything to change, will you please identify the configuration file along with the setting?

                    Thanks so much! PB

                    Console Log:

                    06:47:42,908 INFO [Server] Starting JBoss (MX MicroKernel)...
                    06:47:42,909 INFO [Server] Release ID: JBoss [Trinity] 4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417)
                    06:47:42,910 INFO [Server] Home Dir: C:\JBoss\jboss-4.2.3.GA
                    06:47:42,910 INFO [Server] Home URL: file:/C:/JBoss/jboss-4.2.3.GA/
                    06:47:42,910 INFO [Server] Patch URL: null
                    06:47:42,911 INFO [Server] Server Name: messaging
                    06:47:42,911 INFO [Server] Server Home Dir: C:\JBoss\jboss-4.2.3.GA\server\messaging
                    06:47:42,911 INFO [Server] Server Home URL: file:/C:/JBoss/jboss-4.2.3.GA/server/messaging/
                    06:47:42,911 INFO [Server] Server Log Dir: C:\JBoss\jboss-4.2.3.GA\server\messaging\log
                    06:47:42,911 INFO [Server] Server Temp Dir: C:\JBoss\jboss-4.2.3.GA\server\messaging\tmp
                    06:47:42,912 INFO [Server] Root Deployment Filename: jboss-service.xml
                    06:47:43,140 INFO [ServerInfo] Java version: 1.6.0_10,Sun Microsystems Inc.
                    06:47:43,140 INFO [ServerInfo] Java VM: Java HotSpot(TM) 64-Bit Server VM 11.0-b15,Sun Microsystems Inc.
                    06:47:43,141 INFO [ServerInfo] OS-System: Windows Vista 6.0,amd64
                    06:47:43,433 INFO [Server] Core system initialized
                    06:47:44,968 INFO [WebService] Using RMI server codebase: http://127.0.0.1:8083/
                    06:47:44,970 INFO [Log4jService$URLWatchTimerTask] Configuring from URL: resource:jboss-log4j.xml
                    06:47:45,712 INFO [TransactionManagerService] JBossTS Transaction Service (JTA version) - JBoss Inc.
                    06:47:45,712 INFO [TransactionManagerService] Setting up property manager MBean and JMX layer
                    06:47:45,931 INFO [TransactionManagerService] Starting recovery manager
                    06:47:46,038 INFO [TransactionManagerService] Recovery manager started
                    06:47:46,038 INFO [TransactionManagerService] Binding TransactionManager JNDI Reference
                    06:47:48,137 INFO [EJB3Deployer] Starting java:comp multiplexer
                    06:47:48,943 INFO [ServerPeer] ServerPeer[0] creating server peer with ID 0
                    06:47:50,782 INFO [NativeServerConfig] JBoss Web Services - Native
                    06:47:50,782 INFO [NativeServerConfig] jbossws-3.0.1-native-2.0.4.GA (build=200803312044)
                    06:47:51,503 INFO [Embedded] Catalina naming disabled
                    06:47:51,629 INFO [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files\Java\jdk1.6.0_10\bin
                    06:47:51,678 INFO [Http11Protocol] Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080
                    06:47:51,684 INFO [AjpProtocol] Initializing Coyote AJP/1.3 on ajp-127.0.0.1-8009
                    06:47:51,684 INFO [Catalina] Initialization processed in 181 ms
                    06:47:51,684 INFO [StandardService] Starting service jboss.web
                    06:47:51,686 INFO [StandardEngine] Starting Servlet Engine: JBossWeb/2.0.1.GA
                    06:47:51,723 INFO [Catalina] Server startup in 39 ms
                    06:47:51,806 INFO [TomcatDeployer] deploy, ctxPath=/, warUrl=.../deploy/jboss-web.deployer/ROOT.war/
                    06:47:52,507 INFO [TomcatDeployer] deploy, ctxPath=/invoker, warUrl=.../deploy/http-invoker.sar/invoker.war/
                    06:47:52,644 INFO [TomcatDeployer] deploy, ctxPath=/jbossws, warUrl=.../deploy/jbossws.sar/jbossws-context.war/
                    06:47:52,946 INFO [TomcatDeployer] deploy, ctxPath=/web-console, warUrl=.../deploy/management/console-mgr.sar/web-console.war/
                    06:47:53,471 INFO [MailService] Mail Service bound to java:/Mail
                    06:47:53,627 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jboss-ha-local-jdbc.rar
                    06:47:53,668 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jboss-ha-xa-jdbc.rar
                    06:47:53,707 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jboss-local-jdbc.rar
                    06:47:53,745 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jboss-xa-jdbc.rar
                    06:47:53,830 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/jms-ra.rar
                    06:47:53,866 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/mail-ra.rar
                    06:47:53,916 INFO [RARDeployment] Required license terms exist, view META-INF/ra.xml in .../deploy/quartz-ra.rar
                    06:47:53,927 INFO [QuartzResourceAdapter] start quartz!!!
                    06:47:54,003 INFO [SimpleThreadPool] Job execution threads will use class loader of thread: main
                    06:47:54,042 INFO [QuartzScheduler] Quartz Scheduler v.1.5.2 created.
                    06:47:54,045 INFO [RAMJobStore] RAMJobStore initialized.
                    06:47:54,045 INFO [StdSchedulerFactory] Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
                    06:47:54,045 INFO [StdSchedulerFactory] Quartz scheduler version: 1.5.2
                    06:47:54,046 INFO [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
                    06:47:54,266 INFO [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=WABDS' to JNDI name 'java:WABDS'
                    06:47:54,455 INFO [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' to JNDI name 'java:JmsXA'
                    06:47:54,568 INFO [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' to JNDI name 'java:DefaultDS'
                    06:47:56,541 INFO [ServerPeer] JBoss Messaging 1.3.0.GA server [0] started
                    06:47:56,873 INFO [ConnectionFactory] Connector bisocket://127.0.0.1:4457 has leasing enabled, lease period 10000 milliseconds
                    06:47:56,873 INFO [ConnectionFactory] [/ConnectionFactory, /XAConnectionFactory, java:/ConnectionFactory, java:/XAConnectionFactory] started
                    06:47:56,874 WARN [ConnectionFactoryJNDIMapper] supportsFailover attribute is true on connection factory: jboss.messaging.connectionfactory:service=ClusteredConnectionFactory but post office is non clustered. So connection factory will *not* support failover
                    06:47:56,874 WARN [ConnectionFactoryJNDIMapper] supportsLoadBalancing attribute is true on connection factory: jboss.messaging.connectionfactory:service=ClusteredConnectionFactory but post office is non clustered. So connection factory will *not* support load balancing
                    06:47:56,876 INFO [ConnectionFactory] Connector bisocket://127.0.0.1:4457 has leasing enabled, lease period 10000 milliseconds
                    06:47:56,876 INFO [ConnectionFactory] [/ClusteredConnectionFactory, /ClusteredXAConnectionFactory, java:/ClusteredConnectionFactory, java:/ClusteredXAConnectionFactory] started
                    06:47:57,029 INFO [QueueService] Queue[/queue/DLQ] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,049 INFO [QueueService] Queue[/queue/ExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,051 INFO [TopicService] Topic[/topic/testTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,052 INFO [TopicService] Topic[/topic/securedTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,053 INFO [TopicService] Topic[/topic/testDurableTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,074 INFO [QueueService] Queue[/queue/testQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,107 INFO [QueueService] Queue[/queue/A] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,132 INFO [QueueService] Queue[/queue/B] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,157 INFO [QueueService] Queue[/queue/C] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,182 INFO [QueueService] Queue[/queue/D] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,207 INFO [QueueService] Queue[/queue/ex] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,268 INFO [QueueService] Queue[/queue/PrivateDLQ] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,298 INFO [QueueService] Queue[/queue/PrivateExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,332 INFO [QueueService] Queue[/queue/QueueWithOwnDLQAndExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,333 INFO [TopicService] Topic[/topic/TopicWithOwnDLQAndExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,348 INFO [QueueService] Queue[/queue/QueueWithOwnRedeliveryDelay] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,349 INFO [TopicService] Topic[/topic/TopicWithOwnRedeliveryDelay] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,374 INFO [QueueService] Queue[/queue/testDistributedQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,375 INFO [TopicService] Topic[/topic/testDistributedTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,399 INFO [QueueService] Queue[/queue/SSMQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
                    06:47:57,914 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.entity.PersistenceUnitDeployment
                    06:47:57,919 INFO [JmxKernelAbstraction] installing MBean: persistence.units:jar=WAB_EJB.jar,unitName=WABPU with dependencies:
                    06:47:57,919 INFO [JmxKernelAbstraction] jboss.jca:name=WABDS,service=DataSourceBinding
                    06:47:57,921 INFO [PersistenceUnitDeployment] Starting persistence unit persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                    06:47:57,949 INFO [Version] Hibernate EntityManager 3.2.1.GA
                    06:47:57,972 INFO [Version] Hibernate Annotations 3.2.1.GA
                    06:47:57,982 INFO [Environment] Hibernate 3.2.4.sp1
                    06:47:57,987 INFO [Environment] hibernate.properties not found
                    06:47:57,990 INFO [Environment] Bytecode provider name : javassist
                    06:47:57,997 INFO [Environment] using JDK 1.4 java.sql.Timestamp handling
                    06:47:58,514 INFO [Configuration] Reading mappings from resource : META-INF/orm.xml
                    06:47:58,516 INFO [Ejb3Configuration] [PersistenceUnit: WABPU] no META-INF/orm.xml found
                    06:47:58,942 INFO [HbmBinder] Mapping class: com.sita.wab.entities.aircraft.AircraftType -> AircraftType
                    06:47:59,085 INFO [HbmBinder] Mapping class: com.sita.wab.entities.aircraft.AircraftTypeRevision -> AircraftTypeRevision
                    06:47:59,087 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineImpl -> Airline
                    06:47:59,096 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineBaggagePriorityImpl -> AirlineBaggagePriority
                    06:47:59,097 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineContactImpl -> AirlineContact
                    06:47:59,099 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineFlightTypeImpl -> AirlineFlightType
                    06:47:59,100 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineLoadInformationCodeImpl -> AirlineLoadInformationCode
                    06:47:59,101 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineMessageDeliveryOptionImpl -> AirlineMessageDeliveryOption
                    06:47:59,102 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineOptionsImpl -> AirlineOptions
                    06:47:59,109 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.AirlineUnitsOfMeasureImpl -> AirlineUnitsOfMeasure
                    06:47:59,110 INFO [HbmBinder] Mapping class: com.sita.wab.entities.airline.SubscriberImpl -> Subscriber
                    06:47:59,111 INFO [HbmBinder] Mapping class: com.sita.wab.entities.masterflight.MasterFlightImpl -> MasterFlight
                    06:47:59,116 INFO [HbmBinder] Mapping class: com.sita.wab.entities.masterflight.MasterFlightLegImpl -> MasterFlightLeg
                    06:47:59,123 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.aircraft.AircraftType.SharedAircraftTypes -> AircraftType
                    06:47:59,124 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.aircraft.AircraftType.AircraftTypeRevisions -> AircraftTypeRevision
                    06:47:59,124 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.aircraft.AircraftTypeRevision.SharedAircraftTypeRevisions -> AircraftTypeRevision
                    06:47:59,124 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AirlineLoadInformationCodes -> AirlineLoadInformationCode
                    06:47:59,124 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AirlineFlightTypes -> AirlineFlightType
                    06:47:59,124 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AirlineMessageDeliveryOption -> AirlineMessageDeliveryOption
                    06:47:59,125 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AirlineBaggagePriority -> AirlineBaggagePriority
                    06:47:59,125 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.AirlineImpl.AircraftTypes -> AircraftType
                    06:47:59,125 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.SubscriberImpl.Airline -> Airline
                    06:47:59,125 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.airline.SubscriberImpl.MasterFlights -> MasterFlight
                    06:47:59,125 INFO [HbmBinder] Mapping collection: com.sita.wab.entities.masterflight.MasterFlightImpl.MasterFlightLegs -> MasterFlightLeg
                    06:47:59,301 INFO [ConnectionProviderFactory] Initializing connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
                    06:47:59,307 INFO [InjectedDataSourceConnectionProvider] Using provided datasource
                    06:47:59,320 INFO [SettingsFactory] RDBMS: MySQL, version: 5.0.67-community-nt
                    06:47:59,320 INFO [SettingsFactory] JDBC driver: MySQL-AB JDBC Driver, version: mysql-connector-java-5.0.7 ( $Date: 2007-03-09 22:13:57 +0100 (Fri, 09 Mar 2007) $, $Revision: 6341 $ )
                    06:47:59,352 INFO [Dialect] Using dialect: org.hibernate.dialect.MySQLDialect
                    06:47:59,360 INFO [TransactionFactoryFactory] Transaction strategy: org.hibernate.ejb.transaction.JoinableCMTTransactionFactory
                    06:47:59,364 INFO [TransactionManagerLookupFactory] instantiating TransactionManagerLookup: org.hibernate.transaction.JBossTransactionManagerLookup
                    06:47:59,367 INFO [TransactionManagerLookupFactory] instantiated TransactionManagerLookup
                    06:47:59,367 INFO [SettingsFactory] Automatic flush during beforeCompletion(): disabled
                    06:47:59,367 INFO [SettingsFactory] Automatic session close at end of transaction: disabled
                    06:47:59,368 INFO [SettingsFactory] JDBC batch size: 15
                    06:47:59,368 INFO [SettingsFactory] JDBC batch updates for versioned data: disabled
                    06:47:59,369 INFO [SettingsFactory] Scrollable result sets: enabled
                    06:47:59,369 INFO [SettingsFactory] JDBC3 getGeneratedKeys(): enabled
                    06:47:59,369 INFO [SettingsFactory] Connection release mode: auto
                    06:47:59,371 INFO [SettingsFactory] Maximum outer join fetch depth: 2
                    06:47:59,372 INFO [SettingsFactory] Default batch fetch size: 1
                    06:47:59,372 INFO [SettingsFactory] Generate SQL with comments: disabled
                    06:47:59,372 INFO [SettingsFactory] Order SQL updates by primary key: disabled
                    06:47:59,372 INFO [SettingsFactory] Order SQL inserts for batching: disabled
                    06:47:59,372 INFO [SettingsFactory] Query translator: org.hibernate.hql.ast.ASTQueryTranslatorFactory
                    06:47:59,379 INFO [ASTQueryTranslatorFactory] Using ASTQueryTranslatorFactory
                    06:47:59,379 INFO [SettingsFactory] Query language substitutions: {}
                    06:47:59,379 INFO [SettingsFactory] JPA-QL strict compliance: enabled
                    06:47:59,379 INFO [SettingsFactory] Second-level cache: enabled
                    06:47:59,379 INFO [SettingsFactory] Query cache: disabled
                    06:47:59,379 INFO [SettingsFactory] Cache provider: org.hibernate.cache.HashtableCacheProvider
                    06:47:59,383 INFO [SettingsFactory] Optimize cache for minimal puts: disabled
                    06:47:59,383 INFO [SettingsFactory] Cache region prefix: WAB_EJB_jar,WABPU
                    06:47:59,383 INFO [SettingsFactory] Structured second-level cache entries: disabled
                    06:47:59,397 INFO [SettingsFactory] Statistics: disabled
                    06:47:59,397 INFO [SettingsFactory] Deleted entity synthetic identifier rollback: disabled
                    06:47:59,397 INFO [SettingsFactory] Default entity-mode: pojo
                    06:47:59,397 INFO [SettingsFactory] Named query checking : enabled
                    06:47:59,452 INFO [SessionFactoryImpl] building session factory
                    06:47:59,838 INFO [SessionFactoryObjectFactory] Factory name: persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                    06:47:59,840 INFO [NamingHelper] JNDI InitialContext properties:{java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory, java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces}
                    06:47:59,847 INFO [SessionFactoryObjectFactory] Bound factory to JNDI name: persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                    06:47:59,847 WARN [SessionFactoryObjectFactory] InitialContext did not implement EventContext
                    06:47:59,848 INFO [NamingHelper] JNDI InitialContext properties:{java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory, java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces}
                    06:48:00,119 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.stateless.StatelessContainer
                    06:48:00,127 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:jar=WAB_EJB.jar,name=AirlineEjbBean,service=EJB3 with dependencies:
                    06:48:00,127 INFO [JmxKernelAbstraction] persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                    06:48:00,195 INFO [EJBContainer] STARTED EJB: com.sita.wab.ejb.airline.AirlineEjbBean ejbName: AirlineEjbBean
                    06:48:00,287 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.stateless.StatelessContainer
                    06:48:00,288 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:jar=WAB_EJB.jar,name=SsimEjbBean,service=EJB3 with dependencies:
                    06:48:00,288 INFO [JmxKernelAbstraction] persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                    06:48:00,295 INFO [EJBContainer] STARTED EJB: com.sita.wab.ejb.ssim.SsimEjbBean ejbName: SsimEjbBean
                    06:48:00,305 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.stateless.StatelessContainer
                    06:48:00,305 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:jar=WAB_EJB.jar,name=SubscriberEjbBean,service=EJB3 with dependencies:
                    06:48:00,305 INFO [JmxKernelAbstraction] persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                    06:48:00,312 INFO [EJBContainer] STARTED EJB: com.sita.wab.ejb.subscriber.SubscriberEjbBean ejbName: SubscriberEjbBean
                    06:48:00,322 INFO [JmxKernelAbstraction] creating wrapper delegate for: org.jboss.ejb3.mdb.MDB
                    06:48:00,326 INFO [JmxKernelAbstraction] installing MBean: jboss.j2ee:jar=WAB_EJB.jar,name=SSMListener,service=EJB3 with dependencies:
                    06:48:00,326 INFO [JmxKernelAbstraction] persistence.units:jar=WAB_EJB.jar,unitName=WABPU
                    06:48:00,333 INFO [EJBContainer] STARTED EJB: com.sita.wab.mdb.ssim.SSMListener ejbName: SSMListener
                    06:48:00,371 INFO [EJB3Deployer] Deployed: file:/C:/JBoss/jboss-4.2.3.GA/server/messaging/deploy/WAB_EJB.jar/
                    06:48:00,411 INFO [TomcatDeployer] deploy, ctxPath=/WAB, warUrl=.../deploy/WAB.war/
                    06:48:00,621 WARN [config] JSF1059: WARNING! The com.sun.faces.verifyObjects feature is to aid developers not using tools. It shouldn''t be enabled if using an IDE, or if this application is being deployed for production as it will impact application start times.
                    06:48:04,473 INFO [CacheManager] Selected [org.ajax4jsf.cache.LRUMapCacheFactory] cache factory
                    06:48:04,482 INFO [LRUMapCacheFactory] Creating LRUMap cache instance using parameters: {facelets.DEVELOPMENT=true, enable-cache=false, javax.faces.CONFIG_FILES=/WEB-INF/faces-config.xml, org.richfaces.CONTROL_SKINNING_CLASSES=enable, org.richfaces.SKIN=blueSky, com.sun.faces.injectionProvider=org.jboss.web.jsf.integration.injection.JBossInjectionProvider, com.sun.faces.validateXml=true, org.richfaces.LoadScriptStrategy=ALL, org.richfaces.LoadStyleStrategy=ALL, facelets.SKIP_COMMENTS=true, com.sun.faces.verifyObjects=true, javax.faces.DEFAULT_SUFFIX=.xhtml}
                    06:48:04,482 INFO [LRUMapCacheFactory] Creating LRUMap cache instance of default capacity
                    06:48:04,508 INFO [CacheManager] Selected [org.ajax4jsf.cache.LRUMapCacheFactory] cache factory
                    06:48:04,509 INFO [LRUMapCacheFactory] Creating LRUMap cache instance using parameters: {facelets.DEVELOPMENT=true, enable-cache=false, javax.faces.CONFIG_FILES=/WEB-INF/faces-config.xml, org.richfaces.CONTROL_SKINNING_CLASSES=enable, org.richfaces.SKIN=blueSky, com.sun.faces.injectionProvider=org.jboss.web.jsf.integration.injection.JBossInjectionProvider, com.sun.faces.validateXml=true, org.richfaces.LoadScriptStrategy=ALL, org.richfaces.LoadStyleStrategy=ALL, facelets.SKIP_COMMENTS=true, com.sun.faces.verifyObjects=true, javax.faces.DEFAULT_SUFFIX=.xhtml}
                    06:48:04,509 INFO [LRUMapCacheFactory] Creating LRUMap cache instance of default capacity
                    06:48:04,546 INFO [TomcatDeployer] deploy, ctxPath=/jmx-console, warUrl=.../deploy/jmx-console.war/
                    06:48:04,718 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-127.0.0.1-8080
                    06:48:04,743 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-127.0.0.1-8009
                    06:48:04,757 INFO [Server] JBoss (MX MicroKernel) [4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417)] Started in 21s:844ms
                    06:48:17,572 INFO [STDOUT] Starting message: ID:JBM-0
                    06:48:18,230 INFO [STDOUT] Starting message: ID:JBM-1
                    06:48:18,650 INFO [STDOUT] Starting message: ID:JBM-2
                    06:48:18,724 INFO [STDOUT] Starting message: ID:JBM-3
                    06:48:19,412 INFO [STDOUT] Starting message: ID:JBM-4
                    06:48:19,506 INFO [STDOUT] Starting message: ID:JBM-5
                    06:48:20,279 INFO [STDOUT] Starting message: ID:JBM-6
                    06:48:20,628 INFO [STDOUT] Starting message: ID:JBM-7
                    06:48:20,729 INFO [STDOUT] Starting message: ID:JBM-8
                    06:48:21,135 INFO [STDOUT] Starting message: ID:JBM-9
                    06:48:21,857 INFO [STDOUT] Starting message: ID:JBM-10
                    06:48:21,985 INFO [STDOUT] Starting message: ID:JBM-11
                    06:48:22,628 INFO [STDOUT] Starting message: ID:JBM-12
                    06:48:22,796 INFO [STDOUT] Starting message: ID:JBM-13
                    06:48:23,182 INFO [STDOUT] Starting message: ID:JBM-14
                    06:48:23,928 INFO [STDOUT] Completed message: ID:JBM-7
                    06:48:23,940 INFO [STDOUT] Starting message: ID:JBM-15
                    06:48:24,049 INFO [STDOUT] Completed message: ID:JBM-3
                    06:48:24,088 INFO [STDOUT] Starting message: ID:JBM-16
                    06:48:25,018 INFO [STDOUT] Completed message: ID:JBM-5
                    06:48:25,019 INFO [STDOUT] Starting message: ID:JBM-17
                    06:48:25,029 INFO [STDOUT] Completed message: ID:JBM-12
                    06:48:25,113 INFO [STDOUT] Starting message: ID:JBM-18
                    06:48:25,447 INFO [STDOUT] Completed message: ID:JBM-8
                    06:48:25,647 INFO [STDOUT] Completed message: ID:JBM-6
                    06:48:25,769 INFO [STDOUT] Completed message: ID:JBM-11
                    06:48:26,266 INFO [STDOUT] Starting message: ID:JBM-19
                    06:48:26,871 INFO [STDOUT] Starting message: ID:JBM-20
                    06:48:27,235 INFO [STDOUT] Completed message: ID:JBM-17
                    06:48:27,302 INFO [STDOUT] Starting message: ID:JBM-21
                    06:48:27,705 INFO [STDOUT] Starting message: ID:JBM-22
                    06:48:29,806 INFO [STDOUT] Completed message: ID:JBM-4
                    06:48:29,820 INFO [STDOUT] Starting message: ID:JBM-23
                    06:48:30,296 INFO [STDOUT] Completed message: ID:JBM-9
                    06:48:30,297 INFO [STDOUT] Starting message: ID:JBM-24
                    06:48:30,591 INFO [STDOUT] Completed message: ID:JBM-13
                    06:48:30,592 INFO [STDOUT] Starting message: ID:JBM-25
                    06:48:31,892 INFO [STDOUT] Completed message: ID:JBM-21
                    06:48:31,893 INFO [STDOUT] Starting message: ID:JBM-26
                    06:48:32,008 INFO [STDOUT] Completed message: ID:JBM-22
                    06:48:32,008 INFO [STDOUT] Starting message: ID:JBM-27
                    06:48:32,608 INFO [STDOUT] Completed message: ID:JBM-2
                    06:48:32,608 INFO [STDOUT] Starting message: ID:JBM-28
                    06:48:34,510 INFO [STDOUT] Completed message: ID:JBM-14
                    06:48:34,512 INFO [STDOUT] Starting message: ID:JBM-29
                    06:48:36,428 INFO [STDOUT] Completed message: ID:JBM-26
                    06:48:36,428 INFO [STDOUT] Starting message: ID:JBM-30
                    06:48:38,340 INFO [STDOUT] Completed message: ID:JBM-27
                    06:48:38,341 INFO [STDOUT] Starting message: ID:JBM-31
                    06:48:41,862 INFO [STDOUT] Completed message: ID:JBM-23
                    06:48:41,862 INFO [STDOUT] Starting message: ID:JBM-32
                    06:48:42,741 INFO [STDOUT] Completed message: ID:JBM-24
                    06:48:42,743 INFO [STDOUT] Starting message: ID:JBM-33
                    06:48:42,785 INFO [STDOUT] Completed message: ID:JBM-10
                    06:48:42,786 INFO [STDOUT] Starting message: ID:JBM-34
                    06:48:42,887 INFO [STDOUT] Completed message: ID:JBM-30
                    06:48:42,889 INFO [STDOUT] Starting message: ID:JBM-35
                    06:48:43,374 INFO [STDOUT] Completed message: ID:JBM-19
                    06:48:43,380 INFO [STDOUT] Starting message: ID:JBM-36
                    06:48:43,551 INFO [STDOUT] Completed message: ID:JBM-15
                    06:48:43,552 INFO [STDOUT] Starting message: ID:JBM-37
                    06:48:43,950 INFO [STDOUT] Completed message: ID:JBM-29
                    06:48:43,950 INFO [STDOUT] Starting message: ID:JBM-38
                    06:48:43,950 INFO [STDOUT] Completed message: ID:JBM-32
                    06:48:43,952 INFO [STDOUT] Starting message: ID:JBM-39
                    06:48:44,660 INFO [STDOUT] Completed message: ID:JBM-35
                    06:48:44,667 INFO [STDOUT] Starting message: ID:JBM-40
                    06:48:44,836 INFO [STDOUT] Completed message: ID:JBM-34
                    06:48:44,839 INFO [STDOUT] Starting message: ID:JBM-41
                    06:48:46,515 INFO [STDOUT] Completed message: ID:JBM-33
                    06:48:46,524 INFO [STDOUT] Starting message: ID:JBM-42
                    06:48:51,955 INFO [STDOUT] Completed message: ID:JBM-16
                    06:48:51,956 INFO [STDOUT] Starting message: ID:JBM-43
                    06:48:54,164 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[46]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:48:54,164 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[44]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:48:54,164 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[45]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:48:54,680 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[47]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:48:54,686 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[48]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:48:54,839 INFO [STDOUT] Completed message: ID:JBM-18
                    06:48:54,840 INFO [STDOUT] Starting message: ID:JBM-49
                    06:48:55,318 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[50]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:48:56,392 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[51]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:48:57,557 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[52]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:49:03,212 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[53]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:49:03,773 INFO [STDOUT] Completed message: ID:JBM-43
                    06:49:03,774 INFO [STDOUT] Starting message: ID:JBM-54
                    06:49:04,302 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[45]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:49:04,300 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[46]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:49:04,938 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[48]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:49:04,938 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[44]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:49:05,502 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[47]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    06:49:06,595 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[50]:PERSISTENT
                    javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                     at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                     at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                     at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                     at $Proxy84.onMessage(Unknown Source)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                     at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
                     at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:804)
                     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                     at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                     at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
                     at java.lang.Thread.run(Thread.java:619)
                    


                    • 7. Re: Jboss hangs after running for a whiel
                      mazz

                      That log you just posted shows that you are having problems with JBoss Messaging. I do not see any issues here with the Transaction Manager. You might want to ask this over in the JBoss Messaging forum - http://www.jboss.com/index.html?module=bb&op=viewforum&f=238

                      BTW: when you see messages like this in your log (you had this in your earlier posts):

                      2008-09-16 21:03:00,116 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fee6:96b7:48ce8118:30f42 invoked while multiple threads active within it.


                      That typically is due to the fact that you started a transaction, but never committed or rolled it back within the alloted transaction timeout. This is usually due to the application taking too much time within a transaction, which causes the TM to abort the tx. Thinking this message is due to a transaction manager problem is a red herring and will have you chasing your tail for a long while looking for something in the wrong place - trust me, I've been down this road :) I would ask your question in the JBoss/Messaging forum and see what they have to say over there.

                      • 8. Re: Jboss hangs after running for a whiel
                        pbaker01

                        Hi Mazz,

                        Thanks for looking at this. It is difficult to determine which forum to post on. I recently posted something similar to this problem on the JBoss Messaging forum and then was told that it was an EJB3 problem and should be posted there. I know that cross posting is frowned on...

                        But thanks for looking at this!!!

                        I'll do a bit more research and see if I can find a home for this problem. PB

                        • 9. Re: Jboss hangs after running for a whiel
                          mazz

                          Hmm... they may be right. Looking at the error:

                          06:48:54,164 ERROR [JmsServerSession] Unexpected error delivering message delegator->JBossMessage[46]:PERSISTENT
                          javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=10000
                           at org.jboss.ejb3.StrictMaxPool.get(StrictMaxPool.java:122)
                           at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
                           at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                           at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
                           at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                           at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                           at $Proxy84.onMessage(Unknown Source)
                          


                          it appears to be happening in the EJB3 MDB stuff.

                          I only mention the other forum because the developers listening to this forum may not be the ones that know the answer - for example, I would not be surprised if the guys who wrote that EJB3 interceptor are listening to the EJB3 forum, but not this forum. It's just targeting your message to the correct audience. Yes, cross-posting the same question on several forums is frowned upon, but, if its an honest mistake or if you are hunting around looking for the people that know the answer, moving from forum to forum is sometimes the only way.

                          • 10. Re: Jboss hangs after running for a whiel
                            imranpariyani

                             

                            "pbaker01" wrote:
                            Hi Mazz,

                            Thanks for looking at this. It is difficult to determine which forum to post on. I recently posted something similar to this problem on the JBoss Messaging forum and then was told that it was an EJB3 problem and should be posted there. I know that cross posting is frowned on...

                            But thanks for looking at this!!!

                            I'll do a bit more research and see if I can find a home for this problem. PB


                            Hi baker

                            well i found the problem after analizing the thread dump on jmx-console .

                            here is the link
                            http://www.jboss.com/index.html?module=bb&op=viewtopic&t=142529

                            Hope that helps

                            Regards

                            imran