4 Replies Latest reply on Jul 26, 2011 9:17 AM by Tom Eicher

    Jboss AS 5.1.0 - Error on server restart.

    Gurvinderpal Narula Newbie

      We are running into an issue with out installs of JBOSS AS 5.1.0 deploy. Our enviroment consists of a single JBOSS AS 5.1.0 server (not clustered) that's configured against Postgres SQL backend database server. We have about a half a dozen queues defined on the server (we're using the default messaging broker - JBoss Messaging). Once the server is up and running it works just fine until we are forced to restart the server.

       

      But most of the time when we restart the server (e.g. to deploy an update to our application), we run into the following exception :

       

      2011-05-18 16:24:54,805 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/RetailPriceRequestQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

      2011-05-18 16:24:54,850 ERROR [org.jboss.messaging.util.ExceptionUtil] (main) Queue[null, name=RetailPriceResponseDLQ] startService

      java.io.EOFException

          at java.io.DataInputStream.readFully(DataInputStream.java:180)

          at java.io.DataInputStream.readUTF(DataInputStream.java:592)

          at java.io.DataInputStream.readUTF(DataInputStream.java:547)

          at org.jboss.messaging.util.StreamUtils.readMap(StreamUtils.java:313)

          at org.jboss.messaging.core.impl.JDBCPersistenceManager.bytesToMap(JDBCPersistenceManager.java:2573)

          at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:603)

          at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:523)

          at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:474)

          at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.executeWithRetry(JDBCSupport.java:512)

          at org.jboss.messaging.core.impl.JDBCPersistenceManager.getMessages(JDBCPersistenceManager.java:640)

          at org.jboss.messaging.core.impl.PagingChannelSupport.processReferences(PagingChannelSupport.java:579)

          at org.jboss.messaging.core.impl.PagingChannelSupport.doLoad(PagingChannelSupport.java:511)

          at org.jboss.messaging.core.impl.PagingChannelSupport.load(PagingChannelSupport.java:213)

          at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:108)

          at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)

          at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:322)

          at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

          at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)

          at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

          at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)

          at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

          at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:189)

          at $Proxy38.start(Unknown Source)

          at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)

          at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)

          at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

          at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

          at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

          at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

          at org.jboss.system.ServiceController.doChange(ServiceController.java:688)

          at org.jboss.system.ServiceController.start(ServiceController.java:460)

          at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)

          at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)

          at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)

          at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)

          at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)

          at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)

          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)

          at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)

          at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)

          at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)

          at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

          at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)

          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

          at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)

          at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)

          at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

          at org.jboss.Main.boot(Main.java:221)

          at org.jboss.Main$1.run(Main.java:556)

          at java.lang.Thread.run(Thread.java:619)

      2011-05-18 16:24:54,861 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/ExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

      2011-05-18 16:24:54,965 ERROR [org.jboss.messaging.util.ExceptionUtil] (main) Queue[null, name=RetailFreightResponseQueue] startService

      java.io.EOFException

          at java.io.DataInputStream.readFully(DataInputStream.java:180)

          at java.io.DataInputStream.readUTF(DataInputStream.java:592)

          at java.io.DataInputStream.readUTF(DataInputStream.java:547)

          at org.jboss.messaging.util.StreamUtils.readMap(StreamUtils.java:313)

          at org.jboss.messaging.core.impl.JDBCPersistenceManager.bytesToMap(JDBCPersistenceManager.java:2573)

          at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:603)

          at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:523)

          at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:474)

          at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.executeWithRetry(JDBCSupport.java:512)

          at org.jboss.messaging.core.impl.JDBCPersistenceManager.getMessages(JDBCPersistenceManager.java:640)

          at org.jboss.messaging.core.impl.PagingChannelSupport.processReferences(PagingChannelSupport.java:579)

          at org.jboss.messaging.core.impl.PagingChannelSupport.doLoad(PagingChannelSupport.java:511)

          at org.jboss.messaging.core.impl.PagingChannelSupport.load(PagingChannelSupport.java:213)

          at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:108)

          at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)

          at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:322)

          at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

          at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)

          at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

          at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)

          at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

          at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:189)

          at $Proxy38.start(Unknown Source)

          at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)

          at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)

          at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

          at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

          at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

          at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

          at org.jboss.system.ServiceController.doChange(ServiceController.java:688)

          at org.jboss.system.ServiceController.start(ServiceController.java:460)

          at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)

          at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)

          at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)

          at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)

          at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)

          at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)

          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

          at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)

          at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)

          at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)

          at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)

          at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

          at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)

          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

          at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)

          at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)

          at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

          at org.jboss.Main.boot(Main.java:221)

          at org.jboss.Main$1.run(Main.java:556)

          at java.lang.Thread.run(Thread.java:619)

      2011-05-18 16:24:54,976 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/RetailFreightResponseExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

      2011-05-18 16:24:54,983 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/RetailPriceResponseQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

      2011-05-18 16:24:54,991 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/RetailPriceResponseExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

      2011-05-18 16:24:55,037 ERROR [org.jboss.messaging.util.ExceptionUtil] (main) Queue[null, name=RetailFreightResponseDLQ] startService

      java.io.EOFException

       

      We are using 'shutdown.sh' to stop the server cleanly (we're not killing the process). And the only way we can recover from the exception above it so drop the backend database and recreate it. We tried to analyze the cause of this error, but we got lost in the source code after we tried to trace down the root cause. The major issue with dropping and recreating the database is that in the process we loose all the messages that are waiting to be (re)processed.

       

      Has anyone run into this issue ? So far this issue has not hit us in our production environment.But it will cause a major issue if (it's most likely  'when' based on the number of times we had this problem) it does impact production.

       

      If someone can please shed some light on how to go about tracing down the cause if this issue, it will be much appreciated.

       

      Thanks in advance and thanks for looking,

       

      Gurvinder