Jboss AS 5.1.0 - Error on server restart.

[groovenarula]

We are running into an issue with out installs of JBOSS AS 5.1.0 deploy. Our enviroment consists of a single JBOSS AS 5.1.0 server (not clustered) that's configured against Postgres SQL backend database server. We have about a half a dozen queues defined on the server (we're using the default messaging broker - JBoss Messaging). Once the server is up and running it works just fine until we are forced to restart the server.

But most of the time when we restart the server (e.g. to deploy an update to our application), we run into the following exception :

2011-05-18 16:24:54,805 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/RetailPriceRequestQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

2011-05-18 16:24:54,850 ERROR [org.jboss.messaging.util.ExceptionUtil] (main) Queue[null, name=RetailPriceResponseDLQ] startService

java.io.EOFException

    at java.io.DataInputStream.readFully(DataInputStream.java:180)

    at java.io.DataInputStream.readUTF(DataInputStream.java:592)

    at java.io.DataInputStream.readUTF(DataInputStream.java:547)

    at org.jboss.messaging.util.StreamUtils.readMap(StreamUtils.java:313)

    at org.jboss.messaging.core.impl.JDBCPersistenceManager.bytesToMap(JDBCPersistenceManager.java:2573)

    at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:603)

    at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:523)

    at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:474)

    at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.executeWithRetry(JDBCSupport.java:512)

    at org.jboss.messaging.core.impl.JDBCPersistenceManager.getMessages(JDBCPersistenceManager.java:640)

    at org.jboss.messaging.core.impl.PagingChannelSupport.processReferences(PagingChannelSupport.java:579)

    at org.jboss.messaging.core.impl.PagingChannelSupport.doLoad(PagingChannelSupport.java:511)

    at org.jboss.messaging.core.impl.PagingChannelSupport.load(PagingChannelSupport.java:213)

    at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:108)

    at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)

    at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:322)

    at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

    at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)

    at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

    at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)

    at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

    at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:189)

    at $Proxy38.start(Unknown Source)

    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)

    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)

    at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

    at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

    at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

    at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

    at org.jboss.system.ServiceController.doChange(ServiceController.java:688)

    at org.jboss.system.ServiceController.start(ServiceController.java:460)

    at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)

    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)

    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)

    at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)

    at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)

    at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)

    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)

    at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)

    at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)

    at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)

    at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

    at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)

    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

    at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)

    at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)

    at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

    at org.jboss.Main.boot(Main.java:221)

    at org.jboss.Main$1.run(Main.java:556)

    at java.lang.Thread.run(Thread.java:619)

2011-05-18 16:24:54,861 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/ExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

2011-05-18 16:24:54,965 ERROR [org.jboss.messaging.util.ExceptionUtil] (main) Queue[null, name=RetailFreightResponseQueue] startService

java.io.EOFException

    at java.io.DataInputStream.readFully(DataInputStream.java:180)

    at java.io.DataInputStream.readUTF(DataInputStream.java:592)

    at java.io.DataInputStream.readUTF(DataInputStream.java:547)

    at org.jboss.messaging.util.StreamUtils.readMap(StreamUtils.java:313)

    at org.jboss.messaging.core.impl.JDBCPersistenceManager.bytesToMap(JDBCPersistenceManager.java:2573)

    at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:603)

    at org.jboss.messaging.core.impl.JDBCPersistenceManager$1GetMessageListTX.doTransaction(JDBCPersistenceManager.java:523)

    at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:474)

    at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.executeWithRetry(JDBCSupport.java:512)

    at org.jboss.messaging.core.impl.JDBCPersistenceManager.getMessages(JDBCPersistenceManager.java:640)

    at org.jboss.messaging.core.impl.PagingChannelSupport.processReferences(PagingChannelSupport.java:579)

    at org.jboss.messaging.core.impl.PagingChannelSupport.doLoad(PagingChannelSupport.java:511)

    at org.jboss.messaging.core.impl.PagingChannelSupport.load(PagingChannelSupport.java:213)

    at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:108)

    at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)

    at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:322)

    at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

    at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)

    at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

    at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)

    at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)

    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

    at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:189)

    at $Proxy38.start(Unknown Source)

    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)

    at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)

    at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

    at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

    at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

    at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

    at org.jboss.system.ServiceController.doChange(ServiceController.java:688)

    at org.jboss.system.ServiceController.start(ServiceController.java:460)

    at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)

    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)

    at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)

    at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)

    at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)

    at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)

    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

    at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

    at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)

    at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)

    at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)

    at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)

    at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

    at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)

    at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

    at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

    at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

    at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

    at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)

    at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)

    at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

    at org.jboss.Main.boot(Main.java:221)

    at org.jboss.Main$1.run(Main.java:556)

    at java.lang.Thread.run(Thread.java:619)

2011-05-18 16:24:54,976 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/RetailFreightResponseExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

2011-05-18 16:24:54,983 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/RetailPriceResponseQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

2011-05-18 16:24:54,991 INFO  [org.jboss.jms.server.destination.QueueService] (main) Queue[/queue/RetailPriceResponseExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000

2011-05-18 16:24:55,037 ERROR [org.jboss.messaging.util.ExceptionUtil] (main) Queue[null, name=RetailFreightResponseDLQ] startService

java.io.EOFException

We are using 'shutdown.sh' to stop the server cleanly (we're not killing the process). And the only way we can recover from the exception above it so drop the backend database and recreate it. We tried to analyze the cause of this error, but we got lost in the source code after we tried to trace down the root cause. The major issue with dropping and recreating the database is that in the process we loose all the messages that are waiting to be (re)processed.

Has anyone run into this issue ? So far this issue has not hit us in our production environment.But it will cause a major issue if (it's most likely  'when' based on the number of times we had this problem) it does impact production.

If someone can please shed some light on how to go about tracing down the cause if this issue, it will be much appreciated.

Thanks in advance and thanks for looking,

Gurvinder

[groovenarula]

Just forgot to give more details of the enviroment. The jboss server runs on a VM Guest RHEL 5.5 (x86_64) with Sun JDK 6 (build 1.6.0_18-b07). Our backend is PostgresSQL (also on RHEL 5.5) ver 9.0.2.

[wolfgangknauf]

Hi,

I did some Google, and hopefully this applies to you as well:

https://issues.jboss.org/browse/JBPAPP-4702

https://issues.jboss.org/browse/SOA-554

Those problems both deal with a Sybase database, but you might check whether the database field for the serialized message is too small in your Postgres, too.

Hope this helps

Wolfgang

[groovenarula]

Wolf,

Thanks for the suggestion. As of today morning I believe I might have found the issue. We were using an older version of the jdbc driver. I upgraded the jdbc driver and the server did restart successfully. We're going to monitor it for a few days. But so far it's looking like it the driver version we were using worked fine with Postgres 8.X but does not work with server ver 9.0. I'll mark this as 'Answered' if we don't run into any other problems.

Thanks again for the suggestions.

Gurvinder

[tomeicher]

Confirm: Same error with trusty old pg8 jdbc driver against new 9.03 postgres. upgraded to postgresql-9.0-801.jdbc3.jar, works now. Thanks for the head start :-) Cheers, Tom.