JMS server fail to work with error org.jboss.mq.SpyJMSExcept

[jimyliu]

Our application runs on JBoss 3.2.4. I ran 2 concurrent users test on this application. After 6-10 hours, the JMS server failed to work until I restart it. This machine has multiple ip addresses.

The error mesg like this:
==========================================
org.jboss.mq.SpyJMSException: Cannot authenticate user; - nested throwable: (java.lang.NullPointerException)
at org.jboss.mq.Connection.authenticate(Connection.java:1161)
at org.jboss.mq.Connection.(Connection.java:248)
at org.jboss.mq.Connection.(Connection.java:325)
at org.jboss.mq.SpyConnection.(SpyConnection.java:66)
at org.jboss.mq.SpyConnectionFactory.createConnection(SpyConnectionFactory.java:87)
at org.jboss.mq.SpyConnectionFactory.createTopicConnection(SpyConnectionFactory.java:136)
at com.broadvision.data.message.MsgSessionMgr.openJMSConnection(MsgSessionMgr.java:51)
at com.broadvision.data.message.MsgSender.openJMSConnection(MsgSender.java:41)
at com.broadvision.data.message.MsgSender.sendMessage(MsgSender.java:144)
at com.broadvision.data.cache.CacheManager.sendMessage(CacheManager.java:287)
at com.broadvision.visitor.utils.UserProfileCacheUtil.flushVisitorData(UserProfileCacheUtil.java:214)
at com.broadvision.visitor.ejb.genericVisitorEjb.GenericVisitorBean.flushVisitors(GenericVisitorBean.java:1080)
at java.lang.reflect.Method.invoke(Native Method)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:683)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:315)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:148)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:120)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:331)
at org.jboss.ejb.Container.invoke(Container.java:723)
at java.lang.reflect.Method.invoke(Native Method)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:61)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:53)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:185)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:97)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:90)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:53)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:100)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
at $Proxy120.flushVisitors(Unknown Source)
at com.broadvision.visitor.utils.GenericVisitorClient.flushVisitors(GenericVisitorClient.java:1145)
at com.broadvision.visitor.client.VisitorManager.flushVisitors(VisitorManager.java:892)
at com.broadvision.visitor.client.Visitor.flushCache(Visitor.java:2265)
at com.broadvision.visitor.client.Visitor.isPasswordValid(Visitor.java:430)
at com.broadvision.visitor.client.Visitor.authenticateAgainstBV(Visitor.java:2406)
at com.broadvision.visitor.client.Visitor.authenticate(Visitor.java:2351)
at com.broadvision.web.action.LoginAction.authenticateUser(LoginAction.java:269)
at com.broadvision.web.action.LoginAction.loginUser(LoginAction.java:210)
at com.broadvision.web.action.LoginAction.execute(LoginAction.java:107)

=============================================

I tried this workaround "run.sh -Djboss.bind.address=<ip address>...
Still fail.
Another finding: When I run same application on another server which has only one ip address, there is no any problem under the same test.

Any suggestion to trace down this problem.

Thanks.

Jimy

[jimyliu]

More info:
- After disabling other network interfaces and just keep one ip address for this machine. The problem did not happen again.
- my server is run on Solaris.
Is this a known issue for JBoss 3.2.4 with multiplte network interfaces ?

Jimy

[genman]

I can't confirm, but there were some fixes WRT server bind address in the JMS area recently. Can you test with a later version (3.2.6) or one built from CVS?

[jimyliu]

Hi genman:

Do you know where I can find those fixes detail that I can have a look ?
I will test 3.2.6 .
Thanks.

Jimy

[jimyliu]

Hi:

After I enabled log4j on org.jboss.mq, here is the error mesg in the log :
=============================================
13:48:23,893 DEBUG [JMSDestinationManager] Exception occured, rolling back transaction:
org.jboss.mq.SpyJMSException: Could not commit tx: 72180; - nested throwable: (java.sql.SQLException: File input/output errorerror o
rg.hsqldb.HsqlException: File input/output error SaveRow java.io.InterruptedIOException closing file - file /net/sun-mrhankey/sun-JBoss/server/framework1/data/hypersonic/localDB.data in statement [DELETE FROM JMS_TRANSACTIONS WHERE TX
ID = ?])
at org.jboss.mq.pm.jdbc2.PersistenceManager.commitPersistentTx(PersistenceManager.java:522)
at org.jboss.mq.pm.Tx.commit(Tx.java:149)
at org.jboss.mq.pm.TxManager.commitTx(TxManager.java:89)
at org.jboss.mq.server.JMSDestinationManager.transact(JMSDestinationManager.java:440)
at org.jboss.mq.server.JMSServerInterceptorSupport.transact(JMSServerInterceptorSupport.java:186)
at org.jboss.mq.security.ServerSecurityInterceptor.transact(ServerSecurityInterceptor.java:182)
at org.jboss.mq.server.TracingInterceptor.transact(TracingInterceptor.java:447)
at org.jboss.mq.server.JMSServerInvoker.transact(JMSServerInvoker.java:186)
at org.jboss.mq.il.jvm.JVMServerIL.transact(JVMServerIL.java:327)
at org.jboss.mq.Connection.send(Connection.java:1206)
at org.jboss.mq.SpyXAResourceManager.commit(SpyXAResourceManager.java:150)
at org.jboss.mq.SpyXAResource.commit(SpyXAResource.java:84)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:327)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:871)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:159)
at org.jboss.mq.SpySession.run(SpySession.java:347)
at org.jboss.jms.asf.StdServerSession.run0(StdServerSession.java:200)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
at java.lang.Thread.run(Thread.java:479)
+ nested throwable:
=========================================

And the hypersonic db file's modified time is just same time when the exception happened.
Is the hypersonic DB failed to be connected or other reason ?

Jimy