Serious idle problem with Core Bridges, it's back

[rnicholson10]

I had a similar problem to this on JBM 1.4. After a certain amount of time it is no longer possible to send message to a queue. I will re-run the test this evening but this time undeploy the bridge to see does it have any effect.

The logging seems a lot more informative than the previous version. Hopefully we can find out what is going on this time.

My original JBM1.4 post is here: http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4237532#4237532

Here is the error:

2009-09-12 15:00:29,877 SEVERE [org.hornetq.core.server.impl.HornetQPacketHandler] (Thread-22 (group:HornetQ-server-threads32863724-1105161)) Failed to create session
java.lang.IllegalStateException: BaseClassLoader@6352f8{vfszip:/opt/jboss-5.1.0.GA-jdk6/server/default-with-hornetq/deploy/phase-test.ear/phase-test.war/} classLoader is not connected to a domain (probably undeployed?) for class org.jboss.security.auth.spi.UsersRolesLoginModule
 at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:793)
 at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:300)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
 at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:247)
 at javax.security.auth.login.LoginContext.invoke(LoginContext.java:731)
 at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
 at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
 at javax.security.auth.login.LoginContext.login(LoginContext.java:579)
 at org.jboss.security.plugins.auth.JaasSecurityManagerBase.defaultLogin(JaasSecurityManagerBase.java:552)
 at org.jboss.security.plugins.auth.JaasSecurityManagerBase.authenticate(JaasSecurityManagerBase.java:486)
 at org.jboss.security.plugins.auth.JaasSecurityManagerBase.isValid(JaasSecurityManagerBase.java:365)
 at org.jboss.security.plugins.JaasSecurityManager.isValid(JaasSecurityManager.java:160)
 at org.hornetq.integration.security.JBossASSecurityManager.validateUser(JBossASSecurityManager.java:79)
 at org.hornetq.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:124)
 at org.hornetq.core.server.impl.HornetQServerImpl.doCreateSession(HornetQServerImpl.java:1411)
 at org.hornetq.core.server.impl.HornetQServerImpl.createSession(HornetQServerImpl.java:563)
 at org.hornetq.core.server.impl.HornetQPacketHandler.doHandleCreateSession(HornetQPacketHandler.java:201)
 at org.hornetq.core.server.impl.HornetQPacketHandler.handleCreateSession(HornetQPacketHandler.java:243)
 at org.hornetq.core.server.impl.HornetQPacketHandler.handlePacket(HornetQPacketHandler.java:111)
 at org.hornetq.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:606)
 at org.hornetq.core.remoting.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:397)
 at org.hornetq.core.remoting.impl.RemotingConnectionImpl.access$000(RemotingConnectionImpl.java:41)
 at org.hornetq.core.remoting.impl.RemotingConnectionImpl$1.run(RemotingConnectionImpl.java:357)
 at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)
2009-09-12 15:00:29,880 ERROR [com.paddypower.phase.input.core.CoreMessageSender] (http-0.0.0.0-8080-1) Could not send message from input sender at source: 1000
javax.jms.JMSException
 at org.hornetq.core.remoting.impl.wireformat.HornetQExceptionMessage.decodeBody(HornetQExceptionMessage.java:79)
 at org.hornetq.core.remoting.impl.wireformat.PacketImpl.decode(PacketImpl.java:214)
 at org.hornetq.core.remoting.impl.PacketDecoder.decode(PacketDecoder.java:435)
 at org.hornetq.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:343)
 at org.hornetq.core.client.impl.ConnectionManagerImpl$DelegatingBufferHandler.bufferReceived(ConnectionManagerImpl.java:1142)
 at org.hornetq.integration.transports.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:64)
 at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
 at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
 at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
 at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:324)
 at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:306)
 at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:223)
 at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:87)
 at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
 at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
 at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
 at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:330)
 at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
 at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:203)
 at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:113)
 at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
 at org.hornetq.integration.transports.netty.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:174)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)
Caused by: HornetQException[errorCode=0 message=null]
 ... 27 more

[/url]

[timfox]

Ross - can you file a JIRA? Then someone will take a look. Thx

[timfox]

Can you also include full instructions on how to replicate this on the JIRA?

[rnicholson10]

You will be glad to hear that I have traced this problem down to faulty hardware. After two months of IT telling me the hardware was good I decided to rule it out and ran my own test across some desktops. Works perfectly, no faults.

Should I do something with the JIRA or should I leave that to you guys?

Cheers,

R.

[timfox]

Good news :)

I'll close the JIRA

[rnicholson10]

The only thing I would be concerned about is that even with the fault, surely the message should still be put on the local queue? If the bridge is having problems that should not effect a local send?

Or am I way off here?

Take what happened as a failure scenario. In this case the message was never placed on the queue. Why should a network fault stop this from happening (still tracing this to controller/switch socket etc.)?

I think it warrants some thought...

[timfox]

Hard to say without replicating the issue.

The error looks like something in the bowels of JBoss application server - it's not HornetQ throwing that exception.

[rnicholson10]

Once the IT lads here have figured out exactly what the fault was I'll post it here. If it's something that can be reproduced or emulated then maybe we can look into it. Otherwise we can leave it be.