I am facing hornetq issue when trying to connect using nettty Connector with latest 2.3.0.CR1 build.
The full stack trace
17:01:43.910 [New I/O worker #2] WARN org.hornetq.core.client - HQ212023: connector.create or connectorFactory.createConnector should never throw an exception, implementation is badly behaved, but we will deal with it anyway.
java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread. at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:342) ~[netty-3.6.2.Final.jar:na] at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:256) ~[netty-3.6.2.Final.jar:na] at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:506) ~[hornetq-core-client-2.3.0.CR1.jar:na] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1225) ~[hornetq-core-client-2.3.0.CR1.jar:na] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1071) ~[hornetq-core-client-2.3.0.CR1.jar:na] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:246) ~[hornetq-core-client-2.3.0.CR1.jar:na] at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:826) ~[hornetq-core-client-2.3.0.CR1.jar:na]
I managed to debug hornetq code and i guess i found the problem. This is related to migration to latest io.netty.3.6.2 migration and NettyConnector code.
The problem with this code in NettyConnector:
remoteDestination = new InetSocketAddress(host, port); InetAddress inetAddress = ((InetSocketAddress) remoteDestination).getAddress(); ChannelFuture future = bootstrap.connect(remoteDestination); future.awaitUninterruptibly(); if (future.isSuccess())
After execution of bootstap.connect actual connection may not occur and next next invocation of awaitUninterruptibly supposed to block current thread and wait until physical connection to occur.
However this blocking call throws IllegalStateException - i guess due to using one shared thread pool executor(VirtualExecutorService).
Currently i am trying to overcome this problem by disabling the deadLocker checker using DefaultChannelFuture.setUseDeadLockChecker( false );
Please confirm that this is a bug? Also that current implementation of initial connect retry mechanism implemented in wrong way. Currently retry logic resides in ServerLocatorImpl class and there is a loop to retry.
However i think that the logic needs to be in NettyConnector itself rather then in ServerLocatorImpl. And instead of waiting awaitUninterruptibly use addListener(), change code to use loop with some sleep between retrying.