1 2 3 4 Previous Next 53 Replies Latest reply: Oct 22, 2009 6:22 AM by Ross Nicholson RSS

    Strange netty error when sending a lot of messages

    Ross Nicholson Apprentice

      Ok, I'm back on the Beta5 release. We are seeing an unusual error with netty. At the time this error occurs we will have 10s of thousands of tcp connections from localhost to localhost all in a TIME_WAIT state (using "netstat -an" to see these).

      tcp 0 0 127.0.0.1:33727 127.0.0.1:5445 TIME_WAIT
      tcp 0 0 127.0.0.1:33721 127.0.0.1:5445 TIME_WAIT
      tcp 0 0 127.0.0.1:33725 127.0.0.1:5445 TIME_WAIT
      tcp 0 0 127.0.0.1:33724 127.0.0.1:5445 TIME_WAIT
      tcp 0 0 127.0.0.1:33722 127.0.0.1:5445 TIME_WAIT
      tcp 0 0 127.0.0.1:33728 127.0.0.1:5445 TIME_WAIT
      tcp 0 0 127.0.0.1:33726 127.0.0.1:5445 TIME_WAIT
      tcp 0 0 127.0.0.1:33723 127.0.0.1:5445 TIME_WAIT
      


      We are sending 65K messages one after the other. Each message is dependent on the last one so they must be sent in sequence. Messages are sent via a core bridge using a message group.

      What we think is happening is that netty cannot open another localhost connection as there are too many in the TIME_WAIT state.

      The first error we get is a JMS exception on the MDB which should be consuming the messages.

      2009-10-09 16:22:26,253 ERROR [STDERR] (pool-19-thread-1) Problem with creating connection or session.
      2009-10-09 16:22:26,253 DEBUG [org.jboss.ejb3.interceptors.aop.InterceptorSequencer] (Thread-340 (group:HornetQ-client-global-threads-21795015)) aroundInvoke [advisedMethod=public void com.paddypower.phase.engine.bean.mdb.EngineMDB.onMessage(javax.jms.Message), unadvisedMethod=public void com.paddypower.phase.engine.bean.mdb.EngineMDB.onMessage(javax.jms.Message), metadata=null, targetObject=com.paddypower.phase.engine.bean.mdb.EngineMDB@1c572d0, arguments=[Ljava.lang.Object;@534e5d]
      2009-10-09 16:22:26,253 ERROR [STDERR] (pool-19-thread-1) Messages processing will be not started. Fix queue problem and redeploy application.
      


      Shortly after we get a lot of errors like the following:

      2009-10-09 16:22:26,371 ERROR [STDERR] (pool-19-thread-15) Problem with creating connection or session.
      2009-10-09 16:22:26,384 ERROR [STDERR] (pool-19-thread-15) Messages processing will be not started. Fix queue problem and redeploy application.
      2009-10-09 16:22:26,384 INFO [STDOUT] (pool-19-thread-15) SentToHandler
      2009-10-09 16:22:26,385 SEVERE [org.hornetq.integration.transports.netty.NettyConnector] (pool-19-thread-9) Failed to create netty connection
      java.net.BindException: Cannot assign requested address
       at sun.nio.ch.Net.connect(Native Method)
       at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:145)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:109)
       at org.jboss.netty.channel.Channels.connect(Channels.java:762)
       at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:195)
       at org.jboss.netty.bootstrap.ClientBootstrap$Connector.channelOpen(ClientBootstrap.java:287)
       at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:197)
       at org.jboss.netty.channel.socket.nio.NioClientSocketChannel.<init>(NioClientSocketChannel.java:88)
       at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.newChannel(NioClientSocketChannelFactory.java:146)
       at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.newChannel(NioClientSocketChannelFactory.java:93)
       at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:235)
       at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:199)
       at org.hornetq.integration.transports.netty.NettyConnector.createConnection(NettyConnector.java:363)
       at org.hornetq.core.client.impl.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:903)
       at org.hornetq.core.client.impl.ConnectionManagerImpl.getConnectionWithRetry(ConnectionManagerImpl.java:783)
       at org.hornetq.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:280)
       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:976)
       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:721)
       at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:710)
       at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:729)
       at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:307)
       at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:302)
       at com.paddypower.phase.engine.core.SingleDestMessageSenderTask.run(SingleDestMessageSenderTask.java:54)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
       at java.lang.Thread.run(Unknown Source)
      
      


      Once more, any ideas?

        1 2 3 4 Previous Next