-
1. Re: Timed out waiting for response when sending packet
clebert.suconic Aug 25, 2010 2:55 PM (in response to bizz)A Commit is supposed to be blocked.
How many threads are you doing and how many transactions?
Maybe your server is out of resources (CPU, maybe) and you may need to configure producer flow control properly. (Credits).
-
2. Re: Timed out waiting for response when sending packet
bizz Aug 25, 2010 3:19 PM (in response to clebert.suconic)I have 10 threads performing send on a producer while 1 thread is committing the session periodicly.
I have about 30 sessions open on 10 different queues, and each session has only 1 producer on it.
I can change the flow and work with a non-transactional session, but I was trying to understand what causes the time out.
-
3. Re: Timed out waiting for response when sending packet
clebert.suconic Aug 25, 2010 3:44 PM (in response to bizz)Use less credits in your producer. You are sending more messages than your server can handle (Network, memory, CPU, disk.. etc)
This will give you more information : http://hornetq.sourceforge.net/docs/hornetq-2.1.2.Final/user-manual/en/html/flow-control.html#d0e3784
-
4. Re: Timed out waiting for response when sending packet
bizz Aug 26, 2010 2:49 AM (in response to clebert.suconic)If, in fact, this is the case, then it won't matter if I will work with a non-transactional session.
Isn't that right?
-
5. Re: Timed out waiting for response when sending packet
leosbitto Aug 26, 2010 4:27 AM (in response to bizz)You should not share the jms session and its producers between threads during the transaction. Only one thread should use the producers to send the messages and then commit the session.
-
6. Re: Timed out waiting for response when sending packet
timfox Aug 26, 2010 5:30 AM (in response to leosbitto)Leos Bitto wrote:
You should not share the jms session and its producers between threads during the transaction. Only one thread should use the producers to send the messages and then commit the session.
+1
Leos is spot on.
If you allow multiple threads to use a session concurrently, all sorts of strange things might start happening.
-
7. Re: Timed out waiting for response when sending packet
bizz Aug 26, 2010 5:39 AM (in response to timfox)Ok, I'll use all of your useful remarks and update you.
Thanks
-
8. Re: Timed out waiting for response when sending packet
bizz Aug 26, 2010 10:14 AM (in response to bizz)Hi again
I changed the producer window size and made sure there is only one thread working on each session.
I still have the same exception when committing the session after consuming a message.
javax.jms.JMSException: Timed out waiting for response when sending packet 43
at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
at org.hornetq.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:514)
at org.hornetq.core.client.impl.DelegatingSession.commit(DelegatingSession.java:156)
at org.hornetq.jms.client.HornetQSession.commit(HornetQSession.java:229)(Is it a coincidence it always happens on the 43rd packet?)
I use transaction so I can manually abort a message using the rollback on a session.
Is there a way to do that on a non-transactional session?
Any suggestions?
Thanks
-
9. Re: Timed out waiting for response when sending packet
clebert.suconic Aug 26, 2010 10:24 AM (in response to bizz)> (Is it a coincidence it always happens on the 43rd packet?)
43 is the ID of the package sent on our protocol. On that case, the commit
We do transactions the time. You must be doing something obviously wrong. Since it seems you're not even stressing your server to get this message. Look at the examples.. there are several cases where we do this.
If you think there's a bug, you should share a testcase.
-
10. Re: Timed out waiting for response when sending packet
bizz Aug 31, 2010 4:42 AM (in response to clebert.suconic)Hi,
Still trying to understand why do I have this exception. Trying to understand what I'm doing wrong and maybe create a testcase for you.
I do have a question though. It was said before that I should not use a producer in more then one thread. Should the same thread be the one creating the session?
Can I create the session in one thread and use it in another? (only one thread uses it after creation).
Thanks again
Levona
-
11. Re: Timed out waiting for response when sending packet
leosbitto Aug 31, 2010 6:09 AM (in response to bizz)Levona Hershtik wrote:
Can I create the session in one thread and use it in another? (only one thread uses it after creation).
I guess that this should be fine if you ensure that creation of the session in one thread happens-before using the session in another thread. Since you use threads, you know what happens-before means in Java, right?
-
12. Re: Timed out waiting for response when sending packet
bizz Aug 31, 2010 11:13 AM (in response to leosbitto)Hi again,
I seem to have found the solution for my problem.
To understand the problem I added ExceptionListener on the connection (Which I should have used before). It seems like after a while running I get this exception:
javax.jms.JMSException: HornetQException[errorCode=2 message=Channel disconnected]
at org.hornetq.jms.client.HornetQConnection$JMSFailureListener.connectionFailed(HornetQConnection.java:603)
at org.hornetq.core.client.impl.FailoverManagerImpl.callFailureListeners(FailoverManagerImpl.java:768)
at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrReconnect(FailoverManagerImpl.java:730)
at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnectionFailure(FailoverManagerImpl.java:580)
at org.hornetq.core.client.impl.FailoverManagerImpl.connectionDestroyed(FailoverManagerImpl.java:291)
at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:679)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: HornetQException[errorCode=2 message=Channel disconnected]So I looked it up and came across this post: https://jira.jboss.org/browse/HORNETQ-433
I tuned the connection-ttl, client-failure-check-period and producer-window-size (as Clebert recommended)
It seems to be helping as long as I make sure the client and server are not overwhelmed.
I also added re-attachment configuration to handle the closed connection, and a graceful handling of re-attachment failure. (Using ExceptionListener)
Thanks again for all the responses,
I did learn a lot from this conversation.
-
13. Re: Timed out waiting for response when sending packet
clebert.suconic Aug 31, 2010 11:29 AM (in response to bizz)You probably have an unreliable IP. We sure give a lot of stress to the server without any issues.
You should probably share how you're creating your sessions.. A testcase.
that would at least give us an idea on what you're doing wrong.
or .. if you're not doing anything wrong.. maybe you found a bug pattern. (allthough such simple testcase would have showed up early already)
-
14. Re: Timed out waiting for response when sending packet
bizz Aug 31, 2010 12:30 PM (in response to clebert.suconic)Hi Clebert,
I'll try to explain my design:
I have workers, which consume messages. These workers use transactions each creates it's own session as follows:
Session session = connection.createSession(true, Session.SESSION_TRANSACTED);
They use rollback and commit on the session according to the work they are performing. All work on a session is done in a single thread.
I also have producers which collect messages to be sent to the server (one for each queue) and sends them.
Each producer also has its own session created as follows:
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Here also, all work on a session is done in a single thread (Not like I did at first)
A single connection is used to create all the sessions for the process.
I'm working with 2.1.2
My design is quite complex but if you think this might be a bug I can try to reproduce this on a simpler testcase.