-
30. Re: What is the expected performance of the STOMP protocol?
mrbeagle Oct 8, 2010 8:03 PM (in response to clebert.suconic)since I'm dequeing from stomp, should there still be a concept of window size?
Here are my test scripts
enqueue from java
php deqeue
I enqueue a few thousand rows from java using jms, then run a script in php to dequeue. After between 700-1000 dequeues it locks up the php script and the hornet server itself becomes unresponsive. If I re-run my JMS enqueue test case during this lock up the enqueues time out with the exception I pasted above.
-
31. Re: What is the expected performance of the STOMP protocol?
clebert.suconic Oct 8, 2010 9:38 PM (in response to mrbeagle)I will have to take a look on early next week.
I will have to look at the changes Tim made over for this.
-
32. Re: What is the expected performance of the STOMP protocol?
timfox Oct 9, 2010 5:18 AM (in response to clebert.suconic)Works fine for me using TRUNK. Here are the last few lines of output from dequeue on my laptop:
MSG: This is a test message: 2984
MSG: This is a test message: 2985
MSG: This is a test message: 2986
MSG: This is a test message: 2987
MSG: This is a test message: 2988
MSG: This is a test message: 2989
MSG: This is a test message: 2990
MSG: This is a test message: 2991
MSG: This is a test message: 2992
MSG: This is a test message: 2993
MSG: This is a test message: 2994
MSG: This is a test message: 2995
MSG: This is a test message: 2996
MSG: This is a test message: 2997
MSG: This is a test message: 2998
MSG: This is a test message: 2999
3000 messages dequeuedMSG: This is a test message: 3000
1.1247551441193 secondsI ran your exact JMS client followed by your php script.
-
33. Re: What is the expected performance of the STOMP protocol?
timfox Oct 9, 2010 5:22 AM (in response to timfox)This doesn't follow. The stack trace you posted shows the client waiting in a call to create consumer:
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java -Xms2048m -Xmx2048m -Dfile.encoding=UTF-8 -classpath /Applications/IntelliJ IDEA 9.0.3.app/lib/idea_rt.jar:/Applications/IntelliJ IDEA 9.0.3.app/plugins/junit/lib/junit-rt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/A/Resources/Deploy.bundle/Contents/Resources/Java/deploy.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/dt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/javaws.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/jce.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/management-agent.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/plugin.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/sa-jdi.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/alt-rt.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/charsets.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/classes.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/jconsole.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/jsse.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/ui.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/apple_provider.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/dnsns.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/localedata.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/sunjce_provider.jar:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/ext/sunpkcs11.jar:/Users/jim/Code/InsightsRaven/viral/target/test-classes:/Users/jim/Code/InsightsRaven/viral/target/classes:/Users/jim/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/Users/jim/.m2/repository/org/jsoup/jsoup/1.2.2/jsoup-1.2.2.jar:/Users/jim/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar:/Users/jim/.m2/repository/com/jolbox/bonecp/0.7.0-rc3/bonecp-0.7.0-rc3.jar:/Users/jim/.m2/repository/com/google/collections/google-collections/1.0/google-collections-1.0.jar:/Users/jim/.m2/repository/org/slf4j/slf4j-api/1.5.10/slf4j-api-1.5.10.jar:/Users/jim/Code/InsightsRaven/insights-utils/target/classes:/Users/jim/.m2/repository/org/apache/thrift/libthrift/0.2.0/libthrift-0.2.0.jar:/Users/jim/.m2/repository/org/apache/httpcomponents/httpclient/4.0.1/httpclient-4.0.1.jar:/Users/jim/.m2/repository/org/apache/httpcomponents/httpcore/4.0.1/httpcore-4.0.1.jar:/Users/jim/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/jim/.m2/repository/commons-codec/commons-codec/1.3/commons-codec-1.3.jar:/Users/jim/.m2/repository/com/gravity/utilities/1.0-SNAPSHOT/utilities-1.0-SNAPSHOT.jar:/Users/jim/.m2/repository/org/slf4j/slf4j-log4j12/1.5.10/slf4j-log4j12-1.5.10.jar:/Users/jim/.m2/repository/log4j/log4j/1.2.14/log4j-1.2.14.jar:/Users/jim/.m2/repository/org/jdom/jdom/1.1/jdom-1.1.jar:/Users/jim/.m2/repository/com/google/code/gson/gson/1.4/gson-1.4.jar:/Users/jim/.m2/repository/org/jredis/jredis/1.0-rc1/jredis-1.0-rc1.jar:/Users/jim/.m2/repository/org/mongodb/mongo-java-driver/2.0/mongo-java-driver-2.0.jar:/Users/jim/.m2/repository/mysql/mysql-connector-java/5.1.12/mysql-connector-java-5.1.12.jar:/Users/jim/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-jms-client/2.1.2.Final/hornetq-jms-client-2.1.2.Final.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-core-client/2.1.2.Final/hornetq-core-client-2.1.2.Final.jar:/Users/jim/.m2/repository/org/hornetq/hornetq-transports/2.0.0.GA/hornetq-transports-2.0.0.GA.jar:/Users/jim/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 com.gravity.insights.viral.HornetTest,testHornetjavax.jms.JMSException: Timed out waiting for response when sending packet 40at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateConsumer(ClientSessionImpl.java:1556)javax.jms.JMSException: Timed out waiting for response when sending packet 40
at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateConsumer(ClientSessionImpl.java:1556)
at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:447)
at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:413)
at org.hornetq.core.client.impl.DelegatingSession.createConsumer(DelegatingSession.java:187)
at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:531)
However, the JMS test client you posted doesn't make any calls to create consumer. So you must be running some other client.
Also, FWIW any changes to flow control are currently disabled in TRUNK anyway.
-
34. Re: What is the expected performance of the STOMP protocol?
mrbeagle Oct 9, 2010 4:09 PM (in response to timfox)Below is notes for 3 scenarios in which we were able to put the serverinto a permanent bad state (multiple threads deadlocked and server nolonger operating correctly--server does not shut down unless a kill signalis sent) with a combination of PHP via Stomp and Java via the HornetQ JMSclient. The problem seems to especially happen while Java is enqueuingquickly and PHP client is stopped and restarted simultaneously (an eventthat we need to support frequently in our environment).PHP client here: http://pastebin.com/Midk7nDCJava client here: http://pastebin.com/8wXaQabXSCENARIO 1:Java client enqueues 300k items, php client dequeus 3000 items and thenhangs. java regularly sends a message count request. At the same timethat php hangs, message count starts to send back an error instead of thecount. On the server side, get the following exception dumped to theconsole:Meanwhile another thread is waiting on a commit:SCENARIO 2: Java Client enqueuing, PHP client dequeuing. Everything goeswell until the PHP client is stopped and restarted. At that point the PHPclient hangs and eventually the connection gets shut down. Java clientstops enqueuing and gets timeout exceptions when it tries to retrieve amessage count. Deadlock is detected below on the server side. If Idisconnect all the clients the deadlock below persists indefinitely. If Ireconnect a fresh Java client, I can enqueue, but neither PHP via Stompnor Java via JMS can dequeue until I restart the server. Each time I tryto add a new Consumer a thread is fired up that looks just like the threadbelow that's waiting for a consumer.Java stack information for the threads listed above:===================================================SCENARIO 3: Tried using NIO server. Problem harder to repro with NIOserver. But still possible. Java enqueuing and dequeuing. PHPdequeueing. Everything runs fine, but then stop and restart PHP. PHPgets nothing, Java can no longer dequeue and gets periodic timeoutnotifications. Below deadlock permanently exists on server until shutdown:Java stack information for the threads listed above:===================================================Below is notes for 3 scenarios in which we were able to put the server
into a permanent bad state (multiple threads deadlocked and server no
longer operating correctly--server does not shut down unless a kill signal
is sent) with a combination of PHP via Stomp and Java via the HornetQ JMS
client. The problem seems to especially happen while Java is enqueuing
quickly and PHP client is stopped and restarted simultaneously (an event
that we need to support frequently in our environment).
PHP client here: http://pastebin.com/Midk7nDC
Java client here: http://pastebin.com/8wXaQabX
SCENARIO 1:
Java client enqueues 300k items, php client dequeus 3000 items and then
hangs. java regularly sends a message count request. At the same time
that php hangs, message count starts to send back an error instead of the
count. On the server side, get the following exception dumped to the
console:
Meanwhile another thread is waiting on a commit:
SCENARIO 2: Java Client enqueuing, PHP client dequeuing. Everything goes
well until the PHP client is stopped and restarted. At that point the PHP
client hangs and eventually the connection gets shut down. Java client
stops enqueuing and gets timeout exceptions when it tries to retrieve a
message count. Deadlock is detected below on the server side. If I
disconnect all the clients the deadlock below persists indefinitely. If I
reconnect a fresh Java client, I can enqueue, but neither PHP via Stomp
nor Java via JMS can dequeue until I restart the server. Each time I try
to add a new Consumer a thread is fired up that looks just like the thread
below that's waiting for a consumer.
Java stack information for the threads listed above:
===================================================
SCENARIO 3: Tried using NIO server. Problem harder to repro with NIO
server. But still possible. Java enqueuing and dequeuing. PHP
dequeueing. Everything runs fine, but then stop and restart PHP. PHP
gets nothing, Java can no longer dequeue and gets periodic timeout
notifications. Below deadlock permanently exists on server until shutdown:
Java stack information for the threads listed above:
===================================================
-
35. Re: What is the expected performance of the STOMP protocol?
timfox Oct 10, 2010 4:31 AM (in response to mrbeagle)You didn't mention you were calling getMessageCount previously. What happens when you don't?
This thread is getting very confused.
-
36. Re: What is the expected performance of the STOMP protocol?
timfox Oct 10, 2010 4:33 AM (in response to timfox)If these are different issues, start different threads or it's almost impossible to figure out what is going on, and what the latest set of actions you are doing is.
-
37. Re: What is the expected performance of the STOMP protocol?
timfox Oct 10, 2010 4:38 AM (in response to timfox)There is a known issue with calling getMessageCount when server is under load, I'm just trying to eliminate that as a possibility.
Anyhow, I'm out now, I'm sure someone can take a look next week.
-
38. Re: What is the expected performance of the STOMP protocol?
clebert.suconic Oct 11, 2010 11:18 AM (in response to timfox)@Jimmy if you could please do as Tim said (removing the excessive calls to getCount()) and we will take a look depending on what you tell us.