I suspect that the problem might be in network layer setup on different OS. What kind of replication in JGroups do you use UDP/TCP? I would suggest running http://www.jgroups.org/manual-3.x/html/user-advanced.html#PerformanceTests on two different setups to determine if there is slowdown related to network/OS. Use JGroups 3.1 and follow directions to diagnose a possible cause.
thank you for your fast reply. I also sustected that the problem might be somewhere there. First in response of your queston, we are using the default configurations that are coming with the JBoss 6.1. I did as you told me with only difference that I used the tests comming with jgroups 2.11 because this is the version supplied with jboss 6 AS. In the test property files there were 3 tests configuration option by transport type:
The JGoupsTransport ran fine on both machines. But it was not the same with the UdpTransport. On my developer machine it was able to recieve response for all sent messages , but on the performance server this was not the case actualy the test never completed because there was no response after the 400 000 messages. Here is the output of the failed test, and I am also attaching it as a file:
log_interval: 100000 msg_size: 1000 config: jconfig.txt transport: org.jgroups.tests.perf.transports.UdpTransport num_senders: 1 sender: true props: udp.xml num_msgs: 1000000 num_members: 1
JGroups version: 2.11.0.GA
-- local_addr is 172.27.202.141:53702
-- waiting for 1 members to join
-- 172.27.202.141:7500 joined
-- READY (1 acks)
-- sending 1000000 1KB messages
++ sent 100000
++ sent 200000
-- received 100000 messages (2192 ms, 45620.44 msgs/sec, 45.62MB/sec)
++ sent 300000
++ sent 400000
-- received 200000 messages (1871 ms, 53447.35 msgs/sec, 53.45MB/sec)
++ sent 500000
++ sent 600000
-- received 300000 messages (2029 ms, 49285.36 msgs/sec, 49.29MB/sec)
++ sent 700000
++ sent 800000
++ sent 900000
-- received 400000 messages (1986 ms, 50352.47 msgs/sec, 50.35MB/sec)
++ sent 1000000
And it never ends.
At the same time if I use the JGroupsTransport everything is OK. But what transport is used in JBoss 6 ? and is it related? Any further ideas based on this output?
sender.log.zip 573 bytes
Default in AS 6.1 is udp stack if I recall correctly! I would say make sure that you are binding to right interfaces (look at JBoss AS parameter options), look at JGroups configuration files in jgroups-channelfactory.sar, those files are used as JGroups configs. To be honest I forgot how to change specific JGroups configuration files in AS 6.1 but the JBoss manual should have relevant information. Also makes sure that you take down any firewalls on/between linux machines.
No it is not the UDP configuration , I got missguided by the fact that the test never finishes. I increased the number of messages and it went way beyond 400000. Just the recieve speed is slower than the send speed which I guess is normal after all the recieve buffer is significantly smaller than the send buffer.
I also tested the udp-sync and the tcp configurations, all of then are very slow on that particular machine.
It is interesting that the jgroups throughoutput according to the performance test is significantly] better than what we see in our developer hardware. Still our test show that the system runs 100-200% slower on the faster hardware.
I am getting confuse, is it an OS issue, or it is something related to locking entities, but if it is locking why they are locked only on this machine ? What is so specific. Where we should search for a probable cause?
Thank you both for your input. At least I learned something new about jgroups.
The problem we are experiencing is not related to Infinispan! And the version we were testing with Ehcache is an old version of the product so the test is irelevant.
When we disabled the second level caching the problem was still there, most probably there is something in the hibernate layer that is playing us tricks.
Thanks again for your input!