CPU with 8 cores 16 logical threads

[apetrov2]

We are observing a very strange behaviour in our Infinispan setup. JBoss 6.1, Infinispan 4.2 used as hibernate 2LC. Our product is demonstrating great performance on our development PCs (windows 7), our PCs are with 4 cores and on our test Linux server.

We are doing load test on a PC with parameters near to the production server parameters. We have a web grinder test and our results are with 100% slower on the fast machine compared to what we experience on our development PCs.

My observation is that all write operations are unbelievable slow(3 times slower). Are there any variables related to the hardware? We are running exactly the same setup the only dfference is the hardware and maybe the operating system, although we deployed it on another PC with Linux and it ran fine. So we believe it is related to the hardware?

The same setup runs OK with Echache, but we would prefer infinispan if we are able to make it work. It is quite unpleasant situation as it apears very late into our development cycle when we are preparing ourselves to go in production

Any ideas?

[vblagojevic]

Hi Alexander,

I suspect that the problem might be in network layer setup on different OS. What kind of replication in JGroups do you use UDP/TCP? I would suggest running http://www.jgroups.org/manual-3.x/html/user-advanced.html#PerformanceTests on two different setups to determine if there is slowdown related to network/OS. Use JGroups 3.1 and follow directions to diagnose a possible cause.

Regards,

Vladimir

[apetrov2]

Hi Vladimir,

thank you for your fast reply. I also sustected that the problem might be somewhere there. First in response of your queston, we are using the default configurations that are coming with the JBoss 6.1. I did as you told me with only difference that I used the tests comming with jgroups 2.11 because this is the version supplied with jboss 6 AS. In the test property files there were 3 tests configuration option by transport type:

org.jgroups.tests.perf.transports.JGroupsTransport

org.jgroups.tests.perf.transports.UdpTransport

org.jgroups.tests.perf.transports.TcpTransport

The JGoupsTransport ran fine on both machines. But it was not the same with the UdpTransport. On my developer machine it was able to recieve response for all sent messages , but on the performance server this was not the case actualy the test never completed because there was no response after the 400 000 messages. Here is the output of the failed test, and I am also attaching it as a file:

log_interval:	100000
msg_size:	1000
config:	jconfig.txt
transport:	org.jgroups.tests.perf.transports.UdpTransport
num_senders:	1
sender:	true
props:	udp.xml
num_msgs:	1000000
num_members:	1

JGroups version: 2.11.0.GA

-- local_addr is 172.27.202.141:53702

-- waiting for 1 members to join

-- 172.27.202.141:7500 joined

-- READY (1 acks)

-- sending 1000000 1KB messages

++ sent 100000

++ sent 200000

-- received 100000 messages (2192 ms, 45620.44 msgs/sec, 45.62MB/sec)

++ sent 300000

++ sent 400000

-- received 200000 messages (1871 ms, 53447.35 msgs/sec, 53.45MB/sec)

++ sent 500000

++ sent 600000

-- received 300000 messages (2029 ms, 49285.36 msgs/sec, 49.29MB/sec)

++ sent 700000

++ sent 800000

++ sent 900000

-- received 400000 messages (1986 ms, 50352.47 msgs/sec, 50.35MB/sec)

++ sent 1000000

And it never ends.

At the same time if I use the JGroupsTransport everything is OK. But what transport is used in JBoss 6 ? and is it related? Any further ideas based on this output?

[vblagojevic]

Default in AS 6.1 is udp stack if I recall correctly! I would say make sure that you are binding to right interfaces (look at JBoss AS parameter options), look at JGroups configuration files in jgroups-channelfactory.sar, those files are used as JGroups configs. To be honest I forgot how to change specific JGroups configuration files in AS 6.1 but the JBoss manual should have relevant information. Also makes sure that you take down any firewalls on/between linux machines.

Regards,

Vladimir

[apetrov2]

No it is not the UDP configuration , I got missguided by the fact that the test never finishes. I increased the number of messages and it went way beyond 400000. Just the recieve speed is slower than the send speed which I guess is normal after all the recieve buffer is significantly smaller than the send buffer.

I also tested the udp-sync and the tcp configurations, all of then are very slow on that particular machine.

It is interesting that the jgroups throughoutput according to the performance test is significantly] better than what we see in our developer hardware. Still our test show that the system runs 100-200% slower on the faster hardware.

I am getting confuse, is it an OS issue, or it is something related to locking entities, but if it is locking why they are locked only on this machine ? What is so specific. Where we should search for a probable cause?

[galder.zamarreno]

Try some of the suggestions in https://community.jboss.org/wiki/PerfTuning

[apetrov2]

Thank you both for your input. At least I learned something new about jgroups.

The problem we are experiencing is not related to Infinispan! And the version we were testing with Ehcache is an old version of the product so the test is irelevant.

When we disabled the second level caching the problem was still there, most probably there is something in the hibernate layer that is playing us tricks.

Thanks again for your input!