JGroups 2.2.9 released

[belaban]

Details at http://www.jboss.com/products/jgroups.

This should be API-compatible with JGroups 2.2.7 and 2.2.8, but we haven't yet tested it with JBoss 3.2.x and 4.0.x. You can replace your jgroups JAR with the one from 2.2.9beta, but this is not yet officially supported.
Once JGroups 2.2.9 goes final, QA will certify it with the 3.2, 4.x and 5.x branches.
Bela

[henriknyberg]

Hello,
I go to the link you specified and click on downloads, but I can't find the 2.2.9 beta version. Is it possible for you to supply an exact download link please?

I would like to try out the 2.2.9 beta since the problem with classnotfoundexception I reported in:

http://jboss.com/index.html?module=bb&op=viewtopic&t=70190

appears to be the same problem as reported in:

http://jboss.com/?module=bb&op=viewtopic&t=69157

which apparently is resolved in JGroups 2.2.9

[belaban]

The link is http://sourceforge.net/project/showfiles.php?group_id=6081.

I don't think the problem of not finding the RequestCorrelator.Header will be solved with switching to 2.2.9.
The second problem will definitely be solved, this was related to JRockit (works on SUN's JDK). The problem can also be solved by simply deleting all entries in jg-magic-map.xml which refer to array types.
Bela

[henriknyberg]

Hello Bela,
using 2.2.9 beta seems to resolve the classloader issue, however the replication is still not working due the following error:

2005-10-05 12:35:56,321 : ERROR [UDP mcast receiver] [] 16 org.jgroups.protocols.UDP - failed unmarshalling message
java.io.EOFException
 at java.io.DataInputStream.readShort(DataInputStream.java:323)
 at org.jgroups.Message.readFrom(Message.java:594)
 at org.jgroups.protocols.TP.bufferToMessage(TP.java:847)
 at org.jgroups.protocols.TP.handleIncomingPacket(TP.java:706)
 at org.jgroups.protocols.TP.receive(TP.java:656)
 at org.jgroups.protocols.UDP.run(UDP.java:275)
 at java.lang.Thread.run(Thread.java:536)

I'm using JBossCache 1.2.3 and jgroups 2.2.9beta (Oracle 9ias cluster)

Best regards, Henrik

[belaban]

Can you run a simple test with only JGroups to see whether that works ?
The URL is at http://wiki.jboss.org/wiki/Wiki.jsp?page=TestingJBoss

[henriknyberg]

Yes, the demo works fine. /Henrik

[belaban]

Are you using different versions of JGroups in the same cluster ?

[henriknyberg]

Sorry, my bad. We had a couple of servers on the network with older versions of jgroups running.

JGroups 2.2.9 beta works without errors on Oracle9ias. However, when we load test the application we are seeing some really long GCs, sometimes up to about 1 minute. We haven't seen GCs that long in reference tests with JBossCache 1.2/JGroups 2.2.7. I include parts of the GC details for one JVM below.

Do you have any advice on how to reduce the size of the full GCs as a time consuming full GC is potentially dangerous in a heavily loaded cluster? For example do you recommend a minimum / maximum heap size?

Henrik

7980.880: [GC 386964K->381686K(391680K), 0.0263480 secs]
7981.282: [GC 387310K->259312K(391680K), 0.0353464 secs]
7989.759: [GC 387308K->382540K(391680K), 0.0234982 secs]
7990.270: [GC 388108K->260081K(391616K), 0.1073189 secs]
7998.600: [GC 387946K->382372K(391680K), 0.0274303 secs]
7999.006: [GC 388248K->260499K(391680K), 0.2104827 secs]
8008.546: [GC 388499K->380812K(391680K), 0.0330122 secs]
8009.059: [GC 388876K->260654K(391680K), 0.0373310 secs]
8016.914: [GC 388654K->381819K(391744K), 0.0269610 secs]
8017.350: [GC 388923K->260872K(391744K), 0.0361114 secs]
8025.977: [GC 388996K->383922K(391744K), 0.0357079 secs]
8026.505: [GC 390194K->262229K(391488K), 0.0465966 secs]
8034.697: [GC 389845K->384777K(391488K), 0.0267460 secs]
8035.167: [GC-- 390665K->390973K(391488K), 0.0498293 secs]
8035.217: [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor4]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor41]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor68]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor66]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor5]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor39]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor6]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor14]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor3]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor43]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor44]
[Unloading class sun.reflect.GeneratedConstructorAccessor5]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor67]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor40]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor21]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor26]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor17]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor69]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor18]
[Unloading class sun.reflect.GeneratedMethodAccessor1]
 390973K->173698K(391360K), 71.3795917 secs]
8134.036: [GC 301055K->294782K(391360K), 2.0591646 secs]
8136.878: [GC 302885K->175744K(391104K), 0.2426461 secs]
8145.522: [GC 302592K->292696K(391104K), 0.0309871 secs]
8145.975: [GC 302936K->175917K(391104K), 0.1553561 secs]
8150.412: [GC 302765K->292486K(391104K), 0.0372722 secs]
8150.921: [GC 303558K->176629K(391104K), 0.1419030 secs]
8154.990: [GC 303475K->292839K(391104K), 0.0210634 secs]
8155.443: [GC 304542K->177703K(391040K), 0.1337076 secs]
8159.912: [GC 304423K->292588K(391168K), 0.0207514 secs]
8160.363: [GC 304749K->178122K(391168K), 0.0765179 secs]
8164.437: [GC 305081K->293849K(391168K), 0.0354192 secs]
8164.872: [GC 305944K->178964K(391168K), 0.0812544 secs]
8170.419: [GC 305930K->295510K(391168K), 0.0236516 secs]
8170.949: [GC 306710K->179610K(391104K), 0.0316527 secs]
8177.037: [GC 306455K->298033K(391168K), 0.0323191 secs]
8177.069: [Full GC[Unloading class sun.reflect.GeneratedMethodAccessor2]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor19]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor22]
[Unloading class sun.reflect.GeneratedMethodAccessor3]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor34]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor29]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor33]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor20]
[Unloading class sun.reflect.GeneratedMethodAccessor19]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor31]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor36]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor24]
[Unloading class sun.reflect.GeneratedMethodAccessor16]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor27]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor25]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor32]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor35]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor42]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor30]
 298033K->173099K(391168K), 28.4294364 secs]
8211.966: [GC 300071K->295289K(391168K), 0.0255963 secs]
8212.322: [GC 301548K->174535K(391104K), 0.0273272 secs]
8219.020: [GC 301383K->296205K(391104K), 0.0248468 secs]
8219.289: [GC 301387K->174606K(391040K), 0.0290517 secs]

[belaban]

You are sure you identified that this is caused by JGroups, and not by your application ?
I have run perf tests with 1 billion messages and a max memory of 100MB (!) only and never had any memory problems.
What is your test application doing and can you post the configuration file ?

[henriknyberg]

Hello,
it is not a test application it is our online shop in a load test using a professional load tool (PureLoad). Objects are loaded from the database and stored in the cache based on queries in http requests. I should mention that the long GCs occur mostly during the cache warmup phase and that memory is quite stable after that. I should also mention that we load the application quite heavily, the average GET frequency on a single JBossCache is around 3000 per second.

We are using the fc-fast configuration (as defined in JGroups 2.2.7) which have worked well in production.

Anyway, I don't expect you to be able to diagnose any possible cause. There could be many reasons for this behavior where some of which may be related to the application server or JDK. Since we are looking into upgrading/changing J2EE platform in the near future I think we are going to revisit the JBossCache/JGroups upgrade at that time.

(BTW thank you for the dinner conversation we had in Stockholm in March. It was very interesting to hear about your plans for JGroups in the future.)

[belaban]

OK. If this still pertains to JBossCache, and you can reproduce it, we'd be interested in taking a look. With flow control enabled in JGroups, we should be able to keep garbage collection largely happening inside the young space, so you should never see GCs of 1 minute !
Nice talking to you in Stockholm, I assume you're the person from the company starting with H ? :-)

[henriknyberg]

I assume you're the person from the company starting with H ?

mm-HM...

[belaban]

got you :-)

[collabe]

Details at http://www.jboss.com/products/jgroups.

This should be API-compatible with JGroups 2.2.7 and 2.2.8, but we haven't yet tested it with JBoss 3.2.x and 4.0.x. You can replace your jgroups JAR with the one from 2.2.9beta, but this is not yet officially supported.
Once JGroups 2.2.9 goes final, QA will certify it with the 3.2, 4.x and 5.x branches.
Bela