Increase in Memory Utilisation after Jboss Server Member Leaves and re-joins the Cluster

[shivam37]

Hi

We are using jboss 4.2.3 version running on jdk1.5 update 15, in cluster mode consisting of 4 jboss servers

Recently we observed 1 member left the cluster and re-joined with in a minute as shown in the below snapshot. After this incident the Memory didnt release properly even after 'Full GC' , we had restarted all 4 Jboss server in production in order to resume operations.

Please help us with below queries

1. What could cause a cluster member to go out of cluster?
2. If network glitch would cause this, can we prove it with JBoss/App logs. unfortunately we don’t have heap dump  capture.
3. If a cluster member re-joins shouldn’t it continue to process the messages seamlessly if there is no need for retransmission table sync up?

INFO | jvm 1    | 2015/10/17 14:47:46 | 17 Oct 2015 14:47:46,454  INFO [Thread[Incoming Thread,GLJBOSS,10.0.XX.12:62153,5,Incoming]] GLJBOSS - New cluster view for partition GLJBOSS: 8 ([10.0.XX.11:11099, 10.0.XX.12:11099, 10.0.XX.13:11099] delta: -1)

INFO | jvm 1    | 2015/10/17 14:47:46 | 17 Oct 2015 14:47:46,454  INFO [Thread[AsynchViewChangeHandler Thread,5,jboss]] GLJBOSS - I am (10.0.XX.12:11099) received membershipChanged event:

INFO | jvm 1    | 2015/10/17 14:47:46 | 17 Oct 2015 14:47:46,454  INFO [Thread[AsynchViewChangeHandler Thread,5,jboss]] GLJBOSS - Dead members: 1 ([10.0.XX.14:11099])

INFO | jvm 1    | 2015/10/17 14:47:46 | 17 Oct 2015 14:47:46,454  INFO [Thread[AsynchViewChangeHandler Thread,5,jboss]] GLJBOSS - New Members : 0 ([])

INFO | jvm 1    | 2015/10/17 14:47:46 | 17 Oct 2015 14:47:46,454  INFO [Thread[AsynchViewChangeHandler Thread,5,jboss]] GLJBOSS - All Members : 3 ([10.0.XX.11:11099, 10.0.XX.12:11099, 10.0.XX.13:11099])

INFO | jvm 1    | 2015/10/17 14:48:31 | 17 Oct 2015 14:48:31,567  INFO [Thread[Incoming Thread,GLJBOSS,10.0.XX.12:62153,5,Incoming]] GLJBOSS - New cluster view for partition GLJBOSS: 9 ([10.0.XX.11:11099, 10.0.XX.12:11099, 10.0.XX.13:11099, 10.0.XX.14:11099] delta: 1)

INFO | jvm 1    | 2015/10/17 14:48:31 | 17 Oct 2015 14:48:31,567  INFO [Thread[AsynchViewChangeHandler Thread,5,jboss]] GLJBOSS - I am (10.0.XX.12:11099) received membershipChanged event:

INFO | jvm 1    | 2015/10/17 14:48:31 | 17 Oct 2015 14:48:31,567  INFO [Thread[AsynchViewChangeHandler Thread,5,jboss]] GLJBOSS - Dead members: 0 ([])

INFO | jvm 1    | 2015/10/17 14:48:31 | 17 Oct 2015 14:48:31,567  INFO [Thread[AsynchViewChangeHandler Thread,5,jboss]] GLJBOSS - New Members : 1 ([10.0.XX.14:11099])

INFO | jvm 1    | 2015/10/17 14:48:31 | 17 Oct 2015 14:48:31,567  INFO [Thread[AsynchViewChangeHandler Thread,5,jboss]] GLJBOSS - All Members : 4 ([10.0.XX.11:11099, 10.0.XX.12:11099, 10.0.XX.13:11099, 10.0.XX.14:11099])

Memory after Full GC

2015/10/17 02:46:29 | [Full GC 905876K->550611K(2347712K), 1.1024303 secs]

.

2015/10/17 07:46:34 | [Full GC 724200K->497448K(3078528K), 1.1318268 secs]

.

2015/10/17 11:36:29 | [Full GC 817994K->551993K(2730688K), 1.1420465 secs]

.

2015/10/17 15:37:03 | [Full GC 1185468K->704668K(3116608K), 2.0485422 secs]

.

2015/10/17 23:37:17 | [Full GC 1496768K->1109172K(3020160K), 2.0697918 secs]

.

2015/10/18 15:05:45 | [Full GC 2097266K->1927047K(2439104K), 2.4393695 secs]

.

2015/10/19 15:47:53 | [Full GC 3013137K->3006702K(3084032K), 4.3881563 secs]

Thanks in advance

Siva