Clustering failing under heavy load?

[danhanley]

"danhanley" wrote:
Hi
We're testing a cluster of 3 intel boxes running jBoss 3.2.3 on RedHat 9.
One box in particular is very heavily loaded by our application. Occasionaly (perhaps once a day) the box stops communicating with the rest of the cluster and fills its logs with the following message:

2004-01-22 09:29:36,458 DEBUG [org.javagroups.MDC3TestPartition] [Thu Jan 22 09:29:36 GMT 2004] [ERROR] UDP.sendUdpMessage()
: (unicast) send_sock is null. Message is [dst: 213.86.178.88:44305, src: dev4:32947 (3 headers), size = 1723 bytes], headers are {UNICAST=[UNICAST: DATA, seqno=47],UDP=UDP:group_addr=MDC3TestPartition], MessageDispatcher=[Header: name=MessageDispatcher, type=RSP, id=1074701665445, rsp_expected=false]}

Stopping and restarting jboss clears the problem, but obviously is not something we want to be doing when we go to production.

I havn't seen reference to this message in any of the forums, and was wondering if anyone could explain its cause?

top on the effected box shows:
09:57:20 up 1 day, 17:34, 2 users, load average: 10.84, 10.12, 7.13
63 processes: 62 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 27.0% user 10.1% system 0.0% nice 0.0% iowait 62.4% idle
CPU1 states: 10.2% user 0.2% system 0.0% nice 0.0% iowait 89.1% idle
Mem: 1029764k av, 1020436k used, 9328k free, 0k shrd, 9364k buff
843376k actv, 48520k in_d, 21140k in_c
Swap: 2096472k av, 1048236k used, 1048236k free 80484k cached

[monu]

hi,

I am also getting similar issue. I am using jboss-3.2.2.
While starting jboss, our application starts some threads which pull messages from JMS queue and perfrom some business logic on it, Messages are file persistence message, so if Queue has got many message stored and when jboss restarts , our application starts processing those message thats generate load on machine and because of that server doesn't enters in cluster.

Regards,
Monu

[belaban]

"danhanley" wrote:

"danhanley" wrote:
Hi
We're testing a cluster of 3 intel boxes running jBoss 3.2.3 on RedHat 9.
One box in particular is very heavily loaded by our application. Occasionaly (perhaps once a day) the box stops communicating with the rest of the cluster and fills its logs with the following message:

2004-01-22 09:29:36,458 DEBUG [org.javagroups.MDC3TestPartition] [Thu Jan 22 09:29:36 GMT 2004] [ERROR] UDP.sendUdpMessage()
: (unicast) send_sock is null. Message is [dst: 213.86.178.88:44305, src: dev4:32947 (3 headers), size = 1723 bytes], headers are {UNICAST=[UNICAST: DATA, seqno=47],UDP=UDP:group_addr=MDC3TestPartition], MessageDispatcher=[Header: name=MessageDispatcher, type=RSP, id=1074701665445, rsp_expected=false]}

Stopping and restarting jboss clears the problem, but obviously is not something we want to be doing when we go to production.

Do you have the logs for this ? Also, can you post your cluster-service.xml ? You may have to increase the values for FD, or use FD_SOCK instead.

Bela