MySQL in Cluster question| JBoss.org Content Archive (Read Only)

15. Re: MySQL in Cluster question

timfox Feb 11, 2009 5:22 AM (in response to jbabad)

"jbabad" wrote:
Hi,

Apologies for giving this one a nudge, but I've just had the scenario again where I'm sending in messages to one half of a Jboss/MySQL cluster and a receiver connected to the other half of the cluster doesn't pick up the messages unless I fail over the "sender" side of the cluster.

Thanks again for people's time.
Joseph.

Have you read this wiki?

http://www.jboss.org/community/docs/DOC-10095

Also did you run the clustered examples from the distro successfully?

16. Re: MySQL in Cluster question

jbabad Feb 11, 2009 6:28 AM (in response to jbabad)

Thanks for the info.
Yes, I had seen the wiki, and I've been using the ndb persistence file.
I've run the distributed-queue example from the 1.4.1-Beta download - I think that should be closest to what's in JBoss5 - and have substituted the JBoss5 client jars in there..

What I'm seeing is that messages are only read from the node that they are sent in on
i.e sender node 0, receiver node 1 - message is not picked up,
Next time is sender node 1, receiver node 0, then it picks up the previous message
So messages are not being pulled across the cluster - which is similar to some of the other problems I'm seeing. But, based on what I've seen before this doesn't always happen.

Any pointers?
Thanks.

17. Re: MySQL in Cluster question

timfox Feb 11, 2009 6:49 AM (in response to jbabad)

The distributed queue examples should demonstrate messages being pulled from one node to another.

If the example doesn't work then then the most likely cause is a problem in your environment.

18. Re: MySQL in Cluster question

jbabad Feb 11, 2009 7:31 AM (in response to jbabad)

If the example doesn't work then then the most likely cause is a problem in your environment.

OK. That's been my question - where to look for anything obvious.

The problem doesn't happen all the time - I can see that.
I'm printing out the publisher's and consumer's serverID and can see that they're different . I have LoadBalancing=true so that two calls to the connection factory get two different connections. It appears to be that sometimes I bring up the cluster and it's OK; other times starting the 2 instances gives me these problems.

At first I thought it was specifically related to higher message load, but obviously not if the test program fails.

Any takers on this one? I appreciate people are busy, so any suggestions are welcome.

19. Re: MySQL in Cluster question

jbabad Feb 13, 2009 4:32 AM (in response to jbabad)

I was wondering if there are any takers for this one?

There doesn't seem to be anything obviously wrong with our setup (in terms of the mysql-d.xml and ndb-persistence-service.xml).
What we're seeing is
1. Periodically messages will node be pulled across from one node to another. This can happen straight after starting up the JBoss cluster. Sender application sends messages in, but the receiver doesn't pick them up. (Yes, we've seen this even running the sample distributed queue application. Again, though this doesn't happen all the time)
2. After the above, if we stop the "Sender" JBoss instance and fail the messages over to the other node in the cluster, the receiver gets the messages.
3. Another scenario - under load (e.g. trying to send in 40000 messages of 20k characters) both sender and receiver applications will throw repeated exceptions from the FailoverCommandCenter indicating that there's been failover of the underlying JMS connection.

The set-up is that we're running the JBoss/MySQL cluster on 2 VMs

Any suggestions?
Thanks,

20. Re: MySQL in Cluster question

timfox Feb 13, 2009 4:39 AM (in response to jbabad)

Sounds to me like your nodes aren't forming a cluster either due to some network issue between them, or some problem in your config.

To see if they are forming a cluster just start one up, you'll see a load of JGroups output on the console. Then start the other one up, as the other one starts up you should see the first one recognise the second one has joined the group in the console output.

Also have a look at the JGroups docs, which has stuff about diagnosing network problems.

Best bet for you is to use your support contract if you have one - the support team are experienced in taking people through installation / environment / config issues.

If you don't have a support contract it's very hard for us to provide in depth debugging of a remote setup. Any help on the forums is done off our own backs / in our free time and the main work that we get paid for takes priority.

21. Re: MySQL in Cluster question

jbabad Feb 13, 2009 4:57 AM (in response to jbabad)

Thanks. Appreciate the response... maybe you can persuade the powers that be to get a support contract ;)

I'll take a look at the jgroups, but one thing that comes to mind are the effects (good or otherwise) of running the cluster in Virtual environments - are there any known issues with this? Given that we're running both JBoss and MySQL in VMs? I've not seen anything obvious, but in some searches I'm sure I've seen passing comments on this.

22. Re: MySQL in Cluster question

jbabad Feb 13, 2009 7:37 AM (in response to jbabad)

I'm labouring this one a bit more, I know... but having seen this in the server.log... I'm trying my luck...


2009-02-13 10:45:14,913 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#6[192.168.8.125:47389]) org.jboss.messaging.core.impl.FailoverWaiter@128b200 blocking on the failover lock, waiting for failover to start
2009-02-13 10:45:15,750 DEBUG [org.jgroups.protocols.MERGE2] (Timer-4,192.168.8.117:7900) 192.168.8.117:7900 is looking for merge candidates, found initial_mbrs=[[own_addr=192.168.8.118:7900, coord_addr=192.168.8.117:7900, is_server=true], [own_addr=192.168.8.117:7900, coord_addr=192.168.8.117:7900, is_server=true]]
2009-02-13 10:45:16,360 DEBUG [org.jgroups.protocols.MERGE2] (Timer-9,192.168.8.117:49704) 192.168.8.117:49704 is looking for merge candidates, found initial_mbrs=[[own_addr=192.168.8.118:56013, coord_addr=192.168.8.117:49704, is_server=true], [own_addr=192.168.8.117:49704, coord_addr=192.168.8.117:49704, is_server=true]]

Then later on....


2009-02-13 10:46:14,914 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#6[192.168.8.125:47389]) org.jboss.messaging.core.impl.FailoverWaiter@128b200 releasing the failover lock, checking again whether failover started ...
 2009-02-13 10:46:14,914 DEBUG [org.jboss.messaging.core.impl.FailoverWaiter] (WorkerThread#6[192.168.8.125:47389]) Timed out waiting for failover to start
2009-02-13 10:46:14,915 DEBUG [org.jboss.jms.server.remoting.JMSServerInvocationHandler] (WorkerThread#4[192.168.8.125:47391]) removing callback handler ServerInvokerCallbackHandler[5c4o83h-nlkeqt-fr4rzxwx-1-fr4sagqo-s2g+5c4o83h-nlkeqt-fr4rzxwx-1-fr4sagr5-s2l]
2009-02-13 10:46:14,915 DEBUG [org.jboss.remoting.InvokerRegistry] (WorkerThread#4[192.168.8.125:47391]) removed SocketClientInvoker[11aed12, bisocket://192.168.8.125:342501498] from registry
2009-02-13 10:46:14,917 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] (WorkerThread#4[192.168.8.125:47391]) SocketClientInvoker[11aed12, bisocket://192.168.8.125:342501498] disconnecting ...
2009-02-13 10:46:14,919 DEBUG [org.jboss.jms.server.remoting.ServerSocketWrapper] (WorkerThread#4[192.168.8.125:47391]) wrote CLOSING byte

This has happened, but my client apps still continue to send/receive messages after this. So I assume that it just picked up the same JMS connection and continued?

Thanks again.