Fail-over design questions

[hendra_netm]

Hello JBoss Messaging Developers,
I have a question about what will happen to the crashed server when I bring it back again?

For example, I have two servers Server0 and Server1. Server0 is crash. All connections and messages in Server0 will be failed over to Server1.
When I bring server0 back, I think server0 wil have no client connections, and this will lead to imbalance load between server. Is this correct?

I also want to ask why fail-over action is only trigerred by crashed condition?
I have a case where I need to shut down JBoss Messaging Server for operational thing. I want when I shut down one server, the connection will be failed over to another node, and when I put the server back, that server will share again the load of messages delivery with other servers. Is this scenario possible?

Thank you in advance,
Hendra

[clebert.suconic]

I guess we should consider an option where we can shutdown and failover clients.

When I bring server0 back, I think server0 wil have no client connections, and this will lead to imbalance load between server. Is this correct?

This is correct at this point.

Server0 will assume new connections.. .but we don't have a recover of the failover at this point. This is something we have discussed... and it might be scheduled for future releases.

[timfox]

"hendra_netm" wrote:
Hello JBoss Messaging Developers,
I have a question about what will happen to the crashed server when I bring it back again?

For example, I have two servers Server0 and Server1. Server0 is crash. All connections and messages in Server0 will be failed over to Server1.
When I bring server0 back, I think server0 wil have no client connections, and this will lead to imbalance load between server. Is this correct?

I also want to ask why fail-over action is only trigerred by crashed condition?
I have a case where I need to shut down JBoss Messaging Server for operational thing. I want when I shut down one server, the connection will be failed over to another node, and when I put the server back, that server will share again the load of messages delivery with other servers. Is this scenario possible?

When you failover, non persistent messages would be lost. Therefore we can't just fail over to redistribute load, since people would get upset if suddenly their non persistent messages disappeared.

There are certain situations where it is possible to redistribute a connection, like, if there are no unacked messages in the session, but this gets complex.

[hendra_netm]

"timfox" wrote:

When you failover, non persistent messages would be lost. Therefore we can't just fail over to redistribute load, since people would get upset if suddenly their non persistent messages disappeared.

There are certain situations where it is possible to redistribute a connection, like, if there are no unacked messages in the session, but this gets complex.

Sorry for unclear question. I mean, when I shut down the server the messages and the connection will be not failed over to another node. The fail-over policy is only triggered by crashed condition not normal shut down condition, isn't it?

If a server is shut down, because for example I want to add some components in my server, the messages and connections will not be failed over. Why does shut down not get the same policy like when the server is crashed? Is there any problem that make you differ the crashed and shut down situation?

Thank you for your respond.

Regards,
Hendra

[timfox]

"hendra_netm" wrote:


If a server is shut down, because for example I want to add some components in my server, the messages and connections will not be failed over. Why does shut down not get the same policy like when the server is crashed? Is there any problem that make you differ the crashed and shut down situation?

As I said in my previous reply, we cannot do this because you would lose non persistent messages, and customers would not be very happy.

[hendra_netm]

"timfox" wrote:

As I said in my previous reply, we cannot do this because you would lose non persistent messages, and customers would not be very happy.

I thought that JBoss messaging will use replication for that case. "Persistent level reliability guarantee without persistence" feature means that I can use non-persistent message and still get no message loss, doesn't it?

Anyway, you would lose non-persistent messages as well in crashed situation. Or not?

[timfox]

In memory persistent message replication is not implemented yet, but when it does it would be used for persistent messages not non persistent messages.

Yes, you might lose np messages in crash, but you don't want to just randomly lose them at other non crash times, just because the server wants to move the connection.

I don't think that would be acceptable to the majority of users.

[timfox]

Although the JMS spec technically allows non persistent messages to be lost at any time, we try and make a "best effort", i.e. we try not to lose them unless failure occurs.

[hendra_netm]

"timfox" wrote:
In memory persistent message replication is not implemented yet, but when it does it would be used for persistent messages not non persistent messages.

Yes, you might lose np messages in crash, but you don't want to just randomly lose them at other non crash times, just because the server wants to move the connection.

I don't think that would be acceptable to the majority of users.

I see. Then how are users able to maintenance their server without disturbing the service?

For example, I need to maintenance my server without disturbing my services. So I need to shut down the server but I don't want any message loss.

In normal situation (without clustering), I would stop the producer and let the consumer get all messages before shutting down the server. With clustering, I don't know which clients connect to one specific server that I want to shut down due to HA-JNDI.

Is there possible that clients check which server node that they connect to?

Regards,
Hendra

[hendra_netm]

I prefer that failover in shut down condition can become option. So when I want to do maintenance, I only need to use persistent messages and shut it down.

I think it is better than simulate crash condition in server to trigger the failover

[hendra_netm]

"timfox" wrote:
When you failover, non persistent messages would be lost. Therefore we can't just fail over to redistribute load, since people would get upset if suddenly their non persistent messages disappeared.

Sorry, I still don't get your answer.

The intention of failover over shutdown is not to redistristibute load. It is done to keep the service available. So users still can send messages without knowing that there is a maintenance take place.

Shut down is done by administrators/operators. They know what will happen to non persistent messages when they shut down the server. I think it is unlikely that they will be surprised on their own action.

Anyway, with or without failover, shut down a server will make non persistent messages dissapeared. Why is this failover going to make them upset?

[timfox]

If you want to do this, just shutdown the server.

The client connections will then break, and they'll failover to another node, no extra code is necessary.

[timfox]

As long as your clients are ok with losing np messages then this should work, but in the general case I don't think is acceptable.

I would prefer a situation where you can flick a switch on the jmx console to prevent any new connections being created on a node, then query the console to find out who (ip address? client id? machine name?) the clients are, then the sys admin can kindly find those clients and cleanly shut them down.

[timfox]

"hendra_netm" wrote:

Anyway, with or without failover, shut down a server will make non persistent messages dissapeared.

Not necessarily, as mentioned in my last post, there are other ways to do it that don't involve the loss of np messages.

[hendra_netm]

"timfox" wrote:
As long as your clients are ok with losing np messages then this should work, but in the general case I don't think is acceptable.

I would prefer a situation where you can flick a switch on the jmx console to prevent any new connections being created on a node, then query the console to find out who (ip address? client id? machine name?) the clients are, then the sys admin can kindly find those clients and cleanly shut them down.

Well, that will be very great if that can be done. Actually that is the perfect solution of my problem. I just know that jmx console can do those.

I will take a look on how this can work.

Thank you very much for your time.

Regards,
Hendra