Scalability of portal without verifying that fail over works properly does not add much value. Hence, before I do more of scalability testing, I would like to verify that fail over works. Once that is verified then I can extend the same test to be used for scalability testing.

One simple way to do fail over testing in portal case I think is this:

1. Create a portlet (extend NullPortletPage that we have for load testing) which
a. For a new session, sets a session attribute called "SEQUENCE_NUMBER" to 0
b. On subsequent requests, the portlet simply increases the sequence number.

2. Client makes a request to n-node cluster and it also keeps track of SEQUENCE_NUMBER and asserts that this number matches.

3. In between request, I randomly pick one or more nodes, bring them down and bring them up after some times.

Any thoughts on this?

Another simple test I think will be to change portlet window state to maximized and minimized alternatively and assert on client that response string has correct portlet window state.