Test framework for HA Shared Nodes

[jmesnil]

With the new HA branch, nodes using a shared store uses FileLock and must be run in their own VMS.
Currently, the tests circumvents that by running all the nodes in the same VM and use a fake FileLock.

For integration tests, we must use the real FileLock implementation, run the nodes in separate VMs, and be able to start/stop/crash the servers.

Has someone experience with http://jboss.org/jbossdtf?
Does it fit the bill or is it overkill?

Another simpler alternative would be to use our own SpawnedVMSupport to run the HornetQ servers in separate VMS with a bit of code to STOP/KILL the servers

[jmesnil]

I've reused the code in the JORAM tests to spawn VM and adapt it to make it easy to manage HornetQ server.

All you need is to extend RemoteServerConfiguration class:

   static class SharedLiveServerConfiguration extends RemoteServerConfiguration
   {
      @Override
      public Configuration getConfiguration()
      {
         Configuration config = new ConfigurationImpl();
         config.setSecurityEnabled(false);
         config.setJournalType(JournalType.NIO);
         config.setSharedStore(true);
         config.setClustered(true);
         config.getAcceptorConfigurations().add(createTransportConfiguration(true, true, generateParams(0, true)));
         config.getConnectorConfigurations().put("self",
                                                 createTransportConfiguration(true, false, generateParams(0, true)));
         config.getClusterConfigurations().add(new ClusterConnectionConfiguration("cluster",
                                                                                  "foo",
                                                                                  "self",
                                                                                  -1,
                                                                                  false,
                                                                                  false,
                                                                                  1,
                                                                                  1,
                                                                                  new ArrayList<String>()));
         return config;
      }
   }

   static class SharedBackupServerConfiguration extends RemoteServerConfiguration
   {
      @Override
      public Configuration getConfiguration()
      {
         ...
      }
   }

and then in the tests, use RemoteHornetQServer static methods to start/stop/crash the server:

   public void testCrashLiveServer() throws Exception
   {
      Process liveServer = null;
      Process backupServer = null;
      try
      {
         liveServer = RemoteHornetQServer.start(SharedLiveServerConfiguration.class.getName());
         backupServer = RemoteHornetQServer.start(SharedBackupServerConfiguration.class.getName());

         // write the tests from the client POV

         RemoteHornetQServer.crash(liveServer);
         assertTrue(liveServer.exitValue() == 1);
         liveServer = null;

         // check failover 
      }
      finally
      {
         if (liveServer != null)
         {
            RemoteHornetQServer.stop(liveServer);

         }
         if (backupServer != null)
         {
            RemoteHornetQServer.stop(backupServer);

         }
      }

   }

With a bit more polish and magic, we could adapt it to reuse all the failover tests to test either:

* using invm servers + fake filelock

* using remote servers + real filelock

from the same base tests.

I've added a first test in FailoverWithSharedStoreTest using these methods.