Extending the question,

our Cluster on question has 2 physical servers, which means the group has 2 instances on 1 server and another 2 Instances on another server.

This is the main difference on the samples that we found showing how to Work with Singleton Services on Jboss.

Thanks on advance on any help.

Hello Dieter,

It is also my suspicion, but the configuration is an out of the box (based on domain / host and host-slave provided by the distribution).

I suspect that for multiple machine cluster there is something to do "a little more" for the singleton work properly.

"Do you use the quickstart related to your EAP version?"

Yes, we are using the code "as is", the main difference is the name of the instances related.

"So in your case maybe the cluster is not build correct. How your configuration look like? "

Our configuration is the main issue, is the domain.xml starting on one server (Master + 2 Servers) and the host-slave.xml starting on the second server (2 Servers).

The main question resides on what services the Cluster uses to register the singleton on the second machine (host-slave.xml), one machine is registering the service (singleton) and the other machine is doing the same but my guess is that one machine is not aware of another machine when the service (singleton) is being registered.

Do you have any idea which services i shall verify (which means which jboss services the singleton is dependent) for multiple machine cluster?.

The log of the ssingleton being registered we can recover tomorrow.

Regards.

Hello Dieter,

Between the log messages on the members of the cluser i found something like bellow. But nothing that is pointing to a problem on configuration, in matter of fact we don't know exactly what to look on administrative console about the SingletonService's registered. I mean, we can identify the SingletonService (EJB) on administrative console, but nothing to point which server was elected to execute the Job.

Do you know if there are any JBoss-Cli command which we can execute to retrieve the status of the SingletonServices in Runtime (like which server was elected to execute the SingletonJob)?

19:09:46,679 ERROR [stderr] (MSC service thread 1-1) 0 [MSC service thread 1-1] INFO br.gov.sp.prodesp.saog.scheduler.timer.activator.HATimerServiceActivator - HATimerService will be installed!

19:09:47,853 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000078: Starting JGroups Channel

19:09:47,876 INFO  [stdout] (ServerService Thread Pool -- 65)

19:09:47,876 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

19:09:47,876 INFO  [stdout] (ServerService Thread Pool -- 65) GMS: address=araujos-jb3:Jb3Srv01SAOG/singleton, cluster=singleton, physical address=10.200.116.81:55200

19:09:47,876 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

19:52:56,467 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000078: Starting JGroups Channel

19:52:56,506 INFO  [stdout] (ServerService Thread Pool -- 65)

19:52:56,506 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

19:52:56,506 INFO  [stdout] (ServerService Thread Pool -- 65) GMS: address=araujos-jb3:Jb3Srv02SAOG/singleton, cluster=singleton, physical address=10.200.116.81:55350

19:52:56,506 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

19:52:57,055 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000094: Received new cluster view: [araujos-jb3:Jb3Srv01SAOG/singleton|1] [araujos-jb3:Jb3Srv01SAOG/singleton, araujos-jb3:Jb3Srv02SAOG/singleton]

19:52:57,079 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000079: Cache local address is araujos-jb3:Jb3Srv02SAOG/singleton, physical addresses are [10.200.116.81:55350]

19:52:57,097 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 65) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.10.Final

19:09:47,853 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000078: Starting JGroups Channel

19:09:47,876 INFO  [stdout] (ServerService Thread Pool -- 65)

19:09:47,876 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

19:09:47,876 INFO  [stdout] (ServerService Thread Pool -- 65) GMS: address=araujos-jb3:Jb3Srv01SAOG/singleton, cluster=singleton, physical address=10.200.116.81:55200

19:09:47,876 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

19:09:57,599 INFO  [stdout] (ServerService Thread Pool -- 64)

19:09:57,599 INFO  [stdout] (ServerService Thread Pool -- 64) -------------------------------------------------------------------

19:09:57,599 INFO  [stdout] (ServerService Thread Pool -- 64) GMS: address=araujos-jb2:Jb2Srv02SAOG/singleton, cluster=singleton, physical address=10.200.116.80:55350

19:09:57,600 INFO  [stdout] (ServerService Thread Pool -- 64) -------------------------------------------------------------------

19:09:57,975 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000094: Received new cluster view: [araujos-jb2:Jb2Srv01SAOG/singleton|1] [araujos-jb2:Jb2Srv01SAOG/singleton, araujos-jb2:Jb2Srv02SAOG/singleton]

19:09:57,991 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000079: Cache local address is araujos-jb2:Jb2Srv02SAOG/singleton, physical addresses are [10.200.116.80:55350]

19:02:13,253 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 89) MODCLUSTER000021: All pending requests drained from default-host:/saog-core-service in 0.0 seconds

19:02:13,298 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: araujos-jb2:Jb2Srv02SAOG/singleton elected as the singleton provider of the jboss.saog.ha.singleton.timer service

19:02:13,298 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010340: This node will now operate as the singleton provider of the jboss.saog.ha.singleton.timer service

19:09:52,738 INFO  [org.jboss.as.webservices] (MSC service thread 1-2) JBAS015539: Starting service jboss.ws.port-component-link

19:09:52,830 INFO  [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 65) JBAS010342: araujos-jb3:Jb3Srv01SAOG/singleton elected as the singleton provider of the jboss.saog.ha.singleton.timer service

19:09:52,830 INFO  [org.jboss.as.clustering.singleton] (ServerService Thread Pool -- 65) JBAS010340: This node will now operate as the singleton provider of the jboss.saog.ha.singleton.timer service

19:02:13,195 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010342: araujos-jb3:Jb3Srv02SAOG/singleton elected as the singleton provider of the jboss.saog.ha.singleton.timer service

19:02:13,224 INFO  [org.jboss.as.clustering.singleton] (notification-thread-0) JBAS010340: This node will now operate as the singleton provider of the jboss.saog.ha.singleton.timer service

To have an idea on what happens on practice this is the "production log of the application"... As you notice the application running on 4 servers register 2 executions of the Batch Service (1 on server running on Machine A and 1 Running on Machine B), seems that "something is missing", we checked the configuration and on practice the configuration is an out of box config based on xml's provided by JBOSS.

As you notice at 10:30 the batch was executed just once, at 11:00 the same batch was executed twice, it seems that was logged even the batch executed 3 times (which means just 1 server was not elected to execute the batch).

So,

One of the server during the reitialization show the following message,

19:09:56,995 INFO  [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 1

Let´s suppose that the 4 servers were restarted, then we must have the following sequence of messages on log... (bellow is just a sample...)

XXXXX [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 1 (Server 1 started)

XXXXX [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 2 (Server 2 started)

XXXXX [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 3 (Server3 started)

XXXXX [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 4 (Server 4  started)

The real log

Machine1:

Server 1

19:09:56,995 INFO  [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 1

Server 2

19:09:58,130 INFO  [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 2

Machine 2:

Server 1 :

19:09:49,918 INFO  [org.jboss.as.clustering] (MSC service thread 1-4) JBAS010238: Number of cluster members: 1

Server 2:

19:52:57,175 INFO  [org.jboss.as.clustering] (MSC service thread 1-1) JBAS010238: Number of cluster members: 2

Isn't the log pointing to some problem on JGroups config (like multicast address) ?

I am curious now to understand how we can identify the problem on Cluster configuration. Since we deploy the same application on the group (Machine 1 and Machine 2) but the group is forming 2 clusters.

Regards.

Hello Dieter,

Regarding the log which shall log the cluster registering, that cames on doubt,

our topology resides on a MainServer (host-controller) and 2 slave machines.

So, about the loggging shall we receive the registering of the cluster members on our Machine 1 (that belongs to the group) or the HostController (which does not makes sense since does not participate on the group) also in case of the group, which machine is machine 1 since the group is bootted by a script (first machine to start is machine1 or there is an order)?

And some tests using netstat:

araujos-jb3:/opt/jboss-eap-6.3/slave/servers/Jb3Srv01SAOG/log # netstat -an |grep 45700

araujos-jb3:/opt/jboss-eap-6.3/slave/servers/Jb3Srv01SAOG/log # netstat -an |grep 45688

udp        0      0 230.0.0.4:45688         0.0.0.0:*

udp        0      0 230.0.0.4:45688         0.0.0.0:*

The doubt resides over MPING, how we can check if the discovery process of the MPING is running?

Bellow is the configuration of the slaves and jgroups.

Here is our JGROUPS configuration on our Master Machine:

<socket-binding-group name="full-ha-sockets" default-interface="public">

            <socket-binding name="ajp" port="8009"/>

            <socket-binding name="http" port="8080"/>

            <socket-binding name="https" port="8443"/>

            <socket-binding name="jacorb" interface="unsecure" port="3528"/>

            <socket-binding name="jacorb-ssl" interface="unsecure" port="3529"/>

            <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>

            <socket-binding name="jgroups-tcp" port="7600"/>

            <socket-binding name="jgroups-tcp-fd" port="57600"/>

            <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>

            <socket-binding name="jgroups-udp-fd" port="54200"/>

            <socket-binding name="messaging" port="5445"/>

            <socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9876}"/>

            <socket-binding name="messaging-throughput" port="5455"/>

            <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>

            <socket-binding name="remoting" port="4447"/>

            <socket-binding name="txn-recovery-environment" port="4712"/>

            <socket-binding name="txn-status-manager" port="4713"/>

        </socket-binding-group>

Bellow is the configuration of our slaves machines.

Machine 1

<?xml version='1.0' encoding='UTF-8'?>

<host xmlns="urn:jboss:domain:1.6">

    <management>

        <security-realms>

            <security-realm name="ManagementRealm">

                <server-identities>

                     <!-- Replace this with either a base64 password of your own, or use a vault with a vault expression -->

                     <secret value="ZGV2RUFQQHByMGQmc3A="/>

                </server-identities>

                <authentication>

                    <local default-user="$local" skip-group-loading="true"/>

                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>

                </authentication>

                <authorization map-groups-to-roles="false">

                    <properties path="mgmt-groups.properties" relative-to="jboss.domain.config.dir"/>

                </authorization>              

            </security-realm>

            <security-realm name="ApplicationRealm">

                <authentication>

                    <local default-user="$local" allowed-users="*" skip-group-loading="true" />

                    <properties path="application-users.properties" relative-to="jboss.domain.config.dir" />

                </authentication>

                <authorization>

                    <properties path="application-roles.properties" relative-to="jboss.domain.config.dir"/>

                </authorization>

            </security-realm>

        </security-realms>

        <audit-log>

            <formatters>

                <json-formatter name="json-formatter"/>

            </formatters>

            <handlers>

                <file-handler name="host-file" formatter="json-formatter" relative-to="jboss.domain.data.dir" path="audit-log.log"/>

                <file-handler name="server-file" formatter="json-formatter" relative-to="jboss.server.data.dir" path="audit-log.log"/>

            </handlers>

            <logger log-boot="true" log-read-only="false" enabled="false">

                <handlers>

                    <handler name="host-file"/>

                </handlers>

            </logger>

            <server-logger log-boot="true" log-read-only="false" enabled="false">

                <handlers>

                    <handler name="server-file"/>

                </handlers>

            </server-logger>

        </audit-log>

        <management-interfaces>

            <native-interface security-realm="ManagementRealm">

                <socket interface="management" port="${jboss.management.native.port:9999}"/>

            </native-interface>

        </management-interfaces>

    </management>

    <domain-controller>

       <remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm" username="admin"/>

    </domain-controller>

    <interfaces>

        <interface name="management">

            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

        </interface>

        <interface name="public">

           <inet-address value="${jboss.bind.address:127.0.0.1}"/>

        </interface>

        <interface name="unsecure">

            <!-- Used for IIOP sockets in the standard configuration.

                 To secure JacORB you need to setup SSL -->

            <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>

        </interface>

    </interfaces>

    <jvms>

      <jvm name="default">

            <heap size="64m" max-size="256m"/>

            <permgen size="256m" max-size="256m"/>

              <jvm-options>

                  <option value="-server"/>

  <!-- <option value="-agentpath:/opt/dynatrace-6.1.0/agent/lib64/libdtagent.so=name=SAOG_JBoss,server=10.200.112.75:9998"/>-->

              </jvm-options>

         </jvm>

    </jvms>

    <servers>

        <server name="Jb2Srv01SAOG" group="saog-server-group" auto-start="true"/>

        <server name="Jb2Srv02SAOG" group="saog-server-group" auto-start="true">

            <!-- server-two avoids port conflicts by incrementing the ports in

                 the default socket-group declared in the server-group -->

            <socket-bindings port-offset="150"/>

        </server>

    </servers>

</host>

Machine 2

<?xml version='1.0' encoding='UTF-8'?>

<host xmlns="urn:jboss:domain:1.6">

    <management>

        <security-realms>

            <security-realm name="ManagementRealm">

                <server-identities>

                     <!-- Replace this with either a base64 password of your own, or use a vault with a vault expression -->

                     <secret value="ZGV2RUFQQHByMGQmc3A="/>

                </server-identities>

                <authentication>

                    <local default-user="$local" skip-group-loading="true"/>

                    <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>

                </authentication>

                <authorization map-groups-to-roles="false">

                    <properties path="mgmt-groups.properties" relative-to="jboss.domain.config.dir"/>

                </authorization>              

            </security-realm>

            <security-realm name="ApplicationRealm">

                <authentication>

                    <local default-user="$local" allowed-users="*" skip-group-loading="true" />

                    <properties path="application-users.properties" relative-to="jboss.domain.config.dir" />

                </authentication>

                <authorization>

                    <properties path="application-roles.properties" relative-to="jboss.domain.config.dir"/>

                </authorization>

            </security-realm>

        </security-realms>

        <audit-log>

            <formatters>

                <json-formatter name="json-formatter"/>

            </formatters>

            <handlers>

                <file-handler name="host-file" formatter="json-formatter" relative-to="jboss.domain.data.dir" path="audit-log.log"/>

                <file-handler name="server-file" formatter="json-formatter" relative-to="jboss.server.data.dir" path="audit-log.log"/>

            </handlers>

            <logger log-boot="true" log-read-only="false" enabled="false">

                <handlers>

                    <handler name="host-file"/>

                </handlers>

            </logger>

            <server-logger log-boot="true" log-read-only="false" enabled="false">

                <handlers>

                    <handler name="server-file"/>

                </handlers>

            </server-logger>

        </audit-log>

        <management-interfaces>

            <native-interface security-realm="ManagementRealm">

                <socket interface="management" port="${jboss.management.native.port:9999}"/>

            </native-interface>

        </management-interfaces>

    </management>

    <domain-controller>

       <remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm" username="admin"/>

    </domain-controller>

    <interfaces>

        <interface name="management">

            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

        </interface>

        <interface name="public">

           <inet-address value="${jboss.bind.address:127.0.0.1}"/>

        </interface>

        <interface name="unsecure">

            <!-- Used for IIOP sockets in the standard configuration.

                 To secure JacORB you need to setup SSL -->

            <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>

        </interface>

    </interfaces>

    <jvms>

      <jvm name="default">

            <heap size="64m" max-size="256m"/>

            <permgen size="256m" max-size="256m"/>

              <jvm-options>

                  <option value="-server"/>

              </jvm-options>

         </jvm>

    </jvms>

    <servers>

        <server name="Jb3Srv01SAOG" group="saog-server-group" auto-start="true"/>

        <server name="Jb3Srv02SAOG" group="saog-server-group" auto-start="true">

            <!-- server-two avoids port conflicts by incrementing the ports in

                 the default socket-group declared in the server-group -->

            <socket-bindings port-offset="150"/>

        </server>

    </servers>

</host>

Regards,

Problem Solved,

Executed the Tests for JGroups, we understand the problem with MultiCast UDP here (constraints of network configuration) changed to TCP and it seems that worked fine.

Tks for your help Dieter.

Regards.;