[udit-mishra-5113a21a]

Hey pruivo

No cheers for me

As per your comment I have changed the multicast-port in "jgroups-mping" and "jgroups-udp", in UDIT's domain.xml, but it did not work, the issue persists. Could you explain, why?

[nadirx]

Hi,

I have created a working configuration for this to explain it a bit better. It requires two files:

Infinispan 8.2 Domain configuration for multiple sites · GitHub

Infinispan 8.2 Host configuration for multiple sites · GitHub

Download the two files and start the domain controller as follows:

./bin/domain.sh --domain-config=domain-xsite.xml --host-config=host-xsite.xml

This will give you two server-groups (earth and moon) each with two servers (earth-one, earth-two; moon-one, moon-two) and with earth's default cache backed up to moon's default cache.

Note that each server group is using a dedicated profile. Both earth's and moon's profiles include a common profile (clustered) and they each have a special socket-binding-group.

Each server group is a UDP multicast cluster, earth using address 234.99.54.14 and moon using address 234.100.54.14.

They communicate with each other using a TCP channel which uses the TCP MPING protocol for discovery on multicast address 234.101.54.14.

Unfortunately the console still has a bit of trouble displaying the information correctly.

Although the above configuration shows a self-contained two-site configuration, in practice it would be more likely that you would have two distinct domains, one per site, managing that site's servers.

[udit-mishra-5113a21a]

Hello nadirx ,

First of all thank you for replying. I understood your configuration. I already have made those changes.

One missing thing was to change the IP of jgroups-mping, which also I did.

I understand the configuration. Its just that I have two distinct domain.xml for two DC on two different clusters. You have made changes in one domain.xml only, so you had to create two socket binding groups, profile etc.

My both server nodes, ServerOne from UDIT and ServerOne from JBOSS join the bridge 'xsite' . I can see those messages on console. Web based server management console also shows this. I have attached a file WebUI.png, you can see there. The problem only occurs when I try to put some key into default cache of UDIT. On putting itself, it starts showing error on console -

[Server:ServerOne] 11:11:01,220 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerWorker-8-2) Udit:ServerOne: no route to JBOSS: dropping message

[Server:ServerOne] 11:11:11,222 WARN  [org.infinispan.xsite.BackupSenderImpl] (HotRodServerWorker-8-2) ISPN000202: Problems backing up data for cache default to site JBOSS: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from JBOSS (sync, timeout=10000)

Let me attach both the domain.xml files as well. I do not understand why it can not find JBOSS. Is it something due to network issue?

The IP address of UDIT is 192.168.156.111 and the IP address of JBOSS is 192.168.154.191

[udit-mishra-5113a21a]

Hey nadirx ,

I enabled log at TRACE level for program which connects to the server, perform put and get operations, and here I attach the log file. It will give you more insight into the issue.

One question, how to check that TCP MPING is working or not?

[pruivo]

Hi Udit,

I did a mistake in my reply.

jgroups-upd must have different multicast addresses and/or ports.

jgroups-mping must have the same multicast address and port.

Sorry about that. Can you give it a try?

Cheers,

Pedro

[udit-mishra-5113a21a]

Hii pruivo ,

Its exactly like that only. As per nadirx 's comment, I already made those changes.

UDIT (domain.xml)

<socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:234.101.54.14}" multicast-port="45700"/>

<socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45688"/>

JBOSS (domain.xml)

<socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:234.101.54.14}" multicast-port="45700"/>

<socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:234.100.54.14}" multicast-port="45688"/>

Now what do you say??

[nadirx]

Running locally I see the following:

[Server:earth-one] 09:09:08,482 INFO [org.jgroups.protocols.relay.RELAY2] (Timer-3,master:earth-one) _master:earth-one:EARTH: joined bridge cluster 'xsite'

[Server:moon-one] 09:09:12,335 INFO [org.jgroups.protocols.relay.RELAY2] (Timer-2,master:moon-one) _master:moon-one:MOON: joined bridge cluster 'xsite'

To verify that RELAY2 configuration is working, use a tool like JConsole or VisualVM with the MBeans plugin installed. Run it locally on the machine you want to check. Select the appropriate processes with the relay masters (in the example above: earth-one and moon-one), visualize the "jgroups:type=protocol,cluster="cluster",protocol=RELAY2" MBean and run the "printRoutes()" operation on it. It should display something like this:

EARTH --> _master:earth-one:EARTH

MOON --> _master:moon-one:MOON

If you don't see that information then it means that the xsite channel is not discovering the two nodes.

[udit-mishra-5113a21a]

Hey nadirx ,

I already told you that I see joining messages.

I tried with JConsole on each machine and I can see the route in Udit as  -

UDIT--> _Udit:ServerOne:UDIT

and in JBOSS as  -

JBOSS--> _JBoss1:ServerOne:JBOSS

I have attached those images as well.

Is there anything suspicious ?

[nadirx]

The two "sites" are not finding each other. You should see both entries on both sites.

For the xsite channel it is probably best to use another discovery protocol, instead of multicast. For example, using TCP_PING: http://www.jgroups.org/manual/html/user-advanced.html#TCPPING

Replace <protocol type="MPING" socket-binding="jgroups-mping"> with the following:

<protocol type="TCPPING">
  <property name="initial_hosts">192.168.10.1[7600],192.168.10.2[7600]</property>
  <property name="port_range">10</property>
  <property name="timeout">3000</property>
  <property name="num_initial_members">2</property>
</protocol>

Where initial hosts contains a static list of IP addresses for well known hosts on either side.

[udit-mishra-5113a21a]

Ohh I see..

I did what you told, and now I am able to see both the entries. Also I checked it programatically by putting entries into the cache in UDIT and later retrieving it from JBOSS.

Thank you so much nadirx Thanks a lot

One small question, I have applied SSL everywhere. Right now, I have made intra-cluster commnunication secured. Also opened secure port for web management console. Also, the hotrod client to server communication is on SSL.

But, now this communication between UDIT to JBOSS will be secure or not? If not, please tell me how to make it secure?

Can I put the property encrypt inside tcp stack as well? and Can I resue the secret key which I use for intra - cluster traffic encryption ?

i.e. can I put this entry -

<protocol type="ENCRYPT"> 
            <property name="key_store_name">${jboss.domain.config.dir}/defaultStore.keystore</property> 
            <property name="store_password">changeit</property> 
            <property name="alias">myKey</property> 
</protocol>

in tcp stack as well, provided I have copied the defaultStore.keystore to JBOSS as well?

Yes, it worked, I tried it and it worked.