JGroups UDP multicast issue in WildFly

[valsaraj007]

Recently we faced JGroups UDP multicast issue in WildFly. The issue started when the servers are restarted after deployment. There is no WildFly configuration change. WildFly is in domain mode with 4 nodes.

• When we start nodes, first 3 gets started and the last one fails.
• When we start the failed node by changing bind address to 127.0.0.1, it works. When we start the failed node by changing the jboss.default.multicast.address to different IP, it starts fine.

<stack name="udp">

                    <transport type="UDP" socket-binding="jgroups-udp"/>

                    <protocol type="PING"/>

                    </protocol>

                    <protocol type="MERGE3"/>

                    <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>

                    <protocol type="FD_ALL"/>

                    <protocol type="VERIFY_SUSPECT"/>

                    <protocol type="pbcast.NAKACK2"/>

                    <protocol type="UNICAST3"/>

                    <protocol type="pbcast.STABLE"/>

                    <protocol type="pbcast.GMS"/>

                    <protocol type="UFC"/>

                    <protocol type="MFC"/>

                    <protocol type="FRAG2"/>

                    <protocol type="RSVP"/>

                </stack>

What may be the issue?

[pferraro]

Logs?

[pferraro]

Which WF version is this?  Which OS/JDK?

[valsaraj007]

We are using WildFly-8.2.1 and JDK 8 on CentOS 7.

This was working fine but after when we restarted this issue started. It looks like some limit related to UDP multicast. When we start 4 nodes, first nodes are success and last ones failed. This happens to all nodes based on the order we start.

I check switching from UDP to TCP, then there was no error.

But when switched back to UDP issue started again.

Any thoughts?

[valsaraj007]

pferraro

Is there any kind on limit with UDP communication? Any related settings  we need to check in OS or WildFly?

[pferraro]

8.2.1 is ancient.  Can you try with a recent version?  You are likely running into an issue with channel lifecycle, which is very likely resolved in a newer release.

[valsaraj007]

pferraro

We were running with this for 3 years and without any change, this issue started by this week.

Even after clean restart after deleting data and tmp, the same issue comes.

Checked setting max owners for EJB dist cache as 1 so that it wont transfer state via UDP, so the issue fixed.

The web dist cache have no issue with owners 2.

Would you clarify what is channel lifecycle issue?

Any work around to check?

Another thing is that it works fine when changed from udp to tcp. 

Thanks!

[pferraro]

If I had to guess, it sounds like your network is dropping multicast packets.  Use TCP instead.

WF8 was the first release to use fork channels (i.e. multiple services sharing the same JGroups channel).  It took a few releases to work out the kinks.

However, given that your application worked fine for 3 years, I suspect your switch is to blame.  Try restarting it.