Running Infinispan server in a Docker Swarm

[karlvr]

I would like to run Infinispan Server in a Docker Swarm as a service, using an overlay network for the Infinispan Server instances to communicate, and connecting to the Infinispan Server using hotrod from outside of the swarm.

The last part is the part I'm having trouble with as the overlay network is limited to containers within the swarm, and my client does not run in Docker at all.

I can publish ports when I create the service. This appears to work fine for Hotrod clients to access across the ingress network; one of the Infinispan servers handles the connection and then exchanges the topology with the client. If I use the external-host attribute under the hotrod client definition, then the client gets told about all of the servers using the address I put in external-host.

Therein lies the problem. What address do I put in external-host? I don't know the IP address of the container's host machine, but I could perhaps jump through some hoops to find it out. Would it be okay for multiple Infinispan server nodes running on a host to publish the same Hotrod address, then let Docker's ingress routing choose a random Infinispan host to connect the hotrod client to?

I can't find a recommended configuration / approach to running Infinispan in a swarm like this. Can it be done?

(The reason I need to access Infinispan outside of the swarm is for our development environment, where Tomcat runs on our development machines and wants to access the Infinispan cluster running in Docker)

Any help or advice greatly appreciated.

[sebastian.laskawiec]

Unfortunately I'm not a Docker Swarm expert but I have some experience with Kubernetes, so I will assume things work similarly in both those environments.

The Infinispan Hot Rod client can operate in 3 modes. When a topology-aware or hash-distribution-aware client connects to the cluster, it download information about the servers to optimize querying. The idea is to retrieve data from the server that has it and to avoid additional hops. This requires all servers to be reachable by the client. Unfortunately this is not true if you hide them inside Docker Swarm and expose only a Service (which I'm assuming is a load balancer and/or virtual IP) to access it.

Currently the only way is to use client basic mode and use the service IP for it. There is an ongoing work to implement this in Infinispan but it's not high priority.

[karlvr]

Thank you very much for the quick reply!

I presume that using client basic mode would potentially result in additional hops, as the client wouldn't necessarily query the owner. Is that right?

As there's work currently ongoing to implement this, I guess it isn't possible to operate the Java client in basic mode right now? Assuming you could tolerate the additional hops.

It seems like swarming isn't really viable with Infinispan at the moment where one needs to Hotrod into the cluster from outside the swarm. Maybe I'll try routing packets into the swarm and see if I can make that work!

[sebastian.laskawiec]

I presume that using client basic mode would potentially result in additional hops, as the client wouldn't necessarily query the owner. Is that right?

That's correct.

As there's work currently ongoing to implement this, I guess it isn't possible to operate the Java client in basic mode right now? Assuming you could tolerate the additional hops.

Of course you can. Just use this method in ConfigurationBuilder.

It seems like swarming isn't really viable with Infinispan at the moment where one needs to Hotrod into the cluster from outside the swarm. Maybe I'll try routing packets into the swarm and see if I can make that work!

That sounds like an idea! Even though the single hop has a lot of advantages, it really depends on your data and your network configuration how much boost you will get. I highly recommend to measure both configuration and maybe using basic mode will not be that bad in your case.