Infinispan server standalone as a docker swarm service using bind mount to persist file-store to host

[prateekk.in]

I am deploying the Infinispan server 9.3.2.Final (https://hub.docker.com/r/jboss/infinispan-server/) in a service stack (using docker compose v3 format) on a docker host in a swarm cluster (on premise for now).

It will run in standalone mode using a custom xml file as input (which will create a local cache) and then the other services in the application stack will connect to it using RemoteCacheManager (the ConfigurationBuilder host will take the service name as input).

In case this infinispan service were to crash (and docker swarm will launch it again), I'd like the file-store in the service to persist and be available on the docker host file path. This will ensure that when the service is automatically restarted, it will pre-load itself from the file-store that stayed safe on the docker host's file system (and will again be available as a bind mount).

The problem is that in standalone mode, in the xml file, the file-store path= is relative to the /opt/jboss/infinispan-server/standalone/data and its not possible to specify this as a volume mount on the host (as the host would not have the full folder structure on its file system that the container expects).

I tried using <global state><persistent location="/path/to/where"></global-state> but still my file store remains relative to /opt/jboss/infinispan-server/standalone/data. Am hoping someone could please advise on how this can be done, here is our deployment:

$docker stack deploy -c docker-compose-ourApp.yml ourAppStack

version: "3"
services:
  infinispan:
    image: jboss/infinispan-server:9.3.2.Final
    command: standalone -c custom/infinispan_server.xml
    environment:
     - APP_USER=ourApp
     - APP_PASS=changeme
     - MGMT_USER=ourApp
     - MGMT_PASS=changeme
    ports:
      - "9990:9990"
    volumes:
      - "/c/Users/someone/DockerMapped/datafiles/configuration/:/opt/jboss/infinispan-server/standalone/configuration/custom/"
    networks:
      - ourOverlayNetworkName
networks:
   ourOverlayNetworkName:

Here is the relevant part from /c/Users/someone/DockerMapped/datafiles/configuration/infinispan_server.xml

<subsystem

    xmlns="urn:infinispan:server:core:9.3" default-cache-container="local">

    <cache-container name="local" default-cache="default" statistics="true">

        <global-state/>

        <local-cache name="default"/>

        <local-cache name="APP_CACHE">

            <expiration lifespan="-1"/>

            <file-store path="ourAPP/APP_CACHE" max-entries="-1" purge="false" passivation="false"/>

            <memory>

                <binary size="100000000" eviction="MEMORY"/>

            </memory>

        </local-cache>

    </cache-container>

</subsystem>

From some googling I think it might be possible via some of these but I am not able to get it working:

• some settings in global-state like for e.g:

<global-state>

            <persistent-location path="/path/to/where" />

            <temporary-location path="/path/to/where/tmp"/>        

</global-state>

• Volume mount: by creating a volume in docker (say named cachedata:) and then providing a volume mount in the compose file.

Thanks,

_prateek

[gustavonalle]

Could you try something like:

<global-state>

   <persistent-location path="/relative/path/" relative-to="/path/to/store" />

</global-state>

[nadirx]

I'll clear up the distinction:

`global-state` is where Infinispan stores global persistent state such as internal caches and node UUIDs.

Normally you should not be touching the above.

User caches instead use the location path and relative-to to say where they are going to store data.

The `relative-to` attributes are not explicit paths, but server environment variables, such as `jboss.server.data.dir` (which happens to be the default value).

[prateekk.in]

Thanks gustavonalle, I am trying to get it working but am still getting IllegalArgumentException (no path /custom found or /tmp not found) - for instance with

<global-state>

<persistent-location path="/tmp/" relative-to="/custom" />

</global-state>

If possible could you please suggest the possible values for path and relative-to inside the global-state.

As a first step, I'd want to persist the file-store to say /tmp/ourApp.

The container can be passed an environment variable via docker compose (-e "TEMP=/tmp) and it anyway has a jvm property $(java.io.tmpdir} pointed at /tmp

[prateekk.in]

Thanks nadirx, I am hoping this can work but when I try running the container I still get IllegalArgument exception (Could not find a path called '${java.io.tmpdir}').

<subsystem

    xmlns="urn:infinispan:server:core:9.3" default-cache-container="local">

    <cache-container name="local" default-cache="default" statistics="true">

        <global-state/>

        <local-cache name="default"/>

        <local-cache name="APP_CACHE">

            <expiration lifespan="-1"/>

            <file-store relative-to="${java.io.tmpdir}" path="ourAPP/APP_CACHE" max-entries="-1" purge="false" passivation="false"/>

            <memory>

                <binary size="100000000" eviction="MEMORY"/>

            </memory>

        </local-cache>

    </cache-container>

</subsystem>

If possible could you please suggest the possible values for relative-to since it seems I need to add this to our file-store element (urn:infinispan:config:9.3 ).

For instance, I'd want to persist the file-store to say /tmp/ourApp/APP_CACHE/APP_CACHE.dat

I tried relative-to="$TEMP"

(and separately these too)

"TEMP"

"/tmp" but still failed to start.

The container can be passed an environment variable via docker compose (-e "TEMP=/tmp) and it anyway has a jvm property $(java.io.tmpdir} pointed at /tmp and run with:

$docker run -d -p 9990:9990 -e "TEMP=/tmp" -e "APP_USER=ourApp" -e "APP_PASS=changeme" -e "MGMT_USER=ourApp" -e "MGMT_PASS=changeme" -v /c/Users/someone/DockerMapped/ourapp/datafiles/configuration/:/opt/jboss/infinispan-server/standalone/configuration/custom/ jboss/infinispan-server:9.3.2.Final standalone -c custom/infinispan_server.xml

Later I will add a -v /c/Users/someone/DockerMapped/ourapp/datafiles/configuration/persisted/:/tmp/ourApp in the run command/compose file as the bind mount to docker host.

[nadirx]

Sorry, I should have explained better.

First of all, just adding an empty <file-store/> element to a cache will automatically use "datagrid-infinispan/[container-name]" as path and "${jboss.server.data.dir}" as relative-to. The base name of the file is the name of the cache. Thus:

[jboss.server.data.dir]/datagrid-infinispan/[container-name]/[cache-name].dat

which for the default cache in the local container in standalone mode would be:

standalone/data/datagrid-infinispan/local/default.dat

The path concepts are detailed in the WildFly docs:

General configuration concepts - WildFly 10 - Project Documentation Editor

In particular relative-to is either one of the predefined names (e.g. jboss.server.data.dir) or a custom name path you can define using a <path> element.

Supposing you want to put your cachestores under /var/lib/infinispan, you'd need to define a global path and then point your stores to use it as "relative-to":

Example:

<server xmlns="urn:jboss:domain:8.0">

    <extensions>

    ...

    </extensions>

    <paths>

       <path name="cachestore.root" path="/var/lib/infinispan"/>

    </paths>

    ...

    <profile>

        ...

        <subsystem xmlns="urn:infinispan:server:core:9.4" default-cache-container="local">

            <cache-container name="local">

                <local-cache name="namedCache">

                   <file-store relative-to="cachestore.root"/>

                </local-cache>

            </cache-container>

        </subsystem>

        ...

    </profile>

    ...

</server>

This will create the cachestore for the namedCache cache as:

/var/lib/infinispan/datagrid-infinispan/container/namedCache.dat

If you set the file-store path attribute, it will be used in place of the default "datagrid-infinispan/container":

<file-store relative-to="cachestore.root" path="/" />

Will create the cachestore as:

/var/lib/infinispan/namedCache.dat