5 Replies Latest reply on Oct 10, 2018 2:52 AM by Tristan Tarrant

    Infinispan server standalone as a docker swarm service using bind mount to persist file-store to host

    Prateek Kumar Newbie

      I am deploying the Infinispan server 9.3.2.Final (https://hub.docker.com/r/jboss/infinispan-server/) in a service stack (using docker compose v3 format) on a docker host in a swarm cluster (on premise for now).

       

      It will run in standalone mode using a custom xml file as input (which will create a local cache) and then the other services in the application stack will connect to it using RemoteCacheManager (the ConfigurationBuilder host will take the service name as input).

       

      In case this infinispan service were to crash (and docker swarm will launch it again), I'd like the file-store in the service to persist and be available on the docker host file path. This will ensure that when the service is automatically restarted, it will pre-load itself from the file-store that stayed safe on the docker host's file system (and will again be available as a bind mount).

       

      The problem is that in standalone mode, in the xml file, the file-store path= is relative to the /opt/jboss/infinispan-server/standalone/data and its not possible to specify this as a volume mount on the host (as the host would not have the full folder structure on its file system that the container expects).

       

      I tried using <global state><persistent location="/path/to/where"></global-state> but still my file store remains relative to /opt/jboss/infinispan-server/standalone/data. Am hoping someone could please advise on how this can be done, here is our deployment:

       

      $docker stack deploy -c docker-compose-ourApp.yml ourAppStack

       

      version: "3"
      services:
        infinispan:
          image: jboss/infinispan-server:9.3.2.Final
          command: standalone -c custom/infinispan_server.xml
          environment:
           - APP_USER=ourApp
           - APP_PASS=changeme
           - MGMT_USER=ourApp
           - MGMT_PASS=changeme
          ports:
            - "9990:9990"
          volumes:
            - "/c/Users/someone/DockerMapped/datafiles/configuration/:/opt/jboss/infinispan-server/standalone/configuration/custom/"
          networks:
            - ourOverlayNetworkName
      networks:
         ourOverlayNetworkName:

       

      Here is the relevant part from /c/Users/someone/DockerMapped/datafiles/configuration/infinispan_server.xml

       

      <subsystem

          xmlns="urn:infinispan:server:core:9.3" default-cache-container="local">

          <cache-container name="local" default-cache="default" statistics="true">

              <global-state/>

              <local-cache name="default"/>

              <local-cache name="APP_CACHE">

                  <expiration lifespan="-1"/>

                  <file-store path="ourAPP/APP_CACHE" max-entries="-1" purge="false" passivation="false"/>

                  <memory>

                      <binary size="100000000" eviction="MEMORY"/>

                  </memory>

              </local-cache>

          </cache-container>

      </subsystem>

       

      From some googling I think it might be possible via some of these but I am not able to get it working:

      • some settings in global-state like for e.g:

      <global-state>

                  <persistent-location path="/path/to/where" />

                  <temporary-location path="/path/to/where/tmp"/>        

      </global-state>

       

      • Volume mount: by creating a volume in docker (say named cachedata:) and then providing a volume mount in the compose file.

       

      Thanks,

      _prateek

        • 1. Re: Infinispan server standalone as a docker swarm service using bind mount to persist file-store to host
          Gustavo Fernandes Apprentice

          Could you try something like:

           

          <global-state>

             <persistent-location path="/relative/path/" relative-to="/path/to/store" />

          </global-state>

          1 of 1 people found this helpful
          • 2. Re: Infinispan server standalone as a docker swarm service using bind mount to persist file-store to host
            Tristan Tarrant Master

            I'll clear up the distinction:

            `global-state` is where Infinispan stores global persistent state such as internal caches and node UUIDs.

            Normally you should not be touching the above.

            User caches instead use the location path and relative-to to say where they are going to store data.

            The `relative-to` attributes are not explicit paths, but server environment variables, such as `jboss.server.data.dir` (which happens to be the default value).

            1 of 1 people found this helpful
            • 3. Re: Infinispan server standalone as a docker swarm service using bind mount to persist file-store to host
              Prateek Kumar Newbie

              Thanks gustavonalle, I am trying to get it working but am still getting IllegalArgumentException (no path /custom found or /tmp not found) - for instance with

               

              <global-state>

              <persistent-location path="/tmp/" relative-to="/custom" />

              </global-state>

               

              If possible could you please suggest the possible values for path and relative-to inside the global-state.

               

              As a first step, I'd want to persist the file-store to say /tmp/ourApp.

              The container can be passed an environment variable via docker compose (-e "TEMP=/tmp) and it anyway has a jvm property $(java.io.tmpdir} pointed at /tmp

              • 4. Re: Infinispan server standalone as a docker swarm service using bind mount to persist file-store to host
                Prateek Kumar Newbie

                Thanks nadirx, I am hoping this can work but when I try running the container I still get IllegalArgument exception (Could not find a path called '${java.io.tmpdir}').

                 

                <subsystem

                    xmlns="urn:infinispan:server:core:9.3" default-cache-container="local">

                    <cache-container name="local" default-cache="default" statistics="true">

                        <global-state/>

                        <local-cache name="default"/>

                        <local-cache name="APP_CACHE">

                            <expiration lifespan="-1"/>

                            <file-store relative-to="${java.io.tmpdir}" path="ourAPP/APP_CACHE" max-entries="-1" purge="false" passivation="false"/>

                            <memory>

                                <binary size="100000000" eviction="MEMORY"/>

                            </memory>

                        </local-cache>

                    </cache-container>

                </subsystem>

                 

                If possible could you please suggest the possible values for relative-to since it seems I need to add this to our file-store element (urn:infinispan:config:9.3 ).

                For instance, I'd want to persist the file-store to say /tmp/ourApp/APP_CACHE/APP_CACHE.dat

                 

                I tried relative-to="$TEMP"

                (and separately these too)

                "TEMP"

                "/tmp" but still failed to start.

                 

                The container can be passed an environment variable via docker compose (-e "TEMP=/tmp) and it anyway has a jvm property $(java.io.tmpdir} pointed at /tmp and run with:

                 

                $docker run -d -p 9990:9990 -e "TEMP=/tmp" -e "APP_USER=ourApp" -e "APP_PASS=changeme" -e "MGMT_USER=ourApp" -e "MGMT_PASS=changeme" -v /c/Users/someone/DockerMapped/ourapp/datafiles/configuration/:/opt/jboss/infinispan-server/standalone/configuration/custom/ jboss/infinispan-server:9.3.2.Final standalone -c custom/infinispan_server.xml

                 

                Later I will add a -v /c/Users/someone/DockerMapped/ourapp/datafiles/configuration/persisted/:/tmp/ourApp in the run command/compose file as the bind mount to docker host.

                • 5. Re: Infinispan server standalone as a docker swarm service using bind mount to persist file-store to host
                  Tristan Tarrant Master

                  Sorry, I should have explained better.

                  First of all, just adding an empty <file-store/> element to a cache will automatically use "datagrid-infinispan/[container-name]" as path and "${jboss.server.data.dir}" as relative-to. The base name of the file is the name of the cache. Thus:

                   

                  [jboss.server.data.dir]/datagrid-infinispan/[container-name]/[cache-name].dat

                   

                  which for the default cache in the local container in standalone mode would be:

                   

                  standalone/data/datagrid-infinispan/local/default.dat

                   

                  The path concepts are detailed in the WildFly docs:

                  General configuration concepts - WildFly 10 - Project Documentation Editor

                  In particular relative-to is either one of the predefined names (e.g. jboss.server.data.dir) or a custom name path you can define using a <path> element.

                   

                  Supposing you want to put your cachestores under /var/lib/infinispan, you'd need to define a global path and then point your stores to use it as "relative-to":

                   

                  Example:

                   

                  <server xmlns="urn:jboss:domain:8.0">

                      <extensions>

                      ...

                      </extensions>

                      <paths>

                         <path name="cachestore.root" path="/var/lib/infinispan"/>

                      </paths>

                      ...

                      <profile>

                          ...

                          <subsystem xmlns="urn:infinispan:server:core:9.4" default-cache-container="local">

                              <cache-container name="local">

                                  <local-cache name="namedCache">

                                     <file-store relative-to="cachestore.root"/>

                                  </local-cache>

                              </cache-container>

                          </subsystem>

                          ...

                      </profile>

                      ...

                  </server>

                   

                   

                  This will create the cachestore for the namedCache cache as:

                   

                  /var/lib/infinispan/datagrid-infinispan/container/namedCache.dat

                   

                  If you set the file-store path attribute, it will be used in place of the default "datagrid-infinispan/container":

                   

                  <file-store relative-to="cachestore.root" path="/" />

                   

                  Will create the cachestore as:

                   

                  /var/lib/infinispan/namedCache.dat

                  1 of 1 people found this helpful