Issue in JBOSS7.0 Session Replication using DS

[srajeshacm]

We are using infinispan 8.0 with string-keyed-jdbc-store for session replication but getting below exception

09:16:10,967 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (Eopsbam_Thread734#) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.persistence.keymappers.UnsupportedKeyTypeException: Unsupported key type: 'org.wildfly.clustering.web.infinispan.session.SessionCreationMetaDataKey' on key: VRzcnUcSPipuDx-3gxyD38r5AKGTijKXkoP1_5zF

        at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.key2Str(JdbcStringBasedStore.java:405)

        at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.load(JdbcStringBasedStore.java:192)

Configuration :

<cache-container name="web" default-cache="database" module="org.wildfly.clustering.web.infinispan">

                <transport channel="ee-web" lock-timeout="60000"/>

                <local-cache name="database">

                    <string-keyed-jdbc-store data-source="hubds" passivation="true" preload="false" purge="false" shared="false" singleton="true">

                        <property name="database-Type">

                            oracle

                        </property>

                         <string-keyed-table prefix="A">

                            <id-column name="id" type="VARCHAR2(500)"/>

                            <data-column name="datum" type="VARCHAR2(200)"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </string-keyed-table>

                    </string-keyed-jdbc-store>

                </local-cache>

                <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                    <locking isolation="REPEATABLE_READ"/>

                    <transaction mode="BATCH"/>

                    <file-store/>

                </distributed-cache>

            </cache-container>

Please help ASAP

[pferraro]

The cache keys used by EAP 7.0 for storing web sessions are not strings, thus you cannot use the string-keyed-jdbc-store for distributed web sessions - you would need to use the binary-keyed-jdbc-store.

This limitation is addressed in EAP 7.1, where the binary-keyed-jdbc-store and mixed-keyed-jdbc-store are deprecated.  EAP 7.1 provides the requisite org.infinispan.persistence.keymappers.TwoWayKey2StringMapper to store the POJO keys as strings.

[srajeshacm]

Paul,

Thanks for Your timly response and sharing the information.

Even we tried with binary-keyed-jdbc-store, table got created but session details were not persisted into the db. Also We didn't get any exception

Configuration:

<cache-container name="web" default-cache="database" module="org.wildfly.clustering.web.infinispan">

                <transport channel="ee-web" lock-timeout="60000"/>

                <local-cache name="database">

                    <binary-keyed-jdbc-store data-source="hubds" passivation="true" preload="false" purge="false" shared="false" singleton="true">

                        <property name="database-Type">

                            oracle

                        </property>

                       <binary-keyed-table prefix="A">

                            <id-column name="id" type="VARCHAR2(500)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </binary-keyed-table>

                    </binary-keyed-jdbc-store>

                </local-cache>

                <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                    <locking isolation="REPEATABLE_READ"/>

                    <transaction mode="BATCH"/>

                    <file-store/>

                </distributed-cache>

            </cache-container>

[pferraro]

srajeshacm You've configured the jdbc store with passivation="true".   This means that sessions will only be persisted if the number of session in memory has exceeded the "max-active-sessions" as defined in jboss-web.xml.

I also noticed that you have the jdbc store configured with singleton="true".  I don't think you want that.  This means that only 1 node in the cluster will communicate with the database.  Instead, you probably want to use shared="true".

[eopsscb]

Hi Ferraro,

Continuing the thread from rajesh...

We have the mixed-keyed-jdbc-store in infinispan version 1.4 and the sessions are getting persisted (String) in DB without any issue in jboss eap 6.2. Can we upgrade or downgrade the infinispan version to have DB persistence (String) in jboss eap 7.0? . We tried with binary and all options u mentioned above with no luck.

JBOSS EAP 6.2 cache configurations (working fine)

        <subsystem xmlns="urn:jboss:domain:infinispan:1.4">

            <cache-container name="web" aliases="standard-session-cache" default-cache="database" module="org.jboss.as.clustering.web.infinispan">

                <transport lock-timeout="60000"/>

                <invalidation-cache name="database" mode="ASYNC" batching="true">

                    <mixed-keyed-jdbc-store datasource="java:/jdbc/DefaultDS" preload="true" passivation="false" purge="false">

                        <property name="databaseType">

                            oracle

                        </property>

                        <binary-keyed-table prefix="JBB">

                            <id-column name="id" type="VARCHAR2(200)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </binary-keyed-table>

                        <string-keyed-table prefix="JBS">

                           <id-column name="id" type="VARCHAR2(200)"/>

                           <data-column name="datum" type="BLOB"/>

                           <timestamp-column name="version" type="NUMBER"/>

                        </string-keyed-table>

                     </mixed-keyed-jdbc-store>

                </invalidation-cache>

           </cache-container>

JBOSS EAP7.0 cache configurations (Not working)

      <subsystem xmlns="urn:jboss:domain:infinispan:4.0">

            <cache-container name="web" default-cache="database" module="org.wildfly.clustering.web.infinispan">

                <transport channel="ee-web" lock-timeout="60000"/>

                <local-cache name="database">

                    <binary-keyed-jdbc-store data-source="hubds" passivation="false" preload="false" purge="false" shared="true" singleton="false">

                        <property name="database-Type">

                            oracle

                        </property>

                        <binary-keyed-table prefix="A">

                            <id-column name="id" type="VARCHAR2(500)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </binary-keyed-table>

                    </binary-keyed-jdbc-store>

                </local-cache>

            </cache-container>

[pferraro]

eopsscb

We have the mixed-keyed-jdbc-store in infinispan version 1.4 and the sessions are getting persisted (String) in DB without any issue in jboss eap 6.2. Can we upgrade or downgrade the infinispan version to have DB persistence (String) in jboss eap 7.0? . We tried with binary and all options u mentioned above with no luck.

No.  The distributed session manager was completely rewritten in EAP7.  Using SESSION granularity, a given web session is persisted in 3 entries, one containing creation meta data (only written once), another containing access meta data (written every request), and another containing session attributes (written as needed).  In this way, we minimize the size of data required to be serialized and replicated/persisted on a given request.

JBOSS EAP7.0 cache configurations (Not working)

<subsystem xmlns="urn:jboss:domain:infinispan:4.0">

<cache-container name="web" default-cache="database" module="org.wildfly.clustering.web.infinispan">

<transport channel="ee-web" lock-timeout="60000"/>

<local-cache name="database">

<binary-keyed-jdbc-store data-source="hubds" passivation="false" preload="false" purge="false" shared="true" singleton="false">

<property name="database-Type">

oracle

</property>

<binary-keyed-table prefix="A">

<id-column name="id" type="VARCHAR2(500)"/>

<data-column name="datum" type="BLOB"/>

<timestamp-column name="version" type="NUMBER"/>

</binary-keyed-table>

</binary-keyed-jdbc-store>

</local-cache>

</cache-container>

Can you be more specific?  What do you mean by "not working"?  Are there errors in the log?

A few comments about your configuration:

1. If your application is load balanced across multiple EAP nodes, you'll want to configure your "database" cache as an <invalidation-cache/>, instead of a <local-cache/>.  This will ensure that other nodes do not retain out-dated cache entries in memory.  If you only use a single EAP node, then a <local-cache/> is fine, and you can get rid of your <transport/> config.
2. You have not defined your database type correctly.  Your config should look like:

<binary-keyed-jdbc-store data-source="hubds" dialect="ORACLE" passivation="false" purge="false" shared="true">...</binary-keyed-jdbc-store>

[eopsscb]

Hi Ferraro,

1.

Thanks a lot for the suggestions, now it worked well with local cache mode and the session persisting in binary mode with below config. Still we left with one issue in local-cache mode, the session from the table are not purged post invalidation. We enabled purge flag as true with no luck. Please advise how to purge the old sessions from the tables.

            <cache-container name="web" default-cache="database" module="org.wildfly.clustering.web.infinispan">

               <local-cache name="database" >

                    <binary-keyed-jdbc-store data-source="EOPSSEC" dialect="ORACLE" passivation="false" preload="false" purge="true" shared="true" singleton="false">

                        <property name="database-Type">

                            oracle

                        </property>

                        <binary-keyed-table prefix="SESS">

                            <id-column name="id" type="VARCHAR2(500)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </binary-keyed-table>

                    </binary-keyed-jdbc-store>

                </local-cache>

            </cache-container>

2.

We also tried with invalidation-cache mode as per your suggestion with the below configuration.

                <invalidation-cache name="database" mode="ASYNC">

                   <binary-keyed-jdbc-store data-source="EOPSSEC" dialect="ORACLE" passivation="false" preload="false" purge="false" shared="true" singleton="false">

                      <property name="database-Type">

                            oracle

                        </property>

                        <binary-keyed-table prefix="SESS">

                            <id-column name="id" type="VARCHAR2(500)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </binary-keyed-table>

                    </binary-keyed-jdbc-store>

                </invalidation-cache>

            </cache-container>

With this config, during startup, we are getting the below issue as fetchPersistentState not available, request your support to fix this as well.

ocal address is UAT_EOPSBAM_CL02, physical addresses are [127.0.0.1:12209]

12:12:38,722 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-6) ISPN000128: Infinispan version: Infinispan 'Mahou' 8.1.6.Final-redhat-1

12:12:38,729 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 83) MSC000001: Failed to start service jboss.infinispan.web.database.config: org.jboss.msc.service.StartException in service jboss.infinispan.web.database.config: org.infinispan.commons.CacheConfigurationException: ISPN000420: Cannot enable 'fetchPersistentState' in invalidation caches!

        at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_60]

        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]

        at org.jboss.threads.JBossThread.run(JBossThread.java:320)

Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000420: Cannot enable 'fetchPersistentState' in invalidation caches!

        at org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder.validate(AbstractStoreConfigurationBuilder.java:181)

        at org.infinispan.persistence.jdbc.configuration.AbstractJdbcStoreConfigurationBuilder.validate(AbstractJdbcStoreConfigurationBuilder.java:86)

        at org.infinispan.configuration.cache.PersistenceConfigurationBuilder.validate(PersistenceConfigurationBuilder.java:106)

        at org.infinispan.configuration.cache.ConfigurationBuilder.validate(ConfigurationBuilder.java:203)

        at org.infinispan.configuration.cache.ConfigurationBuilder.build(ConfigurationBuilder.java:246)

        at org.infinispan.configuration.cache.ConfigurationBuilder.build(ConfigurationBuilder.java:236)

        at org.wildfly.clustering.infinispan.spi.service.ConfigurationBuilder.start(ConfigurationBuilder.java:79)

        at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)

        ... 4 more

[pferraro]

I've opened [JBEAP-11563] Cache store should auto-disable fetch-state when used with invalidation-cache - JBoss Issue Tracker

[pferraro]

The above issue is meant to auto-disable fetching of state for invalidation caches.  In the meantime, use the following:

<binary-keyed-jdbc-store data-source="EOPSSEC" dialect="ORACLE" passivation="false" preload="false" purge="false" shared="true" singleton="false" fetch-state="false">...</binary-keyed-jdbc-store>

[eopsscb]

Hi Ferraro,

Many thanks, it worked finally. Kindly let me know when 7.1 would be released with string based persistence.

[pferraro]

Follow the Alpha/Beta/CR/GA releases here:

Red Hat JBoss Enterprise Application Platform Download | Red Hat Developers

[srajeshacm]

Ferraro,

Thanks a lot for your continuous support and explanations.

Everything working as  expected but session tables were not purged for invalidated session on server restart. is there any properties to be enabled to purge all invalidation session details from the session tables?

Thanks in advance.

[pferraro]

This is controlled by the purge="true" attribute of the cache store. However, if your cache store is shared, however, then it would inadvisable to purge it on server restart, as that would delete sessions persisted by all active nodes in your cluster.

[venkatesh.2707]

Hi Paul ,

Following up to the  above configuration , we are getting replication time out issue with horizontal cluster setup during Performance testing .

Error :

10:38:02,725 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-80) ISPN000136: Error executing command PutKeyValueCommand, writing keys [-qJgQeGX6ow0rs5kMNjERlY7rpAMh-8XfYOxn19h]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for UAT_EOPSHUB_CL02

11:33:14,461 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (thread-19,ee,UAT_EOPSHUB_CL02) ISPN000136: Error executing command InvalidateCommand, writing keys [LtZXTU-27Hsp-hw7Jn55PA-r299SK5F4CelwGWV1]: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key LtZXTU-27Hsp-hw7Jn55PA-r299SK5F4CelwGWV1 and requestor CommandUUID{address=UAT_EOPSHUB_CL03, id=8261}. Lock is held by CommandUUID{address=UAT_EOPSHUB_CL01, id=8520}

Configuration :JBOSS EAP7.0 cache configurations (Not working)

            <cache-container name="web" default-cache="database" module="org.wildfly.clustering.web.infinispan">

                <transport lock-timeout="60000"/>

                <invalidation-cache name="database" mode="ASYNC">

                    <binary-keyed-jdbc-store data-source="EOPSSEC" dialect="ORACLE" fetch-state="false" passivation="false" preload="false" purge="false" shared="true" singleton="false">

                        <property name="database-Type">

                            oracle

                        </property>

                        <binary-keyed-table prefix="SESS">

                            <id-column name="id" type="VARCHAR2(500)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </binary-keyed-table>

                    </binary-keyed-jdbc-store>

                </invalidation-cache>

            </cache-container>

We have tried with this configuration :

         </cache-container>

            <cache-container name="web" default-cache="database" module="org.wildfly.clustering.web.infinispan">

                <transport channel="ee-web" lock-timeout="240000"/>

                <invalidation-cache name="database" mode="ASYNC">

                    <binary-keyed-jdbc-store data-source="EOPSSEC" dialect="ORACLE" fetch-state="false" passivation="false" preload="false" purge="false" shared="true" singleton="false">

                        <property name="database-Type">

                            oracle

                        </property>

                        <binary-keyed-table prefix="SESS">

                            <id-column name="id" type="VARCHAR2(500)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </binary-keyed-table>

                    </binary-keyed-jdbc-store>

                </invalidation-cache>

            </cache-container>

<channels default="ee">

    <channel name="ee" stack="tcp"/>

    <channel name="ee-server" stack="tcpping"/>

    <channel name="ee-web" stack="tcpping"/>

    <channel name="ee-ejb" stack="tcpping"/>

    <channel name="ee-hibernate" stack="tcpping"/>

</channels>

Now we are not getting Replication timeout error But the below message is throwing repeatedly in all standalone cluster members.

00:03:36,998 WARN  [org.jgroups.protocols.TCP] (thread-1,ee,UAT_EOPSHUB_CL01) JGRP000012: discarded message from different cluster ee-web (our cluster is ee). Sender was UAT_EOPSHUB_CL01 (received 5 identical messages from UAT_EOPSHUB_CL01 in the last 69144 ms)

00:04:05,301 WARN  [org.jgroups.protocols.TCP] (thread-2,ee,UAT_EOPSHUB_CL01) JGRP000012: discarded message from different cluster ee-web (our cluster is ee). Sender was eac206fd-6e0b-2fcd-b402-e9a8e664b60e (received 6 identical messages from eac206fd-6e0b-2fcd-b402-e9a8e664b60e in the last 78029 ms)

Can you please let us know we can use the above configuration and how to get rid of this message ? and why its occuring ?

Thankyou,

Can you Please help ASAP ....