Combine Embedded and Server as Clustered Infinispan

[pranjal.jaju]

as a proof of concept, I try to build an infinispan cluster with an existing application which starts an embedded cache and one or more standalone infinispan servers.

The reasosing behind this is, that I want to show, that there is a way of zero-configuration automatic creation of a cluster by simply starting freshly downloaded infinispan standalone servers. So my application runs an embedded cache which will be automatically "joined" by new nodes.

I'm using the very default configuration for infinispan and jgroups (see below).

The effect is, that two or more of my applications with embedded cache "see each other" and two or more standalone infinispan servers see each other. But none of my nodes "see" the standalone nodes and they don't see my nodes.

I use infinispan 7.2.3.

My Embedded Configuration is Like-

    builder = new ConfigurationBuilder();

  GlobalConfigurationBuilder globalBuilder = GlobalConfigurationBuilder.defaultClusteredBuilder();

  if (enableDiskPersistance)

  {

  builder.jmxStatistics().enable().persistence().passivation(true).addSingleFileStore().location("data/cacheData").async().threadPoolSize(15).flushLockTimeout(15000).transaction().transactionMode(TransactionMode.TRANSACTIONAL);

  builder.eviction().strategy(EvictionStrategy.LIRS).maxEntries(10000L).expiration().maxIdle(-1, TimeUnit.SECONDS);

  builder.clustering().cacheMode(CacheMode.DIST_ASYNC).l1().enable();

  }

  else

  {

  builder.jmxStatistics().enable().transaction().transactionMode(TransactionMode.TRANSACTIONAL);

  builder.clustering().cacheMode(CacheMode.DIST_ASYNC).hash().numOwners(2).numSegments(20).l1().enable().compatibility().enable().locking().lockAcquisitionTimeout(30000).concurrencyLevel(1000);

  }

  globalBuilder.transport().clusterName("clustered").nodeName( "embededNode" )

  .addProperty( "configurationFile", "jgroups.xml").globalJmxStatistics().enable().allowDuplicateDomains(true).jmxDomain("CLEARINSIGHT").build();

  cacheManager = new DefaultCacheManager(globalBuilder.build(),builder.build());

  org.infinispan.Cache<String, Object> cache = cacheManager.getCache( "default" );

and Jgroups.xml is like -

     <config xmlns="urn:org:jgroups"

            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

            xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-3.2.xsd">

        <UDP

         mcast_addr="${jgroups.udp.mcast_addr:234.99.54.14}"

         mcast_port="${jgroups.udp.mcast_port:45688}"

         tos="8"

         ucast_recv_buf_size="20m"

         ucast_send_buf_size="640k"

         mcast_recv_buf_size="25m"

         mcast_send_buf_size="640k"

         max_bundle_size="31k"

         ip_ttl="${jgroups.udp.ip_ttl:2}"

         enable_diagnostics="false"

         bundler_type="sender-sends-with-timer"

         thread_naming_pattern="pl"

         thread_pool.enabled="true"

         thread_pool.min_threads="2"

         thread_pool.max_threads="30"

         thread_pool.keep_alive_time="60000"

         thread_pool.queue_enabled="true"

         thread_pool.queue_max_size="100"

         thread_pool.rejection_policy="Discard"

         oob_thread_pool.enabled="true"

         oob_thread_pool.min_threads="2"

         oob_thread_pool.max_threads="30"

         oob_thread_pool.keep_alive_time="60000"

         oob_thread_pool.queue_enabled="false"

         oob_thread_pool.queue_max_size="100"

         oob_thread_pool.rejection_policy="Discard"

         internal_thread_pool.enabled="true"

         internal_thread_pool.min_threads="2"

         internal_thread_pool.max_threads="10"

         internal_thread_pool.keep_alive_time="60000"

         internal_thread_pool.queue_enabled="true"

         internal_thread_pool.queue_max_size="100"

         internal_thread_pool.rejection_policy="Discard"

         />

        <PING timeout="3000" num_initial_members="2"/>

        <MERGE2 max_interval="30000" min_interval="10000"/>

        <FD_SOCK/>

        <FD_ALL timeout="15000" interval="3000"/>

        <VERIFY_SUSPECT timeout="1500"/>

        <pbcast.NAKACK2

                    xmit_interval="1000"

                    xmit_table_num_rows="100"

                    xmit_table_msgs_per_row="10000"

                    xmit_table_max_compaction_time="10000"

                    max_msg_batch_size="100"/>

        <UNICAST3

              xmit_interval="500"

              xmit_table_num_rows="20"

              xmit_table_msgs_per_row="10000"

              xmit_table_max_compaction_time="10000"

              max_msg_batch_size="100"

              conn_expiry_timeout="0"/>

        <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="1m"/>

        <pbcast.GMS print_local_addr="false" join_timeout="3000" view_bundling="true"/>

        <tom.TOA/> <!-- the TOA is only needed for total order transactions-->

        <UFC max_credits="2m" min_threshold="0.40"/>

        <MFC max_credits="2m" min_threshold="0.40"/>

        <FRAG2 frag_size="30k" />

        <RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" />

    </config>

And My Cluster.xml is -

    <server xmlns="urn:jboss:domain:2.1">

    <extensions>

        <extension module="org.infinispan.server.endpoint"/>

        <extension module="org.jboss.as.clustering.infinispan"/>

        <extension module="org.jboss.as.clustering.jgroups"/>

        <extension module="org.jboss.as.connector"/>

        <extension module="org.jboss.as.deployment-scanner"/>

        <extension module="org.jboss.as.jdr"/>

        <extension module="org.jboss.as.jmx"/>

        <extension module="org.jboss.as.logging"/>

        <extension module="org.jboss.as.modcluster"/>

        <extension module="org.jboss.as.naming"/>

        <extension module="org.jboss.as.remoting"/>

        <extension module="org.jboss.as.security"/>

        <extension module="org.jboss.as.threads"/>

        <extension module="org.jboss.as.transactions"/>

        <extension module="org.wildfly.extension.io"/>

    </extensions>

    <management>

        <security-realms>

            <security-realm name="ManagementRealm">

                <authentication>

                    <local default-user="$local" skip-group-loading="true"/>

                    <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>

                </authentication>

                <authorization map-groups-to-roles="false">

                    <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>

                </authorization>

            </security-realm>

            <security-realm name="ApplicationRealm">

                <authentication>

                    <local default-user="$local" allowed-users="*" skip-group-loading="true"/>

                    <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>

                </authentication>

                <authorization>

                    <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>

                </authorization>

            </security-realm>

        </security-realms>

        <audit-log>

            <formatters>

                <json-formatter name="json-formatter"/>

            </formatters>

            <handlers>

                <file-handler name="file" formatter="json-formatter" relative-to="jboss.server.data.dir" path="audit-log.log"/>

            </handlers>

            <logger log-boot="true" log-read-only="false" enabled="false">

                <handlers>

                    <handler name="file"/>

                </handlers>

            </logger>

        </audit-log>

        <management-interfaces>

            <http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">

                <socket-binding http="management-http"/>

            </http-interface>

        </management-interfaces>

        <access-control provider="simple">

            <role-mapping>

                <role name="SuperUser">

                    <include>

                        <user name="$local"/>

                    </include>

                </role>

            </role-mapping>

        </access-control>

    </management>

    <profile>

        <subsystem xmlns="urn:jboss:domain:logging:2.0">

            <console-handler name="CONSOLE">

                <level name="INFO"/>

                <formatter>

                    <named-formatter name="COLOR-PATTERN"/>

                </formatter>

            </console-handler>

            <periodic-rotating-file-handler name="FILE" autoflush="true">

                <formatter>

                    <named-formatter name="PATTERN"/>

                </formatter>

                <file relative-to="jboss.server.log.dir" path="server.log"/>

                <suffix value=".yyyy-MM-dd"/>

                <append value="true"/>

            </periodic-rotating-file-handler>

            <logger category="com.arjuna">

                <level name="WARN"/>

            </logger>

            <logger category="org.apache.tomcat.util.modeler">

                <level name="WARN"/>

            </logger>

            <logger category="org.jboss.as.config">

                <level name="DEBUG"/>

            </logger>

            <logger category="sun.rmi">

                <level name="WARN"/>

            </logger>

            <logger category="jacorb">

                <level name="WARN"/>

            </logger>

            <logger category="jacorb.config">

                <level name="ERROR"/>

            </logger>

            <root-logger>

                <level name="INFO"/>

                <handlers>

                    <handler name="CONSOLE"/>

                    <handler name="FILE"/>

                </handlers>

            </root-logger>

            <formatter name="PATTERN">

                <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>

            </formatter>

            <formatter name="COLOR-PATTERN">

                <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>

            </formatter>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">

            <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>

        </subsystem>

        <subsystem xmlns="urn:infinispan:server:endpoint:7.2">

            <hotrod-connector socket-binding="hotrod" cache-container="clustered">

                <topology-state-transfer lazy-retrieval="false" lock-timeout="1000" replication-timeout="5000"/>

            </hotrod-connector>

            <memcached-connector socket-binding="memcached" cache-container="clustered"/>

            <rest-connector socket-binding="rest" cache-container="clustered" security-domain="other" auth-method="BASIC"/>

            <websocket-connector socket-binding="websocket" cache-container="clustered"/>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:datasources:2.0">

            <datasources/>

        </subsystem>

        <subsystem xmlns="urn:infinispan:server:core:7.2" default-cache-container="clustered">

            <cache-container name="clustered" default-cache="default" statistics="true">

                <transport executor="infinispan-transport" lock-timeout="60000"/>

                <distributed-cache name="default" mode="SYNC" segments="20" owners="2" remote-timeout="30000" start="EAGER">

                    <locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                    <transaction mode="NONE"/>

                </distributed-cache>

                <distributed-cache name="memcachedCache" mode="SYNC" segments="20" owners="2" remote-timeout="30000" start="EAGER">

                    <locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                    <transaction mode="NONE"/>

                </distributed-cache>

                <distributed-cache name="namedCache" mode="SYNC" start="EAGER"/>

                <distributed-cache name="transactionalCache" mode="SYNC" start="EAGER">

                    <transaction mode="NON_XA" locking="PESSIMISTIC"/>

                </distributed-cache>

            </cache-container>

            <cache-container name="security"/>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:io:1.1">

            <worker name="default"/>

            <buffer-pool name="default"/>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:jca:2.0">

            <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>

            <bean-validation enabled="true"/>

            <default-workmanager>

                <short-running-threads>

                    <core-threads count="50"/>

                    <queue-length count="50"/>

                    <max-threads count="50"/>

                    <keepalive-time time="10" unit="seconds"/>

                </short-running-threads>

                <long-running-threads>

                    <core-threads count="50"/>

                    <queue-length count="50"/>

                    <max-threads count="50"/>

                    <keepalive-time time="10" unit="seconds"/>

                </long-running-threads>

            </default-workmanager>

            <cached-connection-manager/>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>

        <subsystem xmlns="urn:infinispan:server:jgroups:7.0" default-stack="${jboss.default.jgroups.stack:udp}">

            <stack name="udp">

                <transport type="UDP" socket-binding="jgroups-udp"/>

                <protocol type="PING"/>

                <protocol type="MERGE3"/>

                <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>

                <protocol type="FD_ALL"/>

                <protocol type="VERIFY_SUSPECT"/>

                <protocol type="pbcast.NAKACK2"/>

                <protocol type="UNICAST3"/>

                <protocol type="pbcast.STABLE"/>

                <protocol type="pbcast.GMS"/>

                <protocol type="UFC"/>

                <protocol type="MFC"/>

                <protocol type="FRAG2"/>

            </stack>

            <stack name="tcp">

                <transport type="TCP" socket-binding="jgroups-tcp"/>

                <protocol type="MPING" socket-binding="jgroups-mping"/>

                <protocol type="MERGE3"/>

                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                <protocol type="FD_ALL"/>

                <protocol type="VERIFY_SUSPECT"/>

                <protocol type="pbcast.NAKACK2">

                    <property name="use_mcast_xmit">false</property>

                </protocol>

                <protocol type="UNICAST3"/>

                <protocol type="pbcast.STABLE"/>

                <protocol type="pbcast.GMS"/>

                <protocol type="MFC"/>

                <protocol type="FRAG2"/>

            </stack>

            <stack name="s3">

                <transport type="TCP" socket-binding="jgroups-tcp"/>

                <protocol type="S3_PING">

                    <property name="location">${jgroups.s3.bucket:}</property>

                    <property name="access_key">${jgroups.s3.access_key:}</property>

                    <property name="secret_access_key">${jgroups.s3.secret_access_key:}</property>

                    <property name="pre_signed_delete_url">${jgroups.s3.pre_signed_delete_url:}</property>

                    <property name="pre_signed_put_url">${jgroups.s3.pre_signed_put_url:}</property>

                    <property name="prefix">${jgroups.s3.prefix:}</property>

                </protocol>

                <protocol type="MERGE3"/>

                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                <protocol type="FD_ALL"/>

                <protocol type="VERIFY_SUSPECT"/>

                <protocol type="pbcast.NAKACK2">

                    <property name="use_mcast_xmit">false</property>

                </protocol>

                <protocol type="UNICAST3"/>

                <protocol type="pbcast.STABLE"/>

                <protocol type="pbcast.GMS"/>

                <protocol type="MFC"/>

                <protocol type="FRAG2"/>

            </stack>

            <stack name="google">

                <transport type="TCP" socket-binding="jgroups-tcp"/>

                <protocol type="GOOGLE_PING">

                    <property name="location">${jgroups.google.bucket:}</property>

                    <property name="access_key">${jgroups.google.access_key:}</property>

                    <property name="secret_access_key">${jgroups.google.secret_access_key:}</property>

                </protocol>

                <protocol type="MERGE3"/>

                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                <protocol type="FD_ALL"/>

                <protocol type="VERIFY_SUSPECT"/>

                <protocol type="pbcast.NAKACK2">

                    <property name="use_mcast_xmit">false</property>

                </protocol>

                <protocol type="UNICAST3"/>

                <protocol type="pbcast.STABLE"/>

                <protocol type="pbcast.GMS"/>

                <protocol type="MFC"/>

                <protocol type="FRAG2"/>

            </stack>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:jmx:1.3">

            <expose-resolved-model/>

            <expose-expression-model/>

            <remoting-connector/>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:modcluster:1.2">

            <mod-cluster-config advertise-socket="modcluster" connector="ajp">

                <dynamic-load-provider>

                    <load-metric type="cpu"/>

                </dynamic-load-provider>

            </mod-cluster-config>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:naming:2.0">

            <remote-naming/>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:remoting:2.0">

            <endpoint worker="default"/>

            <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:security:1.2">

            <security-domains>

                <security-domain name="other" cache-type="default">

                    <authentication>

                        <login-module code="Remoting" flag="optional">

                            <module-option name="password-stacking" value="useFirstPass"/>

                        </login-module>

                        <login-module code="RealmDirect" flag="required">

                            <module-option name="password-stacking" value="useFirstPass"/>

                        </login-module>

                    </authentication>

                </security-domain>

                <security-domain name="jboss-web-policy" cache-type="default">

                    <authorization>

                        <policy-module code="Delegating" flag="required"/>

                    </authorization>

                </security-domain>

            </security-domains>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:threads:1.1">

            <thread-factory name="infinispan-factory" group-name="infinispan" priority="5" thread-name-pattern="%G %f-%t"/>

            <unbounded-queue-thread-pool name="infinispan-transport">

                <max-threads count="25"/>

                <keepalive-time time="0" unit="milliseconds"/>

                <thread-factory name="infinispan-factory"/>

            </unbounded-queue-thread-pool>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:transactions:2.0">

            <core-environment>

                <process-id>

                    <uuid/>

                </process-id>

            </core-environment>

            <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>

        </subsystem>

    </profile>

    <interfaces>

        <interface name="management">

            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

        </interface>

        <interface name="public">

            <inet-address value="${jboss.bind.address:127.0.0.1}"/>

        </interface>

    </interfaces>

    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>

        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>

        <socket-binding name="hotrod" port="11222"/>

        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45700"/>

        <socket-binding name="jgroups-tcp" port="7600"/>

        <socket-binding name="jgroups-tcp-fd" port="57600"/>

        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45688"/>

        <socket-binding name="jgroups-udp-fd" port="54200"/>

        <socket-binding name="memcached" port="11211"/>

        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.115" multicast-port="23364"/>

        <socket-binding name="rest" port="8080"/>

        <socket-binding name="txn-recovery-environment" port="4712"/>

        <socket-binding name="txn-status-manager" port="4713"/>

        <socket-binding name="websocket" port="8181"/>

    </socket-binding-group>

</server>

i am able to connect both servers but getting Warning log Stack Trace Like -

    2015-07-08 11:02:52,463 ERROR [org.jgroups.blocks.RequestCorrelator] (OOB-4,shared=udp) failed unmarshalling buffer into return value: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:754) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:95) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:948) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.infinispan.distribution.ch.impl.ReplicatedConsistentHash$Externalizer.doReadObject(ReplicatedConsistentHash.java:327) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.distribution.ch.impl.ReplicatedConsistentHash$Externalizer.doReadObject(ReplicatedConsistentHash.java:314) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.commons.marshall.InstanceReusingAdvancedExternalizer.readObject(InstanceReusingAdvancedExternalizer.java:102) [infinispan-commons.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153) [infinispan-core.jar:7.2.3.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.infinispan.topology.CacheTopology$Externalizer.doReadObject(CacheTopology.java:201) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.topology.CacheTopology$Externalizer.doReadObject(CacheTopology.java:186) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.commons.marshall.InstanceReusingAdvancedExternalizer.readObject(InstanceReusingAdvancedExternalizer.java:102) [infinispan-commons.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153) [infinispan-core.jar:7.2.3.Final]

[rvansa]

The most common issue with connecting nodes is IPv4/IPv6 on dual-stack machines. Please add -Djava.net.preferIPv4Stack=true on both embedded node and make sure this flag is used for the server.

[pranjal.jaju]

I had Already done this my nodes are connecting as one node can discover another but Continues to Give Exceptions

Providing The Stack Trace -  (On Embedded Node if Start Server as Coordinator i.e. started First)

Jul 09, 2015 1:38:48 PM org.jgroups.blocks.RequestCorrelator receiveMessage

SEVERE: failed unmarshalling buffer into return value

java.io.UTFDataFormatException: Invalid byte

  at org.jboss.marshalling.UTFUtils.readUTFBytes(UTFUtils.java:162)

  at org.jboss.marshalling.river.RiverUnmarshaller.readUTF(RiverUnmarshaller.java:1833)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:959)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

  at org.infinispan.topology.CacheTopology$Externalizer.doReadObject(CacheTopology.java:204)

  at org.infinispan.topology.CacheTopology$Externalizer.doReadObject(CacheTopology.java:186)

  at org.infinispan.commons.marshall.InstanceReusingAdvancedExternalizer.readObject(InstanceReusingAdvancedExternalizer.java:102)

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436)

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227)

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

  at org.infinispan.topology.CacheStatusResponse$Externalizer.readObject(CacheStatusResponse.java:73)

  at org.infinispan.topology.CacheStatusResponse$Externalizer.readObject(CacheStatusResponse.java:60)

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436)

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227)

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

  at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:92)

  at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:77)

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436)

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227)

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

  at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:134)

  at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)

  at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)

  at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)

  at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)

  at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)

  at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)

  at org.jgroups.JChannel.up(JChannel.java:739)

  at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)

  at org.jgroups.protocols.RSVP.up(RSVP.java:201)

  at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)

  at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)

  at org.jgroups.protocols.FlowControl.up(FlowControl.java:383)

  at org.jgroups.protocols.tom.TOA.up(TOA.java:121)

  at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)

  at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)

  at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)

  at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)

  at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)

  at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)

  at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)

  at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)

  at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)

  at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)

  at org.jgroups.protocols.Discovery.up(Discovery.java:291)

  at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)

  at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

  at java.lang.Thread.run(Thread.java:745)

and If Embeded Node As Coordinator then on Server Node :

2015-07-09 13:43:47,261 ERROR [org.jgroups.blocks.RequestCorrelator] (OOB-13,shared=udp) failed unmarshalling buffer into return value: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:754) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:95) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:948) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.infinispan.distribution.ch.impl.ReplicatedConsistentHash$Externalizer.doReadObject(ReplicatedConsistentHash.java:327) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.distribution.ch.impl.ReplicatedConsistentHash$Externalizer.doReadObject(ReplicatedConsistentHash.java:314) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.commons.marshall.InstanceReusingAdvancedExternalizer.readObject(InstanceReusingAdvancedExternalizer.java:102) [infinispan-commons.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153) [infinispan-core.jar:7.2.3.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.infinispan.topology.CacheTopology$Externalizer.doReadObject(CacheTopology.java:201) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.topology.CacheTopology$Externalizer.doReadObject(CacheTopology.java:186) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.commons.marshall.InstanceReusingAdvancedExternalizer.readObject(InstanceReusingAdvancedExternalizer.java:102) [infinispan-commons.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153) [infinispan-core.jar:7.2.3.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.infinispan.topology.CacheStatusResponse$Externalizer.readObject(CacheStatusResponse.java:72) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.topology.CacheStatusResponse$Externalizer.readObject(CacheStatusResponse.java:60) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153) [infinispan-core.jar:7.2.3.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:92) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:77) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:153) [infinispan-core.jar:7.2.3.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-osgi-1.4.10.Final.jar:1.4.10.Final]

  at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:134) [infinispan-commons.jar:7.2.3.Final]

  at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101) [infinispan-core.jar:7.2.3.Final]

  at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80) [infinispan-commons.jar:7.2.3.Final]

  at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28) [infinispan-core.jar:7.2.3.Final]

  at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.JChannel.up(JChannel.java:739) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.FlowControl.up(FlowControl.java:394) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.FlowControl.up(FlowControl.java:383) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.MERGE3.up(MERGE3.java:288) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.Discovery.up(Discovery.java:291) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2842) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796) [jgroups-3.6.2.Final.jar:3.6.2.Final]

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]

  at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]

[dan.berindei]

Are you sure the embedded node also uses Infinispan 7.2.3.Final? The CacheTopology externalizer last changed in 7.0.0.alpha4, so it looks like the embedded node is using a version older than that.

[pranjal.jaju]

Yes I am 100% sure i am using Same Version 7.2.3.Final of infinispan in embedded mode.

[dan.berindei]

Unfortunately that was my only guess. If you can post a Maven project reproducing the problem here, I'll try to look deeper into it.

[pranjal.jaju]

Maven Project is Available at : cachelibrary.rar - Google Drive

You can check this out here.