7 Replies Latest reply on Jul 10, 2013 9:04 AM by mike-evans

    startServer.sh replacement in 5.3

    mike-evans

      I've been playing with Infinispan as a potential replacement for memcached in our Clearwater project.  I was testing with 5.2.1, but wanted to upgrade to 5.3.0 to see if it fixes some performance variability and stability issues we were hitting, but our start-up script (which was based on startServer.sh) no longer works.  As far as I can tell from release notes, startServer.sh has been retired in 5.3, so what is the suggested alternative way of starting Infinispan as a standalone server?  The user manual section on running as a memcached server still refers to startServer.sh.

       

      The output from running our script is as follows.

       

      Exception in thread "main" java.lang.NoClassDefFoundError: org/infinispan/server/core/Main

      Caused by: java.lang.ClassNotFoundException: org.infinispan.server.core.Main

              at java.net.URLClassLoader$1.run(URLClassLoader.java:217)

              at java.security.AccessController.doPrivileged(Native Method)

              at java.net.URLClassLoader.findClass(URLClassLoader.java:205)

              at java.lang.ClassLoader.loadClass(ClassLoader.java:321)

              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)

              at java.lang.ClassLoader.loadClass(ClassLoader.java:266)

      Could not find the main class: org.infinispan.server.core.Main. Program will exit.

      3.0.Final-all/modules/websocket/runtime-classpath.txt

       

      Our start-up scripts is as follows ...

      -----

      #!/bin/bash

       

      source "`dirname "$0"`/functions.sh"

       

      add_classpath ${ISPN_HOME}/etc

      add_classpath ${ISPN_HOME}/modules/memcached/*.jar

      add_classpath ${ISPN_HOME}/modules/hotrod/*.jar

      add_classpath ${ISPN_HOME}/modules/websocket/*.jar

      add_classpath ${ISPN_HOME}/modules/cli-server/*.jar

      add_classpath ${ISPN_HOME}/modules/memcached/runtime-classpath.txt

      add_classpath ${ISPN_HOME}/modules/hotrod/runtime-classpath.txt

      add_classpath ${ISPN_HOME}/modules/websocket/runtime-classpath.txt

      add_classpath ${ISPN_HOME}/modules/cli-server/runtime-classpath.txt

       

      add_jvm_args $JVM_PARAMS

      add_jvm_args '-Djgroups.bind_addr=10.190.149.173'

      add_jvm_args '-Djava.net.preferIPv4Stack=true'

       

      # RHQ monitoring options

      add_jvm_args '-Dcom.sun.management.jmxremote.ssl=false'

      add_jvm_args '-Dcom.sun.management.jmxremote.authenticate=false'

      add_jvm_args -Dcom.sun.management.jmxremote.port=$(find_tcp_port)

      add_jvm_args -DCFGPath=${ISPN_HOME}/etc/clearwater

       

      # Workaround for JDK6 NPE: http://bugs.sun.com/view_bug.do?bug_id=6427854

      add_jvm_args '-Dsun.nio.ch.bugLevel=""'

       

      # Sample JPDA settings for remote socket debugging

      #add_jvm_args "-Xrunjdwp:transport=dt_socket,address=8686,server=y,suspend=n"

       

      # LOG4J configuration

      # LOG4J_CONFIG=file:///${ISPN_HOME}/etc/log4j.xml

       

      add_program_args -c "${ISPN_HOME}/etc/clearwater/cache-config.xml" -r memcached -p 11211

       

      start org.infinispan.server.core.Main

      ---

        • 1. Re: startServer.sh replacement in 5.3
          nadirx

          Hi Mike,

           

          sorry about the outdated information. You now need to use Infinispan Server: https://docs.jboss.org/author/display/ISPN/Infinispan+Server

          • 2. Re: startServer.sh replacement in 5.3
            mike-evans

            Tristan

             

            Thanks for the quick reply.  It looks like the configuration files have completely changed.  Is there any information or tools available to map across the old format to the new?  It took a lot of trial and error to find a clustered configuration that worked for us in a cloud environment and I'd rather not have to go through the same process again just to continue kicking the tyres - especially given our performance results so far aren't looking promising.

             

            Mike

            • 3. Re: startServer.sh replacement in 5.3
              nadirx

              No, we currently don't have an automated tool: we should be able to have one in when 6.0 is out.

               

              If you want to share your configuration with me, I can do the conversion.

               

              Tristan

              • 4. Re: startServer.sh replacement in 5.3
                mike-evans

                It would be great if you could give me some pointers on the conversion at least - I was guessing that I need to somehow merge my cache configuration file and jgroups configuration into the single cluster.xml in the new format, but I wasn't clear whether there were differences in the details.

                 

                The configuration files are as follows.

                 

                cache_config.xml

                ----

                <?xml version="1.0" encoding="UTF-8"?>

                <!--

                  ~ JBoss, Home of Professional Open Source

                  ~ Copyright 2009 Red Hat Inc. and/or its affiliates and other

                  ~ contributors as indicated by the @author tags. All rights reserved.

                  ~ See the copyright.txt in the distribution for a full listing of

                  ~ individual contributors.

                  ~

                  ~ This is free software; you can redistribute it and/or modify it

                  ~ under the terms of the GNU Lesser General Public License as

                  ~ published by the Free Software Foundation; either version 2.1 of

                  ~ the License, or (at your option) any later version.

                  ~

                  ~ This software is distributed in the hope that it will be useful,

                  ~ but WITHOUT ANY WARRANTY; without even the implied warranty of

                  ~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU

                  ~ Lesser General Public License for more details.

                  ~

                  ~ You should have received a copy of the GNU Lesser General Public

                  ~ License along with this software; if not, write to the Free

                  ~ Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA

                  ~ 02110-1301 USA, or see the FSF site: http://www.fsf.org.

                  -->

                <infinispan

                      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                      xsi:schemaLocation="urn:infinispan:config:5.2 http://www.infinispan.org/schemas/infinispan-config-5.2.xsd"

                      xmlns="urn:infinispan:config:5.2">

                 

                   <global>

                      <transport clusterName="infinispan-cluster"

                                 distributedSyncTimeout="50000"

                                 transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport">

                         <properties>

                            <property name="configurationFile" value="${CFGPath}/jgroups-tcpping.xml"/>

                         </properties>

                      </transport>

                      <globalJmxStatistics enabled="true"/>

                   </global>

                 

                   <default>

                      <jmxStatistics enabled="true"/>

                      <clustering mode="distribution">

                         <l1 enabled="true" lifespan="60000"/>

                         <hash numOwners="2" />

                         <async/>

                      </clustering>

                   </default>

                </infinispan>

                ---

                 

                and jgroup-tcpping.xml

                ---

                <config xmlns="urn:org:jgroups"

                        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-3.0.xsd">

                    <TCP bind_port="7800" />

                    <TCPPING timeout="3000"

                             initial_hosts="10.190.149.173[7800],10.72.50.110[7800]"

                             port_range="1"

                             num_initial_members="1"/>

                    <VERIFY_SUSPECT timeout="1500"  />

                    <pbcast.NAKACK use_mcast_xmit="false"

                                   retransmit_timeout="300,600,1200,2400,4800"

                                   discard_delivered_msgs="true"/>

                    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                                   max_bytes="400000"/>

                    <pbcast.GMS print_local_addr="true" join_timeout="3000"

                                view_bundling="true"/>

                    <MERGE2 max_interval="30000"

                            min_interval="10000"/>

                    <FD_SOCK/>

                    <FD_ALL/>

                    <BARRIER />

                    <UNICAST />

                    <UFC max_credits="2M"

                         min_threshold="0.4"/>

                    <MFC max_credits="2M"

                         min_threshold="0.4"/>

                    <FRAG2 frag_size="60K"  />

                    <pbcast.STATE_TRANSFER />

                </config>

                ---

                • 5. Re: startServer.sh replacement in 5.3
                  nadirx

                  Hi Mike,

                   

                  the configuration below is a quick conversion. I've removed parts of the jgroups configuration which are implicit.

                  As for disappointing performance, could you tell us what you want achieve (read/write ratio, target performance, etc) ?

                   

                  {code:xml}

                  <?xml version='1.0' encoding='UTF-8'?>

                   

                  <server xmlns="urn:jboss:domain:1.4">

                      <extensions>

                          <extension module="org.infinispan.server.endpoint"/>

                          <extension module="org.jboss.as.clustering.infinispan"/>

                          <extension module="org.jboss.as.clustering.jgroups"/>

                          <extension module="org.jboss.as.connector"/>

                          <extension module="org.jboss.as.jdr"/>

                          <extension module="org.jboss.as.jmx"/>

                          <extension module="org.jboss.as.logging"/>

                          <extension module="org.jboss.as.naming"/>

                          <extension module="org.jboss.as.remoting"/>

                          <extension module="org.jboss.as.security"/>

                          <extension module="org.jboss.as.threads"/>

                          <extension module="org.jboss.as.transactions"/>

                          <extension module="org.jboss.as.web"/>

                      </extensions>

                      <management>

                          <security-realms>

                              <security-realm name="ManagementRealm">

                                  <authentication>

                                      <local default-user="$local"/>

                                      <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>

                                  </authentication>

                              </security-realm>

                              <security-realm name="ApplicationRealm">

                                  <authentication>

                                      <local default-user="$local" allowed-users="*"/>

                                      <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>

                                  </authentication>

                              </security-realm>

                          </security-realms>

                          <management-interfaces>

                              <native-interface security-realm="ManagementRealm">

                                  <socket-binding native="management-native"/>

                              </native-interface>

                              <http-interface security-realm="ManagementRealm">

                                  <socket-binding http="management-http"/>

                              </http-interface>

                          </management-interfaces>

                      </management>

                      <profile>

                          <subsystem xmlns="urn:jboss:domain:logging:1.2">

                              <console-handler name="CONSOLE">

                                  <level name="INFO"/>

                                  <formatter>

                                      <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>

                                  </formatter>

                              </console-handler>

                              <periodic-rotating-file-handler name="FILE" autoflush="true">

                                  <formatter>

                                      <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>

                                  </formatter>

                                  <file relative-to="jboss.server.log.dir" path="server.log"/>

                                  <suffix value=".yyyy-MM-dd"/>

                                  <append value="true"/>

                              </periodic-rotating-file-handler>

                              <logger category="com.arjuna">

                                  <level name="WARN"/>

                              </logger>

                              <logger category="org.apache.tomcat.util.modeler">

                                  <level name="WARN"/>

                              </logger>

                              <logger category="org.jboss.as.config">

                                  <level name="DEBUG"/>

                              </logger>

                              <logger category="sun.rmi">

                                  <level name="WARN"/>

                              </logger>

                              <logger category="jacorb">

                                  <level name="WARN"/>

                              </logger>

                              <logger category="jacorb.config">

                                  <level name="ERROR"/>

                              </logger>

                              <root-logger>

                                  <level name="INFO"/>

                                  <handlers>

                                      <handler name="CONSOLE"/>

                                      <handler name="FILE"/>

                                  </handlers>

                              </root-logger>

                          </subsystem>

                          <subsystem xmlns="urn:infinispan:server:endpoint:5.3">

                              <memcached-connector socket-binding="memcached" cache-container="clustered"/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:datasources:1.1">

                              <datasources/>

                          </subsystem>

                          <subsystem xmlns="urn:infinispan:server:core:5.3" default-cache-container="clustered">

                              <cache-container name="clustered" default-cache="default">

                                  <transport executor="infinispan-transport" lock-timeout="60000"/>

                                  <distributed-cache name="default" mode="SYNC" segments="20" owners="2" remote-timeout="30000" start="EAGER">

                                      <locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                                      <transaction mode="NONE"/>

                                  </distributed-cache>

                                  <distributed-cache name="memcachedCache" mode="SYNC" segments="20" owners="2" remote-timeout="30000" start="EAGER">

                                      <locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                                      <transaction mode="NONE"/>

                                  </distributed-cache>

                              </cache-container>

                              <cache-container name="security"/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:jca:1.1">

                              <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>

                              <bean-validation enabled="true"/>

                              <default-workmanager>

                                  <short-running-threads>

                                      <core-threads count="50"/>

                                      <queue-length count="50"/>

                                      <max-threads count="50"/>

                                      <keepalive-time time="10" unit="seconds"/>

                                  </short-running-threads>

                                  <long-running-threads>

                                      <core-threads count="50"/>

                                      <queue-length count="50"/>

                                      <max-threads count="50"/>

                                      <keepalive-time time="10" unit="seconds"/>

                                  </long-running-threads>

                              </default-workmanager>

                              <cached-connection-manager/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>

                          <subsystem xmlns="urn:jboss:domain:jgroups:1.2" default-stack="${jboss.default.jgroups.stack:tcp}">

                              <stack name="udp">

                                  <transport type="UDP" socket-binding="jgroups-udp"/>

                                  <protocol type="PING"/>

                                  <protocol type="MERGE2"/>

                                  <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>

                                  <protocol type="FD_ALL"/>

                                  <protocol type="pbcast.NAKACK"/>

                                  <protocol type="UNICAST2"/>

                                  <protocol type="pbcast.STABLE"/>

                                  <protocol type="pbcast.GMS"/>

                                  <protocol type="UFC"/>

                                  <protocol type="MFC"/>

                                  <protocol type="FRAG2"/>

                                  <protocol type="RSVP"/>

                              </stack>

                              <stack name="tcp">

                                  <transport type="TCP" socket-binding="jgroups-tcp"/>

                                  <protocol type="TCPPING">

                                     <property name="timeout">3000</property>

                                     <property name="initial_hosts">10.190.149.173[7800],10.72.50.110[7800]</property>

                                     <property name="port_range">1</property>

                                     <property name="num_initial_members">1</property>

                                  </protocol>

                                  <protocol type="MERGE2">

                                     <property name="max_interval">30000</property>

                                     <property name="min_interval">10000</property>

                                  </protocol>

                                  <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                                  <protocol type="FD"/>

                                  <protocol type="VERIFY_SUSPECT"/>

                                  <protocol type="pbcast.NAKACK">

                                      <property name="use_mcast_xmit">false</property>

                                  </protocol>

                                  <protocol type="UNICAST2"/>

                                  <protocol type="pbcast.STABLE">

                                      <property name="stability_delay">1000</property>

                                      <property name="desired_avg_gossip">50000</property>

                                      <property name="max_bytes">400000</property>

                                  </protocol>

                                  <protocol type="pbcast.GMS">

                                      <property name="print_local_addr">true</property>

                                      <property name="join_timeout">3000</property>

                                      <property name="view_bundling">true</property>

                                  </protocol>

                                  <protocol type="UFC">

                                      <property name="max_credits">2M</property>

                                      <property name="min_threshold">0.4</property>

                                  </protocol>

                                  <protocol type="MFC">

                                      <property name="max_credits">2M</property>

                                      <property name="min_threshold">0.4</property>

                                  </protocol>

                                  <protocol type="FRAG2">

                                      <property name="frag_size">60K</property>

                                  </protocol>

                                  <protocol type="RSVP"/>

                              </stack>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:jmx:1.1">

                              <show-model value="true"/>

                              <remoting-connector/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:naming:1.2"/>

                          <subsystem xmlns="urn:jboss:domain:remoting:1.1">

                              <connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm"/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:security:1.2">

                              <security-domains>

                                  <security-domain name="other" cache-type="infinispan">

                                      <authentication>

                                          <login-module code="Remoting" flag="optional">

                                              <module-option name="password-stacking" value="useFirstPass"/>

                                          </login-module>

                                          <login-module code="RealmUsersRoles" flag="required">

                                              <module-option name="usersProperties" value="${jboss.server.config.dir}/application-users.properties"/>

                                              <module-option name="rolesProperties" value="${jboss.server.config.dir}/application-roles.properties"/>

                                              <module-option name="realm" value="ApplicationRealm"/>

                                              <module-option name="password-stacking" value="useFirstPass"/>

                                          </login-module>

                                      </authentication>

                                  </security-domain>

                                  <security-domain name="jboss-web-policy" cache-type="infinispan">

                                      <authorization>

                                          <policy-module code="Delegating" flag="required"/>

                                      </authorization>

                                  </security-domain>

                              </security-domains>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:threads:1.1">

                              <thread-factory name="infinispan-factory" group-name="infinispan" priority="5"/>

                              <unbounded-queue-thread-pool name="infinispan-transport">

                                  <max-threads count="25"/>

                                  <keepalive-time time="0" unit="milliseconds"/>

                                  <thread-factory name="infinispan-factory"/>

                              </unbounded-queue-thread-pool>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:transactions:1.2">

                              <core-environment>

                                  <process-id>

                                      <uuid/>

                                  </process-id>

                              </core-environment>

                              <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>

                              <coordinator-environment default-timeout="300"/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">

                              <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>

                              <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>

                              <virtual-server name="default-host" enable-welcome-root="false">

                                  <alias name="localhost"/>

                                  <alias name="example.com"/>

                              </virtual-server>

                          </subsystem>

                      </profile>

                      <interfaces>

                          <interface name="management">

                              <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

                          </interface>

                          <interface name="public">

                              <inet-address value="${jboss.bind.address:127.0.0.1}"/>

                          </interface>

                      </interfaces>

                      <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

                          <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>

                          <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>

                          <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>

                          <socket-binding name="ajp" port="8009"/>

                          <socket-binding name="http" port="8080"/>

                          <socket-binding name="https" port="8443"/>

                          <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45700"/>

                          <socket-binding name="jgroups-tcp" port="7080"/>

                          <socket-binding name="jgroups-tcp-fd" port="57600"/>

                          <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45688"/>

                          <socket-binding name="jgroups-udp-fd" port="54200"/>

                          <socket-binding name="memcached" port="11211"/>

                          <socket-binding name="remoting" port="4447"/>

                          <socket-binding name="txn-recovery-environment" port="4712"/>

                          <socket-binding name="txn-status-manager" port="4713"/>

                      </socket-binding-group>

                  </server>

                  {code}

                  • 6. Re: startServer.sh replacement in 5.3
                    mike-evans

                    Tristan

                     

                    Thanks for this.  I've been trying the configuration out today and it seems to load okay, but I'm not seeing my two nodes cluster together.  So, if I write data to one of the nodes I see no CPU loading on the other node at all, and if I then try and read from the other node I get no data returned.

                     

                    I can't see anything obvious in the configurations that would explain this - both the jgroups configuration and the default cache configuration looks consistent with how I had it set up with the 5.2.1 code.  Any thoughts on how I can go about debugging this?  Are there any console commands that will allow me to query the clustering state?

                     

                    Mike

                    • 7. Re: startServer.sh replacement in 5.3
                      mike-evans

                      Tristan

                       

                      I've now managed to get this working after a fashion.  The problem was in the socket bindings - the JGroups TCP port was 7080 instead of 7800 (so was inconsistent with the JGroups TCP config) and it was only binding to 127.0.0.1 not the external IP address.

                       

                      The other thing I need to do is switch back to async replication and I should be able to get some comparative numbers against 5.2.1.

                       

                      Thanks for your help on this.

                       

                      Mike