5 Replies Latest reply on Apr 27, 2015 6:36 AM by imckinle

    rtgov overlord with elasticsearch cluster

    lazyman

      hi guys. Can someone help me with rtgov overlord installation with custom elasticsearch cluster? I've installed rtgov on jboss eap 6.3 (standalone), then created ES cluster with two nodes - cluster name is "something" then updated standalone-full.xml, but it's not working...jboss server log contains:

      20:44:25,023 WARN [org.elasticsearch.client.transport] (elasticsearch[Trapper][generic][T#1]) [Trapper] node [#transport#-1][oldman][inet[/192.168.33.10:9300]] not part of the cluster Cluster [elasticsearch], ignoring...

      Kibana thinks "Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above."

      http://localhost:18080/rtgov-ui/elasticsearch/_nodes returns 503

       

      This is my standalone rtgov configuration:

       

      <oc:configuration xmlns:oc="urn:jboss:domain:overlord-configuration:1.0" name="overlord-rtgov">

      <oc:properties>

         <oc:property name="collectionEnabled" value="true"/>

         <oc:property name="ActivityServerLogger.maxThreads" value="10"/>

         <oc:property name="BatchedActivityUnitLogger.maxUnitCount" value="1000"/>

         <oc:property name="BatchedActivityUnitLogger.maxTimeInterval" value="500"/>

         <oc:property name="ActiveCollectionManager.houseKeepingInterval" value="10000"/>

         <oc:property name="ActivityStore.class" value="org.overlord.rtgov.activity.store.elasticsearch.ElasticsearchActivityStore"/>

         <oc:property name="SituationStore.class" value="org.overlord.rtgov.analytics.situation.store.elasticsearch.ElasticsearchSituationStore"/>

         <oc:property name="Elasticsearch.hosts" value="192.168.33.10:9300,192.168.33.11:9300"/>

      </oc:properties>

      </oc:configuration>

      <oc:configuration xmlns:oc="urn:jboss:domain:overlord-configuration:1.0" name="overlord-rtgov-elasticsearch">

      <oc:properties>

         <oc:property name="cluster.name" value="something"/>

         <oc:property name="node.data" value="false"/>

         <oc:property name="node.client" value="true"/>

         <oc:property name="node.local" value="false"/>

      </oc:properties>

      </oc:configuration>

       

      Thanks.

        • 1. Re: rtgov overlord with elasticsearch cluster
          imckinle

          what version are your elastic search nodes?

          • 2. Re: rtgov overlord with elasticsearch cluster
            imckinle

            addtioanlly from your log i see that rtgov is trying to connect to a cluster called "Elasticsearch"

            20:44:25,023 WARN [org.elasticsearch.client.transport] (elasticsearch[Trapper][generic][T#1]) [Trapper] node [#transport#-1][oldman][inet[/192.168.33.10:9300]] not part of the cluster Cluster [elasticsearch], ignoring...


            as a quick test, to rule out version issues. Please change the cluster name of your nodes to "Elasticsearch"

            cluster.name:Elasticsearch



            • 3. Re: rtgov overlord with elasticsearch cluster
              lazyman

              sorry for delayed answer, well I'm using elasticsearch 1.3.2 (same as embedded in rtgov 2.0.0.Final)

               

              cluster.name=elasticsearch doesn't help, also I can't set more "hosts" for cluster. I have 2 elastic nodes with node.data=true

              Current attempt was:

              <oc:configuration xmlns:oc="urn:jboss:domain:overlord-configuration:1.0" name="overlord-rtgov">

              <oc:properties>

                 <oc:property name="collectionEnabled" value="true"/>

                 <oc:property name="ActivityServerLogger.maxThreads" value="10"/>

                 <oc:property name="BatchedActivityUnitLogger.maxUnitCount" value="1000"/>

                 <oc:property name="BatchedActivityUnitLogger.maxTimeInterval" value="500"/>

                 <oc:property name="ActiveCollectionManager.houseKeepingInterval" value="10000"/>

                 <oc:property name="ActivityStore.class" value="org.overlord.rtgov.activity.store.elasticsearch.ElasticsearchActivityStore"/>

                 <oc:property name="SituationStore.class" value="org.overlord.rtgov.analytics.situation.store.elasticsearch.ElasticsearchSituationStore"/>

                 <oc:property name="Elasticsearch.hosts" value="es0.example.com:9301,es1.example.com:9300,es2.example.com:9300"/>

                 <oc:property name="Elasticsearch.server" value="es0.example.com:9201"/>

              </oc:properties>

              </oc:configuration>

              <oc:configuration xmlns:oc="urn:jboss:domain:overlord-configuration:1.0" name="overlord-rtgov-elasticsearch">

              <oc:properties>

                 <oc:property name="cluster.name" value="elasticsearch"/>

                 <oc:property name="node.data" value="false"/>

                 <oc:property name="node.client" value="false"/>

                 <oc:property name="node.local" value="true"/>

                   <oc:property name="node.master" value="false"/>

                   <oc:property name="network.host" value="192.168.1.6"/>

                   <oc:property name="transport.port" value="9301"/>

                   <oc:property name="http.port" value="9201"/>

                   <oc:property name="discovery.zen.ping.multicast.enabled" value="false"/>

                   <oc:property name="discovery.zen.ping.unicast.hosts" value="es0.example.com,es1.example.com,es2.example.com"/>

              </oc:properties>

               

              But it doesn't work, embedded node can't find other nodes. I've checked /etc/hosts, tried telnet every port (9300, 9200). I've tried to set logging to trace:

              <logger category="org.elasticsearch">

                 <level name="TRACE"/>

              </logger>

              <logger category="org.overlord">

                 <level name="TRACE"/>

              </logger>


              I even tried to remove configuration and use two property files:

              JAVA_OPTS="$JAVA_OPTS -Delasticsearch.config=$JBOSS_CONFIG_DIR/elasticsearch.properties"

              JAVA_OPTS="$JAVA_OPTS -Drtgov.config.file.name=$JBOSS_CONFIG_DIR/rtgov.properties"

               

              using property files is bad in this situation, because multivalue properties doesn't work. Does anybody use it in clustered environment?


              • 4. Re: rtgov overlord with elasticsearch cluster
                lazyman

                with this:

                 

                <oc:property name="Elasticsearch.hosts" value="es0.example.com:9301,es1.example.com:9300,es2.example.com:9300"/>


                it won't even start.

                • 5. Re: rtgov overlord with elasticsearch cluster
                  imckinle

                  Hi Viliam,

                   

                  This is interesting, Could you try the following configuration.  This configuration results in the following

                  • Client is cluster aware and can route its data to the responsible shards avoiding a double hop
                  • ES property node.client=true and node.data=false
                  • No data is stored locally

                   

                  this configuration assumes the target ES cluster is not changed. with the same cluster name "elasticsearch"

                  If your elasticsearch cluster name is something else then please add that config  "cluster.name" to overlord-rtgov-elasticsearch section

                   

                   

                                  <configuration name="overlord-rtgov"> 

                                      <properties> 

                                          <property name="collectionEnabled" value="true"/> 

                                          <property name="ActivityServerLogger.maxThreads" value="10"/> 

                                          <property name="BatchedActivityUnitLogger.maxUnitCount" value="1000"/> 

                                          <property name="BatchedActivityUnitLogger.maxTimeInterval" value="500"/> 

                                          <property name="ActiveCollectionManager.houseKeepingInterval" value="10000"/> 

                                          <property name="ActivityStore.class" value="org.overlord.rtgov.activity.store.elasticsearch.ElasticsearchActivityStore"/> 

                                          <property name="SituationStore.class" value="org.overlord.rtgov.analytics.situation.store.elasticsearch.ElasticsearchSituationStore"/> 

                                      </properties> 

                                  </configuration> 

                                  <configuration name="overlord-rtgov-elasticsearch"> 

                                      <properties> 

                                          <property name="path.data" value="../elasticsearch"/> 

                                          <property name="node.data" value="false"/> 

                                          <property name="node.client" value="true"/> 

                                          <property name="node.local" value="false"/> 

                                      </properties> 

                                  </configuration>