rtgov overlord with elasticsearch cluster

[lazyman]

hi guys. Can someone help me with rtgov overlord installation with custom elasticsearch cluster? I've installed rtgov on jboss eap 6.3 (standalone), then created ES cluster with two nodes - cluster name is "something" then updated standalone-full.xml, but it's not working...jboss server log contains:

20:44:25,023 WARN [org.elasticsearch.client.transport] (elasticsearch[Trapper][generic][T#1]) [Trapper] node [#transport#-1][oldman][inet[/192.168.33.10:9300]] not part of the cluster Cluster [elasticsearch], ignoring...

Kibana thinks "Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above."

http://localhost:18080/rtgov-ui/elasticsearch/_nodes returns 503

This is my standalone rtgov configuration:

<oc:configuration xmlns:oc="urn:jboss:domain:overlord-configuration:1.0" name="overlord-rtgov">

<oc:properties>

   <oc:property name="collectionEnabled" value="true"/>

   <oc:property name="ActivityServerLogger.maxThreads" value="10"/>

   <oc:property name="BatchedActivityUnitLogger.maxUnitCount" value="1000"/>

   <oc:property name="BatchedActivityUnitLogger.maxTimeInterval" value="500"/>

   <oc:property name="ActiveCollectionManager.houseKeepingInterval" value="10000"/>

   <oc:property name="ActivityStore.class" value="org.overlord.rtgov.activity.store.elasticsearch.ElasticsearchActivityStore"/>

   <oc:property name="SituationStore.class" value="org.overlord.rtgov.analytics.situation.store.elasticsearch.ElasticsearchSituationStore"/>

   <oc:property name="Elasticsearch.hosts" value="192.168.33.10:9300,192.168.33.11:9300"/>

</oc:properties>

</oc:configuration>

<oc:configuration xmlns:oc="urn:jboss:domain:overlord-configuration:1.0" name="overlord-rtgov-elasticsearch">

<oc:properties>

   <oc:property name="cluster.name" value="something"/>

   <oc:property name="node.data" value="false"/>

   <oc:property name="node.client" value="true"/>

   <oc:property name="node.local" value="false"/>

</oc:properties>

</oc:configuration>

Thanks.

[imckinle]

what version are your elastic search nodes?

[imckinle]

addtioanlly from your log i see that rtgov is trying to connect to a cluster called "Elasticsearch"

20:44:25,023 WARN [org.elasticsearch.client.transport] (elasticsearch[Trapper][generic][T#1]) [Trapper] node [#transport#-1][oldman][inet[/192.168.33.10:9300]] not part of the cluster Cluster [elasticsearch], ignoring...

as a quick test, to rule out version issues. Please change the cluster name of your nodes to "Elasticsearch"

cluster.name:Elasticsearch

[lazyman]

sorry for delayed answer, well I'm using elasticsearch 1.3.2 (same as embedded in rtgov 2.0.0.Final)

cluster.name=elasticsearch doesn't help, also I can't set more "hosts" for cluster. I have 2 elastic nodes with node.data=true

Current attempt was:

<oc:configuration xmlns:oc="urn:jboss:domain:overlord-configuration:1.0" name="overlord-rtgov">

<oc:properties>

   <oc:property name="collectionEnabled" value="true"/>

   <oc:property name="ActivityServerLogger.maxThreads" value="10"/>

   <oc:property name="BatchedActivityUnitLogger.maxUnitCount" value="1000"/>

   <oc:property name="BatchedActivityUnitLogger.maxTimeInterval" value="500"/>

   <oc:property name="ActiveCollectionManager.houseKeepingInterval" value="10000"/>

   <oc:property name="ActivityStore.class" value="org.overlord.rtgov.activity.store.elasticsearch.ElasticsearchActivityStore"/>

   <oc:property name="SituationStore.class" value="org.overlord.rtgov.analytics.situation.store.elasticsearch.ElasticsearchSituationStore"/>

   <oc:property name="Elasticsearch.hosts" value="es0.example.com:9301,es1.example.com:9300,es2.example.com:9300"/>

   <oc:property name="Elasticsearch.server" value="es0.example.com:9201"/>

</oc:properties>

</oc:configuration>

<oc:configuration xmlns:oc="urn:jboss:domain:overlord-configuration:1.0" name="overlord-rtgov-elasticsearch">

<oc:properties>

   <oc:property name="cluster.name" value="elasticsearch"/>

   <oc:property name="node.data" value="false"/>

   <oc:property name="node.client" value="false"/>

   <oc:property name="node.local" value="true"/>

     <oc:property name="node.master" value="false"/>

     <oc:property name="network.host" value="192.168.1.6"/>

     <oc:property name="transport.port" value="9301"/>

     <oc:property name="http.port" value="9201"/>

     <oc:property name="discovery.zen.ping.multicast.enabled" value="false"/>

     <oc:property name="discovery.zen.ping.unicast.hosts" value="es0.example.com,es1.example.com,es2.example.com"/>

</oc:properties>

But it doesn't work, embedded node can't find other nodes. I've checked /etc/hosts, tried telnet every port (9300, 9200). I've tried to set logging to trace:

<logger category="org.elasticsearch">

   <level name="TRACE"/>

</logger>

<logger category="org.overlord">

   <level name="TRACE"/>

</logger>

I even tried to remove configuration and use two property files:

JAVA_OPTS="$JAVA_OPTS -Delasticsearch.config=$JBOSS_CONFIG_DIR/elasticsearch.properties"

JAVA_OPTS="$JAVA_OPTS -Drtgov.config.file.name=$JBOSS_CONFIG_DIR/rtgov.properties"

using property files is bad in this situation, because multivalue properties doesn't work. Does anybody use it in clustered environment?

[lazyman]

with this:

<oc:property name="Elasticsearch.hosts" value="es0.example.com:9301,es1.example.com:9300,es2.example.com:9300"/>

it won't even start.

[imckinle]

Hi Viliam,

This is interesting, Could you try the following configuration.  This configuration results in the following

• Client is cluster aware and can route its data to the responsible shards avoiding a double hop
• ES property node.client=true and node.data=false
• No data is stored locally
  

this configuration assumes the target ES cluster is not changed. with the same cluster name "elasticsearch"

If your elasticsearch cluster name is something else then please add that config  "cluster.name" to overlord-rtgov-elasticsearch section

                <configuration name="overlord-rtgov"> 

                    <properties> 

                        <property name="collectionEnabled" value="true"/> 

                        <property name="ActivityServerLogger.maxThreads" value="10"/> 

                        <property name="BatchedActivityUnitLogger.maxUnitCount" value="1000"/> 

                        <property name="BatchedActivityUnitLogger.maxTimeInterval" value="500"/> 

                        <property name="ActiveCollectionManager.houseKeepingInterval" value="10000"/> 

                        <property name="ActivityStore.class" value="org.overlord.rtgov.activity.store.elasticsearch.ElasticsearchActivityStore"/> 

                        <property name="SituationStore.class" value="org.overlord.rtgov.analytics.situation.store.elasticsearch.ElasticsearchSituationStore"/> 

                    </properties> 

                </configuration> 

                <configuration name="overlord-rtgov-elasticsearch"> 

                    <properties> 

                        <property name="path.data" value="../elasticsearch"/> 

                        <property name="node.data" value="false"/> 

                        <property name="node.client" value="true"/> 

                        <property name="node.local" value="false"/> 

                    </properties> 

                </configuration>