9 Replies Latest reply on Sep 17, 2015 10:09 PM by shawkins

    Jboss+Teiid Cluster

    anilnair

      Hi,

      We are using Jboss 6.4.0 alpha with teiid 8.11. I am trying to come up with a  staging Infrastructure setup for this combination.

      I am a newbie and please apologies for my ignorance if any.

      I have setup the Jboss cluster using this link https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto?_sscc=t

      As mentioned in the link I have used mod_cluster + apache httpd on master as our cluster controller.

      I have setup the cluster with one master and one slave and I am able to connect using the teiid jdbc driver url "jdbc:teiid:SerenityEPO@mm://master:31000,slave:31000";

      I was trying to test the following scenario  and have the following question and request your inputs

      1. When my master is completely down with slave still running all my request gets errored  out? Error I am getting is TEIID20021 No valid host available. Attempted connections to:  Shouldnt the slave process all the request  when master is down.So if this is correct behavior then when master system is completely down then the entire cluster is stale(no use).
      2. When I stop the Server-three of the Other server group through the console my request are getting processed.
      3. On my master console I see the following line

                [Server:server-one] 21:01:28,293 INFO  [org.teiid.RUNTIME] (MSC service thread 1-10) TEIID50012 Teiid JDBC - Name = jdbc, Host = MASTER001.abc.com,  Port = 31000, SSL = OFF, security-domains = teiid-security

                but on slave I do not see this line and hence when I try to connect to the slave directly  I get the message : connection refused,

                I have seen the configuration for this port in the domain.xml of the master since the slave does not have I am not sure how it works

         4.  On the slave console after start up I see this error and have been seeing this in all the clusters that i have been trying to setup (though my vdb,datasource all deployments are successful) and this only on the slave.

      [Server:server-one] 20:59:08,034 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 57) MSC000001: Failed to start service jboss.infinispan.teiid-cache.resultset-repl: org.jboss.msc.service.StartException in service jboss.infinispan.teiid-cache.resultset-repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

      [Server:server-one]     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.5.0.Final-redhat-15.jar:7.5.0.Final-redhat-15]

      [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_60]

      [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_60]

      [Server:server-one]     at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]

      [Server:server-one]     at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]

      [Server:server-one] Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

      [Server:server-one]     at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)

      [Server:server-one]     at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)

      [Server:server-one]     at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)

      [Server:server-one]     at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)

      [Server:server-one]     at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)

      [Server:server-one]     at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217)

      [Server:server-one]     at org.infinispan.CacheImpl.start(CacheImpl.java:582)

      [Server:server-one]     at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686)

      [Server:server-one]     at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)

      [Server:server-one]     at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)

      [Server:server-one]     at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)

      [Server:server-one]     at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:113)

      [Server:server-one]     at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:104)

      [Server:server-one]     at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)

      [Server:server-one]     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [jboss-as-clustering-common-7.5.0.Final-redhat-15.jar:7.5.0.Final-redhat-15]

      [Server:server-one]     ... 4 more

      [Server:server-one] Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-one/teiid-cache

      [Server:server-one]     at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)

      [Server:server-one]     at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)

      [Server:server-one]     at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)

      [Server:server-one]     at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:293)

      [Server:server-one]     at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:107)

      [Server:server-one]     at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:118)

      [Server:server-one]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_60]

      [Server:server-one]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_60]

      [Server:server-one]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_60]

      [Server:server-one]     at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_60]

      [Server:server-one]     at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)

      [Server:server-one]     ... 18 more

      [Server:server-one] Caused by: org.jgroups.TimeoutException: timeout sending message to master:server-one/teiid-cache

      [Server:server-one]     at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:392)

      [Server:server-one]     at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)

      [Server:server-one]     at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)

      20:59:10,795 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss Teiid + Teiid 8.11 (AS 7.5.0.Final-redhat-15) started (with errors) in 65464ms - Started 331 of 442 services (12 services failed or missing dependencies, 139 services are lazy, passive or on-demand)

       

      Thanks

      Anil

        • 1. Re: Jboss+Teiid Cluster
          rareddy

          Anil,

           

          Looks like your setup itself is somehow not valid and your slave process is not up and running. This is evident from your efforts to connect to it directly and as well as the exception in (4). IMO, you did not install Teiid at all in the Slave node, you only installed on the Master one, thus also explains why you are not seeing the INFO line about JDBC port. So, double check that.


          I have no experience with mod_cluster and httpd, so I am not sure how it works for JDBC based connections.  Other users have success with HAProxy, instructions at http://www.tenfourty.com/tag/teiid/


          When you load balance, you do not need to provide the URL like "jdbc:teiid:SerenityEPO@mm://master:31000,slave:31000", you need something like "jdbc:teiid:SerenityEPO@mm://proxy:31000" where proxy is your load balancer, that could be HAProxy or mod_cluster or F5


          Ramesh..

          1 of 1 people found this helpful
          • 2. Re: Jboss+Teiid Cluster
            van.halbert

            mod_cluster (mod_cluster - JBoss Community) is an http load balancer, not a tcp load balancer.  

            1 of 1 people found this helpful
            • 3. Re: Jboss+Teiid Cluster
              anilnair

              Thank you Ramesh and Van.

              I have re-installed the cluster and works fine

              btw I have one follow up question.

              Previously I created rpm  from a pre installed Jboss+ teiid  and was using this rom  to create Master and slaves to avoid manual steps of installation of teiid and seems like it does not work and was getting erorr(mentioned above) on the slaves

              So now I have created jboss rpm  and after I create a jboss cluster(Master and slaves) by pointing to Master I install teiid. Once I do this I see cluster working.

              Is this the right approach?

              Thanks

              Anil

              • 4. Re: Jboss+Teiid Cluster
                rareddy

                Anil,

                 

                The install package is same for Master and Slaves. However the configuration is different when it comes to Master and Slave. I would say you can have one RPM for install, but then have scripts or additional files each cluster member, typically host.xml file. You only need to modify the domain.xml on the master.

                 

                Ramesh..

                • 5. Re: Jboss+Teiid Cluster
                  anilnair

                  Hi Ramesh,

                  Thank you ..

                  This is what I previously did like I created RPM out of preinstalled Jboss+Teiid .

                  I installed master  where I kept the domain.xml  file and I installed slaves deleted domain.xml and make the necessary configuration in the host.xml file out of the same RPM.

                  In this approach master was perfectly fine where as on slaves I used to get the error

                  [Server:server-one] 20:59:08,034 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 57) MSC000001: Failed to start service jboss.infinispan.teiid-cache.resultset-repl: org.jboss.msc.service.StartException in service jboss.infinispan.teiid-cache.resultset-repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

                  [Server:server-one]     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.5.0.Final-redhat-15.jar:7.5.0.Final-redhat-15]


                  Thanks

                  Anil

                  • 6. Re: Jboss+Teiid Cluster
                    shawkins

                    That error I believe is indicative of a low level timeout in JGroups - similar to ERROR org.infinispan.remoting.InboundInvocationHandlerImpl - Defined caches

                     

                    Do you have a root exception here that shows a timeout?

                    • 7. Re: Jboss+Teiid Cluster
                      rareddy

                      Anil,

                       

                      I forgot to mention, when install the master first time, you need to run "<joss-eap>/bin/scripts/teiid-domain-mode-install.cli" script using "jboss-cli.sh" to install the Teiid in domain mode. Like

                       

                      unzip eap

                      unzip Teiid

                      ./bin/domain.sh  (start the server)

                      ./jboss-cli.sj --file=scripts/teiid-domain-mode-install.cli (install Teiid in domain mode)

                       

                      then you need to configure the host.xml for master and slaves individually. That will create single cluster key for caching then when you start all nodes will find each other

                       

                      Ramesh..

                      • 8. Re: Jboss+Teiid Cluster
                        anilnair

                        Hi Ramesh and Steven,

                        Once again thank your for your time

                        Please find my  answers

                        I installed  one server Jboss and then I installed teiid as you had mentioned and I took the entire files and created rpm.

                        Now when I wanted to give it to IT I gave this RPM and told them to install this rpm

                        1. for master  master I told them not do any changes once they installed the rpm

                        2. for slaves I told them to remove the domain.xml file and edit host.xml file and point the Domain controller to the master and also add the secret key.for security idenities

                        When they did this the slave was able to communicate with the master. All were fine till now.

                        So when I was executing the query I was getting the error that I had mentioned above. and magically today when i started both master and slaves  I am not able to get that error and that too on the same cluster.

                        So the question was is the above approach the right one so that IT team has full package and they just have to change the some configuration values depending on  master or slave.

                         

                        Steven

                        The only other error that i was receiving was

                        [Server:server-one] 14:43:35,100 ERROR [stderr] (MSC service thread 1-9) ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.sun.script.javascript.RhinoScriptEngineFactory not found

                        [Server:server-one] 14:43:35,228 ERROR [stderr] (MSC service thread 1-13) ScriptEngineManager providers.next(): javax.script.ScriptEngineFactory: Provider com.sun.script.javascript.RhinoScriptEngineFactory not found

                        and

                        [Server:server-three] 14:43:38,567 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Teiid + Teiid 8.11 (AS 7.5.0.Final-redhat-15) started in 5709ms - Started 257 of 352 services (130 services are lazy, passive or on-demand)

                        [Server:server-three] 14:43:40,111 ERROR [org.hornetq.core.server] (Old I/O client worker ([id: 0x4cd57eb7, /10.45.121.95:46582 => /10.45.121.121:5695])) HQ224066: Stopping ClusterManager. As it failed to authenticate with the cluster: HQ119099: Unable to authenticate cluster user: HORNETQ.CLUSTER.ADMIN.USER

                        [Server:server-three] 14:43:40,126 INFO  [org.hornetq.core.server] (Thread-8 (HornetQ-server-HornetQServerImpl::serverUUID=30645b12-4dc4-11e5-a30d-89689da43af6-1345265425)) HQ221029: stopped bridge sf.my-cluster.7912217a-5cb7-11e5-8eb8-f115a397cdc8

                        [Server:server-three] 14:43:40,124 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: -301569548, [id: 0xee0669f4, /10.45.121.95:5695])) HQ224018: Failed to create session: HornetQException[errorType=CLUSTER_SECURITY_EXCEPTION message=HQ119099: Unable to authenticate cluster user: HORNETQ.CLUSTER.ADMIN.USER]

                        [Server:server-three]   at org.hornetq.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:123) [hornetq-server-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.server.impl.HornetQServerImpl.createSession(HornetQServerImpl.java:979) [hornetq-server-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handleCreateSession(HornetQPacketHandler.java:150) [hornetq-server-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handlePacket(HornetQPacketHandler.java:78) [hornetq-server-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:641) [hornetq-core-client-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:557) [hornetq-core-client-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:533) [hornetq-core-client-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:587) [hornetq-server-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.24.Final-redhat-1.jar:2.3.24.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]

                        [Server:server-three]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_60]

                        [Server:server-three]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_60]

                        [Server:server-three]   at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]

                        And these errors  are still present but not affecting execution of queries.Should I be concerned about these or is there a way to fix those


                        Thanks'

                        Anil

                        • 9. Re: Jboss+Teiid Cluster
                          shawkins

                          The script engine error is an AS/EAP issue - [TEIID-3642] RhinoScriptEngineFactory be removed in Java 8 - JBoss Issue Tracker

                           

                          We should have nothing to do with the HornetQ error.  You may need to consult their community.  However as you have seen it shouldn't effect Teiid query execution.

                          1 of 1 people found this helpful