6.1 EA - Startup issues
We create a fresh fabric using the following command in an ERB file; you can infer the relevant substitution values from their name:
fabric:create --clean --new-user <%= fabric_admin_user %> --new-user-role admin --new-user-password <%= fabric_admin_pass %> --zookeeper-password <%= zookeeper_password %> --resolver manualip --manual-ip <%= vip_ip %>
Following this the fabric is configured for communication with a Sonatype Nexus instance via which Maven artifacts can be resolved. The host has no internet connection, so this step is essential for us:
fabric:profile-edit --pid org.fusesource.fabric.agent/patch.repositories=<%= mvn_nexus_repos %> default
fabric:profile-edit --pid org.fusesource.fabric.agent/org.ops4j.pax.url.mvn.repositories=<%= mvn_nexus_repos %> default
fabric:profile-edit --pid org.fusesource.fabric.maven/remoteRepositories=<%= mvn_nexus_repos %> fabric
Finally, we add the HawtIO web interface and Zookeeper commands to the 'fabric' profile:
fabric:profile-edit --features fabric-zookeeper-commands,hawtio fabric
Following this, the first time JBoss Fuse starts we experience the following error repeated endlessly, with JBoss Fuse exhausting the host CPU and using approx 1/4 of its memory. The problem only disappears if you manually stop-then-start the Fuse instance. The problem doesn't occur every time - about 1/3 starts of a freshly-deployed fabric with the above configuration will produce the issue.
11:52:43,888 | INFO | eMQ Broker: root | ActiveMQServiceFactory$$anon$1 | 109 - org.springframework.context - 3.1.3.RELEASE | Refreshing org.fusesource.mq.fabric.ActiveMQServiceFactory$$anon$1@341e3e4e: startup date [Thu Aug 01 11:52:43 BST 2013]; root of context hierarchy
11:52:43,887 | INFO | eMQ Broker: root | ActiveMQServiceFactory | 130 - org.jboss.amq.mq-fabric - 6.1.0.redhat-003 | Broker root failed to start. Will try again in 10 seconds
11:52:44,532 | INFO | eMQ Broker: root | ActiveMQServiceFactory | 130 - org.jboss.amq.mq-fabric - 6.1.0.redhat-003 | Exception on start: javax.management.InstanceAlreadyExistsException: org.apache.activemq:type=Broker,brokerName=root
javax.management.InstanceAlreadyExistsException: org.apache.activemq:type=Broker,brokerName=root
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)[:1.6.0_24]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)[:1.6.0_24]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)[:1.6.0_24]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)[:1.6.0_24]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)[:1.6.0_24]
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:516)[:1.6.0_24]
at org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)[116:org.apache.activemq.activemq-osgi:5.9.0.redhat-610003]
at org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)[116:org.apache.activemq.activemq-osgi:5.9.0.redhat-610003]
at org.apache.activemq.broker.BrokerService.startManagementContext(BrokerService.java:2361)[116:org.apache.activemq.activemq-osgi:5.9.0.redhat-610003]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:562)[116:org.apache.activemq.activemq-osgi:5.9.0.redhat-610003]
at org.fusesource.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration$$anon$3.run(ActiveMQServiceFactory.scala:321)[130:org.jboss.amq.mq-fabric:6.1.0.redhat-003]
Here's what top shows for the process:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21319 itfus020 20 0 3573m 990m 20m S 99.6 25.1 3:44.16 java
Running $FUSE/bin/stop fails to stop the process; we have to manually kill -9 it.
Typically after creating our initial fabric ('root' ensemble node) we add two additional ensemble nodes as follows:
container-create-ssh --host itfus021 --user itfus021 root2
container-create-ssh --host itfus022 --user itfus022 root3
ensemble-add root2 root3
Following this we see another CPU and memory grind, the karaf shell becomes unresponsive. Initial error output in the logs is as follows:
18:24:05,335 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Changes to perform:
18:24:05,335 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Bundles to uninstall:
18:24:05,335 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Bundles to update:
18:24:05,335 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Bundles to install:
18:24:05,335 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Stopping bundles:
18:24:05,335 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Uninstalling bundles:
18:24:05,335 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Updating bundles:
18:24:05,335 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Installing bundles:
18:24:05,369 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Refreshing bundles:
18:24:05,369 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Starting bundles:
18:24:05,857 | INFO | agent-1-thread-1 | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | Done.
18:24:06,178 | INFO | ry:/0.0.0.0:2182 | NIOServerCnxn | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Closed socket connection for client /10.97.6.186:39597 which had sessionid 0x1403ae860110001
18:24:06,178 | INFO | d-57-EventThread | ClientCnxn | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | EventThread shut down
18:24:06,178 | INFO | Thread-57 | ZooKeeper | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Session: 0x1403ae860110001 closed
18:24:06,241 | INFO | hChildrenCache-0 | FabricConfigAdminBridge | 236 - org.fusesource.fabric.fabric-configadmin - 7.3.0.redhat-003 | Updating configuration org.fusesource.fabric.agent
18:24:06,256 | INFO | ce.fabric.agent) | DeploymentAgent | 239 - org.fusesource.fabric.fabric-agent - 7.3.0.redhat-003 | DeploymentAgent updated with {repository.karaf-enterprise=mvn:org.apache.karaf.assemblies.features/enterprise/2.3.0.redhat-610003/xml/features, feature.camel-snmp=camel-snmp, feature.fabric-zookeeper-commands=fabric-zookeeper-commands, org.ops4j.pax.url.mvn.defaultrepositories=file:/home/itfus020/jboss-fuse-6.1.0.redhat-003/system@snapshots@id=karaf-default,file:/home/itfus020/jboss-fuse-6.1.0.redhat-003/local-repo@snapshots@id=karaf-local, feature.hawtio=hawtio, feature.fabric-jaas=fabric-jaas, fabric.zookeeper.pid=org.fusesource.fabric.agent, feature.camel-bindy=camel-bindy, feature.camel-spring=camel-spring, feature.camel-blueprint=camel-blueprint, lastmodified=1375377846171, feature.camel-jdbc=camel-jdbc, feature.fabric-core=fabric-core, service.pid=org.fusesource.fabric.agent, feature.camel-csv=camel-csv, feature.mq-activemq_7.3.0.redhat-003=mq-fabric, feature.camel-jaxb=camel-jaxb, feature.camel-jetty=camel-jetty, feature.fabric-bundle=fabric-bundle, feature.camel-jasypt=camel-jasypt, org.ops4j.pax.url.mvn.repositories=http://maven/nexus/content/groups/all-repos@snapshots@id=ocado-all-repos, feature.fabric-partition=fabric-partition, feature.fabric-web=fabric-web, feature.fabric-maven-proxy=fabric-maven-proxy, repository.activemq=mvn:org.apache.activemq/activemq-karaf/5.9.0.redhat-610003/xml/features, repository.fuse-fabric=mvn:org.fusesource.fabric/fuse-fabric/7.3.0.redhat-003/xml/features, repository.apache-camel=mvn:org.apache.camel.karaf/apache-camel/2.12.0.redhat-610003/xml/features, feature.camel-jms=camel-jms, feature.cxf=cxf, feature.camel-saxon=camel-saxon, feature.camel-spring-javaconfig=camel-spring-javaconfig, feature.fabric-commands=fabric-commands, obr.resolve.optional.imports=true, feature.war=war, feature.activemq-camel=activemq-camel, feature.karaf=karaf, feature.camel-routebox=camel-routebox, feature.camel-ftp=camel-ftp, feature.fabric-agent=fabric-agent, feature.jolokia=jolokia, feature.fabric-git=fabric-git, feature.camel-core=camel-core, feature.fabric-camel=fabric-camel, repository.apache-servicemix-nmr=mvn:org.apache.servicemix.nmr/apache-servicemix-nmr/1.6.0.redhat-610003/xml/features, feature.camel-exec=camel-exec, feature.camel-ognl=camel-ognl, repository.jboss-fuse=mvn:org.jboss.fuse/jboss-fuse/6.1.0.redhat-003/xml/features, patch.repositories=http://maven/nexus/content/groups/all-repos@snapshots@id=ocado-all-repos, repository.karaf-standard=mvn:org.apache.karaf.assemblies.features/standard/2.3.0.redhat-610003/xml/features, feature.jasypt-encryption=jasypt-encryption, feature.camel-script=camel-script}
18:24:06,261 | INFO | hChildrenCache-0 | FabricConfigAdminBridge | 236 - org.fusesource.fabric.fabric-configadmin - 7.3.0.redhat-003 | Updating configuration org.fusesource.fabric.zookeeper
18:24:06,277 | INFO | sid:0 cport:-1): | PrepRequestProcessor | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Processed session termination for sessionid: 0x1403ad80fe60002
18:24:06,282 | INFO | hChildrenCache-2 | Bridge | 243 - org.fusesource.fabric.fabric-git-zkbridge - 7.3.0.redhat-003 | Git/zk bridge is active
18:24:06,284 | INFO | ry:/0.0.0.0:2181 | NIOServerCnxn | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Closed socket connection for client /10.97.6.187:59696 which had sessionid 0x1403ad80fe60002
18:24:06,286 | INFO | hChildrenCache-4 | GitHttpServerRegistrationHandler | 247 - org.fusesource.fabric.fabric-git-server - 7.3.0.redhat-003 | Git repo is the master
18:24:06,323 | INFO | abric.zookeeper) | ConnectionState | 53 - org.fusesource.fabric.fabric-zookeeper - 7.3.0.redhat-003 | lost marked
18:24:06,324 | INFO | abric.zookeeper) | ConnectionStateManager | 53 - org.fusesource.fabric.fabric-zookeeper - 7.3.0.redhat-003 | State change: LOST
18:24:06,325 | INFO | tp1350866493-510 | ConnectionState | 53 - org.fusesource.fabric.fabric-zookeeper - 7.3.0.redhat-003 | resetting after loss
18:24:06,328 | INFO | sid:0 cport:-1): | PrepRequestProcessor | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Processed session termination for sessionid: 0x1403ad80fe60000
18:24:06,330 | INFO | hChildrenCache-2 | Bridge | 243 - org.fusesource.fabric.fabric-git-zkbridge - 7.3.0.redhat-003 | Git/zk bridge is active
18:24:06,332 | INFO | ry:/0.0.0.0:2181 | NIOServerCnxn | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Closed socket connection for client /10.97.6.186:54970 which had sessionid 0x1403ad80fe60000
18:24:06,333 | INFO | tp1350866493-510 | ZooKeeper | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Session: 0x1403ad80fe60000 closed
18:24:06,333 | INFO | tp1350866493-510 | ZooKeeper | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Initiating client connection, connectString=10.97.6.186:2182,itfus021.ocado.com:2181,itfus022.ocado.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@71c03ae1
18:24:06,333 | INFO | er])-EventThread | ClientCnxn | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | EventThread shut down
18:24:06,346 | INFO | 510-SendThread() | ClientCnxn | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Opening socket connection to server /10.97.6.186:2182
18:24:06,349 | INFO | hChildrenCache-1 | ActiveMQServiceFactory | 130 - org.jboss.amq.mq-fabric - 6.1.0.redhat-003 | Broker root is now a slave, stopping the broker.
18:24:06,349 | INFO | hChildrenCache-1 | ActiveMQServiceFactory | 130 - org.jboss.amq.mq-fabric - 6.1.0.redhat-003 | Broker root is being stopped.
18:24:06,358 | INFO | .ocado.com:2182) | ZooKeeperSaslClient | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
18:24:06,359 | INFO | ry:/0.0.0.0:2182 | NIOServerCnxnFactory | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Accepted socket connection from /10.97.6.186:39598
18:24:06,359 | INFO | .ocado.com:2182) | ClientCnxn | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Socket connection established to itfus020.ocado.com/10.97.6.186:2182, initiating session
18:24:06,363 | INFO | ry:/0.0.0.0:2182 | ZooKeeperServer | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Client attempting to establish new session at /10.97.6.186:39598
18:24:06,371 | INFO | ommitProcessor:1 | ZooKeeperServer | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Established session 0x1403ae860110002 with negotiated timeout 40000 for client /10.97.6.186:39598
18:24:06,371 | INFO | ry:/0.0.0.0:2182 | ZooKeeperServer | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | got auth packet /10.97.6.186:39598
18:24:06,371 | INFO | ry:/0.0.0.0:2182 | ZooKeeperServer | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | auth success /10.97.6.186:39598
18:24:06,372 | INFO | ry:/0.0.0.0:2182 | ZooKeeperServer | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | got auth packet /10.97.6.186:39598
18:24:06,372 | INFO | ry:/0.0.0.0:2182 | ZooKeeperServer | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | auth success /10.97.6.186:39598
18:24:06,372 | INFO | eMQ Broker: root | BrokerService | 116 - org.apache.activemq.activemq-osgi - 5.9.0.redhat-610003 | Apache ActiveMQ 5.9.0.redhat-610003 (root, ID:ithffu20-45878-1375376788561-0:1) is shutting down
18:24:06,375 | INFO | .ocado.com:2182) | ClientCnxn | 52 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.3.0.redhat-003 | Session establishment complete on server itfus020.ocado.com/10.97.6.186:2182, sessionid = 0x1403ae860110002, negotiated timeout = 40000
18:24:06,375 | INFO | -510-EventThread | ConnectionStateManager | 53 - org.fusesource.fabric.fabric-zookeeper - 7.3.0.redhat-003 | State change: RECONNECTED
18:24:06,391 | ERROR | hChildrenCache-2 | TreeCache | 53 - org.fusesource.fabric.fabric-zookeeper - 7.3.0.redhat-003 |
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /fabric/registry/clusters/gitzkbridge/00000000001
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)[52:org.fusesource.fabric.fabric-linkedin-zookeeper:7.3.0.redhat-003]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)[52:org.fusesource.fabric.fabric-linkedin-zookeeper:7.3.0.redhat-003]
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1246)[52:org.fusesource.fabric.fabric-linkedin-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:260)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:256)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:252)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:239)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:39)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.update(ZooKeeperGroup.scala:118)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingleton.update(ClusteredSingleton.scala:227)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.git.zkbridge.Bridge.changed(Bridge.java:161)[243:org.fusesource.fabric.fabric-git-zkbridge:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1$$anonfun$apply$mcV$sp$3.apply(ChangeListenerSupport.scala:76)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1$$anonfun$apply$mcV$sp$3.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)[57:scala-library:2.9.1]
at scala.collection.immutable.List.foreach(List.scala:45)[57:scala-library:2.9.1]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply$mcV$sp(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$class.check_elapsed_time(ChangeListenerSupport.scala:84)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher.check_elapsed_time(ClusteredSingleton.scala:82)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$class.fireChanged(ChangeListenerSupport.scala:74)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher.fireChanged(ClusteredSingleton.scala:82)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher.changed_decoded(ClusteredSingleton.scala:157)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher$$anon$2.changed(ClusteredSingleton.scala:107)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1$$anonfun$apply$mcV$sp$3.apply(ChangeListenerSupport.scala:76)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1$$anonfun$apply$mcV$sp$3.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)[57:scala-library:2.9.1]
at scala.collection.immutable.List.foreach(List.scala:45)[57:scala-library:2.9.1]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply$mcV$sp(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$class.check_elapsed_time(ChangeListenerSupport.scala:84)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.check_elapsed_time(ZooKeeperGroup.scala:62)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$class.fireChanged(ChangeListenerSupport.scala:74)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.fireChanged(ZooKeeperGroup.scala:62)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.fire_cluster_change(ZooKeeperGroup.scala:148)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.childEvent(ZooKeeperGroup.scala:80)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache$5.apply(TreeCache.java:539)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache$5.apply(TreeCache.java:535)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)[70:com.google.guava:13.0.1]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:533)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeEventOperation.invoke(TreeEventOperation.java:35)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache.mainLoop(TreeCache.java:840)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache.access$600(TreeCache.java:73)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:303)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)[:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)[:1.6.0_24]
at java.lang.Thread.run(Thread.java:679)[:1.6.0_24]
18:24:06,392 | ERROR | hChildrenCache-1 | TreeCache | 53 - org.fusesource.fabric.fabric-zookeeper - 7.3.0.redhat-003 |
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /fabric/registry/clusters/fusemq/default
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)[52:org.fusesource.fabric.fabric-linkedin-zookeeper:7.3.0.redhat-003]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)[52:org.fusesource.fabric.fabric-linkedin-zookeeper:7.3.0.redhat-003]
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1566)[52:org.fusesource.fabric.fabric-linkedin-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache.refresh(TreeCache.java:526)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeRefreshOperation.invoke(TreeRefreshOperation.java:39)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache.mainLoop(TreeCache.java:840)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache.access$600(TreeCache.java:73)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:303)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)[:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)[:1.6.0_24]
at java.lang.Thread.run(Thread.java:679)[:1.6.0_24]
18:24:06,394 | INFO | hChildrenCache-2 | Bridge | 243 - org.fusesource.fabric.fabric-git-zkbridge - 7.3.0.redhat-003 | Git/zk bridge is inactive
18:24:06,393 | INFO | onStateManager-0 | GitHttpServerRegistrationHandler | 247 - org.fusesource.fabric.fabric-git-server - 7.3.0.redhat-003 | Git repo is not the master
18:24:06,397 | ERROR | onStateManager-0 | ListenerContainer | 53 - org.fusesource.fabric.fabric-zookeeper - 7.3.0.redhat-003 | Listener (org.fusesource.fabric.groups.internal.ZooKeeperGroup@4232483e) threw an exception
java.lang.IllegalArgumentException: Has not joined locally: 0000000001
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.update(ZooKeeperGroup.scala:121)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingleton.update(ClusteredSingleton.scala:227)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.git.http.GitHttpServerRegistrationHandler.changed(GitHttpServerRegistrationHandler.java:188)[247:org.fusesource.fabric.fabric-git-server:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1$$anonfun$apply$mcV$sp$3.apply(ChangeListenerSupport.scala:76)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1$$anonfun$apply$mcV$sp$3.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)[57:scala-library:2.9.1]
at scala.collection.immutable.List.foreach(List.scala:45)[57:scala-library:2.9.1]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply$mcV$sp(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireChanged$1.apply(ChangeListenerSupport.scala:75)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$class.check_elapsed_time(ChangeListenerSupport.scala:84)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher.check_elapsed_time(ClusteredSingleton.scala:82)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$class.fireChanged(ChangeListenerSupport.scala:74)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher.fireChanged(ClusteredSingleton.scala:82)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher.changed_decoded(ClusteredSingleton.scala:157)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher$$anon$2.changed(ClusteredSingleton.scala:107)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.ClusteredSingletonWatcher$$anon$2.disconnected(ClusteredSingleton.scala:116)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireDisconnected$1$$anonfun$apply$mcV$sp$2.apply(ChangeListenerSupport.scala:66)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireDisconnected$1$$anonfun$apply$mcV$sp$2.apply(ChangeListenerSupport.scala:65)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)[57:scala-library:2.9.1]
at scala.collection.immutable.List.foreach(List.scala:45)[57:scala-library:2.9.1]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireDisconnected$1.apply$mcV$sp(ChangeListenerSupport.scala:65)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireDisconnected$1.apply(ChangeListenerSupport.scala:65)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$$anonfun$fireDisconnected$1.apply(ChangeListenerSupport.scala:65)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$class.check_elapsed_time(ChangeListenerSupport.scala:84)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.check_elapsed_time(ZooKeeperGroup.scala:62)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ChangeListenerSupport$class.fireDisconnected(ChangeListenerSupport.scala:64)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.fireDisconnected(ZooKeeperGroup.scala:62)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.onDisconnected(ZooKeeperGroup.scala:107)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.fusesource.fabric.groups.internal.ZooKeeperGroup.stateChanged(ZooKeeperGroup.scala:101)[129:org.fusesource.fabric.fabric-groups:7.3.0.redhat-003]
at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:184)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:180)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)[70:com.google.guava:13.0.1]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:177)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:40)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:104)[53:org.fusesource.fabric.fabric-zookeeper:7.3.0.redhat-003]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)[:1.6.0_24]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)[:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)[:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)[:1.6.0_24]
at java.lang.Thread.run(Thread.java:679)[:1.6.0_24]
After this initial output, the following error is repeated endlessly. The CPU will stay at 99% usage most of the time, occasionally dipping down to ~75% but then quickly climbing again.
18:24:50,851 | ERROR | tp1350866493-510 | ConnectionState | 53 - org.fusesource.fabric.fabric-zookeeper - 7.3.0.redhat-003 | Connection timed out for connection string (10.97.6.186:2182,itfus021.ocado.com:2181,itfus022.ocado.com:2181) and timeout (15000) / elapsed (15996)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:102)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:108)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:449)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38)
at org.fusesource.fabric.zookeeper.utils.ZooKeeperUtils.getChildren(ZooKeeperUtils.java:122)
at org.fusesource.fabric.service.ZooKeeperDataStore.getContainers(ZooKeeperDataStore.java:163)
at org.fusesource.fabric.service.FabricServiceImpl.getContainers(FabricServiceImpl.java:181)
at org.fusesource.fabric.api.jmx.FabricManager.containers(FabricManager.java:343)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.6.0_24]
at java.lang.reflect.Method.invoke(Method.java:616)[:1.6.0_24]
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)[:1.6.0_24]
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)[:1.6.0_24]
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)[:1.6.0_24]
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)[:1.6.0_24]
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)[:1.6.0_24]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)[:1.6.0_24]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)[:1.6.0_24]
at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:102)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:91)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:388)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:150)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:195)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:131)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.http.AgentServlet$2.handleRequest(AgentServlet.java:283)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.http.AgentServlet.handle(AgentServlet.java:225)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:197)[246:io.hawt.hawtio-web:1.2.0.M1]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)[98:org.apache.geronimo.specs.geronimo-servlet_2.5_spec:1.2]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)[98:org.apache.geronimo.specs.geronimo-servlet_2.5_spec:1.2]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)[83:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329)[83:org.eclipse.jetty.servlet:7.6.7.v20120910]
at io.hawt.web.CORSFilter.doFilter(CORSFilter.java:42)[246:io.hawt.hawtio-web:1.2.0.M1]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)[83:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:193)[103:org.ops4j.pax.web.pax-web-runtime:1.1.13]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300)[83:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)[83:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[101:org.ops4j.pax.web.pax-web-jetty:1.1.13]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[85:org.eclipse.jetty.security:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[101:org.ops4j.pax.web.pax-web-jetty:1.1.13]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[83:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:81)[101:org.ops4j.pax.web.pax-web-jetty:1.1.13]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.Server.handle(Server.java:363)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:931)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:992)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856)[82:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)[82:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[87:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[84:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[84:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[93:org.eclipse.jetty.util:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[93:org.eclipse.jetty.util:7.6.7.v20120910]
at java.lang.Thread.run(Thread.java:679)[:1.6.0_24]
