Skip navigation

4 Replies Latest reply on Aug 3, 2017 10:09 AM by ketanmistry

HAWKAPM600000: Maximum retry reached.

ketanmistry May 11, 2017 8:09 AM

Installed Hawkular APM 0.14.3 distribution on RHEL and started standalone server.

Pointed my Zipkin instrumentation to it and generated some load - nothing. I set the standalone server in debug mode and can see this error:

2017-05-11 12:09:09,647 WARN [org.hawkular.apm.server.api.log] (Thread-189 (ActiveMQ-client-global-threads--1391354641)) HAWKAPM600000: Maximum retry reached. Last exception to occur ....: org.hawkular.apm.server.api.task.RetryAttemptException: URL is null, span id = 845E8F893976CA60

at org.hawkular.apm.server.processor.zipkin.NodeDetailsDeriver.processOneToOne(NodeDetailsDeriver.java:69)

at org.hawkular.apm.server.processor.zipkin.NodeDetailsDeriver.processOneToOne(NodeDetailsDeriver.java:39)

at org.hawkular.apm.server.api.task.ProcessingUnit.handle(ProcessingUnit.java:146)

at org.hawkular.apm.server.jms.RetryCapableMDB.process(RetryCapableMDB.java:191)

at org.hawkular.apm.server.jms.RetryCapableMDB.onMessage(RetryCapableMDB.java:154)

at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

at java.lang.reflect.Method.invoke(Method.java:508)

at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)

at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)

at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)

at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)

at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)

at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:254)

at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:329)

at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:239)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)

at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)

at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)

at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)

at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)

at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)

at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)

at org.hawkular.apm.server.jms.span.NodeDetailsDeriverMDB$$$view6.onMessage(Unknown Source)

at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

at java.lang.reflect.Method.invoke(Method.java:508)

at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:139)

at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73)

at org.hawkular.apm.server.jms.span.NodeDetailsDeriverMDB$$$endpoint1.onMessage(Unknown Source)

at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:310)

at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:932)

at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:47)

at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1045)

at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:100)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.lang.Thread.run(Thread.java:785)

Any ideas?

1. Re: HAWKAPM600000: Maximum retry reached.

objectiser May 12, 2017 2:59 AM (in response to ketanmistry)

Sorry for the delay in replying.

Hawkular APM identifies endpoints using a URI, which is the basis upon which aggregations are performed for those endpoints. Currently the only standard tags in zipkin that provide those URIs are 'http.uri', 'http.url' and 'http.path'. So if client/server spans are being reported without those tags, then it will result in this problem.
Actions
2. Re: HAWKAPM600000: Maximum retry reached.

ketanmistry May 12, 2017 5:43 AM (in response to objectiser)

Thanks Gary.

I have a spring-boot app which is using the Spring Cloud Sleuth library, I will look at the tagging available.
Actions
3. Re: HAWKAPM600000: Maximum retry reached.

howard.em Aug 2, 2017 9:48 AM (in response to ketanmistry)

Hi,

I'm using Spring Cloud Sleuth and having the same issue you faced. Did you find a solution about how integrate Sleuth with Hawkular?

Best
Actions
4. Re: HAWKAPM600000: Maximum retry reached.

ketanmistry Aug 3, 2017 10:09 AM (in response to howard.em)

I had a quick look at tags in Spring Cloud Sleuth, but gave up in the end.
Actions

Go to original post