How to block HTTP/S calls during JBoss startup
sepadev Dec 5, 2014 3:11 AMOn a productive server I'm having a deadlock situation during startup. Clients send HTTP requests which mess up the normal startup order and startup never completes. I expected to find a deadlock in threaddump, but this was not the case. I was able to startup server if i blocked http by firewall during startup.
There is a suspicious http thread, which initiates the startup of some ejb singletons, but waits forever. Weird: thread locks 0x00000007e23c9d58 but then waits for it later (see below.. locked and then waiting).
"http-example.com-443-2" daemon prio=10 tid=0x00007f3cdc003000 nid=0x709 in Object.wait() [0x00007f3d71efc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e23c9d58> (a org.jboss.as.ejb3.component.singleton.SingletonComponent)
at java.lang.Object.wait(Object.java:503)
at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:109)
- locked <0x00000007e23c9d58> (a org.jboss.as.ejb3.component.singleton.SingletonComponent)
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:127)
at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:116)
- locked <0x00000007e25417a0> (a java.lang.Object)
at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:48)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:228)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:304)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:190)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
at com.example.core.config.ResourceConfigurationService$$$view37.getResourceConfigurations(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:264)
at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52)
at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137)
at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:260)
at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:111)
at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105)
at com.example.core.config.DomainConfigurationService$NodeConfigurationService$ResourceConfigurationService$ResourceServiceConfigurationService$-541784333$Proxy$_$$_Weld$Proxy$.getResourceConfigurations(DomainConfigurationService$NodeConfigurationService$ResourceConfigurationService$ResourceServiceConfigurationService$-541784333$Proxy$_$$_Weld$Proxy$.java)
at com.example.core.fetch.ResourceFetchService.init(ResourceFetchService.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:264)
at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52)
at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137)
at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:260)
at org.jboss.weld.introspector.jlr.WeldMethodImpl.invoke(WeldMethodImpl.java:174)
at org.jboss.weld.bean.AbstractClassBean.defaultPostConstruct(AbstractClassBean.java:516)
at org.jboss.weld.bean.ManagedBean$ManagedBeanInjectionTarget.postConstruct(ManagedBean.java:174)
at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:291)
at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:107)
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:616)
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:681)
at org.jboss.as.weld.injection.InjectableField.inject(InjectableField.java:56)
at org.jboss.as.weld.injection.WeldEEInjection.inject(WeldEEInjection.java:81)
at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:60)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
at org.jboss.as.web.deployment.component.WebComponentInstantiator$1.(WebComponentInstantiator.java:57)
at org.jboss.as.web.deployment.component.WebComponentInstantiator.getReference(WebComponentInstantiator.java:55)
at org.jboss.as.web.deployment.WebInjectionContainer.instantiate(WebInjectionContainer.java:99)
at org.jboss.as.web.deployment.WebInjectionContainer.newInstance(WebInjectionContainer.java:78)
at org.jboss.as.web.deployment.WebInjectionContainer.newInstance(WebInjectionContainer.java:72)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1156)
- locked <0x00000007e0018618> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:952)
- locked <0x00000007e0018618> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:188)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489)
at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:745)
My preferred solution is to somehow tell tomcat to reject or pause http requests as long as Server was not started up. The other way is to find a potential error in my ejb startup chain.
Server: JBoss 7.1.1
OS: Ubuntu 14.04