1 Reply Latest reply on Dec 5, 2014 7:35 AM by sepadev

    How to block HTTP/S calls during JBoss startup

    sepadev Newbie

      On a productive server I'm having a deadlock situation during startup. Clients send HTTP requests which mess up the normal startup order and startup never completes. I expected to find a deadlock in threaddump, but this was not the case. I was able to startup server if i blocked http by firewall during startup.

       

      There is a suspicious http thread, which initiates the startup of some ejb singletons, but waits forever. Weird: thread locks 0x00000007e23c9d58 but then waits for it later (see below.. locked and then waiting).

       

      "http-example.com-443-2" daemon prio=10 tid=0x00007f3cdc003000 nid=0x709 in Object.wait() [0x00007f3d71efc000]
        java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00000007e23c9d58> (a org.jboss.as.ejb3.component.singleton.SingletonComponent)
      at java.lang.Object.wait(Object.java:503)
      at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:109)
      - locked <0x00000007e23c9d58> (a org.jboss.as.ejb3.component.singleton.SingletonComponent)
      at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:127)
      at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
      at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:116)
      - locked <0x00000007e25417a0> (a java.lang.Object)
      at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:48)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:228)
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:304)
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:190)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
      at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
      at com.example.core.config.ResourceConfigurationService$$$view37.getResourceConfigurations(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:264)
      at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52)
      at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137)
      at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:260)
      at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:111)
      at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
      at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105)
      at com.example.core.config.DomainConfigurationService$NodeConfigurationService$ResourceConfigurationService$ResourceServiceConfigurationService$-541784333$Proxy$_$$_Weld$Proxy$.getResourceConfigurations(DomainConfigurationService$NodeConfigurationService$ResourceConfigurationService$ResourceServiceConfigurationService$-541784333$Proxy$_$$_Weld$Proxy$.java)
      at com.example.core.fetch.ResourceFetchService.init(ResourceFetchService.java:159)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:264)
      at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52)
      at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137)
      at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:260)
      at org.jboss.weld.introspector.jlr.WeldMethodImpl.invoke(WeldMethodImpl.java:174)
      at org.jboss.weld.bean.AbstractClassBean.defaultPostConstruct(AbstractClassBean.java:516)
      at org.jboss.weld.bean.ManagedBean$ManagedBeanInjectionTarget.postConstruct(ManagedBean.java:174)
      at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:291)
      at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:107)
      at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:616)
      at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:681)
      at org.jboss.as.weld.injection.InjectableField.inject(InjectableField.java:56)
      at org.jboss.as.weld.injection.WeldEEInjection.inject(WeldEEInjection.java:81)
      at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:60)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
      at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
      at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
      at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
      at org.jboss.as.web.deployment.component.WebComponentInstantiator$1.(WebComponentInstantiator.java:57)
      at org.jboss.as.web.deployment.component.WebComponentInstantiator.getReference(WebComponentInstantiator.java:55)
      at org.jboss.as.web.deployment.WebInjectionContainer.instantiate(WebInjectionContainer.java:99)
      at org.jboss.as.web.deployment.WebInjectionContainer.newInstance(WebInjectionContainer.java:78)
      at org.jboss.as.web.deployment.WebInjectionContainer.newInstance(WebInjectionContainer.java:72)
      at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1156)
      - locked <0x00000007e0018618> (a org.apache.catalina.core.StandardWrapper)
      at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:952)
      - locked <0x00000007e0018618> (a org.apache.catalina.core.StandardWrapper)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:188)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489)
      at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
      at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
      at java.lang.Thread.run(Thread.java:745)

       

      My preferred solution is to somehow tell tomcat to reject or pause http requests as long as Server was not started up. The other way is to find a potential error in my ejb startup chain.

       

      Server: JBoss 7.1.1

      OS: Ubuntu 14.04