0 Replies Latest reply on Jan 31, 2018 7:22 AM by Ghousuddin Mohammed

    Wildfly 10.1.0 Final - server getting stalled (Stopping...)

    Ghousuddin Mohammed Newbie

      Wildfly server is getting stalled whenever I restart, to debug the issue downloaded JBoss source code, and identified hanging threads in debug mode, which are waiting for each other to complete.

      Below is the stack trace, any help or pointers appreciated.

       

      Thread [ServerService Thread Pool -- 61] (Suspended)

      waiting for: PrintStream  (id=486)

      owned by: Daemon Thread [Starting ActiveMQ Broker] (Running)

      PrintStream.println(String) line: 806

      StdioContext$1(StdioContext$DelegatingPrintStream).println(String) line: 474

      SpringBootBanner.printBanner(Environment, Class<?>, PrintStream) line: 49

      SpringApplicationBannerPrinter.print(Environment, Class<?>, PrintStream) line: 71

      SpringApplication.printBanner(ConfigurableEnvironment) line: 505

      SpringApplication.run(String...) line: 298

      MainApplication(SpringBootServletInitializer).run(SpringApplication) line: 151

      MainApplication(SpringBootServletInitializer).createRootApplicationContext(ServletContext) line: 131

      MainApplication(SpringBootServletInitializer).onStartup(ServletContext) line: 86

      SpringServletContainerInitializer.onStartup(Set<Class<?>>, ServletContext) line: 169

      DeploymentManagerImpl$1.call(HttpServerExchange, Object) line: 186

      DeploymentManagerImpl$1.call(HttpServerExchange, Object) line: 171

      ServletRequestContextThreadSetupAction$1.call(HttpServerExchange, C) line: 42

      ContextClassLoaderSetupAction$1.call(HttpServerExchange, C) line: 43

      LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44

      LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44

      LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44

      LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44

      LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44

      DeploymentManagerImpl.deploy() line: 234

      UndertowDeploymentService.startContext() line: 100

      UndertowDeploymentService$1.run() line: 82

      Executors$RunnableAdapter<T>.call() line: 511

      FutureTask<V>.run() line: 266

      ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142

      ThreadPoolExecutor$Worker.run() line: 617

      JBossThread(Thread).run() line: 745

      JBossThread.run() line: 320

       

      Thread [MSC service thread 1-1] (Suspended)

      waiting for: PrintStream  (id=486)

      owned by: Daemon Thread [Starting ActiveMQ Broker] (Running)

      PrintStream.write(byte[], int, int) line: 479

      StdioContext$1(StdioContext$DelegatingPrintStream).write(byte[], int, int) line: 264

      StdioContext$1(FilterOutputStream).write(byte[]) line: 97

      ConsoleTarget$1.write(byte[]) line: 37

      PatternLayoutEncoder(LayoutWrappingEncoder<E>).doEncode(E) line: 131

      ConsoleAppender<E>(OutputStreamAppender<E>).writeOut(E) line: 187

      ConsoleAppender<E>(OutputStreamAppender<E>).subAppend(E) line: 212

      ConsoleAppender<E>(OutputStreamAppender<E>).append(E) line: 100

      ConsoleAppender<E>(UnsynchronizedAppenderBase<E>).doAppend(E) line: 84

      AppenderAttachableImpl<E>.appendLoopOnAppenders(E) line: 51

      Logger.appendLoopOnAppenders(ILoggingEvent) line: 270

      Logger.callAppenders(ILoggingEvent) line: 257

      Logger.buildLoggingEventAndAppend(String, Marker, Level, String, Object[], Throwable) line: 421

      Logger.filterAndLog_0_Or3Plus(String, Marker, Level, String, Object[], Throwable) line: 383

      Logger.log(Marker, String, int, String, Object[], Throwable) line: 765

      SLF4JBridgeHandler.callLocationAwareLogger(LocationAwareLogger, LogRecord) line: 221

      SLF4JBridgeHandler.publish(LogRecord) line: 303

      LoggerNode.publish(ExtLogRecord) line: 314

      LoggerNode.publish(ExtLogRecord) line: 322

      LoggerNode.publish(ExtLogRecord) line: 322

      LoggerNode.publish(ExtLogRecord) line: 322

      LoggerNode.publish(ExtLogRecord) line: 322

      LoggerNode.publish(ExtLogRecord) line: 322

      LoggerNode.publish(ExtLogRecord) line: 322

      Logger.logRaw(ExtLogRecord) line: 850

      Logger.log(String, Level, String, ExtLogRecord$FormatStyle, Object[], Throwable) line: 802

      JBossLogManagerLogger.doLogf(Logger$Level, String, String, Object[], Throwable) line: 53

      JBossLogManagerLogger(Logger).logf(String, Logger$Level, Throwable, String, Object) line: 2398

      ConnectorLogger_$logger.startedDriverService(String) line: 164

      DriverService.start(StartContext) line: 68

      ServiceControllerImpl$StartTask.startService(Service<? extends S>, StartContext) line: 1948

      ServiceControllerImpl$StartTask.run() line: 1881

      ServiceContainerImpl$ContainerExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: 1142

      ThreadPoolExecutor$Worker.run() line: 617

      ServiceContainerImpl$ServiceThread(Thread).run() line: 745

       

      Daemon Thread [Starting ActiveMQ Broker] (Suspended)

      owns: StringBuilder  (id=484)

      owns: SingleByte$Decoder  (id=485)

      owns: PrintStream  (id=486)

      waited by: Thread [ServerService Thread Pool -- 61] (Running)

      waited by: Thread [MSC service thread 1-1] (Running)

      owns: OutputStreamWriter  (id=487)

      owns: ConsoleAppender  (id=488)

      owns: RootLogger  (id=489)

      owns: ConcurrentHashMap<K,V>  (id=490)

      owns: Object  (id=491)

      owns: ActiveMQResourceAdapter  (id=492)

      Unsafe.park(boolean, long) line: not available [native method]

      LockSupport.park(Object) line: 175

      ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).parkAndCheckInterrupt() line: 836

      ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquireQueued(AbstractQueuedSynchronizer$Node, int) line: 870

      ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquire(int) line: 1199

      ReentrantLock$NonfairSync.lock() line: 209

      ReentrantLock.lock() line: 285

      ConsoleAppender<E>(OutputStreamAppender<E>).subAppend(E) line: 210

      ConsoleAppender<E>(OutputStreamAppender<E>).append(E) line: 100

      ConsoleAppender<E>(UnsynchronizedAppenderBase<E>).doAppend(E) line: 84

      AppenderAttachableImpl<E>.appendLoopOnAppenders(E) line: 51

      Logger.appendLoopOnAppenders(ILoggingEvent) line: 270

      Logger.callAppenders(ILoggingEvent) line: 257

      Logger.buildLoggingEventAndAppend(String, Marker, Level, String, Object[], Throwable) line: 421

      Logger.filterAndLog_0_Or3Plus(String, Marker, Level, String, Object[], Throwable) line: 383

      Logger.log(Marker, String, int, String, Object[], Throwable) line: 765

      SLF4JBridgeHandler.callLocationAwareLogger(LocationAwareLogger, LogRecord) line: 221

      SLF4JBridgeHandler.publish(LogRecord) line: 303

      LoggerNode.publish(ExtLogRecord) line: 314

      LoggerNode.publish(ExtLogRecord) line: 322

      Logger.logRaw(ExtLogRecord) line: 850

      Logger.log(Level, String) line: 596

      LoggingWriter(AbstractLoggingWriter).write(char[], int, int) line: 71

      LoggingOutputStream(WriterOutputStream).finish() line: 143

      LoggingOutputStream(WriterOutputStream).flush() line: 164

      PrintStream.write(byte[], int, int) line: 482

      StdioContext$1(StdioContext$DelegatingPrintStream).write(byte[], int, int) line: 264

      StreamEncoder.writeBytes() line: 221

      StreamEncoder.implFlushBuffer() line: 291

      StreamEncoder.implFlush() line: 295

      StreamEncoder.flush() line: 141

      OutputStreamWriter.flush() line: 229

      QuietWriter.flush() line: 59

      ConsoleAppender(WriterAppender).subAppend(LoggingEvent) line: 324

      ConsoleAppender(WriterAppender).append(LoggingEvent) line: 162

      ConsoleAppender(AppenderSkeleton).doAppend(LoggingEvent) line: 251

      AppenderAttachableImpl.appendLoopOnAppenders(LoggingEvent) line: 66

      Logger(Category).callAppenders(LoggingEvent) line: 206

      Logger(Category).forcedLog(String, Priority, Object, Throwable) line: 391

      Logger(Category).log(String, Priority, Object, Throwable) line: 856

      Log4jLoggerAdapter.info(String, Object...) line: 363

      XBeanBrokerService(BrokerService).doStartBroker() line: 760

      XBeanBrokerService(BrokerService).startBroker(boolean) line: 720

      XBeanBrokerService(BrokerService).start() line: 623

      XBeanBrokerService.afterPropertiesSet() line: 73

      NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]

      NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62

      DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43

      Method.invoke(Object, Object...) line: 498

      DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeCustomInitMethod(String, Object, RootBeanDefinition) line: 1700

      DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1639

      DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1568

      DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).doCreateBean(String, RootBeanDefinition, Object[]) line: 539

      DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 476

      AbstractBeanFactory$1.getObject() line: 303

      DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory<?>) line: 230

      DefaultListableBeanFactory(AbstractBeanFactory).doGetBean(String, Class<T>, Object[], boolean) line: 299

      DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 194

      DefaultListableBeanFactory.preInstantiateSingletons() line: 755

      XBeanBrokerFactory$1(AbstractApplicationContext).finishBeanFactoryInitialization(ConfigurableListableBeanFactory) line: 762

      XBeanBrokerFactory$1(AbstractApplicationContext).refresh() line: 480

      XBeanBrokerFactory$1(ResourceXmlApplicationContext).<init>(Resource, List) line: 64

      XBeanBrokerFactory$1(ResourceXmlApplicationContext).<init>(Resource) line: 52

      XBeanBrokerFactory$1.<init>(XBeanBrokerFactory, Resource) line: 104

      XBeanBrokerFactory.createApplicationContext(String) line: 104

      XBeanBrokerFactory.createBroker(URI) line: 67

      BrokerFactory.createBroker(URI, boolean) line: 71

      BrokerFactory.createBroker(URI) line: 54

      ActiveMQResourceAdapter$1.run() line: 88

       

      Seems Issue is with logging, gone through below few links didn't find a proper solution.

      https://issues.jboss.org/browse/WFCORE-707

      https://issues.jboss.org/browse/WFCORE-40