Wildfly 10.1.0 Final - server getting stalled (Stopping...)
Wildfly server is getting stalled whenever I restart, to debug the issue downloaded JBoss source code, and identified hanging threads in debug mode, which are waiting for each other to complete.
Below is the stack trace, any help or pointers appreciated.
Thread [ServerService Thread Pool -- 61] (Suspended)
waiting for: PrintStream (id=486)
owned by: Daemon Thread [Starting ActiveMQ Broker] (Running)
PrintStream.println(String) line: 806
StdioContext$1(StdioContext$DelegatingPrintStream).println(String) line: 474
SpringBootBanner.printBanner(Environment, Class<?>, PrintStream) line: 49
SpringApplicationBannerPrinter.print(Environment, Class<?>, PrintStream) line: 71
SpringApplication.printBanner(ConfigurableEnvironment) line: 505
SpringApplication.run(String...) line: 298
MainApplication(SpringBootServletInitializer).run(SpringApplication) line: 151
MainApplication(SpringBootServletInitializer).createRootApplicationContext(ServletContext) line: 131
MainApplication(SpringBootServletInitializer).onStartup(ServletContext) line: 86
SpringServletContainerInitializer.onStartup(Set<Class<?>>, ServletContext) line: 169
DeploymentManagerImpl$1.call(HttpServerExchange, Object) line: 186
DeploymentManagerImpl$1.call(HttpServerExchange, Object) line: 171
ServletRequestContextThreadSetupAction$1.call(HttpServerExchange, C) line: 42
ContextClassLoaderSetupAction$1.call(HttpServerExchange, C) line: 43
LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44
LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44
LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44
LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44
LegacyThreadSetupActionWrapper$1.call(HttpServerExchange, C) line: 44
DeploymentManagerImpl.deploy() line: 234
UndertowDeploymentService.startContext() line: 100
UndertowDeploymentService$1.run() line: 82
Executors$RunnableAdapter<T>.call() line: 511
FutureTask<V>.run() line: 266
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142
ThreadPoolExecutor$Worker.run() line: 617
JBossThread(Thread).run() line: 745
JBossThread.run() line: 320
Thread [MSC service thread 1-1] (Suspended)
waiting for: PrintStream (id=486)
owned by: Daemon Thread [Starting ActiveMQ Broker] (Running)
PrintStream.write(byte[], int, int) line: 479
StdioContext$1(StdioContext$DelegatingPrintStream).write(byte[], int, int) line: 264
StdioContext$1(FilterOutputStream).write(byte[]) line: 97
ConsoleTarget$1.write(byte[]) line: 37
PatternLayoutEncoder(LayoutWrappingEncoder<E>).doEncode(E) line: 131
ConsoleAppender<E>(OutputStreamAppender<E>).writeOut(E) line: 187
ConsoleAppender<E>(OutputStreamAppender<E>).subAppend(E) line: 212
ConsoleAppender<E>(OutputStreamAppender<E>).append(E) line: 100
ConsoleAppender<E>(UnsynchronizedAppenderBase<E>).doAppend(E) line: 84
AppenderAttachableImpl<E>.appendLoopOnAppenders(E) line: 51
Logger.appendLoopOnAppenders(ILoggingEvent) line: 270
Logger.callAppenders(ILoggingEvent) line: 257
Logger.buildLoggingEventAndAppend(String, Marker, Level, String, Object[], Throwable) line: 421
Logger.filterAndLog_0_Or3Plus(String, Marker, Level, String, Object[], Throwable) line: 383
Logger.log(Marker, String, int, String, Object[], Throwable) line: 765
SLF4JBridgeHandler.callLocationAwareLogger(LocationAwareLogger, LogRecord) line: 221
SLF4JBridgeHandler.publish(LogRecord) line: 303
LoggerNode.publish(ExtLogRecord) line: 314
LoggerNode.publish(ExtLogRecord) line: 322
LoggerNode.publish(ExtLogRecord) line: 322
LoggerNode.publish(ExtLogRecord) line: 322
LoggerNode.publish(ExtLogRecord) line: 322
LoggerNode.publish(ExtLogRecord) line: 322
LoggerNode.publish(ExtLogRecord) line: 322
Logger.logRaw(ExtLogRecord) line: 850
Logger.log(String, Level, String, ExtLogRecord$FormatStyle, Object[], Throwable) line: 802
JBossLogManagerLogger.doLogf(Logger$Level, String, String, Object[], Throwable) line: 53
JBossLogManagerLogger(Logger).logf(String, Logger$Level, Throwable, String, Object) line: 2398
ConnectorLogger_$logger.startedDriverService(String) line: 164
DriverService.start(StartContext) line: 68
ServiceControllerImpl$StartTask.startService(Service<? extends S>, StartContext) line: 1948
ServiceControllerImpl$StartTask.run() line: 1881
ServiceContainerImpl$ContainerExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: 1142
ThreadPoolExecutor$Worker.run() line: 617
ServiceContainerImpl$ServiceThread(Thread).run() line: 745
Daemon Thread [Starting ActiveMQ Broker] (Suspended)
owns: StringBuilder (id=484)
owns: SingleByte$Decoder (id=485)
owns: PrintStream (id=486)
waited by: Thread [ServerService Thread Pool -- 61] (Running)
waited by: Thread [MSC service thread 1-1] (Running)
owns: OutputStreamWriter (id=487)
owns: ConsoleAppender (id=488)
owns: RootLogger (id=489)
owns: ConcurrentHashMap<K,V> (id=490)
owns: Object (id=491)
owns: ActiveMQResourceAdapter (id=492)
Unsafe.park(boolean, long) line: not available [native method]
LockSupport.park(Object) line: 175
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).parkAndCheckInterrupt() line: 836
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquireQueued(AbstractQueuedSynchronizer$Node, int) line: 870
ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquire(int) line: 1199
ReentrantLock$NonfairSync.lock() line: 209
ReentrantLock.lock() line: 285
ConsoleAppender<E>(OutputStreamAppender<E>).subAppend(E) line: 210
ConsoleAppender<E>(OutputStreamAppender<E>).append(E) line: 100
ConsoleAppender<E>(UnsynchronizedAppenderBase<E>).doAppend(E) line: 84
AppenderAttachableImpl<E>.appendLoopOnAppenders(E) line: 51
Logger.appendLoopOnAppenders(ILoggingEvent) line: 270
Logger.callAppenders(ILoggingEvent) line: 257
Logger.buildLoggingEventAndAppend(String, Marker, Level, String, Object[], Throwable) line: 421
Logger.filterAndLog_0_Or3Plus(String, Marker, Level, String, Object[], Throwable) line: 383
Logger.log(Marker, String, int, String, Object[], Throwable) line: 765
SLF4JBridgeHandler.callLocationAwareLogger(LocationAwareLogger, LogRecord) line: 221
SLF4JBridgeHandler.publish(LogRecord) line: 303
LoggerNode.publish(ExtLogRecord) line: 314
LoggerNode.publish(ExtLogRecord) line: 322
Logger.logRaw(ExtLogRecord) line: 850
Logger.log(Level, String) line: 596
LoggingWriter(AbstractLoggingWriter).write(char[], int, int) line: 71
LoggingOutputStream(WriterOutputStream).finish() line: 143
LoggingOutputStream(WriterOutputStream).flush() line: 164
PrintStream.write(byte[], int, int) line: 482
StdioContext$1(StdioContext$DelegatingPrintStream).write(byte[], int, int) line: 264
StreamEncoder.writeBytes() line: 221
StreamEncoder.implFlushBuffer() line: 291
StreamEncoder.implFlush() line: 295
StreamEncoder.flush() line: 141
OutputStreamWriter.flush() line: 229
QuietWriter.flush() line: 59
ConsoleAppender(WriterAppender).subAppend(LoggingEvent) line: 324
ConsoleAppender(WriterAppender).append(LoggingEvent) line: 162
ConsoleAppender(AppenderSkeleton).doAppend(LoggingEvent) line: 251
AppenderAttachableImpl.appendLoopOnAppenders(LoggingEvent) line: 66
Logger(Category).callAppenders(LoggingEvent) line: 206
Logger(Category).forcedLog(String, Priority, Object, Throwable) line: 391
Logger(Category).log(String, Priority, Object, Throwable) line: 856
Log4jLoggerAdapter.info(String, Object...) line: 363
XBeanBrokerService(BrokerService).doStartBroker() line: 760
XBeanBrokerService(BrokerService).startBroker(boolean) line: 720
XBeanBrokerService(BrokerService).start() line: 623
XBeanBrokerService.afterPropertiesSet() line: 73
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 498
DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeCustomInitMethod(String, Object, RootBeanDefinition) line: 1700
DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1639
DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1568
DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).doCreateBean(String, RootBeanDefinition, Object[]) line: 539
DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 476
AbstractBeanFactory$1.getObject() line: 303
DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory<?>) line: 230
DefaultListableBeanFactory(AbstractBeanFactory).doGetBean(String, Class<T>, Object[], boolean) line: 299
DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 194
DefaultListableBeanFactory.preInstantiateSingletons() line: 755
XBeanBrokerFactory$1(AbstractApplicationContext).finishBeanFactoryInitialization(ConfigurableListableBeanFactory) line: 762
XBeanBrokerFactory$1(AbstractApplicationContext).refresh() line: 480
XBeanBrokerFactory$1(ResourceXmlApplicationContext).<init>(Resource, List) line: 64
XBeanBrokerFactory$1(ResourceXmlApplicationContext).<init>(Resource) line: 52
XBeanBrokerFactory$1.<init>(XBeanBrokerFactory, Resource) line: 104
XBeanBrokerFactory.createApplicationContext(String) line: 104
XBeanBrokerFactory.createBroker(URI) line: 67
BrokerFactory.createBroker(URI, boolean) line: 71
BrokerFactory.createBroker(URI) line: 54
ActiveMQResourceAdapter$1.run() line: 88
Seems Issue is with logging, gone through below few links didn't find a proper solution.