Worker thread initialization failure

[boniek]

Whenever I run jboss on my Solaris system i get following exceptions:

2008-11-13 17:45:34,898 TRACE [org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread] Sleeping...
2008-11-13 17:45:35,997 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] process(): region: /_default_/
2008-11-13 17:45:35,997 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] processed 0 node events in region: /_default_/
2008-11-13 17:45:35,997 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] Recycle queue is empty
2008-11-13 17:45:36,390 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] Begin run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@6dbcef
2008-11-13 17:45:36,390 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] End run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@6dbcef
2008-11-13 17:45:36,943 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] Begin run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@621bd7
2008-11-13 17:45:36,943 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] End run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@621bd7
2008-11-13 17:45:38,670 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] Begin run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@1f5e1ea
2008-11-13 17:45:38,670 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] End run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@12eb04e
2008-11-13 17:45:39,097 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] process(): region: /_default_/
2008-11-13 17:45:39,097 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] processed 0 node events in region: /_default_/
2008-11-13 17:45:39,097 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] Recycle queue is empty
2008-11-13 17:45:39,907 TRACE [org.jboss.mx.loading.RepositoryClassLoader] loadClass org.jboss.mx.loading.UnifiedClassLoader3@182eca8{ url=file:/opt/jboss-4.3.0.GA_CP02/server/production/deploy/deploy.last/ ,addedOrder=47} name=org.jboss.net.protocol.file.FileURLLister, loadClassDepth=0
2008-11-13 17:45:39,907 TRACE [org.jboss.mx.loading.RepositoryClassLoader] Loaded class from cache, org.jboss.net.protocol.file.FileURLLister@bfb235<CodeSource: (file:/opt/jboss-4.3.0.GA_CP02/lib/jboss-common.jar <no signer certificates>)>
2008-11-13 17:45:39,907 TRACE [org.jboss.mx.loading.RepositoryClassLoader] loadClass org.jboss.mx.loading.UnifiedClassLoader3@182eca8{ url=file:/opt/jboss-4.3.0.GA_CP02/server/production/deploy/deploy.last/ ,addedOrder=47} name=org.jboss.net.protocol.file.FileURLLister class=class org.jboss.net.protocol.file.FileURLLister cl=org.jboss.system.server.NoAnnotationURLClassLoader@16546ef
2008-11-13 17:45:39,908 TRACE [org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread] Sleeping...
2008-11-13 17:45:40,073 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] Begin run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@135779
2008-11-13 17:45:40,073 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] End run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@1f5e1ea
2008-11-13 17:45:41,007 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] process(): region: /_default_/
2008-11-13 17:45:41,007 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] processed 0 node events in region: /_default_/
2008-11-13 17:45:41,007 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] Recycle queue is empty
2008-11-13 17:45:44,107 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] process(): region: /_default_/
2008-11-13 17:45:44,107 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] processed 0 node events in region: /_default_/
2008-11-13 17:45:44,107 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] Recycle queue is empty
2008-11-13 17:45:44,917 TRACE [org.jboss.mx.loading.RepositoryClassLoader] loadClass org.jboss.mx.loading.UnifiedClassLoader3@182eca8{ url=file:/opt/jboss-4.3.0.GA_CP02/server/production/deploy/deploy.last/ ,addedOrder=47} name=org.jboss.net.protocol.file.FileURLLister, loadClassDepth=0
2008-11-13 17:45:44,917 TRACE [org.jboss.mx.loading.RepositoryClassLoader] Loaded class from cache, org.jboss.net.protocol.file.FileURLLister@bfb235<CodeSource: (file:/opt/jboss-4.3.0.GA_CP02/lib/jboss-common.jar <no signer certificates>)>
2008-11-13 17:45:44,917 TRACE [org.jboss.mx.loading.RepositoryClassLoader] loadClass org.jboss.mx.loading.UnifiedClassLoader3@182eca8{ url=file:/opt/jboss-4.3.0.GA_CP02/server/production/deploy/deploy.last/ ,addedOrder=47} name=org.jboss.net.protocol.file.FileURLLister class=class org.jboss.net.protocol.file.FileURLLister cl=org.jboss.system.server.NoAnnotationURLClassLoader@16546ef
2008-11-13 17:45:44,918 TRACE [org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread] Sleeping...
2008-11-13 17:45:46,017 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] process(): region: /_default_/
2008-11-13 17:45:46,017 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] processed 0 node events in region: /_default_/
2008-11-13 17:45:46,017 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] Recycle queue is empty
2008-11-13 17:45:46,870 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] Begin run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@27f221
2008-11-13 17:45:46,871 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] End run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@27f221
2008-11-13 17:45:47,454 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] Begin run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@b536e6
2008-11-13 17:45:47,454 TRACE [org.jboss.util.threadpool.RunnableTaskWrapper] End run, wrapper=org.jboss.util.threadpool.RunnableTaskWrapper@b536e6
2008-11-13 17:45:47,890 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] passed through ServerSocketRefresh.release()
2008-11-13 17:45:47,890 TRACE [org.jboss.remoting.serialization.impl.java.JavaSerializationManager] Creating ObjectOutputStream
2008-11-13 17:45:47,890 TRACE [org.jboss.remoting.serialization.impl.java.JavaSerializationManager] Creating ObjectInputStreamWithClassLoader
2008-11-13 17:45:47,890 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor41.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:755)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:375)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2281)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2750)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:780)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
at org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.<init>(ObjectInputStreamWithClassLoader.java:95)
at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:54)
at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:72)
at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:55)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:179)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:162)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(ClientSocketWrapper.java:66)
at org.jboss.remoting.transport.socket.ServerSocketWrapper.<init>(ServerSocketWrapper.java:46)
... 6 more
2008-11-13 17:45:48,963 TRACE [org.jboss.remoting.transport.socket.SocketServerInvoker] passed through ServerSocketRefresh.release()
2008-11-13 17:45:48,963 TRACE [org.jboss.remoting.serialization.impl.java.JavaSerializationManager] Creating ObjectOutputStream
2008-11-13 17:45:48,963 TRACE [org.jboss.remoting.serialization.impl.java.JavaSerializationManager] Creating ObjectInputStreamWithClassLoader
2008-11-13 17:45:48,963 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor41.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:755)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:375)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2281)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2750)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:780)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
at org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.<init>(ObjectInputStreamWithClassLoader.java:95)
at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:54)
at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:72)
at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:55)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:179)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:162)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(ClientSocketWrapper.java:66)
at org.jboss.remoting.transport.socket.ServerSocketWrapper.<init>(ServerSocketWrapper.java:46)
... 6 more

uname -a:

SunOS valkyrie.srv 5.10 Generic_137112-02 i86pc i386 i86pc

JBoss:

4.3.0.GA_CP02

No applications are deployed. Exceptions occur very frequently flooding my logs.

[boniek]

JBoss is running production profile.

[ron_sigal]

It looks like a client has connected to the Remoting server, which attempts to create an ObjectInputStream to read the forthcoming invocation. But the client shut down before writing the header bytes to the corresponding ObjectOutputStream. Look and see what is happening on the client side when these EOFExceptions occur.

By the way, there is no AS 4.3.0.GA_CP02. Do you mean EAP 4.3.0.GA_CP02? In the latter case, you might get more timely help from the customer portal. The support guys are really good, and in the unusual event that they can't solve your problem, it will come back to me anyway. :)

[boniek]

I think I tracked the problem. My load-balancer was probing port 3873 (one of many) to see if it is up or not to determine if machine is up and working. Probing was done by acquiring connection on this port and if succesful - closing it. Removing probe solved the problem. Thanks for translating exception for me:)

[ron_sigal]

Interesting. Thanks for the information.