Nessus scan makes JBoss 5.1.0.GA stop processing web requests
d-rock Feb 16, 2010 3:47 PMWe recently started getting an issue with our production JBoss server where a Nessus security scan stops the JBoss server from responding to web service requests. We've been doing Nessus scans for many months now, so I'm assuming that there's a new plugin that's causing the issue. When it happens we see this in the logs:
{code}
2010-02-16 02:03:07,511 ERROR [org.apache.coyote.ajp.AjpMessage] (ajp-0.0.0.0-8009-1) Invalid message received with signature 18245
2010-02-16 02:03:28,589 ERROR [org.jboss.invocation.pooled.server.ServerThread] (PooledInvokerThread-172.20.2.5-0) Failed to initialize
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2281)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2750)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:780)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
at org.jboss.invocation.pooled.interfaces.OptimizedObjectInputStream.<init>(OptimizedObjectInputStream.java:147)
at org.jboss.invocation.pooled.server.ServerThread.dorun(ServerThread.java:265)
at org.jboss.invocation.pooled.server.ServerThread.run(ServerThread.java:156)
2010-02-16 02:03:28,589 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#34[172.20.2.5:33997]) WorkerThread#34[172.20.2.5:33997] exception occurred during first invocation
java.io.EOFException
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:693)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:524)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
2010-02-16 02:03:28,589 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#51[172.20.2.5:35756]) WorkerThread#51[172.20.2.5:35756] exception occurred during first invocation
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:909)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:491)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2281)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2750)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:780)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
at org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.<init>(ObjectInputStreamWithClassLoader.java:100)
at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:54)
at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:75)
at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:58)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:179)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:162)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(ClientSocketWrapper.java:66)
at org.jboss.remoting.transport.socket.ServerSocketWrapper.<init>(ServerSocketWrapper.java:46)
... 7 more
2010-02-16 02:05:27,403 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#51[172.20.2.5:34921]) WorkerThread#51[172.20.2.5:34921] exception occurred during first invocation
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:909)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:491)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
Caused by: java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1784)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:691)
at org.jboss.remoting.marshal.serializable.SerializableMarshaller.getMarshallingStream(SerializableMarshaller.java:90)
at org.jboss.remoting.marshal.serializable.SerializableMarshaller.getMarshallingStream(SerializableMarshaller.java:72)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createOutputStream(ClientSocketWrapper.java:198)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:161)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(ClientSocketWrapper.java:66)
at org.jboss.remoting.transport.socket.ServerSocketWrapper.<init>(ServerSocketWrapper.java:46)
... 7 more
2010-02-16 02:05:27,418 ERROR [org.jboss.invocation.pooled.server.ServerThread] (PooledInvokerThread-172.20.2.5-1) Failed to initialize
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1784)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:691)
at org.jboss.invocation.pooled.server.ServerThread.dorun(ServerThread.java:263)
at org.jboss.invocation.pooled.server.ServerThread.run(ServerThread.java:156)
2010-02-16 02:05:27,418 ERROR [org.jboss.remoting.transport.socket.ServerThread] (WorkerThread#44[172.20.2.5:36827]) WorkerThread#44[172.20.2.5:36827] exception occurred during first invocation
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:1038)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:673)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:524)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
{code}
After that, the JVM process is still running, and we can cleanly shut down JBoss, but it seems that the web stuff doesn't work anymore. I'm at a loss, so I've temporarily disabled the Nessus scan and that seems to have stopped the issue. Not scanning the server isn't a long-term solution, however, so I'd like to figure out what's going on here. I have a dev JBoss server that I can test against if anyone has any ideas...
Thanks,
Derek