[brian.stansberry]

If we're going to open sockets, we need to make the binding configurable so admins can lock down what is used. 127.0.0.1:0 is fine as a default though.

Haha, I started off as opposed to using stdio, but now feel oddly supportive of it. Not sure why.

In any case, even if we use sockets for the inter-process messaging traffic, these thread dumps are going to go to the PM via stdout/ManagedProcess$OutputStreamHandler. So we need to decide what to do with them. Convert them to log messages in the PM log, a la the above? Dump the directly to PM's stdout? (I'm checking if that will actually just result in log messages in the PM log file.) Maintain a file per managed process?

[jason.greene]

The big issue is of course security. We likely will have to add some kind of auth protocol to this. Perhaps some kind of token is written to a tmp file that can be read by the other child processes.

[dmlloyd]

Brian Stansberry wrote:

If we're going to open sockets, we need to make the binding configurable so admins can lock down what is used. 127.0.0.1:0 is fine as a default though.

Haha, I started off as opposed to using stdio, but now feel oddly supportive of it. Not sure why.

In any case, even if we use sockets for the inter-process messaging traffic, these thread dumps are going to go to the PM via stdout/ManagedProcess$OutputStreamHandler. So we need to decide what to do with them. Convert them to log messages in the PM log, a la the above? Dump the directly to PM's stdout? (I'm checking if that will actually just result in log messages in the PM log file.) Maintain a file per managed process?

I think just continuing with the idea of using a log category per server is fine.  Then the PM's logging.properties can configure it however the user wants.  I think people will be looking for this info on the console though.

Also, it's worth verifying that hitting ctrl-/ on the PM will or won't cascade to all child processes, causing a huge mass of thread dumps to occur.

[dmlloyd]

Jason Greene wrote:

The big issue is of course security. We likely will have to add some kind of auth protocol to this. Perhaps some kind of token is written to a tmp file that can be read by the other child processes.

Yes, a security token should be used, but it should just be passed into the child process on stdin or as a CLI argument.

[jason.greene]

CLI is a no go. Someone can do ps -ef and get the token. STDIN sounds best

[kabirkhan]

Brian Stansberry wrote:

David Lloyd wrote:

Might be easier to use specific exit values: 0 means shut down on purpose, anything else means crash?

That certainly seems simpler.

Either way the SM should be notified if a Server exits, regardless of the cause: if the server is not configured to be shut down but it returns 0, the SM needs to kick it back alive again.  Also the same respawn logic has to apply; otherwise a poorly-placed System.exit(0) can bog down the system with fork/execs.

+1 on notifying the SM and the restart rules. Whether the SM should automatically restart the server following getting a signal that one shut down cleanly, I'm not so sure. The signal should imply some human action was taken to cause the shutdown, and if that's the case I think requiring human action to restart makes sense. The SM should squawk in the logs though if the shutdown happens via something other than the intended managment API.

I have handled exit code = 0 to not respawn in http://github.com/kabir/jboss-as/commit/8862da78064ca95b5ff8d2e866e38093e2a12294

I'll look at notifying the SM next week when I get to implementing the SM<->Server protocol.

I have made a start on socket-base PM-process communication but should probably hold off until some consensus on if we want that is reached?

[jason.greene]

I'll look at notifying the SM next week when I get to implementing the SM<->Server protocol.

I have made a start on socket-base PM-process communication but should probably hold off until some consensus on if we want that is reached?

I think we all agree that the line input idea is regrettably dead. Brian's findings sort of put the nail in the coffin, which is a shame because I really like the idea. I spoke with Anil on IRC and basically he thinks that the proposed design an ephermal port on a local interface with a security token would pass the various security certifications. So I think we finally have a way forward.

Anyone disagree (silence is acceptance!)

[jason.greene]

I think we all agree that the line input idea is regrettably dead. Brian's findings sort of put the nail in the coffin, which is a shame because I really like the idea.

BTW I would just like to note that it is very unsurprising that the key data defeating this FANTASTIC idea came from Brian

On another note, I think we could have easily handled this stack trace scenario. It's manually triggered so the overhead to the protocol is minimal. It just pointed out that we really don't know what can go to STDOUT and when.

[brian.stansberry]

Jason Greene wrote:

BTW I would just like to note that it is very unsurprising that the key data defeating this FANTASTIC idea came from Brian

I'm in mourning. I was starting to like the idea. But you're right; we don't really know what all may come through stdout. So sockets it is.

[brian.stansberry]

David Lloyd wrote:

Also, it's worth verifying that hitting ctrl-/ on the PM will or won't cascade to all child processes, causing a huge mass of thread dumps to occur.

A kill -3 to the PM pid results in a thread dump for the PM process to the console and nothing else.

[kabirkhan]

I've got the servers starting up with sockets instead of stdio now, but I am seeing some errors in the logs. They are exactly the same in my branch using sockets as they are in master which uses stdio, so maybe they are expected at this stage?

When running standalone.sh, I get this error in boot.log:

11:59:41,868 ERROR [server] Failed to activate deployment my-app.ear: java.lang.RuntimeException: Deployment root does not exist."/Users/kabir/sourcecontrol/jboss-as7/git/jboss-as/build/target/server/deployments/my-app.ear"
at org.jboss.as.model.ServerGroupDeploymentElement.activate(ServerGroupDeploymentElement.java:218)
at org.jboss.as.model.Standalone.activate(Standalone.java:464)
at org.jboss.as.server.AbstractServer.start(AbstractServer.java:100)
at org.jboss.as.server.StandaloneServer.start(StandaloneServer.java:68)
at org.jboss.as.server.Main.boot(Main.java:93)
at org.jboss.as.server.Main.main(Main.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_20]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [:1.6.0_20]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_20]
at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_20]
at org.jboss.modules.Module.run(Module.java:373) [jboss-modules.jar:1.0.0.Beta3]
at org.jboss.modules.Main.main(Main.java:171) [jboss-modules.jar:1.0.0.Beta3]
 
11:59:41,871 INFO  [model] Activating server group deployment: my-war.ear:22cfd207b9b90e0014a4
11:59:41,872 ERROR [server] Failed to activate deployment my-war.ear: java.lang.RuntimeException: Deployment root does not exist."/Users/kabir/sourcecontrol/jboss-as7/git/jboss-as/build/target/server/deployments/my-war.ear"
at org.jboss.as.model.ServerGroupDeploymentElement.activate(ServerGroupDeploymentElement.java:218)
at org.jboss.as.model.Standalone.activate(Standalone.java:464)
at org.jboss.as.server.AbstractServer.start(AbstractServer.java:100)
at org.jboss.as.server.StandaloneServer.start(StandaloneServer.java:68)
at org.jboss.as.server.Main.boot(Main.java:93)
at org.jboss.as.server.Main.main(Main.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_20]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [:1.6.0_20]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_20]
at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_20]
at org.jboss.modules.Module.run(Module.java:373) [jboss-modules.jar:1.0.0.Beta3]
at org.jboss.modules.Main.main(Main.java:171) [jboss-modules.jar:1.0.0.Beta3]
 
11:59:41,874 INFO  [model] Activating server group deployment: my-ejb.jar:7de777924ef61549b1af
11:59:41,874 ERROR [server] Failed to activate deployment my-ejb.jar: java.lang.RuntimeException: Deployment root does not exist."/Users/kabir/sourcecontrol/jboss-as7/git/jboss-as/build/target/server/deployments/my-ejb.jar"
at org.jboss.as.model.ServerGroupDeploymentElement.activate(ServerGroupDeploymentElement.java:218)
at org.jboss.as.model.Standalone.activate(Standalone.java:464)
at org.jboss.as.server.AbstractServer.start(AbstractServer.java:100)
at org.jboss.as.server.StandaloneServer.start(StandaloneServer.java:68)
at org.jboss.as.server.Main.boot(Main.java:93)
at org.jboss.as.server.Main.main(Main.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_20]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [:1.6.0_20]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_20]
at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_20]
at org.jboss.modules.Module.run(Module.java:373) [jboss-modules.jar:1.0.0.Beta3]
at org.jboss.modules.Main.main(Main.java:171) [jboss-modules.jar:1.0.0.Beta3]
 

For run.sh I get this error in server-one/boot.log (and similar for server-two):

11:59:57,844 INFO  [server] Server Available to start
11:59:57,959 ERROR [server] Caught exception handling message from ServerManager: java.lang.RuntimeException: Failed to read server command
at org.jboss.as.server.MessageHandler.handleMessage(MessageHandler.java:67)
at org.jboss.as.server.ServerCommunicationHandler$Controller.run(ServerCommunicationHandler.java:202)
at java.lang.Thread.run(Thread.java:637) [:1.6.0_20]
Caused by: java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to java.lang.String
at org.jboss.marshalling.ModularClassTable.readClass(ModularClassTable.java:70)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:816)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1144)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNestedObject(RiverUnmarshaller.java:165)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:671)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1615)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1531)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadMapObject(RiverUnmarshaller.java:195)
at org.jboss.marshalling.river.RiverUnmarshaller.readMapData(RiverUnmarshaller.java:745)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:671)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1615)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1531)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1615)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1531)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1615)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1531)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1615)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1531)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadCollectionObject(RiverUnmarshaller.java:180)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObjectArray(RiverUnmarshaller.java:1440)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadArray(RiverUnmarshaller.java:1479)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:326)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1615)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1531)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1495)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1175)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:72)
at org.jboss.as.server.MessageHandler.readServerCommand(MessageHandler.java:93)
at org.jboss.as.server.MessageHandler.handleMessage(MessageHandler.java:65)
... 2 more
Caused by: an exception which occurred:
in object of type java.util.TreeMap comparator
in field subsystems
in object of type org.jboss.as.model.ProfileElement
in map value at index [249] of size [2]
in field profiles
in object of type org.jboss.as.model.Domain
in field this$0
in object of type org.jboss.as.model.Domain$7
in field includedProfileResolver
in object of type org.jboss.as.model.ProfileElement
in field profile
in object of type org.jboss.as.model.Standalone
in element at index [0] of size [1]
in field args
in object of type org.jboss.as.server.manager.ServerCommand

This seems to be in response to the START server message command which passes through the Standalone element.

[aloubyansky]

Deployment errors in standalone's log are expected. For the domain, I haven't seen this one, but I haven't pulled from master since yesterday at least.

[dmlloyd]

This issue was caused by https://jira.jboss.org/browse/JBMAR-112 which Jason fixed last week.