Admin Console Architecture (medium-long)

[ccrouch]

This topic is going to discuss one facet of the integration between the JBoss Application Server and the Admin Console. Any constructive feedback or suggestions on the issues I raise below would be most welcome.

In summary, the first releases of the Admin Console will focus on providing a web-based client that will allow users to review and update various common services in a single instance of JBoss AS. Example features are the ability to create/read/update/delete datasources, manage JMS settings and create/read/update and delete destinations.

The architecture for the application will be as follows:

1) Web-tier. A typical web-app, nothing scary or overly elaborate. Another post will describe the particular framework that will be used.

2) Integration between web-tier and JBoss AS. Planned to be a straightforward interface based service layer, with POJO implementations allowing full unit/integration testing of the back-end without any dependency on the web UI.

3) JBoss AS. [The topic of this post]. I'm referring here to the facilities currently available or to be developed within the JBoss AS that will enable the Admin Console to do its job. In more detail...


One of the more obvious requirements for the Admin Console is that changes to settings made through it must persist across restarts of the JBoss AS instance. AFAIK we really have two possible options when dealing with persistence of configuration changes:

1) Assuming all interesting settings are available via JMX we could just make "everything" an XMBean (http://jboss.org/wiki/Wiki.jsp?page=XMBean).
In theory this would allow you to make changes directly via JMX and those changes would be persisted transparently for us by the Persistence Service (http://www.jboss.org/wiki/Wiki.jsp?page=XMBeanAttributePersistenceService). Unfortunately it is not possible to make every MBean an XMBean, for example Tomcat creates MBeans programmatically which we can't get at. Also Dimitris points out here (http://www.jboss.org/wiki/Wiki.jsp?page=ExtentionsToJBoss32xDeploymentLayer) that he ran into problems making MBeans that generate or consume notifications into XMBeans. Finally using any form of MBean persistence will break the link to the underlying configuration, e.g blah-ds.xml. Once a change has been made to an XMBean the version kept in the persistent store always wins over the configuration .xml file.

2) Use the DeploymentService (http://www.jboss.org/wiki/Wiki.jsp?page=DeploymentService) to update the actual configurations in the ./deploy directory. This service enables clients to generate fresh configuration packages using Velocity templates. Attributes can be passed in and will be inserted into the appropriate place in a .xml configuration file. The configuration package will then be copied into ./deploy. The tricky part is where to get the current configuration settings from in order to display them in the Admin Console?. Again there seems to be two options:

i) Via JMX? This is not always straightforard. For example, how do I get a list of installed Datasources, making sure to avoid any other JCA Managed Connection Factories which have nothing to do with a database?
ii) Via parsing the configuration files? The flexibility offered when configuring services would make the task of parsing the xml tough, right?

Is the best solution for fixing 2) just to give the Admin Console its own database? So when it first starts it doesn't know of any deployed services. But after you have created, say, a datasource through the Admin Console, it will remember the settings, so that when you come back to the Admin Console to view the datasource it will display the "correct" values. I say"correct" here to highlight the fact that this solution will not be aware of changes to the configuration made directly in the .xml. It will only know of changes made through the console itself. Is this an acceptable option? If not, do you have any ideas for a better alternative?

Thanks for your time
Charles

[dimitris]

Elaborating on the last alternative:

Yes, and to relieve the WebApp from the persistence burden the DeploymentService could be enhanced to "remember" the properties of a module it has produced. So then it'd be very easy to "edit" any deployment created through the console.

Trying also to see how to best package things, we could seperate the deployments that can be modified by the console, by making it copy stuff to a subdirectory of deploy e.g:

./deploy
./deploy/console

or if we want to have a full seperation of the console related stuff, we could have:

./console/
- ./templates
- ./deploy
- ./undeploy
- ./properties

and simply point URLDeploymentScanner to both:

./deploy
./console/deploy

A problem I see in general with the console is the dependency to tomcat, since the console is a webapp itself. Will tomcat be under control of the console? If tomcat is redeployed what happens to the console webapp?

[ccrouch]

Elaborating on the last alternative:

Yes, and to relieve the WebApp from the persistence burden the DeploymentService could be enhanced to "remember" the properties of a module it has produced. So then it'd be very easy to "edit" any deployment created through the console.

So we would just add a method to the DeploymentService, say getCurrentProperties, which takes a module name and returns a Map of property names and values? The DeploymentService would have stored this Map somewhere when the module was first created?

Trying also to see how to best package things, we could seperate the deployments that can be modified by the console, by making it copy stuff to a subdirectory of deploy e.g:

./deploy
./deploy/console

or if we want to have a full seperation of the console related stuff, we could have:

./console/
- ./templates
- ./deploy
- ./undeploy
- ./properties

and simply point URLDeploymentScanner to both:

./deploy
./console/deploy

I like the idea of keeping all the deployments rooted under ./deploy, i.e. the first suggestion. That way people don't have to remember another directory structure in which to look for deployments. The other DeploymentService working directories could then just go under ./console as you described.

A problem I see in general with the console is the dependency to tomcat, since the console is a webapp itself. Will tomcat be under control of the console? If tomcat is redeployed what happens to the console webapp?

Great question! Tomcat is meant to be one of the services which the Admin Console manages. Clearly the approach of using the DeploymentService to update the instance of Tomcat underlying the Admin Console is not feasible. [The Admin Console would ask the DeploymentService to create a new Tomcat configuration and copy it over the existing one, this would have the effect of undeploying/deploying Tomcat, thus killing the Admin Console]. At best maybe we could aim for *any* change to Tomcat requiring a server restart? But then we need a mechanism to support some sort of "delayed" deployment, i.e. create a new Tomcat configuration but only deploy it when the server restarts, when we can be sure no-one is connected to the Admin Console.

Thanks
Charles

[rauschuber]

A problem I see in general with the console is the dependency to tomcat, since the console is a webapp itself. Will tomcat be under control of the console? If tomcat is redeployed what happens to the console webapp?

Great question! Tomcat is meant to be one of the services which the Admin Console manages. Clearly the approach of using the DeploymentService to update the instance of Tomcat underlying the Admin Console is not feasible. [The Admin Console would ask the DeploymentService to create a new Tomcat configuration and copy it over the existing one, this would have the effect of undeploying/deploying Tomcat, thus killing the Admin Console]. At best maybe we could aim for *any* change to Tomcat requiring a server restart? But then we need a mechanism to support some sort of "delayed" deployment, i.e. create a new Tomcat configuration but only deploy it when the server restarts, when we can be sure no-one is connected to the Admin Console.

Will there also be a text-based console? If so, it would not be vulnerable to Tomcat changes. Maybe making immediate changes to Tomcat with the understanding that you lose your web console would be "good enough" for the first pass. The text-based console and the delayed deployment could be part of the second pass.

[ccrouch]

"rauschuber" wrote:

Will there also be a text-based console? If so, it would not be vulnerable to Tomcat changes. Maybe making immediate changes to Tomcat with the understanding that you lose your web console would be "good enough" for the first pass. The text-based console and the delayed deployment could be part of the second pass.

The current focus is on the web-based Admin Console. However the "services" which the web-app uses will just be POJO's sitting on top of JBoss so these could be deployed in a .sar along with a text-based console, thus removing the dependency on Tomcat. I am tending to agree with you, that in the first pass enabling the updating of Tomcat directly via the DeploymentService may be the best option.

Thanks

[mazz]

wrt a text-based console. Rich F. has a requirement on the JBoss Network that there be a command-line interface to the Network admin stuff. Things like "deploy new app", "undeploy app", "start instance", "stop instance", etc. should be also be controllable via command-line/scripts/text console.

Today, I have a command line interface to Command Framework that I'm using to perform remote commands on an remote instance (specifically, today, I have the ability to send a remote command via JBoss/Remoting to start/stop JBoss instances). This stuff should be fairly extensible so we can add commands and their client-side API pretty easily. I would love to be able to see if I can hook my stuff to your POJOs for deployment.

I suspect we can use this capability to perform your text-console things since your stuff are just .sar deployed POJOs (my stuff is also just a .sar deployment on the server side with a command-client.jar client-side API).

I'm getting alot of warm and fuzzies that the JBoss Network will be able to piggy-back and reuse alot of the stuff you guys are doing. Let's remain in close contact so we can track each other.

[dimitris]

So is the capability of remotely starting/stoping a jboss instance implemented or investigated?

How it will be done?

Replying to Chris' post, we would persist the properties of any module we create (and overwrite them every time we edit/regenerate that module). So we will always have the last good property image, as long nobody screws up with the console-generated modules.

The delayed deployment is also an option. We can store delayed deployments in a different location and upon startup, as long as the deployment service is started from ./conf/jboss-service.xml, it could overwrite those delayed deployments *before* the DeploymentScanner starts...

[ccrouch]

I suspect we can use this capability to perform your text-console things since your stuff are just .sar deployed POJOs (my stuff is also just a .sar deployment on the server side with a command-client.jar client-side API).

For the first phase everything is likely to be bundled into a .war. However, subsequently, I don't see a reason we couldn't split things up to support other clients calling the APIs to update the JBoss configuration.

[starksm64]

We already have the simple twiddle command line tool which is a simple app on top of the jmx adaptor. As long as we have an mbean facade for the operations it would be a relatively simple matter to support access via the command line.

The biggest disconnect will be for deployment operations where content needs to be uploaded to the server.

[mazz]

So is the capability of remotely starting/stoping a jboss instance implemented or investigated?

I have the code on my box (and internal JBossNetwork CVS) - it is implemented on top of JBoss/Remoting. In prototype stage - obviously not released yet (in alpha or otherwise).

Basically, you send the command line to the remote machine and the remote JVM executes the process. Its simple, completely unsecure :-) but flexible. I'm hoping JBoss/Remoting will have security built in (there's already an open source project that looks promising - can wrap ssh over our transport protocol).

That's start - as for stopping, you send a remote stop command over the wire and on the remote end, it uses the Shutdown class to do the killing. Basically its the same as Shutdown except to get to the remote node from the command line client, it doesn't use the JMX adapter - rather it uses the JBoss/Remoting infrastructure (which maintains consistency with all of JBoss Network). Plus, if this code is deployed within the same JBoss instance that is to be stopped, that instance does not need to have remote JNDI/remote JMX enabled for this to work.

I'm picturing some remote command client being able to use the Command Framework to ship over a "configuration" and have the server-side command invoker call your API to actually deploy/undeploy/reconfigure based on the new configuration information. This is how we can implement a remote Admin Console (i.e. what the JBossNEtwork is calling the Domain Admin Console - used for cluster management and multi-node configuration).

Email me for details on how to get the code if you wish to look at it.

[dimitris]

John, in a way is this a kind-of 'java rsh' to let you execute processes on a remote host?

Is there any document/wiki explaining this? I think we should use the forums more to discuss the design; a lot of good ideas may come up.

[mazz]

The Command Framework is more generic than "java rsh"... Its more of "execute something on a remote host" where a "something" can be anything you want (via Java code) - so when I say Command, think "Command Pattern", not "command-line command", does that make sense? :).

However, yes you are right in that I wrote a couple of Commands that do something like a remote java rsh in which it allows you to execute processes on a remote host and stop JBoss instances on a remote host.

Of course, that said, we've come across this:

http://wrapper.tanukisoftware.org/doc/english/introduction.html

We may use that for our starting/stopping capabilities. If anyone has any prior knowledge or has used that product, let us know.

There is no document/wiki/forum on this yet.

[ccrouch]

So to summarize what I understand from the discussion so far:

1) The Admin Console is going to update JBoss AS using the DeploymentService
2) The Admin Console is going to start out not knowing anything about the JBoss instance it is running on, but learns by having services created through it.

Well I have a problem, and solution, with 2).

The concept of the Admin Console "learning" works great for services such as Datasources, i.e. it is reasonable for users to start with an empty list and add new Datasources to it. However the concept breaks down when applied to monolithic services such as JMS: It is not reasonable for users to create the JMS service, e.g. specifying the cache manager and security manager settings, from scratch. The Admin Console must be able to find the existing settings and display them. So how can the Admin Console find these settings?

Answer: JMX. For monolithic services, e.g. JMS, Tomcat, JTA Transaction Service, we can use hard-coded ObjectName's to locate MBeans and from those, all the necessary information should be available. Specifying the ObjectNames in a configuration file would allow us to update them if for some reason they changed between versions of JBossAS.

For example, lookup the MBean "jboss.mq:service=DestinationManager" and from there you can get references to the JMS Cache Manager, Persistence Manager and State Manager (but oddly not the Security Manager?). These MBeans provide all the information you could possible want on the base JMS settings. In addition doing a lookup of "jboss.mq:service=InvocationLayer,type=HTTP" gives you an MBean with with all the JMS HTTPIL settings.

This approach does have an important implication: You should not change settings via the jmx-console if you are also using the Admin Console. The problem is that changes made through the jmx-console, which are mostly transient, could be made permanent. For example,
1) Start JBoss up
2) Go to the JMX console and update the HighMemoryMark on jboss.mq:service=MessageCache
3) Open up the Admin Console and browse to the JMS settings. You will see the change made in 2)
4) In the Admin Console change the MaxMemoryMark on the CacheManager.
5) Commit the change in the Admin Console. You will persist the changes made in both steps 2) and 4). This is unexpected behaviour and is therefore bad.

Thanks