DynamicLoginConfig equivalent for stand-alone EAR deployments

[zbedell]

Greetings all,

I'm exploring what will be required to migrate my organization's many JBoss 4.2.3 application servers up to JBoss 7.x (currently looking at 7.1.1.Final).  One of the main reasons we're looking to move is that we've found that using isolated classloaders for EAR deployments only "mostly" works in 4.2.3, and the new classloading system in JBoss 7 looks like it does a much better job of keeping deployements' classes truly separated.  We generally run anywhere from a few up to a dozen separate EAR files deployed to a single JBoss instance, so being able to have EAR files that are fully self-contained is attractive to us.

To date, we've managed to get our deployments in JBoss 4.2.3 to be fully self-contained except for library JAR files which end up in server/lib.  Specifically, all datasource descriptors, JAAS configuration (including the obfuscated database passwords), system property settings, etc. are all loaded via MBeans spawned from jboss-service.xml's within the various EAR files.  The up shot of all of this is that when we deploy multiple EAR files to a server, we need only deploy the EARs themselves and the superset of all JAR files required by the various apps (with any version conflicts resolved via cooperation between the teams providing the apps).  We don't need to edit any of the global JBoss configuration files (such as conf/jboss-service.xml or conf/login-config.xml) as the EAR-embedded MBeans allow us to influence the server-wide configuration in a far more compartmentalized fashion.  This has eliminated a huge amount of contention where modifying especially login-config.xml in the past required careful coordination between groups sharing an app server and caused complications with maintaining separate single-app versions of the file for individual developers' local testing versus the combined version for use on live servers.

So now looking at JBoss 7.1, I have to admit I'm a little aghast what what I see with respect to keeping per-app configuration out of the server-wide config files.  JBoss 7.0 doesn't appear to have supported *-ds.xml files at all for datasources, so I'm thankful that 7.1.1 appears to, admittedly as second-class citizens versus placing DB configuration directly in configuration/standalone.xml.  The absence of org.jboss.varia.property.SystemPropertiesService was somewhat disconcerting, but fortunately I was able to modify that class from JBoss 4.2.3 to provide the minimum necessary functionality (loading properties from a *.properties file), and deploy that as a global JBoss module.

That leaves one rather major stumbling block in terms of org.jboss.security.auth.login.DynamicLoginConfig.  Presently, we deploy that MBean in each EAR's jboss-service.xml and reference a private login-config.xml also deployed within the EAR file.  Something like:

  <mbean code="org.jboss.security.auth.login.DynamicLoginConfig"

    name="sample-app:service=DynamicLoginConfig,domain=YOUR_EAR_NAME">

    <attribute name="AuthConfig">META-INF/login-config.xml</attribute>

    <depends optional-attribute-name="LoginConfigService">

      jboss.security:service=XMLLoginConfig

    </depends>

    <depends optional-attribute-name="SecurityManagerService">

      jboss.security:service=JaasSecurityManager

    </depends>

  </mbean>

Under JBoss 7.1, that MBean is no longer available, and Googling around and searching the JBoss Community Forums has led to a number of various requests for an equivalent, all of which are pointed towards modifying standalone.xml.  That solution is a huge step backwards in terms of encapsulating application-specific configuration as it now requires deploying an EAR file and also monkeying around with global application server configuration files for each EAR that's deployed.  Testing a single application locally requires a differently modified config file that running all applications on a live server, so inevitable configuration bugs are introduced in maintaining separate versions of a very complicated XML configuration file.

So to finally get to a real question…

Is there any mechanism in JBoss 7.1.1 which can load JAAS configurations from a location outside configuration/standalone.xml?  We need a way for each application to specify its own configuration without interfering with other applications sharing the server.  Something embedded in the EAR would be ideal, though if there were a way to deploy separate XML configuration files to standalone/configuration or similar and have them loaded, that would work.  Having to coordinate edits to a single shared configuration file across applications is inherently troublesome and something we've worked very hard to avoid.

We use per-app JAAS configuration both to setup actual login modules (we have a custom LoginModule which operates against our rather complicated in-house authenticated webservices) and also to provide database credentials in an obfuscated form via org.jboss.resource.security.SecureIdentityLoginModule.  It appears JBoss 7.1's solution for the latter option is to use the vault facility, though that as well looks problematic.  From what I've been able to gather, the vault facility uses a single keystore file to hold encrypted passwords for the entire app server.  Contrasted to the JBoss 4.2.3 mechanism where the password encryption process yields only an encrypted string which can be pasted into config files or system properties without being dependent on a file that's global to the app server.  I suspect that there may be some way to plug in to PicketBox with some other magic ${…} syntax to provide self-contained encrypted passwords, but I've not yet delved in to try and find that.

Philosophically speaking, it seems to me that anything which requires editing global server files to deploy an application is inherently counter to the idea of JEE application encapsulation.  The idea of dropping a single self-contained EAR file into a directory and having it able to configure itself and operate without additional meddling is a very attractive approach.

If anyone has any suggestions on how to resolve this, I'd very much appreciate the help.

Best regards,

Zac Bedell

[wdfink]

You might not care about the standalone.xml file.

I avoid editing it by hand.

A smooth way is to use the CLI interface to set/remove system-properties, datasource definition, Security configuration and deployments.

There is no need to edit the file and the benefit is that in case of installing a new server there is no work by hand and if you use a different version the CLI commands are checked whether it still fit.

So it is simmilar to copy a config file or start a CLI script.

And I recommend to use the CLI (or a custom programm using the management API) to handle this.

BTW, use of *-ds.xml is not recommended to use in production it's a development feature. But I switch to CLI script also

[zbedell]

I've looked into the command line, but that kind of twiddling on the server after it's up is kind of anthesis to what we're trying to accomplish. 

We run a deployment server (WebDAV) where our build server pushes a completely configured JBoss app.  The JBoss files themselves as well as the EARs, all configuration files, etc. are all pushed to that server as part of the build.  For appservers that have multiple application EARs, the build server runs a harness which erases the deployment area for the server and pushes a base JBoss configuration, then runs each applications' build.  As we have things configured with JBoss 4.2.3 right now, that means each application build pushes an EAR (which contains *-ds.xml, login-config.xml, etc.) to server/deploy, its JAR files to server/lib (not ideal, but...), and a single properties file to server/conf/myapp.properties which is loaded by SystemPropertiesService within the EAR/SAR.  The properties file contains things like database connect strings & encrypted passwords which are referenced with ${...} notation inside the *-ds.xml & login-config.xml so that only the properties file should need to change between environments (dev/qa/prod/etc).  The only coordination required between the separate apps is ensuring the JARs pushed to server/lib don't conflict; and we end up using Apache Ivy and coordinating changes to its configuration file for each project to manage which versions of what are pushed.

On the actual appservers, we have a script (small Java app actually that we'll get around to open sourcing one of these days...) that needs to know only the service name, environment (dev/prod/etc), and the server's IP address.  It connects to the deployment server, downloads the entire application + JBoss, runs some sanity checks, hardening scripts, etc., then spawns JBoss and exists.  While it's technically possible to have JBossController (very original name for the script thingy)  bring JBoss up in admin only mode, drive the necessary config changes via CLI, then deploy the apps & get out of the way, that's quite a bit more complexity than we'd like to have at deployment time.  I'm assuming it's probably possible to hook into the CLI via direct Java calls somehow (so as to avoid string parsing / `expect` type programming), but it's still likely to be brittle. 

The other draw back to that approach is that our developers don't use JBossController for local development, so it's important that the deployments work as-is without requring changes made by the deployment script.  If we have to start dual-pathing the configuration for DB, etc., we inevitably end up with things that work locally & break on the live servers since they were never tested, changes not made in both places, etc.

Am I wrong in my thinking that the "JEE way" would tend to prefer a droppable EAR with all dependencies encapsulated rather than EAR+appserver hacking to get the app running?

What's the reasoning behind *-ds.xml not being recommended for production?  I understand that they're not manageable entites when datasources are deployed that way, but the way we use JBoss, we'd basically never want to connect to the management console anyways.  Any time we need to connect to the server and manually monkey with things, something has gone terribly wrong in our process.  Ideally the servers should come up, and run completely hands-off until the next time we restart them to deploy a new version.  Anything less is indicative of a problem as far as we're concerned.

Assuming the approach we're looking for isn't currently supported, I'm by no means averse to cracking open the source & trying to add it.  If that's the case, any pointers for where in the source base to start would be appreciated.

[wdfink]

With AS7 it is not longer possible to do configurations in that way.

From my perspective it is a improvement as there is no longer a mixture of JBoss and custom configuration.

You have to configure only one, or two files in domain mode, to set all needed properties.

Also use of the CLI will make independent from JBoss versions as the cli script will work with the next release if there are changes you want to have. BTW it's also possible to drop the configuration files from an older version into a new.

So in your case the login-config.xml does not longer exists as an separate file you have to configure the standalone/domain.xml.

I know that this is different to former versions and produce a bit effort, but it looks like a way out of the xml configuration hell and the necessity to check lots of configurations in case of installing a new JBoss version.

The "JEE way" don't show how the container should be configured, the idea behind is that the application point to references (e.g. DS name) and the container provide the resource. In that case you are able to change the type of DS without changing the application.

The reason that *-ds.xml is not supported is that you can not use it in domain mode and it is not manageable by the mngmt API.

If you want to have a 'drop and start' configuration you might configure the standalone (or domain/host) xml files and drop it to the server.

So after a while with CLI I use the maven plugin for JBoss or CLI/shell scripts for configuration and I love it. The script contain all necessary details as list of variables and the cli command will push it to the server.

So I can setup my dev or test environment in seconds.

[zbedell]

Respecfully, the new way is very much not an improvement for the way that we operate.  Compare the workflows for a multi-application server (assuming isolated classloading works properly):

JBoss 4.x (assuming functional isolated EAR classloaders which admittedly doesn't work):

1. Build server deploys a "bare metal" JBoss configuration to deployment server.
2. Each application builds a single EAR file which embeds all application code, library JARs, datasource configuration, JAAS security settings, JMS queues, etc.   EAR contains *-ds.xml & login-config.xml which use ${...} references to all environment-specific data (DB host, passwords, etc.), thus only a single copy of the *-ds.xml & login-config.xml is required for all environments.
3. EAR is deployed to deployement server at server/deploy/myapp.ear.
4. Properties file (containing dev/qa/prod settings for DB hosts, password, etc.) is deployed to server/conf/myapp.properties.  Database passwords are self-contained encrypted strings created by org.jboss.resource.security.SecureIdentityLoginModule.  This file contains all configuration as plain-text (asside from the encrypted passwords which are at leat printable ASCII) and can be easily versioned making change analysis trivial.
5. Runtime script (JBossController) downloads the entire JBoss instance from the deployment server, spawns the JBoss process, and exits.  JBoss runs without futher intervention.

In this scenario, no files are in contention between applications.  The EAR & properties files deployed by each app are named specific to the app.  No JBoss-global files need to be edited, so separate applications can co-exist on the same server without requiring coordination between the developers on each project.  Deploying only a subset of the applications (IE just one for development) simply means omitting the EAR & properties files for the other applications.  Note that system properties defined in myapp.properties are by convention prefixed by the application's name so as to prevent conflict in the global system properties scope.

Additionally, all elements of the configuration are plain-text, checked into revision control, and are easily versioned.  Everything relating to the JBoss runtime configuration is driven by automated scripts, all backed by revision control.  Nothing "lives" on the server in terms of configuration.  Any manual changes which might be made to the server are obliterated at next restart (which strongly discourages ad-hoc changes as they tend to be short lived and break on restart).

JBoss 7.x:

1. Build server deploys a base JBoss configuration.
2. Each application builds an EAR which contains code & JARs, but no DB, JAAS, etc. configuration.
3. A human must drive the vault.sh script manually to insert each applications' DB passwords into a keystore.  This keystore must be manually maintained, and separate versions must be maintained for dev/qa/prod and deployed.  A separate version of this file must likely be maintained for running single applications for local development.  This file is a binary blob, thus making versioning opaque and changes difficult to analyze.
4. Configuration must be done via one of the following mechanisms:
   1. A human must manually merge the XML changes into a single configuration/standalone.xml file on the server before deployment.  This file might be kept in revision control, but separate versions must be maintained for the "live" server which contains all applications versus for development which would typically run only a single application at once.  Additionally separate versions must be kept for dev/qa/prod due to database & other configuration differences.  Since these files contain the entire server configuration, significant amounts of configuration are duplicated across versions.  When one of these duplicate elements (IE thread pool sizes, SSL settings, socket bindings, etc.) must be changed, the change must be manually made to tens of files.  Verifying that these changes were made successfully in all configurations becomes difficult, and inevitably files are missed, incorrectly edited, and errrors that should otherwise have been caught during local development could easily manifest only in production.
   2. Applications publish XSLT's to inject their configuration into standalone.xml.  These might be run by the build server at the end of the build process or by startup scripts (IE our JBossController) prior to spawning the appserver.  It may be difficult to get these scripts to run during local development (IE in Eclipse) which would necesitate further fracturing of the configuration between "live" servers and local development.  Writing XSLT is inherently more complicated than writing the XML files directly.
   3. Applications publish JBoss CLI scripts which must be run against JBoss prior to the applications' EAR files being deployed.  Perhaps driving by scripts or JBossController(?).  Again, this would be cumbersome to implement for local development.  The CLI format is again removed from the underlying XML so, an additional domain specific language of sorts must be learned.
   4. A persistent instance of JBoss is allowed to exist on the server which is manually configured via CLI or web GUI to contain each applications' configuration. We consider this option absolutely unacceptable.  No persistent configuration should ever exist which is not directly driven by an automated process backed by revision control.

I understand how maintaining a single XML configuration file rather than having configuration spread about can seem attractive, but in practice it seems to duplicate identical configuration across environments.  The separation of configuration files as implemented in JBoss 4.2 is actually very much in line with the way those configuration elements tend to be administered.  Merging them all into one lump creates a tangled mess of overlapping concerns and makes maintenance a nightmare.  Under 4.2, if *-ds.xml is changed, you know the database changed.  Under 7.x, if standalone.xml is changed, maybe the DB changed, maybe thread pool sizes, who knows without diff'ing the versions.  If the vault keystore changes, maybe a password was added, maybe one modified, there's no way to diff them to tell.

I really can't stress enough that this change in configuration methodology is an enormous step backwards for JBoss.  One of the things that I've long appreciated about JBoss versus the other "big" JEE appservers was that JBoss didn't have much "magic."  Where the big servers tend towards a monolithic server which houses all kinds of important configuration, JBoss allowed a zero-install approach with everything pulled down at startup.  Configuration for each class of object was in a specific kind of file, everything was plain-text on a disk somewhere.  Deploying JBoss was just a matter of copying files and starting the process. 

The shift in thought in JBoss 7.x seems to be more towards monolithic container-managed magic.  Where you access an API to modify things, where the server's files are expected to be long-lived persistent data rather than ephemeral.  The configuration file versioning feature (standalone_xml_history) especially is disturbing as it implies a shift in thinking to where the server's configuration is an entity on to itself.  Contrast this to our model where the server's files are disposable, all configuration is nothing more than XML files on a disk (and in revision control), and the entire configuration is pushed from revision control each time the server is started.  No changes should ever be made to the running server that would necesitate versioning the configuration physically on the server.  Such changes should be made in revision control and pushed out via a build to ensure repeatablility.