JBossDeveloper

I was wondering about a couple of things in regards to JVM configuration, that go beyond the schema.

Are we going to do any validation of the arguments themselves?  I know I have seen support cases where options are specified incorrectly, or in many cases arguments are specified that directly conflict with each other, such as specifying both the parallel collector and the concurrent mark and sweep collector at the same time.

The second question I have is related to options that may have OS speciic configuration.  For example, using large pages and NUMA.

In the case of large memory pages if you specify -XX:+UseLargePages with the 32-bit JVM, the JVM throws IllegalArgumentException and won't start.  In the case of the 64-bit JVM it may fail to allocate large pages and give and error, but then fall back to using regular memory pages (typical case is errno=12).

For the option to use the NUMA aware allocator (-XX:+UseNUMA), on Linux systems you have to start the JVM through the numactl command, so the startup of the JVM would look like the following:

numactl <NUMA policy options> java <other options> -XX:+UseNUMA ....

So, specifying that option won't have the desired affects without the OS level command.

I'm just trying to get the scope of what we think we will do here.  The NUMA case here, I think we need to support in terms of being able to start the JVM, even if we don't strictly validate everything, because all the newest x86 systems are now NUMA systems (Nehamlem from Intel, and Opteron from AMD has always been NUMA).

Andrig Miller wrote:

 

I was wondering about a couple of things in regards to JVM configuration, that go beyond the schema.

 

Are we going to do any validation of the arguments themselves?  I know I have seen support cases where options are specified incorrectly, or in many cases arguments are specified that directly conflict with each other, such as specifying both the parallel collector and the concurrent mark and sweep collector at the same time.

Our curent solution for that is to try and support as many things via XML elements as we can.  I don't think it's realistic to try and validate the raw options; it's much easier to validate at an XML level and define a mapping to the JVM type.

Andrig Miller wrote:

 

The second question I have is related to options that may have OS speciic configuration.  For example, using large pages and NUMA.

 

In the case of large memory pages if you specify -XX:+UseLargePages with the 32-bit JVM, the JVM throws IllegalArgumentException and won't start.  In the case of the 64-bit JVM it may fail to allocate large pages and give and error, but then fall back to using regular memory pages (typical case is errno=12).

 

For the option to use the NUMA aware allocator (-XX:+UseNUMA), on Linux systems you have to start the JVM through the numactl command, so the startup of the JVM would look like the following:

 

numactl <NUMA policy options> java <other options> -XX:+UseNUMA ....

 

So, specifying that option won't have the desired affects without the OS level command.

 

I'm just trying to get the scope of what we think we will do here.  The NUMA case here, I think we need to support in terms of being able to start the JVM, even if we don't strictly validate everything, because all the newest x86 systems are now NUMA systems (Nehamlem from Intel, and Opteron from AMD has always been NUMA).

I think we definitely should support OS commands wrapping the JVM start - this would be really nice for users.  Other wrapper commands I could see being useful are "nice" for setting priority, (non-NUMA ["UMA"?]) processor affinity commands, ulimit, etc.  Large page support is something that we ought to have too, preferably with validation.

David Lloyd wrote:

 

Andrig Miller wrote:

 

I was wondering about a couple of things in regards to JVM configuration, that go beyond the schema.

 

Are we going to do any validation of the arguments themselves?  I know I have seen support cases where options are specified incorrectly, or in many cases arguments are specified that directly conflict with each other, such as specifying both the parallel collector and the concurrent mark and sweep collector at the same time.

Our curent solution for that is to try and support as many things via XML elements as we can.  I don't think it's realistic to try and validate the raw options; it's much easier to validate at an XML level and define a mapping to the JVM type.

 

Andrig Miller wrote:

 

The second question I have is related to options that may have OS speciic configuration.  For example, using large pages and NUMA.

 

In the case of large memory pages if you specify -XX:+UseLargePages with the 32-bit JVM, the JVM throws IllegalArgumentException and won't start.  In the case of the 64-bit JVM it may fail to allocate large pages and give and error, but then fall back to using regular memory pages (typical case is errno=12).

 

For the option to use the NUMA aware allocator (-XX:+UseNUMA), on Linux systems you have to start the JVM through the numactl command, so the startup of the JVM would look like the following:

 

numactl <NUMA policy options> java <other options> -XX:+UseNUMA ....

 

So, specifying that option won't have the desired affects without the OS level command.

 

I'm just trying to get the scope of what we think we will do here.  The NUMA case here, I think we need to support in terms of being able to start the JVM, even if we don't strictly validate everything, because all the newest x86 systems are now NUMA systems (Nehamlem from Intel, and Opteron from AMD has always been NUMA).

I think we definitely should support OS commands wrapping the JVM start - this would be really nice for users.  Other wrapper commands I could see being useful are "nice" for setting priority, (non-NUMA ["UMA"?]) processor affinity commands, ulimit, etc.  Large page support is something that we ought to have too, preferably with validation.

I definitely agree that using the XML elements is the best way to go, because there are just too many things that are different between JVM's, like IBM's and Sun's (and OpenJDK), as well as the things that change between even update releases.  I'm going to augment the description for the Andiamo JIRA with that information, to keep the scope of what we are doing recorded there.

I also agree that it would be really nice for users to be able to have the OS commands that wrap the JVM in their two.

Maybe the schema should be augmented with an optional tag just for that purpose.

On the large page support, its a little difficult to do anything with the schema, and validation could be interesting.  On Linux, the /proc/meminfo contains what you need to validate, but other platforms wouldn't be so easy.  Also, in RHEL 6, they have added "transparent" large page support, which means prior to the JVM starting you might not see anything in /proc/meminfo that tells you something valid.  Also, if you have more than one process using large page memory on the same server, the information in /proc/meminfo isn't granular enough.  Finally, the pages could be configured properly, but the security stuff could be preventing access.  Suffice it say, the validation will be complex.

I just want to mention that schema- and xml-wise I want the configuration to look simple and not too sophisticated for things like jvm options. There are hundreds of them and each of them can be specified as a simple string. IMO, the schema itself shouldn't be aware of any specific option and shouldn't be especially strict.

It should be easy for the user to configure a JVM and it should not require a configuration manual with all the options and their combinations.

If you want validation of JVM configuration - great, but I would do it  on the next level, not in the XSD, and invest in clear error messages instead of XML parser errors.

Alexey Loubyansky wrote:

 

I just want to mention that schema- and xml-wise I want the configuration to look simple and not too sophisticated for things like jvm options. There are hundreds of them and each of them can be specified as a simple string. IMO, the schema itself shouldn't be aware of any specific option and shouldn't be especially strict.

It should be easy for the user to configure a JVM and it should not require a configuration manual with all the options and their combinations.

If you want validation of JVM configuration - great, but I would do it  on the next level, not in the XSD, and invest in clear error messages instead of XML parser errors.

I disagree.  Putting options in the schema helps the user a lot because options vary from JVM to JVM, and on hotspot (for example) most of the options aren't even documented anywhere.  By putting it in the schema we get a few benefits:

1. The user can have a single source of documentation for the complete set of options no matter what JVM they're on; no more pain-in-the-ass secret undocumented options to do common things like large pages, change GC modes, set memory pool sizes, etc.
2. The user can switch JVMs without rewriting all their arguments and doing investigation to figure out what the equivalent option is on the new JVM, with its own set of secret undocumented options.
3. The JVMs can be configured at the domain level even if they have different JVM types on different operating systems (i.e. Windows vs. Mac vs Linux vs Mainframe vs whatever else).  If you make them manually specify each option then this is not even possible.
4. If an option isn't directly supported by a specific we can decide the best action to take based upon the option; either perform the closest equivalent operation, ignore the option, or raise an error (which can do nice things like link to specific documentation on the option).
5. Since everything is defined in the XSD, users editing domain.xml can take advantage of IDE tab-completion and quick-documentation and never have to open a manual to be able to completely configure the JVM instance however they like.

In terms of validation - since we're using stax directly (with our own validation) we can make the error message as clear as we like, tailored to the exact problem.

There are options that are available on all the JVMs

and there are options that are not available on all.

There are also options that can exist only in specific release

(alghouth the same vendor is used), see e.g. this doc:

http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp

Why don't learn from others

* IBM WAS

* OCL Bea

* OCL GF

and waste our time/resources on NIH?

I'd expect someone to analyze thirdparty products

and then pick up the best of all the worlds.

IMHO JVM configuration options should be easy & no magic behind.

* easy - simple XML schema

* no magic

   - there should be no generator that will generate JVM options for users

   - there should be no validation if user specified options combination is correct

   - there should be no schema udpate required if new JVM option is introduced in some vendor JDK

Our users shold be responsible for JVM options validation/configuration. (e.g. using VMWare images to distribute).

Take a look to GF approach:

<java-config

  debug-options="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009"

  system-classpath=""

  classpath-prefix=""

  classpath-suffix=""

>
  <jvm-options>-client</jvm-options>
  <jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
  <jvm-options>-XX:+LogVMOutput</jvm-options>
  <jvm-options>-XX:LogFile=${com.sun.aas.instanceRoot}/logs/jvm.log</jvm-options>
  <jvm-options>-Djava.endorsed.dirs=${installRoot}/modules/endorsed${path.separator}${installRoot}/lib/endorsed</jvm-options>
  <jvm-options>-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy</jvm-options>
</java-config>

Simple, Intuitive, Portable. Even for JVMs that don't exist ATM

I see your idea to have generic Java configuration and share it accross different Server Managers.

But this means too much work and minimal benefits.

Java shouldn't be configured neither on DomainController nor ServerManager level.

It should be configured on server.

There can be some callbacks to pass in min/max memory options from Domain Controller to server, but I wouldn't go futher.

David Lloyd wrote:

I disagree.  Putting options in the schema helps the user a lot because options vary from JVM to JVM, and on hotspot (for example) most of the options aren't even documented anywhere.  By putting it in the schema we get a few benefits:

1. The user can have a single source of documentation for the complete set of options no matter what JVM they're on; no more pain-in-the-ass secret undocumented options to do common things like large pages, change GC modes, set memory pool sizes, etc.
2. The user can switch JVMs without rewriting all their arguments and doing investigation to figure out what the equivalent option is on the new JVM, with its own set of secret undocumented options.
3. The JVMs can be configured at the domain level even if they have different JVM types on different operating systems (i.e. Windows vs. Mac vs Linux vs Mainframe vs whatever else).  If you make them manually specify each option then this is not even possible.
4. If an option isn't directly supported by a specific we can decide the best action to take based upon the option; either perform the closest equivalent operation, ignore the option, or raise an error (which can do nice things like link to specific documentation on the option).
5. Since everything is defined in the XSD, users editing domain.xml can take advantage of IDE tab-completion and quick-documentation and never have to open a manual to be able to completely configure the JVM instance however they like.

 

In terms of validation - since we're using stax directly (with our own validation) we can make the error message as clear as we like, tailored to the exact problem.

This sounds like http://en.wikipedia.org/wiki/God_object to me

Java shouldn't be configured neither on  DomainController nor ServerManager level.

It should be configured  on server.

There can be some callbacks to pass in min/max memory  options from Domain Controller to server, but I wouldn't go futher.

 

To clarify things

JVM configuration can be in domain schema

if there's a prediction that all the nodes in

the domain use the same JVM.

This isn't too restrictive requirement IMHO

because usually (I'd tip 95%)

users/customers have just one JVM in production.

If there are more JVMs available users/customers

would create domain configuration per JVM.

My 2c.

David Lloyd wrote:

 

Alexey Loubyansky wrote:

 

I just want to mention that schema- and xml-wise I want the configuration to look simple and not too sophisticated for things like jvm options. There are hundreds of them and each of them can be specified as a simple string. IMO, the schema itself shouldn't be aware of any specific option and shouldn't be especially strict.

It should be easy for the user to configure a JVM and it should not require a configuration manual with all the options and their combinations.

If you want validation of JVM configuration - great, but I would do it  on the next level, not in the XSD, and invest in clear error messages instead of XML parser errors.

I disagree.  Putting options in the schema helps the user a lot because options vary from JVM to JVM, and on hotspot (for example) most of the options aren't even documented anywhere.  By putting it in the schema we get a few benefits:

1. The user can have a single source of documentation for the complete set of options no matter what JVM they're on; no more pain-in-the-ass secret undocumented options to do common things like large pages, change GC modes, set memory pool sizes, etc.
2. The user can switch JVMs without rewriting all their arguments and doing investigation to figure out what the equivalent option is on the new JVM, with its own set of secret undocumented options.
3. The JVMs can be configured at the domain level even if they have different JVM types on different operating systems (i.e. Windows vs. Mac vs Linux vs Mainframe vs whatever else).  If you make them manually specify each option then this is not even possible.
4. If an option isn't directly supported by a specific we can decide the best action to take based upon the option; either perform the closest equivalent operation, ignore the option, or raise an error (which can do nice things like link to specific documentation on the option).
5. Since everything is defined in the XSD, users editing domain.xml can take advantage of IDE tab-completion and quick-documentation and never have to open a manual to be able to completely configure the JVM instance however they like.

 

In terms of validation - since we're using stax directly (with our own validation) we can make the error message as clear as we like, tailored to the exact problem.

Trying to come up with a standard set of options for similar/equivalent non-standard options on different JVMs sounds nice. But how useful and practical will this be? I am actually interested to know since I've never had to tune the JVM using all the availble options.

I imagine that the real affect (and perhaps meaning to some degree) of similar options can be different on different JVM implementations. To be more specific, let's take EJB container as an example (it's easier for me). E.g. if a user migrates from WebLogic to JBoss and wants to port EJB container config from WL to JBoss. Both ASs provide settings for cache capacities, pools, etc. The thing is it wouldn't right to just use the same values in JBoss. Because the performance costs of creating an instance in both servers are different, as well as the memory footprint. So, you'll have to run performance benchmarks and come up with a JBoss-specific tuning for the same thing. I guess it can also be true for some JVM options when you switch from one JVM to another?

Plus, if you are at the point where you have to use very specific/exotic options then you are supposed to know what you are doing and if you switch to another JVM you'll make a decision based on existence of similar options in that VM (or existence of options that don't exist in the current VM), i.e. you would make sure the similar/equivalent options exist and how they look like. So, you'll know how to specify them anyway. But the values might still be different?

Richard Opalka wrote:

 

Why don't learn from others

* IBM WAS

* OCL Bea

* OCL GF

and waste our time/resources on NIH?

Because they all suck.  Don't go throwing "NIH" at me, that's one of the most moronic excuses for being lazy and not doing something right that I know of.  I'm only interested in solving the problem correctly.

Richard Opalka wrote:

 

IMHO JVM configuration options should be easy & no magic behind.

* easy - simple XML schema

* no magic

   - there should be no generator that will generate JVM options for users

   - there should be no validation if user specified options combination is correct

   - there should be no schema udpate required if new JVM option is introduced in some vendor JDK

The XML schema is simple already.  You're solving a problem that doesn't exist.  We won't require a schema update for new JVM options.  It's simple to implement and it's a unique feature that people will like, end of story.

Richard Opalka wrote:

 

I see your idea to have generic Java configuration and share it accross different Server Managers.

But this means too much work and minimal benefits.

Java shouldn't be configured neither on DomainController nor ServerManager level.

It should be configured on server.

There can be some callbacks to pass in min/max memory options from Domain Controller to server, but I wouldn't go futher.

I think you're wrong about this.  But you won't be convinced so I'm not going to bother.

Richard Opalka wrote:

 

David Lloyd wrote:

I disagree.  Putting options in the schema helps the user a lot because options vary from JVM to JVM, and on hotspot (for example) most of the options aren't even documented anywhere.  By putting it in the schema we get a few benefits:

1. The user can have a single source of documentation for the complete set of options no matter what JVM they're on; no more pain-in-the-ass secret undocumented options to do common things like large pages, change GC modes, set memory pool sizes, etc.
2. The user can switch JVMs without rewriting all their arguments and doing investigation to figure out what the equivalent option is on the new JVM, with its own set of secret undocumented options.
3. The JVMs can be configured at the domain level even if they have different JVM types on different operating systems (i.e. Windows vs. Mac vs Linux vs Mainframe vs whatever else).  If you make them manually specify each option then this is not even possible.
4. If an option isn't directly supported by a specific we can decide the best action to take based upon the option; either perform the closest equivalent operation, ignore the option, or raise an error (which can do nice things like link to specific documentation on the option).
5. Since everything is defined in the XSD, users editing domain.xml can take advantage of IDE tab-completion and quick-documentation and never have to open a manual to be able to completely configure the JVM instance however they like.

 

In terms of validation - since we're using stax directly (with our own validation) we can make the error message as clear as we like, tailored to the exact problem.

 

This sounds like http://en.wikipedia.org/wiki/God_object to me

Come on, really?  All I'm talking about is adding a couple of XML elements.  We already have to have some knowledge about JVM types anyway, just to launch a process.  We might as well formalize and add this nice feature.  It's not like it's particularly difficult.

I can't believe you think that a nice feature list for a minimal coding effort == God object.  That's just ridiculous.

Alexey Loubyansky wrote:

 

Trying to come up with a standard set of options for similar/equivalent non-standard options on different JVMs sounds nice. But how useful and practical will this be? I am actually interested to know since I've never had to tune the JVM using all the availble options.

I'm not saying do this for all options, just some of the main ones that people are interested in.  Heap, stack, and perm gen size for sure.  Andy Miller's large pages and NUMA stuff.  Process modifiers like nice, CPU affinity, ulimit which vary by platform.

We can always keep a generic <option> for things we don't, can't, or won't account for.

Brian Stansberry wrote:

 

I'm going to be lazy and not consider the arguments for and against trying to be helpful and convert generalized schema elements into VM-specific options. I'll just look at the practical issues:

 

1) Is there anything in our formal requirements that requires us to do this? (No)

2) Do we have resources to implement this if not required,while still completing our formal requirements? (No)

3) Does a schema design that doesn't support this prevent us from adding it in the future? That is, would adding it require incompatible changes? (I don't think so but I'll be lazy again and count on someone else to confirm/refute. ;-) )

 

If doing it later doesn't require an incompatible change, I suggest this becomes a feature request JIRA for a later release; we can formally decide later whether we want to implement it.

 

I'm not saying stop discussing the issue, just that we should be clear whether this is a critical decision for AS 7 or not and whether or not any actual work on it should be done now.

I agree with your sentiment. However have a few common aliases like what has been defined heap-sze etc is not a big deal, and will pave the way for more complex mappings/