Production/Uptime tips

[sanne]

Hi,

I'm about to roll in my first JBoss production server at a clients on a linux system. Although there will be sysadmin support, I would like to throw in some minimal criteria for a production environment.

I wondered if anybody had any production tips/startup scripts/uptime scripts etc. to maintain a good installation. I've read the docu, but production usage doesn't seem to be covered.

Maybe put together a production guide?

Sanne

[sanne]

O.k., very basic but:

run.sh > /dev/null 2>&1

Means execute run, throw all the console output away, redirect all the error output (2) to the std out destination (i.e. /dev/null).

in conf/log4j.xml

change the log level to error:

<!-- ================ -->
<!-- Limit categories -->
<!-- ================ -->

<!-- Limit JBoss categories to INFO -->




or even to FATAL

<!-- ================ -->
<!-- Limit categories -->
<!-- ================ -->

<!-- Limit JBoss categories to INFO -->




Which generates less output, and more speed.
Don't log to console:

<!-- ======================= -->
<!-- Setup the Root category -->
<!-- ======================= -->


<appender-ref ref="FILE"/>


Setting the logging to ASYNC is not necessary now since there is almost nothing logged.

[sanne]

Notification of an fatal error:

<!-- The SMTP Appender sends email messages. The threshold parameter should not be changed.
LTCG sends email o nERROR and FATAL errors only. Values must be provided for SMTPHost,
To, From, Subject, and BufferSize. To email each ERROR and FATAL error as it occurred, the
BufferSize parameter should be set to 1. The email will contain the time the FATAL error
occurred, the filename, the line number, and the custom error message. -->












<!-- ======================= -->
<!-- Setup the Root category -->
<!-- ======================= -->


<appender-ref ref="FILE"/>
<appender-ref ref="SMTP"/>



Send an email when a fatal error occurs. So you will know when JBoss has shutdown due to an fatal error.
Setting the buffer size to one means that you get an email for EVERY log line of your stacktrace (+/- 50 for a nice stack trace!). Don't know how to get one decent email. But it works

[jonlee]

You should block sensitive ports from external access and turn off unused listeners.

You shouldn't run JBoss with the root account in case a vulnerability allows someone to break the JVM and access the system as root. This also means some port redirection if Jetty/Tomcat are directly facing the internet/intranet or if you are using an intermediate Linux firewall - see http://www.amitysolutions.com.au/downloads/Tomcat-Firewalls.pdf for greater detail.

Choose between performance or access logging with reverse lookups in Tomcat/Jetty.

If using Jetty, configure your logging timestamps as necessary for your sysadmin - i.e. Timezone setting in the Jetty jboss-service.xml.

Create a proper RedHat startup and shutdown script (when using RedHat) and ensure that the startup and shutdown sequence is correct - don't startup before port redirection/etc is in place and don't shutdown before the firewall is turned off - that kind of thing.

Should also include instructions for installing certificates for SSL as well as file protections so the file can't be changed, etc by unauthorised persons. For most clients we've had, this is usually one of the first things a sysadmin asks about.

Access file archival topic.
Redirecting JSP and content file storage so that cron jobs don't clean off these necessary webapp services.

Upgrading the JDK and environment variables to change for this.

Installation and configuration for Jikes.

Memory and disk space recommendations.

JVM memory settings and -Xrs if JBoss keeps mysteriously shutting down (encountered this a few times).

Troubleshooting.

Can't think of anything else for now.

Sysadmin manuals are always fun.

[sanne]

Hi,

Thanks for your input, this is good advice.
I didn't get one point:

'
Access file archival topic.
Redirecting JSP and content file storage so that cron jobs don't clean off these necessary webapp services.

Upgrading the JDK and environment variables to change for this.
'

Could you rephrase that?

[jonlee]

Sorry. I was looking at that one blankly too. It was late at night - or ealy in the morning.

You probably need some coverage on archiving log files. For example, the Tomcat or Jetty page access logs should be stored somewhere or deleted - depending on the client's requirements - otherwise they just keep building up.

By default, for Jetty at least, the class files for the servlet code generated from a JSP and much of the static content gets stored in /tmp. RedHat Linux and a few others have a cron job that sweeps out the /tmp directory. So you often need to modify the script so you can add JAVA_OPTS with -Djava.io.tmpdir=/jetty so these files get put somewhere that the cron job won't delete - otherwise you get a broken app sometime in the next month or so (which a appserver restart will cure). You'd need to create this /jetty directory from this example.

At some point someone will want to upgrade the JDK - which means you will want to have some instructions on changing JAVA_HOME otherwise next time they start JBoss, it probably won't start.

[jswright]

Lack of at least a simple checklist for production configuration seems like a gaping hole in the available JBoss documentation. I did some searches and couldn't find any. I did notice that there was a recent post to this forum that the for-pay documentation from the JBoss Group doesn't cover production usage either.

Here are things I've discovered so far:

* Running on Linux (probably applies to all Unix), the issue already mentioned of putting the compiled version of JSP page under /tmp burned us for a long time until we figured that out. I found this in the FAQ on http://coredevelopers.net, btw.

* We created Linux startup/shutdown scripts, also as mentioned.

* We added logic to our shutdown script to clean the JBoss and Jetty temporary directories, so that the space consumed doesn't just grow and grow.

* You also need to monitor space in your log directory, and perhaps delete old files there.

* I haven't worked with JBoss 3.2 yet, but 3.0.x the default server is configured with the JMX console with no security. This either needs to be secured or removed.

One thing I've been meaning to do is devote some thought to whether there is some approach to configuration that would make it easier to install dot releases of the app server. For example, so far we typically just use the preconfigured "deploy" directory, but I'm wondering if it might be better to set up one or more of my own deploy directories outside the directory tree from the JBoss distribution.

--jeff

[jonlee]

One other thing I forgot, secure your deploy directory, particularly the datasource scripts as they usually have the database username/password in them - you should also manage access to the Tomcat/Jetty certificate passwords for SSL.

This is where having another deployment directory outside the standard install might be useful. Although, it is also an option to bundle your own production install of JBoss with your applications pre-configured so it is a no-brainer for your clients to re-install if something very tragic occurs - i.e. the hard drive head ploughs into the magnetic surface and throws all the oxide holding your once useful application into the stratosphere.

As for the production checklist - well, I wouldn't want the JBoss Group to have all the fun. Some of us still need to make a living. ;)

And customers like to have a nice bound Operations manual. ;)

[sanne]

Hi Jeff,

I'm unfamiliar with:

* Running on Linux (probably applies to all Unix), the issue already mentioned of putting the compiled version of JSP page under /tmp burned us for a long time until we figured that out. I found this in the FAQ on http://coredevelopers.net, btw.

And couldn't find the faq... Could you rephrase that one?

Just in case you didn't know in jboss service there is a:


deploy/


for adding dirs, and another deployer for adding separate files and dirs:










One could setup an install with all the base services/passwords in a separate dir not readable to the client, or in your case to switch dot releases fast.

Would you have any samples of the start-up and shutdown script, and the monitoring of the dirs?

The JMX console is indeed a problem: don't think the security is there yet, since the shutdown.sh gives not implemented on that one. The only solution would be maybe to make the rmi go over ssl and secure it that way.

The list getting better ...

[sanne]

The list --IS-- getting better ofcourse.

There is also a jmx ant task:

http://jmx4ant.sourceforge.net/

Using a cron job on this task one could do a sanity check on the jmx stack. Basically if the server has started one expects or the whole stack to fail, or the whole stack to collapse. One could for instance poll the server mbean StartDate to see of it is still alive.

The jmx webconsole can easily be secured, I don;t know about remote calls made via RMI, those should be secured too then. Anybody any idea's?

One could also deploy a trivial webapp responding ok and check that.

Sanne

[jswright]

I'm not able to access coredevelopers.net at the moment either. The exact link I had found before was

http://coredevelopers.net/technology/jetty/jetty-jboss-faq.jsp

This was only a few weeks ago.

What I wound up doing was adding something like this to the JBoss run.sh:

JAVA_OPTS="$JAVA_OPTS -Djava.io.tmpdir=$JBOSS_HOME/tmp_jetty"

Then, in my shutdown script, I added:

rm -rf $JBOSS_HOME/server/default/tmp
rm -rf $JBOSS_HOME/tmp_jetty/*

Jetty uses the system property java.io.tmpdir to decide where to put compiled JSPs. Apparently this defaults to /tmp on Unix, and since most systems have a cron job to clean old files out of /tmp you have the files deleted out from under your running web site and the pages just don't show up anymore. You restart and everything is back to normal.

And with the JBoss temporary directory... One server I was working with had been operational for over six months with several apps with large EAR files. The JBoss server/default/tmp directory was wasting a lot of space. You could be a little more elegant and delete old file, but it doesn't really hurt to just rm the whole directory.

Anybody else have more tips to share?

--jeff

[jonlee]

As with most things, eject what you don't use. If you are not using the scheduler, remove it from deployment. Same with the UUID-key-generator. If you are not using hsqldb at all, remove it - be aware that things like JMS use the hsqldb unless you tell it otherwise, so trial things on your test system before you do it in production.

Also hsqldb is by default configured for tcp connection (port 1701). If you want to retain the service and are not accessing it by other processes - you'll want to switch it to in-process or in-memory. Doing this means you can disable the Hypersonic MBean as well. View the hsqldb-ds.xml for more information. We tend to do this rather than removing hsqldb altogether.

Disable listeners you aren't using in your servlet.

If you think your Jetty server is going to sustain some heavy load, you might want to consider using the Jetty Channel Socket Listener. It implements non-blocking IO (you need JDK 1.4), and it means that it can share the socket when your system can't create enough sockets. You need to compile this in from the Jetty sources as the standard JBoss (to support JDK 1.3) does not include this.

[sanne]

Hi jonlee,

Good tip on optimization.
Is the NIO stuff in Jetty stable?
Where'nt there any VM problems or so?

Sanne

[sanne]

Jeff, JBoss doesn't delete the old files in tmp on undeplo/redeploy? This seems like something fixable.
There must be a way of dealing with this. Maybe an MBean which scans all the deployers for their working directory? And delete the dir if it is not uses.

There is also the tip on configuring memory/cache where you should avoid setting a hard limit like 500 on the chache size. After the cache has 500 objects stored, performance begins to degrade! Rule of the thumb is set cache size to +/- 1/3 of available memory.

Any thoughts on securing the JMX RMI adapter? Maybe I'll ask in the JMX forum.

Sanne

[jonlee]

The NIO is actually in JDK 1.4.x so Jetty makes use of it. Before that, the NIO libraries I think were a GNU set and then Sun decided it was a good comms idea.The Channel Socket Listener seems stable enough and has also been around a while. It seems to handle things under stress and doesn't seem to have the issue of blocking more users until persistent HTTP connections expire - although hopefully your system should never be under such stresses normally.