I'm currently migrating a production application from older physical hardware, to a new vmware host.
I've confirmed that all components (java version, jboss version, config files, application and libraries) are the same between platforms. I've also made the platforms as close as possible, recognizing that vmware HD space, RAM and CPU are not the same as 'real' ones. The remaining differences would be that our existing production server is (I think) some flavour of redhat running linux 2.6.8 on reiserfs, and the new vmware host is ubuntu linux 2.6.24 using ext3.
I recognize that there's quite a difference between plain hardware and VMWare hosts. But...
When I start this application on my localhost (also ubuntu) for development it takes about 17 seconds for the whole thing to start. On production it takes about the same. But on the new VM host, it takes a minute and 17 seconds, and nearly the whole minute extra is experienced right here (3rd line from bottom):
========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /usr/foo/java/jboss-3.2.6 JAVA: /usr/foo/java/j2sdk1.4.2_19/bin/java JAVA_OPTS: -server -Xms700m -Xmx700m -Djava.awt.headless=true -Dprogram.name=run.sh CLASSPATH: /usr/foo/java/jboss-3.2.6/bin/run.jar:/usr/foo/java/j2sdk1.4.2_19/lib/tools.jar ========================================================================= me@oldhost:/usr/foo/java/jboss-3.2.6/bin$ 10:06:41,010 INFO [Server] Starting JBoss (MX MicroKernel)... 10:06:41,011 INFO [Server] Release ID: JBoss [WonderLand] 3.2.6 (build: CVSTag=JBoss_3_2_6 date=200410140106) 10:06:41,012 INFO [Server] Home Dir: /usr/foo/java/jboss-3.2.6 10:06:41,012 INFO [Server] Home URL: file:/usr/foo/java/jboss-3.2.6/ 10:06:41,012 INFO [Server] Library URL: file:/usr/foo/java/jboss-3.2.6/lib/ 10:06:41,013 INFO [Server] Patch URL: null 10:06:41,013 INFO [Server] Server Name: default 10:06:41,013 INFO [Server] Server Home Dir: /usr/foo/java/jboss-3.2.6/server/default 10:06:41,013 INFO [Server] Server Home URL: file:/usr/foo/java/jboss-3.2.6/server/default/ 10:06:41,013 INFO [Server] Server Data Dir: /usr/foo/java/jboss-3.2.6/server/default/data 10:06:41,014 INFO [Server] Server Temp Dir: /usr/foo/java/jboss-3.2.6/server/default/tmp 10:06:41,014 INFO [Server] Server Config URL: file:/usr/foo/java/jboss-3.2.6/server/default/conf/ 10:06:41,014 INFO [Server] Server Library URL: file:/usr/foo/java/jboss-3.2.6/server/default/lib/ 10:06:41,014 INFO [Server] Root Deployment Filename: jboss-service.xml 10:06:41,016 INFO [Server] Starting General Purpose Architecture (GPA)... 10:06:41,250 INFO [ServerInfo] Java version: 1.4.2_19,Sun Microsystems Inc. 10:06:41,250 INFO [ServerInfo] Java VM: Java HotSpot(TM) Server VM 1.4.2_19-b04,Sun Microsystems Inc. 10:06:41,250 INFO [ServerInfo] OS-System: Linux 2.6.24-23-generic,i386 10:06:41,422 INFO [Server] Core system initialized 10:07:34,053 INFO [Log4jService$URLWatchTimerTask] Configuring from URL: resource:log4j.xml 10:07:39,116 INFO [WebService] Using RMI server codebase: http://oldhost:8083/
See how it takes nearly a minute after "Core system initialized" to proceed with the rest of the startup? Once it's past this hump, the rest of the load proceeds 'normally' and the app itself seems performant (haven't gotten to true load testing yet).
Has anyone seen this before, or have ideas what it could be, or where I should start looking for my lost minute?