JBoss AS 6.0.0.Final is taking 20+ minutes to start

[tmuldo]

I'm in the process of migrating a JBoss AS 4.2.3 application to JBoss AS 6.0.0.Final (deployed as a WAR file) and I'm afraid that JBoss is taking between 21 and 24 minutes to start.  While I still have errors that I need to resolve, I'm guessing that it should not take that long to start.  My guess is that I'm incurring a hit while JBoss is scanning the many jar files found within my WEB-INF/lib directory and while I've tried to exclude them via {jboss.server.home}/deploy/jboss-scanning.xml, there doesn't seem to be any improvement. I've searched the JBoss forums and wiki pages but I really haven't found anything that would explain why it's taking so long to start. I've attached both my server.log and my jboss-scanning.xml file. Any ideas are suggested.

Thanks in advance, Tom

[jaikiran]

I guess, it's a side effect of 2 things:

1) Known bug https://issues.jboss.org/browse/JBAS-8786

2) The WEB-INF/lib jar files are probably being scanned for annotations.

The first issue needs a fix in 6.1.0. But for the second issue, we can improve the situation through the web.xml. Do you have any jar files in WEB-INF/lib which require annotation scanning? If not, you can set the metadata-complete attribute to true in your web.xml so that annotation scanning is skipped (in a portable way). What does your web.xml look like?

[tmuldo]

My web.xml...

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/j2ee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee web-app_2_4.xsd"
    version="2.4">

  <display-name>FirstBest Application</display-name>

  <!-- =================================================================== -->
  <!-- Spring Context Loader                                                                                              -->
  <!-- =================================================================== -->
  <context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>
        /WEB-INF/webApplicationContext.xml
    </param-value>
  </context-param>
  <context-param>
    <param-name>locatorFactorySelector</param-name>
    <param-value>
        classpath*:beanRefContext.xml
    </param-value>
  </context-param>
  <context-param>
    <param-name>parentContextKey</param-name>
    <param-value>fbs-web</param-value>
  </context-param>
    <servlet>
        <display-name>CXF Servlet</display-name>
        <servlet-name>CXFServlet</servlet-name>
        <servlet-class>
            org.apache.cxf.transport.servlet.CXFServlet
        </servlet-class>
        <load-on-startup>2</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>CXFServlet</servlet-name>
        <url-pattern>/webservices/*</url-pattern>
    </servlet-mapping>

  <listener>
    <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
  </listener>
  <listener>
    <listener-class>com.firstbest.web.ServletContextListener</listener-class>
  </listener>

  <!-- =================================================================== -->
  <!-- Acegi Security                                                      -->
  <!-- =================================================================== -->
  <listener>
    <listener-class>org.acegisecurity.ui.session.HttpSessionEventPublisher</listener-class>
  </listener>

  <filter>
    <filter-name>Acegi Filter Chain Proxy</filter-name>
    <filter-class>org.acegisecurity.util.FilterToBeanProxy</filter-class>
    <init-param>
      <param-name>targetClass</param-name>
      <param-value>org.acegisecurity.util.FilterChainProxy</param-value>
    </init-param>
  </filter>
  <filter-mapping>
    <filter-name>Acegi Filter Chain Proxy</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

  <filter>
    <filter-name>Token Authentication Filter</filter-name>
    <filter-class>com.firstbest.authentication.TokenAuthenticationFilter</filter-class>
  </filter>
  <filter-mapping>
    <filter-name>Token Authentication Filter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

  <!-- =================================================================== -->
  <!-- Spring MVC                                                          -->
  <!-- =================================================================== -->
    <servlet>
      <servlet-name>DispatcherServlet</servlet-name>
      <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
      <load-on-startup>2</load-on-startup>
  </servlet>

  <servlet-mapping>
      <servlet-name>DispatcherServlet</servlet-name>
      <url-pattern>/mvc/*</url-pattern>
  </servlet-mapping>

  <filter>
       <filter-name>ReportDownloadFilter</filter-name>
       <filter-class>com.firstbest.report.ReportDownloadFilter</filter-class>
    </filter>
    <filter-mapping>
       <filter-name>ReportDownloadFilter</filter-name>
     <url-pattern>/mvc/*</url-pattern>
    </filter-mapping>

  <!-- =================================================================== -->
  <!-- Flex Configuration                                                  -->
  <!-- =================================================================== -->
  <context-param>
    <param-name>flex.class.path</param-name>
    <param-value>/WEB-INF/flex/hotfixes,/WEB-INF/flex/jars</param-value>
  </context-param>

  <!-- Http Flex Session attribute and binding listener support -->
  <listener>
    <listener-class>flex.messaging.HttpFlexSession</listener-class>
  </listener>

  <!-- MessageBroker Servlet -->
  <servlet>
    <display-name>MessageBroker Servlet</display-name>
    <servlet-name>MessageBrokerServlet</servlet-name>
    <servlet-class>flex.messaging.MessageBrokerServlet</servlet-class>
    <init-param>
      <param-name>services.configuration.file</param-name>
      <param-value>/WEB-INF/flex/services-config.xml</param-value>
    </init-param>
    <init-param>
      <param-name>flex.write.path</param-name>
      <param-value>/WEB-INF/flex</param-value>
    </init-param>
    <!-- start MessageBroker last since it uses Spring beans -->
    <load-on-startup>4</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>MessageBrokerServlet</servlet-name>
    <url-pattern>/messagebroker/*</url-pattern>
  </servlet-mapping>

  <!-- Attachment Download Servlet -->
  <servlet>
    <servlet-name>AttachmentDownloadServlet</servlet-name>
    <servlet-class>com.firstbest.attachment.AttachmentDownloadServlet</servlet-class>
    <load-on-startup>3</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>AttachmentDownloadServlet</servlet-name>
    <url-pattern>/attachments/*</url-pattern>
  </servlet-mapping>

  <!-- Custom Error Page -->
  <error-page>
    <exception-type>java.lang.Throwable</exception-type>
    <location>/HTTPError.html</location>
  </error-page>
  <error-page>
    <error-code>404</error-code>
    <location>/HTTPError.html</location>
  </error-page>

  <resource-ref>
    <description>FBS main datasource</description>
    <!--
       - The value of res-ref-name must match the value
       - of the mainDataSourceJndiName bean with
       - the "java:comp/env/" prefix removed.
      -->
    <res-ref-name>fbsMainDataSource</res-ref-name>
    <res-type>javax.sql.DataSource</res-type>
    <res-auth>Container</res-auth>
  </resource-ref>

  <!--
     - If the analysisDataSourceJndiName bean has a different
     - value than the mainDataSourceJndiName bean, a resource-ref
     - tag for analysisDataSourceJndiName bean must be defined.
    -->
</web-app>

[jaikiran]

Hmm, so it's a web-app 2.4 version. So annotation scanning will anyway be disabled by default (assuming there's no bug in there). How many jar files do you have in WEB-INF/lib and what's the total size of those jar files?

[tmuldo]

# of jar files: 222

total size of jar files: 207 MB

[jaikiran]

Tom Muldoon wrote:

# of jar files: 222

total size of jar files: 207 MB

Wow, that's a lot! You are definitely running into this https://issues.jboss.org/browse/JBAS-8786 issue. Is the workaround mentioned in that JIRA an option for you?

[tmuldo]

Yeah, it's a rather large application. While I haven't worked with EARs before, I don't mind trying the workaround mentioned in the jira. i.e.

    "the only workaround I see is to repack my war files into ear and copy most of libraries to ear/lib folder".

Stay tuned...

[tmuldo]

A colleague suggested moving my application jars from WEB-INF/lib to {jboss.home.dir}/server/default/lib to get around the unpacking and scanning issue and while I have classloader issues yet to resolve, initial results regarding the startup time are positive (JBoss now takes just a couple of minutes to start).

With that said, my classloader issues (due to the fact that JBoss 6 is classloading a later version of Hibernate (3.6) than is supported by my application) are documented here...

    http://community.jboss.org/message/589284#589284

Tom

PS. I have not yet tried repacking my war file in an ear.

[jaikiran]

Tom Muldoon wrote:

A colleague suggested moving my application jars from WEB-INF/lib to {jboss.home.dir}/server/default/lib to get around the unpacking and scanning issue ...

initial results regarding the startup time are positive (JBoss now takes just a couple of minutes to start).

Yes, that's definitely another workaround for this issue

[jaikiran]

This issue in now fixed in AS6 trunk and is available in the nightly builds http://community.jboss.org/thread/161386. Please give it a try and see if it fixes the problem.