IllegalStateException in EAR Deployment

[ftg314159]

I'm running into a strange case.  I had an EJB app which I built under Ant; the EAR deploys just fine in both 5.1.GA and 6.0.Final.  I switched to building with the maven-ear-plugin, and although the EAR structure looks virtually identical, I now get a ClassNotFoundException trying to deploy.

In the old case, the EAR looks like this:

   (ejb-app).jar

   derby-ds.xml

   META-INF

      application.xml (contains single <module><ejb></ejb></module> for (ejb-app).jar)

      jboss-app.xml (contains single <module><service></service></module> for derby-ds.xml

      MANIFEST.MF (default)

   lib

      util1.jar

      util2.jar

In the new case, the EAR looks like this:

   (ejb-app).jar

   util1.jar

   util2.jar

   derby-ds.xml

   META-INF

      application.xml (contains single <module><ejb></ejb></module> for (ejb-app).jar)

      jboss-app.xml (contains single <module><service></service></module> for derby-ds.xml

      MANIFEST.MF (default)

and the deployment fails with a ClassNotFoundException for a class in util1.jar:

         Deployment "vfs:///data/jboss/jboss-6.0.0.Final/server/default/deploy/dvd-ear-1.0-SNAPSHOT.ear" is in error due to the following reason(s): java.lang.ClassNotFoundException: org.profsoftsvcs.log.LogLogEntry from BaseClassLoader@136a2062{vfs:///data/jboss/jboss-6.0.0.Final/server/default/deploy/dvd-ear-1.0-SNAPSHOT.ear}

I tried several manual experiments:

   1) Move util1.jar and util2.jar to a /lib directory

       Result: JBoss complains that there is no module definition for lib/util1.jar and lib/util2.jar:

         Deployment "vfs:///data/jboss/jboss-6.0.0.Final/server/default/deploy/dvd-ear-1.0-SNAPSHOT.ear" is in error due to the following reason(s): org.jboss.deployers.spi.DeploymentException: No module with relative path lib/ejb-ejb-1.0-SNAPSHOT.jar found  in set of modules for vfs:///data/jboss/jboss-6.0.0.Final/server/default/deploy/dvd-ear-1.0-SNAPSHOT.ear [dvd-ejb-1.0-SNAPSHOT.jar, derby-ds.xml]

   2) Add <module><java></java></module> entries to application.xml for uti1.jar and util2.jar

       Result: JBoss complains that a Context already exists.  Google found this error, but only as a cleanup issue when a prior error is encountered;

                  however, here there is no prior error, just the "Context already exists":

         Caused by: java.lang.IllegalStateException: Context already exists with path 'dvd-ejb-1.0-SNAPSHOT.jar' contexts=[ContextInfoImpl{path= metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=common-java-1.0-SNAPSHOT.jar}, ClassPathEntryImpl{path=dvd-ejb-1.0-SNAPSHOT.jar}, ClassPathEntryImpl{path=ejb-ejb-1.0-SNAPSHOT.jar}, ClassPathEntryImpl{path=}] comparator=org.jboss.system.deployers.LegacyDeploymentContextComparator}, ContextInfoImpl{path=dvd-ejb-1.0-SNAPSHOT.jar metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=dvd-ejb-1.0-SNAPSHOT.jar}]}, ContextInfoImpl{path=ejb-ejb-1.0-SNAPSHOT.jar metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=ejb-ejb-1.0-SNAPSHOT.jar}]}, ContextInfoImpl{path=derby-ds.xml metaData=[] classPath=null}]

   3) Add <library-directory>/</library-directory> to jboss-app.xml

       Result: same as in (2)

I'd love to turn on debug-level logging, but 6.0.0 doesn't appear to have a log4j config file any more.

What I'm trying to do is create an EAR which is as self-contained as possible, i.e. contains its own DataSource definitions, private versions of its utility JARs, etc.  I could probably get this to work by copying the utilX JARs to /lib or /common/lib or ...server/XXX/lib, but that would defeat the purpose.  The goal is to have the user drop the EAR into the deploy directory - period.

Any idea what there is about the new EAR that causes the problem ?

[chazware]

In the maven generated EAR's MANIFEST.INF, were the util jar files listed in the classpath definition line?

[ftg314159]

Ah, yes, I should have mentioned that I tried it both ways, i.e. with and without:

<archive>
    <manifest>
         <addClasspath>true</addClasspath>
    </manifest>
</archive>

It had no effect.

[ashettleroe]

Have you tried to add <defaultLibBundleDir>lib</defaultLibBundleDir> to the configuration for maven-ear-plugin?  This places my non-enterprise modules into a lib directory within the ear.

[ftg314159]

That's good to know; I had seen that tag in the docs, but didn't recognize it for what it did.

From what I can see, it places util1.jar and util2.jar in a /lib directory as claimed, and I would have expected the same result I got in case (1) when I did the same thing manually.  However, what I got was the IllegalStateException mentioned in case (2):

Caused by: java.lang.IllegalStateException: Context already exists with path 'dvd-ejb-1.0-SNAPSHOT.jar' contexts=[ContextInfoImpl{path= metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=dvd-ejb-1.0-SNAPSHOT.jar}, ClassPathEntryImpl{path=}] comparator=org.jboss.system.deployers.LegacyDeploymentContextComparator}, ContextInfoImpl{path=dvd-ejb-1.0-SNAPSHOT.jar metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=dvd-ejb-1.0-SNAPSHOT.jar}]}, ContextInfoImpl{path=derby-ds.xml metaData=[] classPath=null}]

**but*** this time there is no mention of util1/util2 (ejb-ejb and common-java) as there was in case (2), perhaps because the Class-Path in the EAR MANIFEST.MF is adjusted for the change:

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: ftg
Build-Jdk: 1.6.0_22
Class-Path: dvd-ejb-1.0-SNAPSHOT.jar lib/ejb-ejb-1.0-SNAPSHOT.jar lib/
common-java-1.0-SNAPSHOT.jar

Something seems to be wrong with dvd-ejb, since in the original working case it was in the root of the EAR as well.  I'll poke around some more...

[ashettleroe]

In my ear, which is similiar to yours with one ejb jar and one war file, I don't have the Class-Path defined in the manifest.  I don't think you need it if you place the files in the correct spot, as in the old case of your ear file, which I'm hoping the defaultLibBundleDir will help maven create for you.  The class-path maybe being picked up by JBoss and it will deploy the ejb twice, once because it was in the top of the ear and again since it was mentioned in the class-path.  Though I'm guessing here.  Another place JBoss will auto-discovery enterprise mods is the lib directory (disregarding the application.xml), so ensure your ejb jar is only added once to your ear (at the top level.)

I actually setup my pom.xml to create the application.xml from the maven-ear-plugin config as well.

<configuration>
  <finalName>myejb</finalName>
  <defaultLibBundleDir>lib</defaultLibBundleDir>
  <modules>
    <ejbModule>
      <groupId>com.company.myejb</groupId>
      <artifactId>myejb</artifactId>
      <bundleFileName>myejb.jar</bundleFileName>
    </ejbModule>
    <webModule>
      <groupId>com.company.myejb</groupId>
      <artifactId>mywar</artifactId>
      <bundleFileName>mywar.war</bundleFileName>
    </webModule>
  </modules>
  <version>5</version>
</configuration>

[ftg314159]

Good guess, but it still has no effect.  I've ensured that the ejb-jar is only in the root, and I eliminated manifest CLASS-Path: from both the EAR and EJB, but I still get the same error.

However, I think you're on the right track.  I found http://community.jboss.org/thread/2325 where the poster had the same error and worked around it by moving his persistence.xml to a JAR of its own.  My persistence.xml is in the META-INF directory of dvd-ejb (the JAR that's being complained about), so it could be that dvd-ejb is getting deployed once for the EJB classes and once for the persistence.xml.  But that was the case with the Ant build as well.

Or maybe it's your use of finalName and bundleFileName that makes the difference ?  Here's my two POMs that build the EAR file:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <!--

    PSSEJBEARProject is a master project template for all of the Professional
    Software Services EAR creation, and contains elements that all of them
    should inherit.  We inherit from PSSProject.

  -->

  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.profsoftsvcs</groupId>
    <artifactId>PSSProject</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>
  <groupId>org.profsoftsvcs</groupId>
  <artifactId>PSSEJBEARProject</artifactId>
  <packaging>pom</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>PSSEJBEARProject</name>
  <url>http://maven.apache.org</url>

  <build>
    <plugins>

      <!--
        Disable compilation - not needed here.
      -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <executions>
          <execution>
            <phase>none</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>

      </plugin>

      <!--
        Our custom plugins.
      -->
      <plugin>
        <groupId>org.profsoftsvcs</groupId>
        <artifactId>maven-plugins</artifactId>
        <version>1.0-SNAPSHOT</version>
        <executions>

          <!--
            Disable message consolidation - not needed here.
          -->
          <execution>
            <id>consolidate-messages</id>
            <phase>none</phase>
            <goals>
              <goal>consolidate-messages</goal>
            </goals>
          </execution>

          <!--
            Disable deployment descriptor processing - not needed here.
          -->
          <execution>
            <id>create-ejb-descriptor</id>
            <phase>none</phase>
            <goals>
              <goal>create-ejb-descriptor</goal>
            </goals>
          </execution>

        </executions>

      </plugin>

      <!--
        Specifications for the generation of the EAR file.
      -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-ear-plugin</artifactId>
        <version>2.4.2</version>
        <configuration>

          <!--
            Generate an application.xml file.
          -->
          <generateApplicationXml>true</generateApplicationXml>

          <!--
            Put JARs in /lib rather than /.
          -->
          <defaultLibBundleDir>lib</defaultLibBundleDir>
         
          <!--
            JARs which should be included in this EAR.  Everyone
            gets common-java, and ejb-ejb.
          -->
          <modules>
            <jarModule>
              <groupId>org.profsoftsvcs</groupId>
              <artifactId>ejb-ejb</artifactId>
              <includeInApplicationXml>false</includeInApplicationXml>
            </jarModule>
            <jarModule>
              <groupId>org.profsoftsvcs</groupId>
              <artifactId>common-java</artifactId>
              <includeInApplicationXml>false</includeInApplicationXml>
            </jarModule>
          </modules>

          <!--
            The JBoss-specific configuration.  Including this for
            other application servers won't hurt.
          -->
          <jboss>

            <version>5</version>

            <!--
              The JBoss default library directory within the EAR is
              /lib, but the plugin puts all jarModules into the root.
            -->
            <library-directory>/</library-directory>

            <!--
              The DataSources that should be specific to this EAR.
              These will not be visible to other applications in the
              container, and prevent any dependency on our EAR's part
              on global DataSources configured for the entire container.
            -->
            <data-sources>
              <data-source>derby-ds.xml</data-source>
            </data-sources>

          </jboss>

        </configuration>
      </plugin>

    </plugins>

  </build>

  <description>The Professional Software Services "PSSEJBEAR" project is a template for projects which build EARs</description>

</project>

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <!--

    This project builds the EJB JARFile for the PSS "dvd" project.

  -->

  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.profsoftsvcs</groupId>
    <artifactId>PSSEJBEARProject</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>
  <groupId>org.profsoftsvcs</groupId>
  <artifactId>dvd-ear</artifactId>
  <packaging>ear</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>dvd-ear</name>
  <url>http://maven.apache.org</url>

  <build>
    <plugins>

      <!--
        Additions and overrides for the generation of the EAR file.
      -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-ear-plugin</artifactId>
        <configuration>

          <!--
            Specify our EJB JARFile.
          -->
          <modules>
            <ejbModule>
              <groupId>org.profsoftsvcs</groupId>
              <artifactId>dvd-ejb</artifactId>
            </ejbModule>
          </modules>

        </configuration>
      </plugin>

    </plugins>
  </build>

  <dependencies>

    <dependency>
      <groupId>org.profsoftsvcs</groupId>
      <artifactId>dvd-ejb</artifactId>
      <version>1.0-SNAPSHOT</version>
      <type>ejb</type>
    </dependency>

  </dependencies>

  <description>The Professional Software Services "dvd-ear" project creates the EARFile for the "dvd" project</description>

</project>

I checked the source code for org.jboss.deployers.plugins.structure.StructureMetaDataImpl.addContext(StructureMetaDataImpl.java:59) and it's not really clear to me why this is worth throwing an exception over.  I'll try running JBoss under a debugger, and see where the original and the duplicate call to this module come from...

[ftg314159]

Well, so much for that idea.  The 6.0.0.Final source download contains nothing under org.jboss.deployers except org.jboss.deployers.managed.

[jaikiran]

I'm a bit confused about what issue is being discussed. Is there some ClassNotFoundException being thrown? Can you post the entire exception stacktrace (whichever it is)?

[ftg314159]

As it has developed, there are two issues here:

1) JBoss defaults to looking for adjunct JARs in /lib only.  The Maven ear plugin seems to think that the root is the default location for such JARs, and several texts, e.g. "JBoss in Action" seem to agree.  This can be fixed either by directing Maven to use /lib or diirecting JBoss to look in the root instead of /lib.  Minor issue.  This was the original cause of the ClassNotFoundException.

2) In deploying a Maven-built EAR, JBoss is attempting to add a context which has a path that is already represented in one of the existing contexts, and throwing an IllegalStateException as a result.  I can't tell whether this is a valid error, and if so what is causing it at the packaging level.  A manually-built,, i.e. without Maven, EAR with virtually identical structure deploys correctly.  This is the major issue.

There's also a side issue that the 6.0.0.Final source download does not appear to contain all of the org.jboss classes executing in the container.

[ftg314159]

I'm in the process of building the SVN trunk version.  If it works, then it's a bug that's been fixed.  If it still fails, I will have a complete source set to use with a debugger and can look into this further.

[ftg314159]

I located the source for the module throwing the IllegalStateException, and in the SVN trunk the exception has been replaced with the following code:

      // Ignore duplicates

      if (deployers.contains(wrapper))

         return;

So I guess I was not alone in wondering why this condition warranted an exception :-)

[ftg314159]

Well, if this was a bug, the fix has apparently not percolated to all deployers.  In the SVN snapshot version, I get:

Caused by: java.lang.IllegalStateException: Context already exists with path 'dvd-ejb-1.0-SNAPSHOT.jar' contexts=[ContextInfoImpl{path= metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=dvd-ejb-1.0-SNAPSHOT.jar}, ClassPathEntryImpl{path=}] comparator=org.jboss.system.deployers.LegacyDeploymentContextComparator}, ContextInfoImpl{path=dvd-ejb-1.0-SNAPSHOT.jar metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=dvd-ejb-1.0-SNAPSHOT.jar}]}, ContextInfoImpl{path=derby-ds.xml metaData=[] classPath=null}]

    at org.jboss.deployers.plugins.structure.StructureMetaDataImpl.addContext(StructureMetaDataImpl.java:59) [:2.2.0.GA]

which has the same code that was removed from the class which was throwing the original exception.

Please, jaikiran pai, as you seem to be the senior mage in attendance :-) , could you give an opinion as to whether this is an endemic bug in multiple deployers that is in the process of being fixed, or whether the maintainer for the original class just got tired of the exceptions and decided to ignore the condition ?  If the former, I can just sit and wait, although if you could point me to any relevant JIRA reports I'd appreciate it, so I can dig up a workaround if one exists.  If the latter, I'll be happy to enter JIRA issues for all deployers in which I can find this condition, but I don't want to assume that anyone throwing an IllegalStateException with the word "duplicate" in it is an offender --- it would help if I knew what caused the code change in the original module and whether there was any developer discussion about it.

Thanks.

[jaikiran]

I would like to help, but so far I haven't yet understood what issue you are running into Can you please attach a sample application which reproduces this issue and/or post the entire exception stacktrace and the relevant details about the issue?

[ftg314159]

To summarize, deploying my EAR causes an org.jboss.deployers.spi.DeploymentException: Exception determining structure.  The stacktrace is quite large, but the root cause is:

Caused by: java.lang.IllegalStateException: Context already exists with path 'dvd-ejb-1.0-SNAPSHOT.jar' contexts=[ContextInfoImpl{path= metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=dvd-ejb-1.0-SNAPSHOT.jar}, ClassPathEntryImpl{path=}] comparator=org.jboss.system.deployers.LegacyDeploymentContextComparator}, ContextInfoImpl{path=dvd-ejb-1.0-SNAPSHOT.jar metaData=[META-INF - DEFAULT] classPath=[ClassPathEntryImpl{path=dvd-ejb-1.0-SNAPSHOT.jar}]}, ContextInfoImpl{path=derby-ds.xml metaData=[] classPath=null}]

    at org.jboss.deployers.plugins.structure.StructureMetaDataImpl.addContext(StructureMetaDataImpl.java:59) [:2.2.0.GA]

    at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.determineStructure(VFSStructuralDeployersImpl.java:164) [:2.2.0.GA]

    at org.jboss.deployers.vfs.spi.structure.StructureContext.determineChildStructure(StructureContext.java:294) [:2.2.0.GA]

    at org.jboss.deployment.EARStructure.doDetermineStructure(EARStructure.java:276) [:6.0.1-SNAPSHOT]

    ... 32 more

My comments above were a bit confused because for some reason I thought the initial IllegalStateException I received was from org.jboss.deployers.plugins.deployers.DeployersImpl, but it appears that I had it confused with org.jboss.deployers.plugins.structure.StructureMetaDataImpl.  DeployersImpl has the code quoted above which ignores duplicates, but StructureMetaDataImpl checks for duplicates and throws the IllegalStateException. 

Stopping in StructureMetaDataImpl.addContext() with a debugger, I see calls that add a single empty "" path to 4 separate objects, but one of them also gets calls to add "dvd-ejb-1.0-SNAPSHOT.jar", "derby-ds.xml", and then "dvd-ejb-1.0-SNAPSHOT.jar" again, which causes the exception.  The call stack for both of the additions of "dvd-ejb-1.0-SNAPSHOT.jar" is identical.  I'll attach the EAR...