Two persistence.xml in two EJB projects within one EAR don't work

[hades32]

We're in the progress of migrating to Wildfly (from WebSphere) but it doesn't seem to support our current setup.

I created a repro at https://github.com/Hades32/wildfly-issues

The repro in it's current form is simply an EAR containing a JAR in /lib and two EJBs and one WAR each referencing the JAR. The rest of the dependencies is: WAR->EJB1->EJB2.

This works fine as long as no persistence.xml exists. As soon as each EJB gets its own persistence.xml (in our full project they point to different DSs) we're not able to deploy anymore.

Wildfly doesn't seem to be able to resolve dependencies between modules as soon as there are two persistence.xml's. Despite the (repro) project doesn't even use it (yet). If you remove the persistence.xml's the project works fine. From the error (see below) it looks like the services defined in EJB1 try to use the persistence unit defined in EJB2 despite there not being any kind of reference.

...
11:51:12,805 DEBUG [org.hibernate.jpa.internal.EntityManagerFactoryRegistry] (ServerService Thread Pool -- 63) Initializing EntityManagerFactoryRegistry : org.hibernate.jpa.internal.EntityManagerFactoryRegistry@265f9159
11:51:12,806 DEBUG [org.hibernate.jpa.internal.EntityManagerFactoryRegistry] (ServerService Thread Pool -- 63) Registering EntityManagerFactory: com.car2go.server.ear/wftest.server.acre.jar#HW30VserverACREUR
11:51:12,806 DEBUG [org.hibernate.jpa.internal.EntityManagerFactoryRegistry] (ServerService Thread Pool -- 65) Registering EntityManagerFactory: com.car2go.server.ear/wftest.server.acre.jar#HW30VserverACRE
11:51:12,815 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "com.car2go.server.ear")]) - failure description: {"WFLYCTL0180: Services with missing/unavailable dependencies" => [
    "jboss.deployment.unit.\"com.car2go.server.ear\".deploymentCompleteService is missing [jboss.deployment.subunit.\"com.car2go.server.ear\".\"wftest.server.ejb.jar\".deploymentCompleteService, jboss.deployment.subunit.\"com.car2go.server.ear\".\"wftest.server.backend.web.war\".deploymentCompleteService]",
    "jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService.ORB is missing [jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService]",
    "jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService.ValidatorFactory is missing [jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService]",
    "jboss.deployment.subunit.\"com.car2go.server.ear\".\"wftest.server.acre.jar\".component.StationService.START is missing [jboss.persistenceunit.\"com.car2go.server.ear/wftest.server.ejb.jar#HW30Vserver\", jboss.persistenceunit.\"com.car2go.server.ear/wftest.server.ejb.jar#HW30VserverUR\"]",
    "jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService.HandleDelegate is missing [jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService]",
    "jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService.InAppClientContainer is missing [jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService]",
    "jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService.InstanceName is missing [jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService]",
    "jboss.deployment.subunit.\"com.car2go.server.ear\".\"wftest.server.backend.web.war\".INSTALL is missing [jboss.deployment.subunit.\"com.car2go.server.ear\".\"wftest.server.ejb.jar\".deploymentCompleteService]",
    "jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService.Validator is missing [jboss.naming.context.java.comp.\"wftest.server.ear\".\"wftest.server.ejb\".ACREService]"
]}
...
11:51:12,909 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0184:    New missing/unsatisfied dependencies:
      service jboss.deployment.subunit."com.car2go.server.ear"."wftest.server.backend.web.war".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."com.car2go.server.ear".deploymentCompleteService]
      service jboss.deployment.subunit."com.car2go.server.ear"."wftest.server.ejb.jar".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."com.car2go.server.ear".deploymentCompleteService, service jboss.deployment.subunit."com.car2go.server.ear"."wftest.server.backend.web.war".INSTALL]
      service jboss.naming.context.java.comp."wftest.server.ear"."wftest.server.ejb".ACREService (missing) dependents: [service jboss.naming.context.java.comp."wftest.server.ear"."wftest.server.ejb".ACREService.Validator, service jboss.naming.context.java.comp."wftest.server.ear"."wftest.server.ejb".ACREService.InstanceName, service jboss.naming.context.java.comp."wftest.server.ear"."wftest.server.ejb".ACREService.ORB, service jboss.naming.context.java.comp."wftest.server.ear"."wftest.server.ejb".ACREService.ValidatorFactory, WFLYCTL0208: ... and 3 more ]
      service jboss.persistenceunit."com.car2go.server.ear/wftest.server.ejb.jar#HW30Vserver" (missing) dependents: [service jboss.deployment.subunit."com.car2go.server.ear"."wftest.server.acre.jar".component.StationService.START]
      service jboss.persistenceunit."com.car2go.server.ear/wftest.server.ejb.jar#HW30VserverUR" (missing) dependents: [service jboss.deployment.subunit."com.car2go.server.ear"."wftest.server.acre.jar".component.StationService.START]
...

This was tested with Wildfly 9.0 Final

[smarlow]

Tough to follow the failure to the cause.  I enabled TRACE logging (org.jboss.as.jpa), output is here.

It might help to divide and conquer the test case in a different way.  I wonder if the error would reproduce if the below components are removed.  I'm not sure that this would help but might reduce the test case a little further.  From the TRACE output, the components are:

2015-07-23 08:52:27,738 DEBUG [org.jboss.as.jpa] (MSC service thread 1-3) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACREUR" for component wftest.server.ejb.acre.ACREService

2015-07-23 08:52:27,738 DEBUG [org.jboss.as.jpa] (MSC service thread 1-2) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACREUR" for component wftest.server.acre.station.StationService

2015-07-23 08:52:27,738 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACREUR" for component javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-2) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE" for component wftest.server.acre.station.StationService

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACREUR" for component javax.servlet.jsp.jstl.tlv.ScriptFreeTLV

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-3) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE" for component wftest.server.ejb.acre.ACREService

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-2) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver" for component wftest.server.acre.station.StationService

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACREUR" for component javax.faces.webapp.FacetTag

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-3) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver" for component wftest.server.ejb.acre.ACREService

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-2) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30VserverUR" for component wftest.server.acre.station.StationService

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACREUR" for component wftest.server.backend.vehicle.VehicleResource

2015-07-23 08:52:27,739 DEBUG [org.jboss.as.jpa] (MSC service thread 1-3) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30VserverUR" for component wftest.server.ejb.acre.ACREService

2015-07-23 08:52:27,740 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACREUR" for component com.sun.faces.config.ConfigureListener

2015-07-23 08:52:27,740 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE" for component javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV

2015-07-23 08:52:27,740 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE" for component javax.servlet.jsp.jstl.tlv.ScriptFreeTLV

2015-07-23 08:52:27,740 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE" for component javax.faces.webapp.FacetTag

2015-07-23 08:52:27,740 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE" for component wftest.server.backend.vehicle.VehicleResource

2015-07-23 08:52:27,740 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE" for component com.sun.faces.config.ConfigureListener

2015-07-23 08:52:27,740 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver" for component javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV

2015-07-23 08:52:27,741 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver" for component javax.servlet.jsp.jstl.tlv.ScriptFreeTLV

2015-07-23 08:52:27,741 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver" for component javax.faces.webapp.FacetTag

2015-07-23 08:52:27,741 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver" for component wftest.server.backend.vehicle.VehicleResource

2015-07-23 08:52:27,741 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver" for component com.sun.faces.config.ConfigureListener

2015-07-23 08:52:27,741 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30VserverUR" for component javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV

2015-07-23 08:52:27,741 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30VserverUR" for component javax.servlet.jsp.jstl.tlv.ScriptFreeTLV

2015-07-23 08:52:27,741 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30VserverUR" for component javax.faces.webapp.FacetTag

2015-07-23 08:52:27,742 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30VserverUR" for component wftest.server.backend.vehicle.VehicleResource

2015-07-23 08:52:27,742 DEBUG [org.jboss.as.jpa] (MSC service thread 1-7) Adding dependency on PU service service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30VserverUR" for component com.sun.faces.config.ConfigureListener

[hades32]

Honestly I don't even undestand the first line of your log "wftest.server.acre.jar#HW30VserverACREUR" for component wftest.server.ejb.acre.ACREService":

ACREService lives in the ...ejb.jar (sorry for the confusing naming - it's an extract from the original project) and it doesn't (explicitly?) reference the give PU.

Why would this ever happen?

[smarlow]

Internally, we add dependencies between each application EE component and the persistence unit, so that the components are not deployed until after the JPA persistence unit is deployed (since multiple deployment threads can be used).

You mentioned that the deployment failure goes away if you remove the persistence.xml files.  Have you tried only removing the HW30VserverUR persistence unit?

[hades32]

It didn't help, but of course the errors became fewer and I spotted this:

root@vagrant-ubuntu-trusty-64:/vagrant_data/wildfly-issues# grep '#' log
19:49:32,813 INFO  [org.jboss.as.jpa] (ServerService Thread Pool -- 58) WFLYJPA0010: Starting Persistence Unit (phase 1 of 2) Service 'wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE'
19:49:32,815 INFO  [org.jboss.as.jpa] (ServerService Thread Pool -- 59) WFLYJPA0010: Starting Persistence Unit (phase 1 of 2) Service 'wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver'
19:49:34,883 INFO  [org.jboss.as.jpa] (ServerService Thread Pool -- 58) WFLYJPA0010: Starting Persistence Unit (phase 2 of 2) Service 'wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.acre.jar#HW30VserverACRE'
    "jboss.deployment.unit.\"wftest.server.ear-1.31.0-SNAPSHOT.ear\".WeldStartService is missing [jboss.persistenceunit.\"wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver\"]",
    "jboss.deployment.subunit.\"wftest.server.ear-1.31.0-SNAPSHOT.ear\".\"wftest.server.acre.jar\".component.StationService.START is missing [jboss.persistenceunit.\"wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver\"]",
      service jboss.persistenceunit."wftest.server.ear-1.31.0-SNAPSHOT.ear/wftest.server.ejb.jar#HW30Vserver" (missing) dependents: [service jboss.deployment.subunit."wftest.server.ear-1.31.0-SNAPSHOT.ear"."wftest.server.acre.jar".component.StationService.START, service jboss.deployment.unit."wftest.server.ear-1.31.0-SNAPSHOT.ear".WeldStartService]

The phrase Starting Persistence Unit (phase 2 of 2) is only apearing for one of the PUs... That might be the culprint I guess.

[hades32]

Scott Marlow schrieb:

Internally, we add dependencies between each application EE component and the persistence unit, so that the components are not deployed until after the JPA persistence unit is deployed (since multiple deployment threads can be used).

Reading this a second time I wonder: Doesn't this perfectly explain the behavior, and my setup can never work that way? (In WildFly at least)

Your internal logic creates a circular reference as soon as there are to modules containing persistence units... Or did I get this wrong?

After this conclusion and after I found this link (

http://stackoverflow.com/questions/8370043/configure-persistence-units-to-be-available-in-several-jars-of-an-ear ), I created two JARs in /lib each containing one persistence.xml. This worked! I still have to test though, if it works for the full application.

And I still don't know if I'm doing it wrong or it may be a bug

[smarlow]

Could you please push these changes that simplify the test case but still see the error. 

[hades32]

I made a separate branch:

https://github.com/Hades32/wildfly-issues/tree/reduced_error

I tried it now with the full application and it works! (apart from some unrelated and more obvious errors)

So... Is this a bug?

[smarlow]

I don't really know the cause, from the error message that you posted, it sounds like the following didn't start as expected:

component.StationService.START

service jboss.deployment.unit."wftest.server.ear-1.31.0-SNAPSHOT.ear".WeldStartService]

I would like to better understand why.

[hades32]

Well, I'm not sure how to make the test case any simpler...

Did you ever have a working EAR with multiple persistence.xml in EJB JARs? Or this simply a too exotic setup that we have here...?

[smarlow]

Well, clearly, you are hitting on some type of bug here. 

Well, I'm not sure how to make the test case any simpler...

Maybe try removing some of the application code but still have two EJB jars in the EAR (with all four persistence units active).

[hades32]

I have created an issue

https://issues.jboss.org/browse/WFLY-5042