[ctomc]

maybe a silly idea but given how simple your deployment is, why not just convert it to war deployment.

since EE6 you can also have EJB jars in your war/WEB-INF/lib and it works just fine.

so you could really simplify deployment and reduce problem surfaces for that.

so what you do is basicly just package all ejb-jars to WEB-INF/lib as well as all jars in current EAR/lib and it should work without any problems.

it would i think also simplfy your workflow.

[jiseb]

Thanks for the tip, we will probably try this (in paralel with waiting for reoccurence with trace logging enabled).

[jiseb]

We have reproduced the issue with trace logging (issue was again with class TransactionContextImpl). Issue is from archive described above (version number has chnaged but it is still an ear containg war and jar; single war file is being prepared now):

2017-06-21 13:39:02,413 TRACE [org.jboss.modules] (default task-404) Finding class cz.company.loom.host.epas.context.TransactionContextImpl from Module "deployment.EpashostHostConnector-ear-1.11.11-host.ear.EpashostHostConnector-war-1.11.11.war:main" from Service Module Loader

2017-06-21 13:39:02,413 TRACE [org.jboss.modules] (default task-404) Finding local class cz.company.loom.host.epas.context.TransactionContextImpl from Module "deployment.EpashostHostConnector-ear-1.11.11-host.ear.EpashostHostConnector-war-1.11.11.war:main" from Service Module Loader

2017-06-21 13:39:02,413 TRACE [org.jboss.modules] (default task-404) Loading class cz.company.loom.host.epas.context.TransactionContextImpl locally from Module "deployment.EpashostHostConnector-ear-1.11.11-host.ear.EpashostHostConnector-war-1.11.11.war:main" from Service Module Loader

2017-06-21 13:39:02,413 TRACE [org.jboss.modules] (default task-404) No local specification found for class cz.company.loom.host.epas.context.TransactionContextImpl in Module "deployment.EpashostHostConnector-ear-1.11.11-host.ear.EpashostHostConnector-war-1.11.11.war:main" from Service Module Loader

2017-06-21 13:39:02,413 TRACE [org.jboss.modules] (default task-404) Class cz.company.loom.host.epas.context.TransactionContextImpl not found from Module "deployment.EpashostHostConnector-ear-1.11.11-host.ear.EpashostHostConnector-war-1.11.11.war:main" from Service Module Loader

...

2017-06-21 13:39:02,417 ERROR [io.undertow.request] (default task-404) UT005023: Exception handling request to /EpasHostConnector/host/xproto: javax.servlet.ServletException: Error processing request

    at cz.company.loom.host.epas.XProtoHostServlet.doPost(XProtoHostServlet.java:320)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl

    at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:140)

    at cz.company.loom.host.epas.XProtoHostServlet.doPost(XProtoHostServlet.java:308)

    ... 29 more

Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpashostHostConnector-ear-1.11.11-host.ear.EpashostHostConnector-war-1.11.11.war:main" from Service Module Loader]

    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)

    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)

    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)

    ... 31 more

[jaikiran]

Can you attach that entire log file to your post? (Click on the "Use advanced editor" link to open up the editor that allows attaching files).

[jiseb]

Complete log attached (it is quite big, it has to be uploaded as a two-part archive) - it starts right after restart of servers after trace logging enablement (we restarted servers to reproduce the issue). There are several servers running on this WF instance but only one of them is having module EpascustomerHostConnector-ear-1.11.11-host.ear. There are also other modules starting with "Epas" across these servers - they are built from the same source and all of them are having these troubles sometimes. All other modules are working fine.

We have also found out that for other similar modules (that are working fine) we have only servlet classes in war archive and all other classes are in jar file. In problematic (Epas*) modules there ary many classes in war archive (and also many in jar file) and the trouble is always with classes from war. Not sure if this could cause such issue but we might try to move non-servlet classes to jar (in case our current try with war instead of ear will not work).

[jiseb]

Tomaz Cerar wrote:

maybe a silly idea but given how simple your deployment is, why not just convert it to war deployment.

...

Unfortunately this didn't help. After a while we have a similar issue again (this time with a different class from the war file):

26.06.2017;13:43:01.140 ERROR [default task-45]  [epas_bnpp ] - cz/company/loom/host/epas/message/state/MessageCompletionStateReverzal [cz.company.loom.host.epas.XProtoHostServlet:252]

java.lang.NoClassDefFoundError: cz/company/loom/host/epas/message/state/MessageCompletionStateReverzal

    at cz.company.loom.host.epas.message.translate.factory.TranslatorFactoryCompletion.createRequestTranslator(TranslatorFactoryCompletion.java:97) ~[classes:na]

   ...

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]

    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.message.state.MessageCompletionStateReverzal from [Module "deployment.EpascustomerHostConnector-war-1.11.11.2-bnpp.war:main" from Service Module Loader]

    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198) ~[na:na]

    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363) ~[na:na]

    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351) ~[na:na]

    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93) ~[na:na]

    ... 38 common frames omitted

[jaikiran]

Before digging any deeper, can you confirm that the class cz.company.loom.host.epas.context.TransactionContextImpl is indeed present in the EpascustomerHostConnector-war-1.11.11.war that's part of the EpascustomerHostConnector-ear-1.11.11-host.ear that is deployed on the server?

Also, can you post the complete contents of your jboss-deployment-structure.xml for this specific .ear?

[jiseb]

Yes, the class is definetely there (also other classes having such issue are there). It started to work correctly after redeploy (of the same package) so it must contain all classes. It looks like some classes are (sometimes)  being lost during deploy - not sure if we can check it somehow in WildFly.

Here is the content of jboss-deployment-structure.xml:

<jboss-deployment-structure>

    <!-- Make sub deployments isolated by default, so they cannot see each others classes without a Class-Path entry -->

    <ear-subdeployments-isolated>false</ear-subdeployments-isolated>

</jboss-deployment-structure>

[jiseb]

I just want to give an update how we solved the issue. It has been working corretly for one month so we believe it is ok now.

We moved all classes (except 2 servlet classes) from war to jar file. Since that moment we haven't seen such issue. I know it is still just a workaround but it might help someone.

Thank you very much for all your hints in this thread -  they helped us to find the way to the solution.