9 Replies Latest reply on Sep 26, 2017 10:24 AM by claudio4j

Wildfly 10.1Final Manager interface will not show deployments

gturek Sep 13, 2017 9:42 PM

Hello, I have 2 instances of Wildfly 10.1Final (actually 10 patched to 10.1). Even though I can deploy my war file using jenkins and maven, when I log into the

management web app it does not show me the deployed war files (I have 2). The error it gives me in the interface is "Unable to load deployments".

However the error that shows in the logs is as follows:

2017-09-14 13:34:10,588 ERROR [org.jboss.as.controller.management-operation] (External Management R equest Threads -- 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([ ("deployment" => "riskscapemr_rest.war"), ("subsystem" => "jpa"), ("hibernate-persistence-unit" => "riskscapemr_rest.war#riskscapemr"), ("entity-cache" => "nz.org.riskscape.archive.entity.Module") ]): java.lang.NullPointerException at org.hibernate.stat.internal.ConcurrentStatisticsImpl.getSecondLevelCacheStatistics(Concu rrentStatisticsImpl.java:306) at org.hibernate.stat.internal.ConcurrentStatisticsImpl.getSecondLevelCacheStatistics(Concu rrentStatisticsImpl.java:30) at org.jboss.as.jpa.hibernate5.management.HibernateEntityCacheStatistics.getStatistics(Hibe rnateEntityCacheStatistics.java:94) at org.jboss.as.jpa.hibernate5.management.HibernateEntityCacheStatistics$4.invoke(Hibernate EntityCacheStatistics.java:126) at org.jboss.as.jpa.hibernate5.management.HibernateAbstractStatistics.getValue(HibernateAbstractStatistics.java:99) at org.jboss.as.jpa.management.ManagementResourceDefinition$1.handle(ManagementResourceDefinition.java:130) at org.jboss.as.jpa.management.ManagementResourceDefinition$AbstractMetricsHandler.executeRuntimeStep(ManagementResourceDefinition.java:268) at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:53) at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890) at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329) at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400) at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:222) at org.jboss.as.domain.http.server.DomainApiHandler.handleRequest(DomainApiHandler.java:219) at io.undertow.server.handlers.encoding.EncodingHandler.handleRequest(EncodingHandler.java:72) at org.jboss.as.domain.http.server.security.SubjectDoAsHandler$1.run(SubjectDoAsHandler.java:72) at org.jboss.as.domain.http.server.security.SubjectDoAsHandler$1.run(SubjectDoAsHandler.java:68) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:149) at org.jboss.as.domain.http.server.security.SubjectDoAsHandler.handleRequest(SubjectDoAsHandler.java:68) at org.jboss.as.domain.http.server.security.SubjectDoAsHandler.handleRequest(SubjectDoAsHandler.java:63) at io.undertow.server.handlers.BlockingHandler.handleRequest(BlockingHandler.java:56) at org.jboss.as.domain.http.server.DomainApiCheckHandler.handleRequest(DomainApiCheckHandler.java:95) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at org.jboss.threads.JBossThread.run(JBossThread.java:320) This is only happening on one of the two servers, the other works just fine. Only one of the 2 apps is hibernate dependent, the other is not, but the error prevents both being shown. How can I fix this, the management web app is mostly useless because of this. Thank you

1. Re: Wildfly 10.1Final Manager interface will not show deployments

jaikiran Sep 14, 2017 1:10 AM (in response to gturek)

There appears to be multiple issues in here:

1. The NPE in that hibernate code which seems to be fixed in a subsequent version of Hibernate [HHH-11374] ConcurrentStatisticsImpl#getSecondLevelCacheStatistics() throws NPE if second level cache is not activated -…
2. The fact that whatever management operation that the admin console is invoking for listing the deployments causes the whole page to fail if any of the deployments has an issue. I am not too sure what management operation is being invoked on that admin console page, but perhaps there's a way to handle it better by at least showing the deployments for which the management operation succeeded. @harald.pehl might know more about this part.
Actions
2. Re: Wildfly 10.1Final Manager interface will not show deployments

gturek Sep 14, 2017 4:50 AM (in response to gturek)

I ran across that bug report while searching for that second level cache error message, but it was fixed and the version is hibernate I am using (5.2.4.Final)
has the fix. Otherwise the deployment would fail itself (I would think) but it does not, it works fine, and it does so in my other copies of the same server. It's almost as if something got corrupted somewhere in the management app and now it's permanently "broken".
Actions
3. Re: Wildfly 10.1Final Manager interface will not show deployments

ctomc Sep 14, 2017 5:18 AM (in response to gturek)

disable statistics for jpa subsystem and this should go away.

or upgrade hibernate.
Actions
4. Re: Wildfly 10.1Final Manager interface will not show deployments

claudio4j Sep 14, 2017 9:19 AM (in response to jaikiran)

I will investigate this issue, this problem should not let the web console unusable.
Actions
5. Re: Wildfly 10.1Final Manager interface will not show deployments

claudio4j Sep 14, 2017 1:43 PM (in response to jaikiran)

Can you try hibernate 5.2.11 ? as you are using a different version than bundled with Wildfly, would be worth try a recent version with that bug fixed.

Also, can you try the following command on CLI before hibernate upgrade ?

/host=<change>/server=<change>/deployment=riskscapemr_rest.war/subsystem=jpa/hibernate-persistence-unit=riskscapemr_rest.war#riskscapemr/entity-cache=nz.org.riskscape.archive.entity.Module:read-resource(include-runtime)

Does it log an error on server.log ?
Actions
6. Re: Wildfly 10.1Final Manager interface will not show deployments

gir489 Sep 14, 2017 2:49 PM (in response to claudio4j)

I don't know if this is relevant, but I found that when using something that Wildfly already uses (in my case, Log4J) a lot of collisions can occur. We were including log4j.jar as a lib dependency. We have since switched to a META-INF dependency, and not only is it 100X better, it solved an issue we were having with the console always saying that our Log4J calls were coming from STDOUT. So, I'm guessing it allowed Wildfly's logger to directly link to our project. I'm glad we didn't end up porting to Log4J 2 that we had discussed previously, because it seems that Wildfly only Supports Log4J 1's XML style.

You could instead try to declare org.hibernate as a Dependencies in your META-INF/MANIFEST.MF, and then have Wildfly wire in your dependency that way. Make sure to remove your Hibernate JARs from the deployment, but not your build. We had to keep the log4j.jar we were using in a folder ANT could see during build, but wouldn't grab during the phase it makes the exploded WAR directory.
Actions
7. Re: Wildfly 10.1Final Manager interface will not show deployments

gturek Sep 25, 2017 11:01 PM (in response to claudio4j)

I get

[standalone@localhost:10990 /] /host=<change>/server=<change>/deployment=riskscapemr_rest.war/subsystem=jpa/hibernate-persistence-unit=riskscapemr_rest.war#riskscapemr/entity-cache=nz.org.riskscape.archive.entity.Module:read-resource(include-runtime)
The argument value is not specified for include-runtime: 'null'

And the log shows:

2017-09-26 15:52:10,325 ERROR [org.jboss.as.controller.management-operation] (External Management Request Threads -- 2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
    ("deployment" => "riskscapemr_rest.war"),
    ("subsystem" => "jpa"),
    ("hibernate-persistence-unit" => "riskscapemr_rest.war#riskscapemr"),
    ("entity-cache" => "nz.org.riskscape.archive.entity.Application")
]): java.lang.NullPointerException
    at org.hibernate.stat.internal.ConcurrentStatisticsImpl.getSecondLevelCacheStatistics(ConcurrentStatisticsImpl.java:306)
    at org.hibernate.stat.internal.ConcurrentStatisticsImpl.getSecondLevelCacheStatistics(ConcurrentStatisticsImpl.java:30)
    at org.jboss.as.jpa.hibernate5.management.HibernateEntityCacheStatistics.getStatistics(HibernateEntityCacheStatistics.java:94)
    at org.jboss.as.jpa.hibernate5.management.HibernateEntityCacheStatistics$4.invoke(HibernateEntityCacheStatistics.java:126)
    at org.jboss.as.jpa.hibernate5.management.HibernateAbstractStatistics.getValue(HibernateAbstractStatistics.java:99)
    at org.jboss.as.jpa.management.ManagementResourceDefinition$1.handle(ManagementResourceDefinition.java:130)
    at org.jboss.as.jpa.management.ManagementResourceDefinition$AbstractMetricsHandler.executeRuntimeStep(ManagementResourceDefinition.java:268)
    at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:53)
    at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
    at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
    at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
    at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)
    at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)
    at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:222)

......

I could try a new version of Hibernate, but given that this issue only cropped up on one out of 4 separate installations I don't think this is the issue. Something got corrupted somewhere
Actions
8. Re: Wildfly 10.1Final Manager interface will not show deployments

gturek Sep 25, 2017 11:31 PM (in response to ctomc)

According to my console the jpa subsystem does not have an option for statistics. Transactions does, but it's already off. EJB3 statistics option is set to true.
Actions
9. Re: Wildfly 10.1Final Manager interface will not show deployments

claudio4j Sep 26, 2017 10:24 AM (in response to gturek)

Hi, the JPA statistics is enabled/disable in persistence.xml file packaged in the application.
I second Tomaz, disable jpa statistics or upgrade hibernate.
Actions

Go to original post