Random error causes users to not be able to login

[magick93]

Hi

We keep getting random errors which result in users not being able to login.

We cannot constantly reproduce this. We are unable to find the cause. And it is a big problem for us.

So far, the only way we have found to solve this is to restart our jboss server.

Does anyone know why we have this problem? And how we fix it?

The error message we get is:

11:16:17,571 INFO  [app.server.service.UserManagementServiceImpl] (http--0.0.0.0-80-83) Logging in username=carl

11:16:17,603 INFO  [stdout] (http--0.0.0.0-80-83) 3:pass

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83) java.lang.RuntimeException: no marshalling definition available for type:models.shared.User_$$_javassist_4

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.marshalling.client.util.MarshallUtil.getMarshaller(MarshallUtil.java:130)

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.marshalling.client.marshallers.ErraiProtocolEnvelopeMarshaller.marshall(ErraiProtocolEnvelopeMarshaller.java:100)

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.marshalling.client.protocols.ErraiProtocol.encodePayload(ErraiProtocol.java:40)

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.marshalling.server.protocol.ErraiProtocolServer.encodePayloadToByteArrayInputStream(ErraiProtocolServer.java:31)

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.io.BufferHelper.encodeAndWrite(BufferHelper.java:38)

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.io.BufferDeliveryHandler.deliver(BufferDeliveryHandler.java:54)

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.MessageQueueImpl.offer(MessageQueueImpl.java:123)

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.ServerMessageBusImpl.enqueueForDelivery(ServerMessageBusImpl.java:411)

11:16:17,805 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.ServerMessageBusImpl.send(ServerMessageBusImpl.java:379)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.ServerMessageBusImpl.send(ServerMessageBusImpl.java:325)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.service.MessageBusProxy.send(MessageBusProxy.java:75)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.client.api.base.ConversationMessageWrapper.sendNowWith(ConversationMessageWrapper.java:238)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.client.api.base.DefaultMessageBuilder$1.sendNowWith(DefaultMessageBuilder.java:84)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.io.ValueReplyRPCEndpointCallback.callback(ValueReplyRPCEndpointCallback.java:22)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.io.RemoteServiceCallback.callback(RemoteServiceCallback.java:54)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.cdi.server.CDIExtensionPoints$3.callback(CDIExtensionPoints.java:499)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.DeliveryPlan.deliver(DeliveryPlan.java:47)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.ServerMessageBusImpl.sendGlobal(ServerMessageBusImpl.java:284)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.SimpleDispatcher.dispatchGlobal(SimpleDispatcher.java:46)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:92)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:109)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.doPost(DefaultBlockingServlet.java:140)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)

11:16:17,821 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)

11:16:17,836 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83)       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83)       at java.lang.Thread.run(Thread.java:722)

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83) *** Message delivery failure ***

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83) Bus: org.jboss.errai.bus.server.ServerMessageBusImpl@737e05

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83) Message: CommandType=login:java.lang.String:java.lang.String:java.lang.Boolean:, ErrorTo=app.client.shared.service.UserManagementService:RPC.login:jav

a.lang.String:java.lang.String:java.lang.Boolean::14:Errors:RPC, ReplyTo=app.client.shared.service.UserManagementService:RPC.login:java.lang.String:java.lang.String:java.lang.Boolean::14:RespondTo:RPC

, ToSubject=app.client.shared.service.UserManagementService:RPC, MethodParms=[carl, pass, true]

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83) errorMessage: Error calling remote service: app.client.shared.service.UserManagementService:RPC

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83) exception: java.lang.RuntimeException: no marshalling definition available for type:com.magick.models.shared.User_$$_javassist_4

11:16:17,852 ERROR [stderr] (http--0.0.0.0-80-83) disconnect: false

[nva]

I have been struggling with a similar problem for quite some time now. Intermittently the authentication response gets 'lost' and it doesn't arrive on the client to complete the process. It happened with various versions of Errai using a variety of configurations. I haven't been able to pinpoint it to a specific error message. Currently there is nothing conclusive in my logs.

[magick93]

Hi Valentin

Well, it is to some degree, good to hear that someone else is having this problem - but it is a concern that you also not able to consistently reproduce this.

This is such a massively critical issue for us. If users cannot login, and the only way to fix this is to restart jboss, we have a really, really big problem.

What security framework are you using?

We are using Apache Shiro. We, at this stage, do not have any specific info on what or where the error occurs, so I cannot say that it is Shiro.

[nva]

My problem is totally intermittent and I have not been able to isolate it ether. It never happens in dev mode. It's a major headache. Am not using any security framework: authentication is implemented with an Errai AuthenticationAdapter.

In my frustration I even plastered the login screen with all sorts of data that's fetched with RPC. All RPC works all the time, except the authentication. Interestingly, when the authentication 'hangs', logging in with an incorrect password usually returns a response right away. Sometimes the successful authentication result arrives at the client with a 45-90 seconds delay.

Perhaps we should rename this discussion 'Errai approved authentication framework' and use something that works for everyone else?

[mbarkley]

Hi Anton,

So looking at your stack trace, the name of the class in the message stands out. I just googled "javassist" and from the first paragraph description I found: "[Javassist] enables Java programs to define a new class at runtime and to modify a class file when the JVM loads it."

Are you (directly or indirectly) using Javassist? If you are that is probably the cause of that stack trace. Errai production marshallers are generated at compile time, so marshalling dynamically created classes isn't possible.

Cheers.

[magick93]

Hi Max

We are not using javaassist, but I  believe weld is. So maybe this is a bug in the errai usage of CDI?

[magick93]

I see errai-commons has a dependency on java assist.

Here are the uses of java assist in our application.

Please note - this error is causing us a world of pain!

[magick93]

Hi @Valentin Nemeth

Do you use hibernate?

[mbarkley]

Hi Anton,

After some investigation, my best hypothesis is that your code is retrieving a User entity from hibernate, but is receiving a lazily-fetched proxy.

To try and figure out if this is the case you could add logging statements for when the User object you get from the entity manager is an instance of HibernateProxy.

Some possible workarounds:

1. Configure the fetch type for the entity to be eager (the proxy is created for lazily fetched entities).
2. Manually unwrap the entity with ((HibernateProxy) object).getHibernateLazyInitializer().getImplementation() (Note: this needs to be done before the entity manager that fetched the object is closed).

If hibernate isn't the cause of your problem, then you need to track down where your User object is being proxied and figure out how to unwrap it. It's also possible that weld is the culprit.

Hope that helps.

[nva]

Hi Anton,

yes, I am using hibernate.

I added debug logging to every single line of my login process. The user lookup, password check, etc. all complete and the response is placed on the bus. From there the response either arrives severely delayed (up to 2 minutes) or doesn't arrive on the client at all. When this happens I see an increased number of threads in blocking and waiting state in jetty (although the max thread limits are not reached). I had also reports of multiplied RPC calls during this period. Ie. user clicks on a button once only, but the server side runs 3 times.

I still haven't been able to isolate the problem further and correlate it with anything significant.

Cheers,

V.