12 Replies Latest reply on Mar 27, 2008 6:13 AM by brian.stansberry

    ava.lang.ClassCastException:     java.lang.ClassNotFoundExceptio

    northgorky

      reetings.

      This starts as a database error in an EJB (a new Database Table can't be
      found -- I messed up in my Database somewhere). But the then the JCache
      gets into some kind of weird state, and you can't get ANYTHING out of
      it, taking down the entire application. Get ClassCastExceptions and
      ClassNotFoundExceptions on any Cache Access after this point (some are
      attached). The ONLY way I have found to recover so far is to
      cold start JBoss, as all access from all clients are failing at this point.

      This also happens when a Servlet exception occurs when the Cache is
      being accessed (and/or updated). I can generate and post the Servlet
      crash stack trace as well.

      Oh yeah, versions;
      Jboss-4.2.2.GA
      JCache: 1.4.1.SP5
      JDK: "1.5.0_11

      steve@system:/usr/local/java/jboss-4.2.2.GA> java -jar
      server/hrx/lib/jboss-cache-jdk50.jar

      Version: 1.4.1.SP5
      Codename: Cayenne
      CVS: $Id: Version.java 4510 2007-09-26 18:45:19Z
      manik.surtani@jboss.com $
      History: (see http://jira.jboss.com/jira/browse/JBCACHE for details)
      steve@system:/usr/local/java/jboss-4.2.2.GA> java -version
      java version "1.5.0_11"
      Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
      Java HotSpot(TM) Client VM (build 1.5.0_11-b03, mixed mode, sharing)

      Due to an application constraint, only one Thread would be accessing
      (Read or Write, and mostly read) any given node at any time. So I've
      reduced locking as much as possible (I hope).

      Stacktrace can be found at;
      http://j2eeguys.com/~steve/JBoss/JCacheCrash.txt

      Current config file can be found at;
      http://j2eeguys.com/~steve/JBoss/hrx-session-cache-service.xml

      Let me know if you need more details. Log is attached. Not sure what
      else is needed. I can even be persuaded to come out to Atlanta, if that
      would be helpful.

      Regards,
      Steve

        • 1. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
          northgorky

          Sorry about the 'trimming'. I was not paying as much attention as I should have been, and just to make things more interesting, the forum seems to be trimming long titles as well.

          -Steve

          • 2. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
            manik

             

            "northgorky" wrote:

            This starts as a database error in an EJB (a new Database Table can't be
            found -- I messed up in my Database somewhere). But the then the JCache
            gets into some kind of weird state, and you can't get ANYTHING out of
            it, taking down the entire application. Get ClassCastExceptions and
            ClassNotFoundExceptions on any Cache Access after this point (some are
            attached). The ONLY way I have found to recover so far is to
            cold start JBoss, as all access from all clients are failing at this point.


            Surely after your cold start, your table then exists? So on restart do you still see the "table not exists" error?

            • 3. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
              northgorky

              Greetings.

              I went and fixed the 'table not exists' error by adding it in the database. Problem is, any and all actions on any other connection, including this one, that did not access that table also all failed, with the indicated ClassCastException/ClassNotFoundException.

              And its not just table not exists. IllegalArgumentExceptions and NullPointerExceptions both take the Cache down the same way.

              -Steve

              • 4. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                brian.stansberry

                Any static initializers in com.hrnexus.security.ThinUserProxy or in any class it needs to resolve when being initialized (e.g. type of a static field)?

                I've seen cases where a failure in a static initializer gets swallowed but the VM ends up treating the class as unavailable. The problem ends up manifesting itself as a CNFE.

                What the cache is doing in your stack trace is a basic deserialization operation using ObjectInputStream and the thread context classloader. Nothing particularly cache-specific about it.

                Another (vague) possibility is the exception somehow affects how jetty is setting the context class loader on request threads. If the TCCL isn't set to one that has visibility to com.hrnexus.security.ThinUserProxy you won't be able to deserialize.

                • 5. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                  northgorky

                  Hi Brian.

                  Good question. Especially as statics and static initializers are routinely used in this project. So, I had to do some digging to answer this question. Turns out that ThinUserProxy only has 4 properties;
                  int
                  java.lang.String
                  PrimaryKey (with two properties, both java.lang.String)
                  javax.ejb.Handle (extends java.io.Serializable)

                  I'm going to lean towards something happening with Classloaders during rollback, as after the first exception, can't even load the Node (that's what is goinig on in;
                  com.hrnexus.web.SessionCache.getSession(SessionCache.java:133)

                  I don't have the other stack traces handy right now, but any attempt to access anything, including java.lang. objects stored, all fail once this starts happening.

                  I am not sure that this is a Jetty issue for the following reasons;
                  1) ThinUserProxy itself is in a library jar that all the other EJBs and Servlets depend on. It has full application visibility
                  2) Jetty is not directly access JCache -- only through hrnexus. Although, it is highly probably that JCache is accessing the Jetty Classloaders, if I understand your response correctly.

                  I have never needed to sign up to the Jetty Support list before, so signing up there now. They might be able to provide additional insight.

                  Some questions;
                  - What's getting done to/in the classloader that is causing it to fail this spectacularly? Its almost like its been told to clear itself of all classes, and clear the parent classloaders as well.
                  - How do you get a ClassCastException on a ClassNotFoundException? (Seeing this one quite a bit)

                  Regards,
                  Steve

                  • 6. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                    northgorky

                    Greetings.

                    Forgot a question;
                    What can I do to make these errors stop happening?

                    -Steve

                    • 7. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                      gregwilkins

                      Just a quick answer from a jetty point of view.... the TCCL for the webapp is set by jetty in the ContextHandler and is not unset until the finally block is called. So in the stack trace,you can see:

                      12:48:11,673 ERROR [org.jboss.logging.util.LoggerStream.write(LoggerStream.java:152)] at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:726)

                      So as far as jetty is concerned the TCCL should be set. Of course that does not stop something else unsetting or setting it to something different.

                      I've created a jetty task: http://jira.codehaus.org/browse/JETTY-544 to track this issue , so if it does look like a Jetty bug we can pick it up.

                      • 8. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                        brian.stansberry

                        Thanks, Greg.

                        TBH I'd be very surprised if this were a Jetty problem.

                        "northgorky" wrote:

                        - What's getting done to/in the classloader that is causing it to fail this spectacularly? Its almost like its been told to clear itself of all classes, and clear the parent classloaders as well.


                        That's the big question we're trying to figure out.

                        "northgorky" wrote:

                        - How do you get a ClassCastException on a ClassNotFoundException? (Seeing this one quite a bit)


                        Not sure what you mean. You've mentioned other failure types beside what you've linked; please provide details, maybe they will provide a clue.

                        • 9. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                          northgorky

                          Greetings.

                          I've done some more runs. One (labeled pause), just sat for about two minutes, and then crashed on the next click. Write through an exception during an update, but continued working, briefly. Then crashed.

                          If you need more 'Data points', or additional info, let me know. In addition, I am in Texas, and willing to come out to Atlanta, if anyone wants a 'hands on', to through a Debugger on this system.

                          Steve
                          Links for additional logs;
                          http://j2eeguys.com/~steve/JBoss/server.pause.20080326.log
                          http://j2eeguys.com/~steve/JBoss/server.read.20080326.log
                          http://j2eeguys.com/~steve/JBoss/server.write.20080326.log

                          • 10. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                            northgorky

                            Greetings.

                            Don't know if this would affect things, but application structure is;
                            ear
                            - library jars
                            - multiple ejb jars
                            - multiple wars

                            -Steve

                            • 11. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                              brian.stansberry

                              No one relevant is in Atlanta. :-) Unless someone in the community has some ideas (knock on wood.) We work remotely from all over the world. I'm in St. Louis.

                              • 12. Re: ava.lang.ClassCastException:     java.lang.ClassNotFoundExce
                                brian.stansberry

                                Looking at the latest logs, I'm seeing this logged all over the place:

                                2008-03-26 12:33:51,159 ERROR [org.jboss.logging.util.LoggerStream.write(LoggerStream.java:152)] In JettyDeployer, setting webapp.metadata.contextloader=ContextLoader@null


                                Don't know what that means, but it sure sounds relevant. :-) I'd make finding out what's causing that your prime focus.