14 Replies Latest reply on Jul 26, 2005 10:14 PM by gray1

    java.lang.ClassCircularityError first time subscribe to jms

    gray1

      ENVIRONMENT: Windows 2000 SP4, JBossAS-4.0.1 Final, JDK 1.4.2_08
      (I have not yet verified if this problem occurs on later versions of JBoss as our product currently only supports 4.0.1)

      DESCRIPTION
      First time my client connects to my jms server to create two jms subscriptions in quick succession I get the following server-side error. Subsequent subscriptions to the same running server work fine.

      Example 1

      16:53:25,260 ERROR [SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.ReceiveMsg13684176[msgType: m_receive, msgID: 9, error: null]
      java.lang.ClassCircularityError: org/jboss/mq/selectors/Operator
      at org.jboss.mq.selectors.SelectorParser.comparisonExpression(SelectorParser.java:288)
      at org.jboss.mq.selectors.SelectorParser.conditionalExpression(SelectorParser.java:202)
      at org.jboss.mq.selectors.SelectorParser.selectorFactor(SelectorParser.java:167)
      at org.jboss.mq.selectors.SelectorParser.selectorTerm(SelectorParser.java:136)
      at org.jboss.mq.selectors.SelectorParser.selectorExpression(SelectorParser.java:114)
      at org.jboss.mq.selectors.SelectorParser.expression(SelectorParser.java:105)
      at org.jboss.mq.selectors.SelectorParser.parse(SelectorParser.java:58)
      at org.jboss.mq.selectors.SelectorParser.parse(SelectorParser.java:38)
      at org.jboss.mq.selectors.Selector.(Selector.java:74)
      at org.jboss.mq.Subscription.getSelector(Subscription.java:75)
      at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:433)
      at org.jboss.mq.server.JMSTopic.receive(JMSTopic.java:285)
      at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:222)
      at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager.java:673)
      at org.jboss.mq.server.JMSServerInterceptorSupport.receive(JMSServerInterceptorSupport.java:226)
      at org.jboss.mq.security.ServerSecurityInterceptor.receive(ServerSecurityInterceptor.java:100)
      at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:570)
      at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:226)
      at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:149)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:361)
      at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
      at java.lang.Thread.run(Thread.java:534)

      Example 2

      17:12:12,989 ERROR [SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.ReceiveMsg13628909[msgType: m_receive, msgID: 9, error: null]
      java.lang.ClassCircularityError: org/jboss/mq/selectors/Token
      at org.jboss.mq.selectors.SelectorParser.(SelectorParser.java:1422)
      at org.jboss.mq.selectors.SelectorParser.(SelectorParser.java:32)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:274)
      at java.lang.Class.newInstance0(Class.java:308)
      at java.lang.Class.newInstance(Class.java:261)
      at org.jboss.mq.selectors.Selector.(Selector.java:73)
      at org.jboss.mq.Subscription.getSelector(Subscription.java:75)
      at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:433)
      at org.jboss.mq.server.JMSTopic.receive(JMSTopic.java:285)
      at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:222)
      at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager.java:673)
      at org.jboss.mq.server.JMSServerInterceptorSupport.receive(JMSServerInterceptorSupport.java:226)
      at org.jboss.mq.security.ServerSecurityInterceptor.receive(ServerSecurityInterceptor.java:100)
      at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:570)
      at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:226)
      at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:149)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:361)
      at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
      at java.lang.Thread.run(Thread.java:534)


      Looks like some sort of mulit threaded classloading bug to me.

      Interestingly if I slow my code down say by running it in debug mode and stepping through it there is no error. Its only if i run it at full speed it happens.

      This bug is a problem for the first guy that logs on as there is NO CLIENT SIDE EXCEPTION thrown to indicate the connection was unsuccessful. As far as they are concerned they are listening to the topic but they arent really

      I have posted this as bug http://jira.jboss.com/jira/browse/JBAS-2032

        • 1. Re: java.lang.ClassCircularityError first time subscribe to
          gray1

          I have verified this bug also occurs executing on JDK 1.5.0_04

          • 2. Re: java.lang.ClassCircularityError first time subscribe to
            genman

            The only times I've seen this sort of error were on older (like pre-4.0) versions of JBoss and deployments that duplicated the same classes in different .jar files. It could be that you are bundling something twice in your deployment, or JBoss is.

            • 3. Re: java.lang.ClassCircularityError first time subscribe to
              gray1

              Yeah the only references I could find to it in existing forum posts were pre JBoss 4 too. However it is most definitely happening in JBoss 4.0.1 and note its complaining about a JBoss class which I have not touched. It is not a class I have written or put on the server.

              And before someone suggests I have corrupted my server this is with a fresh copy of the default server.

              • 4. Re: java.lang.ClassCircularityError first time subscribe to
                gray1

                I can also make the error go away by turning the logging up to max. I think this acts to slow down the server such that the 2nd jms subscription request does not initiate classloading as the the 1st jms subscription request has already loaded the jboss classes required for jms.

                The relevant part of the log (when it works) is

                10:57:03,414 DEBUG [SocketManager] Begin ReadTask.run
                10:57:03,414 DEBUG [SocketManager] Begin WriteTask.run
                10:57:03,414 DEBUG [SocketManager] Created ObjectOutputStream
                10:57:03,414 DEBUG [SocketManager] Begin ReadTask.run
                10:57:03,414 DEBUG [SocketManager] Begin WriteTask.run
                10:57:03,445 DEBUG [SocketManager] Created ObjectOutputStream
                10:57:03,445 DEBUG [SocketManager] Created ObjectInputStream
                10:57:03,445 DEBUG [SocketManager] Created ObjectInputStream

                Note the 2 successive subscription requests

                • 5. Re: java.lang.ClassCircularityError first time subscribe to
                  gray1

                  I just tested our app against 4.0.2 Final Release (with jbas-1875-patch.zip security patch applied) and JDK 1.5.0_04 in the hopes that might fix it but sadly the error still occurs. Not absolutely every time mind you but enough to be a serious concern.

                  • 6. Re: java.lang.ClassCircularityError first time subscribe to
                    genman


                    This is really a "JMX" bug, so if you can come up with a test case, I would file it on jira.jboss.org and it will get addressed, or at least analyzed for you.

                    • 7. Re: java.lang.ClassCircularityError first time subscribe to

                      He already did and it already got rejected with the standard work around for the problem.

                      And this is a REALLY a Sun bug that is more than 2 years old:
                      http://jira.jboss.com/jira/browse/JBAS-2032

                      • 8. Re: java.lang.ClassCircularityError first time subscribe to

                        Just for completeness, JBoss already jumps through hoops to try to avoid the problem
                        including scheduling classloading across threads and retrying the classloading if it hits
                        the error.

                        But since the problem is non-deterministic (like all non-thread-safe bugs), any
                        work around (except one that avoids concurrent usage) is not guaranteed to work.

                        • 9. Re: java.lang.ClassCircularityError first time subscribe to
                          gray1

                          OK Adrian. Thanks for taking the time to respond. I just read your workaround


                          It can be trivially worked around by adding an MBean to the startup sequence that preloads the problem classes on the main thread, thus avoiding the non-thread-safe checking in Sun's code when two threads compete to load classes.


                          This sounds promising however I have two hurdles to overcome with this.

                          1) I'm new to mbeans. I understand the basics of what they are but I don't know make one to preload a class. If you could provide an example snippet based on the example exception stack trace and suggest which xml file would be best to put it in I would be extremely grateful!

                          2) The class that it throws the circularity error on is not exactly the same each time. I find it easy to reproduce the problem since I am opening two jms subscriptions on the same jms connection. However the actual class varies between a variety of different jms related classes (see two examples I provided for two common ones). Would there be a way of just preloading the whole jboss jms jar?

                          Kindest Regards,

                          Gray


                          • 10. Re: java.lang.ClassCircularityError first time subscribe to
                            gray1

                            After reading the wiki my first attempt looks something like

                            public class JBossClassPreloader {
                             private static final Logger logger = Logger.getLogger(JBossClassPreloader.class);
                            
                             public JBossClassPreloader() {
                             try {
                             // get the classloader associated with the thread
                             ClassLoader cl = Thread.currentThread().getContextClassLoader();
                             // load the class
                             cl.loadClass("org.jboss.mq.selectors.Operator");
                             cl.loadClass("org.jboss.mq.selectors.Token");
                             //repeat for every possible class that jboss mq uses! Eek!
                             } catch (Exception ex) {
                             logger.warn("Exception in class preloader", ex);
                             }
                             }
                            }
                            


                            Then I create a new xml file in the deploy directory that has the following

                            <?xml version="1.0" encoding="UTF-8"?>
                            <!DOCTYPE server>
                            <!-- $Id$ -->
                            
                            <server>
                            
                             <mbean code="mypackage.server.plugin.JBossClassPreloader"
                             name="jboss:service=MyClassPreloader">
                             </mbean>
                            
                            </server>
                            


                            Is this kind of what you had in mind Adrian? Considering how many classes are in the jboss mq jar file I dont think this is going to be practical.

                            Maybe I should create an MBean that presubscribes to one of the topics instead to achieve the same classloading effect? But then how would I make it so the MBean is executed after the mbeans that initialise the topics?



                            • 11. Re: java.lang.ClassCircularityError first time subscribe to
                              gray1

                              OK ignore my previous post as I obviously didn't understand how mbeans work.

                              This is the code that I believe takes the approach Adrian suggested. I have made it detailed for Adrian to check the approach is valid and so anyone else following in my footsteps doesn't have to trawl through the doco like I did.

                              JmsClassPreloader.java

                              package com.acme.server.plugin.mbean;
                              
                              import org.apache.log4j.Logger;
                              import org.jboss.system.ServiceMBeanSupport;
                              
                              public class JmsClassPreloader extends ServiceMBeanSupport implements JmsClassPreloaderMBean {
                               private static final Logger logger = Logger.getLogger(JmsClassPreloader.class);
                              
                               public void start() throws Exception {
                               try {
                               if (logger.isDebugEnabled()) {
                               logger.debug("starting JmsClassPreloader mbean service");
                               }
                               // get the classloader associated with the thread
                               ClassLoader cl = Thread.currentThread().getContextClassLoader();
                               // load the class
                               cl.loadClass("org.jboss.mq.selectors.Operator");
                               cl.loadClass("org.jboss.mq.selectors.Token");
                               //etc for every possible class that jboss mq uses!
                               } catch (Exception ex) {
                               logger.warn("Exception in class preloader", ex);
                               }
                               }
                              }
                              


                              JmsClassPreloaderMBean.java
                              package com.acme.server.plugin.mbean;
                              
                              import org.jboss.system.ServiceMBean;
                              
                              public interface JmsClassPreloaderMBean extends ServiceMBean {
                              
                              }
                              


                              Then I create the following directory structure in my jboss server/deploy directory

                              [acme-com-mbeans.sar]
                              ----acme-mbeans.jar
                              ----[META-INF]
                              --------jboss-service.xml

                              Where acme-mbeans.jar file is a jar containing the above two java class files and jboss-service.xml contains the following

                              <?xml version="1.0" encoding="UTF-8"?>
                              <!DOCTYPE server>
                              <!-- $Id$ -->
                              
                              <server>
                              
                               <mbean code="com.acme.server.plugin.mbean.JmsClassPreloader"
                               name="acme.com:service=JmsClassPreloader">
                               </mbean>
                              
                              </server>
                              


                              Questions

                              Is this what you had in mind Adrian? Considering how many classes are in the jboss mq jar file I dont think this is going to be practical since I have to list every single class in the jar in the mbean start method for it to be loaded (probably 50 or more). Is there a way of just loading a whole jar?

                              If not maybe I should instead create an MBean that presubscribes to one of the topics instead to achieve the same classloading effect. But then am i guaranteed my MBean will execute after the mbeans in the deploy/jms directory that initialise the topics?

                              I await your response in hope,

                              Gray.

                              • 12. Re: java.lang.ClassCircularityError first time subscribe to
                                gray1

                                I should add that while the above code compiles and jboss seems to run it correctly it does not help in the slightest. Despite me preloading the whole org.jboss.mq.selectors.* package (I have them all specified manually one at a time in my preloader) the class circularity problem still shows later when i first subscribe with one of the supposedly preloaded classes manifesting the error.

                                Am I preloading it in the wrong classloader perhaps? Do I need to somehow preload the whole jar?

                                Nothing I try seems to fix this problem and I have run out of ideas.

                                • 13. Re: java.lang.ClassCircularityError first time subscribe to

                                  1) It should be in createService() not start()
                                  although it probably won't make much difference in this case.

                                  2) You need to make sure it loads the classes BEFORE jbossmq starts.
                                  e.g. add the mbean config to the top of jbossmq-service.xml or put in conf/jboss-service.xml

                                  3) Use

                                  Class.forName(name, true, classLoader);

                                  to ensure the class is resolved and initialized along with its referenced classes.
                                  Using ClassLoader.loadClass() might only give you a reference to the reflection
                                  metadata rather than the entire transitive closure.

                                  4) You don't need to load all classes. You only need to load the problem classes.
                                  It is usually seen where there is bidirectional link, e.g. main class <-> inner class
                                  JBossMQ already loads many of the classes itself.

                                  5) Read the documentation, in particular
                                  a) the admin docs - notably chapter 2
                                  b) a good tutorial on classloading
                                  c) the wiki on how to enable to classloader TRACE logging

                                  6) Use search to find what must be hundreds of other discussions on this problem.
                                  You will have to wade through a lot of unanswered ones where people like you
                                  didn't use search. I only answered in the first place to reject your incorrect bug report.
                                  I don't look kindly on people that post bug reports to jump the "help queue".

                                  7) Come back when you don't need hand-holding on debugging this problem and know
                                  where the problem is (assuming that once you understand it, you don't know how to resolve it yourself).

                                  8) If you resolve the problem, post an update for others to find, preferably on the WIKI.

                                  9) If/when you do come back, post full configuration, code, stacktraces and relevant
                                  logging.
                                  I am not going to guess what you did wrong or take your implied assumption that you did it
                                  correctly.
                                  http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossHelp

                                  • 14. Re: java.lang.ClassCircularityError first time subscribe to
                                    gray1

                                    Thanks for the response Adrian.

                                    Regarding your initial points I have done the following

                                    -Moved the code into createService method
                                    -Shifted the order that the mbean loads to be earlier by putting this into conf/jboss-server.xml.
                                    -Used Class.forName(name, true, classLoader) to ensure classes get preloaded property

                                    The results are promising. I have not managed to reproduce the issue once on versions 4.0.1 and 4.0.2 of jboss yet.

                                    Here is the final java code for the benefit anyone that follows

                                    JmsClassPreloader.java

                                    import org.apache.log4j.Logger;
                                    import org.jboss.system.ServiceMBeanSupport;
                                    
                                    public class JmsClassPreloader extends ServiceMBeanSupport implements JmsClassPreloaderMBean {
                                     private static final Logger logger = Logger.getLogger(JmsClassPreloader.class);
                                    
                                     public void createService() throws Exception {
                                     try {
                                     if (logger.isDebugEnabled()) {
                                     logger.debug("starting JmsClassPreloader mbean service");
                                     }
                                    
                                     // get the classloader associated with the thread
                                     ClassLoader cl = Thread.currentThread().getContextClassLoader();
                                    
                                     //load the classes
                                     String name;
                                     name = "org.jboss.mq.selectors.ASCII_CharStream";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.Identifier";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.ISelectorParser";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.Operator";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.ParseException";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.RegExp";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.Selector";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.SelectorParser";
                                     Class clazz = Class.forName(name, true, cl);
                                     //initialise the inner class JJCalls
                                     clazz.newInstance();
                                     name = "org.jboss.mq.selectors.SelectorParserConstants";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.SelectorParserTokenManager";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.Token";
                                     Class.forName(name, true, cl);
                                     name = "org.jboss.mq.selectors.TokenMgrError";
                                     Class.forName(name, true, cl);
                                    
                                     } catch (Exception ex) {
                                     logger.warn("Exception in class preloader", ex);
                                     }
                                     }
                                    }
                                    


                                    JmsClassPreloaderMBean.java
                                    import org.jboss.system.ServiceMBean;
                                    
                                    public interface JmsClassPreloaderMBean extends ServiceMBean {
                                    
                                    }
                                    


                                    Both of these classes are in a jar file in the server/lib directory and I have put the following entry in my server/conf/jboss-service.xml file right up the top

                                    <server>
                                    
                                     <classpath codebase="lib" archives="*"/>
                                    
                                     <mbean code="JmsClassPreloader"
                                     name="acme.com:service=JmsClassPreloader">
                                     </mbean>
                                    
                                     ...
                                    
                                    </server>
                                    


                                    Regarding your point 6 I understand how frustrating it must be when people don't read the forums before posting but I personally don't think this applies to me.

                                    1) Prior to posting I read every single forum post using Search criteria [keyword "ClassCircularityError", All Forums]. This search only received less than a dozen hits all of which refer to version 3 or earlier of jboss and none of which relate to jms.

                                    2) A Search of jira for existing bug reports using "ClassCircularityError" as the search term retrieved 3 hits none of which seemed even vaguely related to my problem.

                                    On those grounds since I was 99% sure it was a genuine bug I felt justified in creating the bug report in addition to the forum post.

                                    After you posted your mbean workaround suggestion I was admittedly guilty of not reading the wiki hence my first blatently incorrect attempts at making an mbean which I quickly corrected (oh to be able to edit or delete one's own posts!)

                                    Thanks again for the help.

                                    Kindest Regards,

                                    Gray.