1 2 Previous Next 27 Replies Latest reply on Jul 19, 2003 8:22 PM by brado

    JBoss ClassLoader implementation is unappropriate for EJB co

    aanno

      The Problem
      --------------
      EJB (jar), WEB (war), and enterprise (ear) archives should be isolated from another. This especially means that:

      (a) independant packed components could use different version of a file with the same qualified name

      (b) independant packed components could call each other (i.e. home, remote, and local interfaces)

      JBoss ClassLoader design will cache all Class instances globally in standard configuration. This violates (a). It is possible to switch to (ear) archive-wise caching. But this violates (b), because the (client interfaces) packed with component 1 will be loaded with a different ClassLoader than the interface definition packed in component 2 (that also implements the bean) if both components are deployed on the same JVM. Violation of (b) normally shows up in a ClassCastException in PortableRemoteObject.narrow().

      In summary it is impossible in JBoss to adhere to (a) and (b) at the same time.

      Proposed solution
      ------------------
      There are two ways around this problem:

      - disable the local call optimisation for call between independant packed components or

      - use a IIOP like scheme for all calls. IIOP will only transfer data members of object instances. It will NOT share Class implementation between independant packed component.

      Kind regards,

      aanno

        • 1. Re: JBoss ClassLoader implementation is unappropriate for EJ
          douglm

          I'm afraid I also have to agree that Jboss3.x is just not a practical platform.

          We build and deploy many, generally small, applications with a high degree of code sharing.

          We have many utility classes, general, ejb, jsp, struts etc.

          We have had a number of occasions to deploy different versions of our own and other organizations classes. For example, some are running with the beta version of struts as we have an urgent need for working file upload.

          We have a fairly high degree of database interaction. Some of those applications are essentially configured clones of each other. Updates often require database changes. In the new world we cannot update each clone independently, about 5 mins each but all at once, about half an hour down time.

          I fully understand that I COULD replace all my applications at once but I'd prefer to limit my liability and update them singly.

          Performance for most is largely irrelevent. Almost none of our machines is stretched for cpu and in an environment where change is constant, machines are much cheaper than the extra people needed to manage the test and redeployment overhead jboss now seems to be imposing.

          Another aspect of the same problem is those applications we get from outside. At the moment we can safely wrap them up with their own jars.

          • 2. Re: JBoss ClassLoader implementation is unappropriate for EJ

            I don't disagree with you except that you say it is
            a problem with jboss?
            You cannot isolate and share classes at
            the same time.

            If you isolate classes you must pay a cost in
            serializing/marshalling objects when invoking between
            different classloader spaces.

            By default JBoss3 avoids the performance cost of
            serialization by sharing classes. In fact from
            entry into the server (RMI, HTTP, etc) to exit
            (DB, other) there is no serialization.

            The cost of serialization is typically an
            order magnitude (a factor of 10) even when it is
            implemented effciently.
            This has nothing to do with network, database, bad
            implementation, etc. It is just the cpu required to
            convert an object tree (pointer references)
            to a byte stream (values) and back again.
            Pass by reference will always be more efficient.

            Did you know if you have myapp1.ear and myapp2.ear
            that need to be isolated from myapp3.ear but not
            from each other, you can specify the same
            <loader-repository> for myapp1.ear and myapp2.ear

            For your proposed solution, you can change the client
            interceptor stack to use ByValueInvokerInterceptor
            instead of InvokerInterceptor.

            2.x would calculate whether these semantics were
            required, but the calculation is not cheap, so
            it has been dropped in 3.x
            The calculation involves checking whether the
            returned interfaces are the same class objects as those
            loaded through the thread context classloader.
            Where they are different, you fail over to
            the ineffienct "by value".

            You could re-implement the 2.x behaviour with a
            ByValueOnlyWhenMyPackingIsIneffcientInvokerInterceptor
            :-)

            Regards,
            Adrian

            • 3. Re: JBoss ClassLoader implementation is unappropriate for EJ
              aanno

              Dear Adrian,

              thank you for your tips with ByValueInvokerInterceptor and ByValueOnlyWhenMyPackingIsIneffcientInvokerInterceptor.

              If I got you right JBoss 2.4 had a different behaviour that would avoid the problem. However, my experience tells me that is the very same, expect I don't no much about JBoss 2.4 on source level.

              My general feeling is that JBoss has violated the idea of a component based model for a long time now and that the standard jboss behaviour should be fixed to the state that normal J2EE experts would expect. I checked the problem with WebSphere 5 myself and can tell you that it simply doesn't exist there. I would also guess it doesn't exist in WebLogic or Oracle (perhaps somebody will check Sun's RI) as well. My point of view is that the ClassLoader problem prevents JBoss from being deployed in a production environment, especially if you haven't full control over ALL the packed archived that are deployed here.

              The ClassLoader implementation in JBoss makes INDEPENDANT component depend on each other in rather obscure ways.

              There is nothing bad with aggressive optimizations. Certainly serialization and deserialization would cost some time. But they would help JBoss to become more standard compliant. (By the way the (de)serialization call would be necessary only for independant packed components - within a component all optimization could be applied.)

              Kind regards,

              aanno

              • 4. Re: JBoss ClassLoader implementation is unappropriate for EJ
                moraelin

                Maybe I can explain better why this is a problem. This is an actual problem which happened here:

                - Application A uses one version of Electric-XML, let's call it E1

                - Bean B uses a completely different version, let's call it E2. It also uses another library L, in a version which we'll call L2.

                - Bean C doesn't use Electric XML at all, but uses a different version of library L. Let's call it L3. It also calls Bean B.

                Both are packed as EAR files, and each should use its own libraries. However, when deployed together on JBoss 3:

                A) with the unified class loader, the wrong library versions are used all over the place. Application A tries to Electric XML version E2, loaded from Bean B instead of its own E1. Since Electric XML changed some classes in between the two versions, Application A dies. Bean B tries to use the L3 version of library L, loaded from Bean C, instead of its own L2. It's incompatible too, so Bean B dies too. And so on.

                When you have a project with a dozen (or two dozen) different EAR files deployed on one single JBoss server, this escalates exponentially into a nightmare. One single EAR file with one single library in a different version, can break every single other application and EJB on the server.

                Sorry, _no_ optimization is worth this kind of a problem.

                B) Ok, let's scope the classloaders, as per the JBoss docs. (I.e., via jboss-app.xml.)

                First of all in and by itself this is a poorly thought out hack, because essentially it says "I don't want to export my classes", instead of being able to say "I want my own classes loaded first." (As is the case in IBM WebSphere 5.0.) Again, one single EAR file who fails to scope its classloader, will break everything else running on that server. In and by itself that is unacceptable for a production environment.

                But ok, let's put a jboss-app.xml file in each and every single EAR file. It should solve everything right?

                Well, wrong. Now all of a sudden when Bean C tries to obtain a reference to Bean B (via narrow()), it will die with a ClassCastException. All of a sudden beans residing in different EAR files can no longer call each other.

                So what did this solve? It would work if the whole application resided in a single EAR file, but in that case: why do I need EJB? If they're going to always stay together like that, why not make them libraries and stick with Tomcat, instead of going through all the EJB trouble?

                C) Yes, I could go ahead and implement my own classloaders and invokers and whatnot, then test them, etc. But you're missing an important point here. Developper time is money. Someone has to pay a lot of money for that time. If JBoss is to be taken seriously in a production environment it should not require investing tens of thousands in development time to work around its bugs, because if it gets at that point, it's cheaper to just buy WebSphere or WebLogic instead.

                • 5. Re: JBoss ClassLoader implementation is unappropriate for EJ

                  Load locally first in the Hierarchical Loader Repository
                  is a planned enhancement.
                  This will allow you to have the same behaviour as
                  the Servlet ClassLoader.

                  To avoid the ClassCastException you need to use
                  the ByValueInvokerInterceptor to force the
                  Remote interface invocations to serialize the invocations.

                  I would be surprised if this were a real production
                  problem.
                  If the changes in the supporting jars
                  really are that different that they cannot use the
                  same version, how are two apps going
                  communicate consistently anyway?

                  This whole problem smells funny to me.
                  If you are designing a component based solution
                  you should share interfaces (contracts) and
                  choose the best implementation for the job.

                  Regards,
                  Adrian

                  • 6. Re: JBoss ClassLoader implementation is unappropriate for EJ
                    moraelin

                    Believe it or not, this _is_ a real production problem. Well, an integration problem so far, since it obviously can't go into production like this even if we wanted to. When you have several teams working on different things, which all must run on the same server, sadly, yes, this is exactly the kind of thing you'll run into.

                    How do those things work with different supporting libraries, is actually quite easy. The key word is: supporting. None of the Home or Remote interfaces, nor any of the data objects, directly expose those supporting libraries to the outside world. They're used strictly internaly by the EJB. They're implementation details. Whether Bean B uses version 5 of a library, or version 4, or even if it uses its very own code instead, is not a detail that Bean C must even know. All that Bean C needs, and all it will ever see, is a clean interface to Bean B. What happens inside Bean B is no longer its problem.

                    Basically it's just very elementary OOP design, nothing horribly sophisticated.

                    Thing is, here we're not talking about some guy running a single app on his home machine. Our final target for this stuff is some big iron on which they'll run two dozen apps written by two dozen different teams, some of them not even in the same country. I think it's pretty safe to say that trying to get them all to use the exact same libraries, is a rather futile exercise.

                    And even if at one moment in time everyone was coerced to stick to some The One True List of allowed version, what will happen in 6 months? Or a year? Does everyone still have to stick to those old libraries? Or do all the existing beans and apps need to be re-compiled, extensively re-tested and re-deployed each time someone else needs a newer version of one of those libraries? See what I mean? That optimization simply isn't worth the inconvenience. That's all I'm saying.

                    • 7. Re: JBoss ClassLoader implementation is unappropriate for EJ

                      It might not be work it for you.

                      But your customer has to buy a cpu that is 10 times as
                      powerful or put 10 times as many boxes in the cluster
                      to cope with the same load.
                      Not cheap on a big iron, I can see why ibm support
                      this :-)

                      And all because you can't organize your packaging.

                      Regards,
                      Adrian

                      • 8. Re: JBoss ClassLoader implementation is unappropriate for EJ
                        moraelin

                        Well, lemme see... they can buy 10 times the CPU power, _or_ they could pay some extra time to all those teams to re-package, re-compile, re-test and re-deploy anything. And pay again for every single app, every time someone needs a newer version of a library. Hmm... I'd go with buying more CPU power myself.

                        Again, in the real world developper time is money. One helluva lot of money. Also, in the real world, a corporation's applications are not just re-compiled and dropped in on a whim: if they're all going to be re-compiled to the same libraries, we're talking at least a month of testing each. Because unlike someone's homesite and guestbook, here you can't afford an "oh well, if it malfunctions, we'll tell them to try again" policy.

                        Basically if your solution is the kind of overhead described above, over and over again... well, trust me, that big iron is pocket change by comparison. Well, actually the big computers they already have anyway, but even if they needed to buy another of those, it would still end up cheaper.

                        This is not supposed to be a flame. It's just basic economics. I _know_ it doesn't feel right that more hardware is a cheaper solution than some ideal, perfectly optimized, 100% perfectly packaged solution. Trust me, it actually _hurts_ to see a stack trace on an EJB server, and realize the kind of _massive_ unneeded overhead that happens there, compared to writing the same thing in C. (Or compared to my old favourite: writing the inner loops in assembly. I used to think that expensive C compilers were doing a remarkably poor job of optimizing, and thuse caused a waste of CPU and RAM.) In an ideal world, I could actually go to these guys and say, "hey, let's dump the whole EJB stupidity and we'll write you a C/asm .so module for Apache, which can run fast enough on a 486." But sadly, it's not an ideal world, so we'll all have to deal with the fact that it's actually cheaper for them to buy more hardware than to pay for twice the man-hours. It hurts, but it's in the end just a fact of life.

                        Basically same issue here with the packaging. An optimization which saves maybe 10% CPU time, but causes this much maintenance overhead, is just not worth it. From the point of view of the guy paying for it, it's just as not worth it as my wanting to write it in assembly instead.

                        And, heh, yeah, I too can understand why IBM and Sun are pushing this stuff. Been saying the same thing myself. (A couple of times even in these forums.) Must do wonders for their hardware sales.

                        • 9. Re: JBoss ClassLoader implementation is unappropriate for EJ
                          aanno

                          Hello,

                          Mainly the discussion has end with:

                          (warjort): all is OK. If you have a problem then:
                          (1) You probably don't know how to package your application(s), and I suspect that you can't programm anyway.
                          (2) Speed is all. JBoss behaviour is no 100% to the specs but fast.
                          (3) If you really vexed, just get the ByValueInvoker in or wait for Hierachical Loader Repository.

                          Moraelin has supported my point of view and added:
                          (1) What libraries (other Classes) are used within a (packed) component is an implementation detail and calling component should not bother.
                          (2) Speed is not all. Develoment time is expensive, too. And there is also the option to buy a piece of software that does more what you expect.

                          I strongly agree with Moraelin. It is questionable if JBoss would run 10 times slower with a more appropriate ClassLoader solution. As I said before optimation are possible within a component. (By the way - this is the main reason why I don't like the ByValueInvoker idea that much.)

                          Hierachical Loader sounds good. But I'm afraid that they will be implemented along the lines of
                          http://sourceforge.net/mailarchive/message.php?msg_id=3852655 . To me this looks like a very complex solution with limited useability.

                          However, discussion has loose sight of the main point a little. I really think that a good software should make easy things going easy and complex things possible. A component model is superior to the traditional object model because dependencies are diminished. JBoss behaviour is definitifly not what a normal JBoss user expects. He may be right or wrong, a lousy programmer or packer/deployer. But deploying this kind of components should be POSSIBLE in STANDARD configuration.

                          The sad part of this is not that I can't get JBoss to go this way. The bad is that it is possible but it needs a JBoss guru to do so. All this because of speed considerations. Guru's should do the optimization! Normal JBoss user should just drop their components in the deploy folder.

                          JBoss could do much better than it is doing right now.

                          Sincerly,

                          aanno

                          • 10. Re: JBoss ClassLoader implementation is unappropriate for EJ
                            aanno

                            Hello,

                            Mainly the discussion has end with:

                            (warjort): all is OK. If you have a problem then:
                            (1) You probably don't know how to package your application(s), and I suspect that you can't programm anyway.
                            (2) Speed is all. JBoss behaviour is no 100% to the specs but fast.
                            (3) If you really vexed, just get the ByValueInvoker in or wait for Hierachical Loader Repository.

                            Moraelin has supported my point of view and added:
                            (1) What libraries (other Classes) are used within a (packed) component is an implementation detail and calling component should not bother.
                            (2) Speed is not all. Develoment time is expensive, too. And there is also the option to buy a piece of software that does more what you expect.

                            I strongly agree with Moraelin. It is questionable if JBoss would run 10 times slower with a more appropriate ClassLoader solution. As I said before optimation are possible within a component. (By the way - this is the main reason why I don't like the ByValueInvoker idea that much.)

                            Hierachical Loader sounds good. But I'm afraid that they will be implemented along the lines of
                            http://sourceforge.net/mailarchive/message.php?msg_id=3852655 . To me this looks like a very complex solution with limited useability.

                            However, discussion has loose sight of the main point a little. I really think that a good software should make easy things going easy and complex things possible. A component model is superior to the traditional object model because dependencies are diminished. JBoss behaviour is definitifly not what a normal JBoss user expects. He may be right or wrong, a lousy programmer or packer/deployer. But deploying this kind of components should be POSSIBLE in STANDARD configuration.

                            The sad part of this is not that I can't get JBoss to go this way. The bad is that it is possible but it needs a JBoss guru to do so. All this because of speed considerations. Guru's should do the optimization! Normal JBoss user should just drop their components in the deploy folder.

                            JBoss could do much better than it is doing right now.

                            Sincerly,

                            aanno

                            • 11. Re: JBoss ClassLoader implementation is unappropriate for EJ
                              aanno

                              Hello,

                              I thought over the problem again this weekend. I now think it could be possible to implement an algorithm in JBoss than will lead to the desired behaviour without sacrifing any performance. Here it is:

                              (1) Construction of Classpath while deploying EJB (jar) file.

                              (a) When an EJB (jar) file is deployed the ejb-jar.xml descriptor is examined. If there is an <ejb-client-jar/> tag, a ClassLoader pointing to this jar is used as EJB interfaces ClassLoader.

                              (b) If there isn't an <ejb-client-jar/> tag, (home, remote, local) interface classes of the xml descriptor are used as root for an class file dependency analysis.

                              (c) Dependency analysis is performed by looking at the class file constant pool (for example with jakarta bcel). (Using reflection is NOT appropriate because it isn't giving us full dependencies.) Dependency analysis will split the classes in the EJB jar file in 2 sets: one closure that is used by the (home, remote, local) interfaces and one used by the implementation only. In order to build two independant ClassLoaders, the EJB jar file is repacked according to the dependency analysis.

                              Result of phase 1 is the construction of 2 independant ClassLoaders: one for the EJB interfaces (home, remote, local) and one for the EJB implementation.

                              (2) Finding the references to other (EJB) components

                              (a) Looking at ejb-jar.xml for <ejb-ref/> and <ejb-local-ref/> tags, the corresponding EJB interfaces ClassLoaders are found.

                              Result of phase 2 is a set of ClassLoaders to the EJB interfaces (home, remote, local) that are used by this EJB.

                              (3) Construction of conclusive ClassLoader for the EJB

                              The 'real' ClassLoader of the EJB consists of (from parent to child):

                              (a) the EJB interfaces ClassLoader

                              (b) the EJB interfaces ClassLoaders of the referred components/EJBs

                              (c) the EJB implementation ClassLoader.

                              I expect this algorithm to solve the ClassLoader problem in nearly all practical situations without escaping to the Hierachical ClassLoader hack. If, however the (home, remote, local) interfaces are directly dependant on classes that exist in different version within a EJB calling chain, the proposed solution certainly did NOT work. In this case, only a ByValue call would be appropriate (perhaps JBoss should allow declaration of this on component and not on container basis).

                              As an improvement the algorithm could be extended to seperate treatment of remote and local interfaces.

                              Is anybody willing to help me to implement this?

                              Sincerly,

                              aanno

                              • 12. Re: JBoss ClassLoader implementation is unappropriate for EJ

                                I have been a JBoss developer since 2000 working with almost every version of JBoss. It has allways had its own quirks, but has been a charm for a developer. Not so with 3.0. The classloading scheme has really turned into a nightmare. Today I would probably agree that my sugested extentions to the repository mechanism should be considered a hack (it is somehow working but also breaks the hierachical nature of the Java classloader). Currently I do not have a good solution.

                                Your solution sounds even more complicated. You must also remember this:

                                1. The classloader architecture of JBoss is not only there for EJB:s but also for MBeans. An attractive solution must also take this into account.

                                2. Have any of you really tested the ByValue invoker to handle calles beween EJB:s. I do not think it will work. Why?. Because JBoss 3.0 vanilla can not handle remote calles beween EJB containers with container managed transactions, since the TransactionImpl is not serializable.

                                One more thing to note is that since 3.0.6 it is not even possible to place interface and parameter classes globaly if they are also available in any of the ear files involved since if any implementation file is lodaded from the ear; each ear is now only one classloader with all classes/jars and so what part of it's URL:s. If a class is first loaded through the ear classloader it will load every class it can find from that one first.

                                In principal I think you are on the right track. Component based deployment outght to work in the way that you may "export" your interfaces to make them available to other components. ClassLoader vise its seems hard tough. We have componenent A, B, C, D. B and C calles A. But D calles B and C. This would if I get it right require D to depend on two classloaders outside of its reach. How would it do that: by heirarchically having them as parents - breaking Java classloading heirarchy? By delegating to them, also breaking java classloading heirarchy? Or by importing them into its own space?

                                Sorry if this was mostly rambling. Just wanted to give some input on this verry frustrating matter.

                                //Peter

                                • 13. 3768573
                                  jamoville
                                  • 14. Re: JBoss ClassLoader implementation is unappropriate for EJ
                                    moraelin

                                    Except for the classloading issues, JBoss has indeed always been a charm from a developper point of view. Well, it's more fun than WebSphere to work with, that's for sure. I (and pretty much everyone else around here) consider it more technically advanced than WebSphere in some areas, too.

                                    Oh well... Maybe by 4.0 we'll actually be able to go productive with it, too. I can only hope :)

                                    1 2 Previous Next