1 2 Previous Next 18 Replies Latest reply on Apr 14, 2009 7:35 AM by Nicklas Karlsson

    Proper hot deployment, a proposal

    Stuart Douglas Master
      There have been a lot of requests on this forum and in the jira for hot deployment of ejb's. I have a bit of a theory on why this has not been done yet, and why it would not be much of a time saver even if it was done. Hot deployment requires that the classloader containing the old classes be discareded, and new classes loaded into a different classloader, and then the deployment process run on these new classes. This means that we would not be hot deploying one ejb at a time but rather all of them in the archive (for a lot of people this would be all their classes), and it would probably take almost as long as a full redeploy by the time the ejb and seam deployers have finished doing their thing. I think the only way to solve this problem is to allow classes to be reloaded in the VM the same way javarebel does. I think I know how this can be acomplished and I have attached an outline below. If no one can see any show stopping technical reasons as to why this won't work I will try and implement it.


      Even though it is easy to replace a class inside the JVM using the java instrumentation api, it does not let you change the signature after the class has been loaded. My idea of how to get around this limitation is to change classes before they are loaded in such a way as we can fake adding methods etc.

      The use the instrumentation API our class must be compiled as a java agent and installed in the JVM at startup using the –javagent option. Transformation and class redefinition must be enabled in the agent’s manifest file.

      This agent will provide an method call to allow classes to be replaced, the actual job of deciding which classes to replace and supplying the bytecode can be done by code outside the agent, and is not very difficult to implement.

      This agent will supply a re-implementation of the reflection api that is aware of the structure of our instrumented classes. As classes are loaded they are transformed so all calls to the reflection api are redirected through this wrapper. The java instrumentation api provides a methods to inspect and transform clasess as they are loaded.

      As classes that might be replace are loaded they have a method with the signature ‘Object myNewMethodImpl(int,Object[]) added to them. It is through this method that we will fake adding new methods to classes. The int parameter refers to the order that the new method was added (the first method we add is index 0), and the object array contains the parameters for the method call. Calls to a method that we have added are transparently redirected to this method, either through our reflection wrapper api or by transforming new classes as they are loaded (existing classes will not have a reference to this method, otherwise it would have been there at compile time and we would not need all this mucking around).

      If the new class definition is missing a method definition that was present on the original class then a definition that throws a MethodNotFoundError is added to the class at load time.

      If a method has been added the agent creates the bytecode for a whole new class, gives it a random name, and loads it into the JVM. This class has a single static method with the same signature and body as the new method on the class, except that it is static and the first argument is a reference to an instance of the class that is being replaced. The new method definition is removed from the original class and the myNewMethodImpl that was added at class load time has code added to it to call the newly created static method if the correct index is passed. Any classes that are loaded by the JVM from this point on will have calls to the newly added method routed through myNewMethodImpl instead.

      Seam and ejb3 hot deploy will have to be layered on to of this through the use of call backs when a class is replaced, when this happens the framework will be responsible for generating new proxies, scanning for observer methods etc.

      Fields, annotations, constructors etc can be added / removed / replaced in a similar manner to the way methods are added.
      Fields would have an array of objects installed in the class definition to hold values of newly added fields, access to this field is turned into array access. To add constructors we would have to add a constructor MyConstructor(int,Object[]) and proceed in a similar manner to the way methods are added. Annotation support is added purely in the reflection wrapper library.

      The state is stored in static variables inside the agent. This state is used by the reflection wrapper and the instrumentation process.  System classes would not be instrumented, there is no need. Some classes (e.g. application server classes) would only need to be instrumented to rewrite reflection calls to the wrapper.

      Can anyone see any technical reason why this would not work?  It is going to involve a lot of bytecode instrumentation and may slow things down a little bit, but if it works it would give seam and ejb3 the same sort of turn around time php developers enjoy.


        • 1. Re: Proper hot deployment, a proposal
          Arbi Sookazian Master

          I'm not going to comment on the technical feasability of your proposed solution (I am not knowledgeable enough in this area :)


          The response from Jevgeny in the other thread was interesting: My Link


          However, why is it that if it truly is possible to achieve perfect EJB3 hot deploy, that JBoss team has not developed this solution yet?  It would really be a major differentiator b/n JBoss AS and the rest of the JEE middleware market/vendors.


          Everybody is always looking for productivity gains.


          Especially in this economy where all corporations are penny-pinching (i.e. time is money).

          • 2. Re: Proper hot deployment, a proposal
            Pete Muir Master

            Stuart Douglas wrote on Apr 08, 2009 12:45:


            There have been a lot of requests on this forum and in the jira for hot deployment of ejb's. I have a bit of a theory on why this has not been done yet, and why it would not be much of a time saver even if it was done. Hot deployment requires that the classloader containing the old classes be discareded, and new classes loaded into a different classloader, and then the deployment process run on these new classes. This means that we would not be hot deploying one ejb at a time but rather all of them in the archive (for a lot of people this would be all their classes), and it would probably take almost as long as a full redeploy by the time the ejb and seam deployers have finished doing their thing.


            Right, the correct way to do this is to maintain a dependency graph, so that when you reload one EJB, you can reload all it dependencies, however you could easily end up with all ejbs being reloaded. Seam gets around this by making the user do this dependency management, and place only classes they are working on in hot-deploy. I believe JBoss MC can do this individual reloads of beans (as it controls the classloader completely).


            Really, you should discuss this on the JBoss MC forum, as this goes deep into the app server...

            • 3. Re: Proper hot deployment, a proposal
              Arbi Sookazian Master

              It should be added to an EJB JSR in the future, such that all EJB containers are required by spec to support EJB hot deployment.


              Perhaps this is one significant advantage the Spring framework has over the Seam framework, in that it does not have this productivity impediment that EJB3 adopting frameworks like Seam do.  Or correct me if I'm wrong.


              Although in Seam it's possible to use JavaBeans exclusively I suppose...

              • 4. Re: Proper hot deployment, a proposal
                Arbi Sookazian Master

                started the following thread on MC forum:  My Link


                I don't understand why it's necessary that hot deployment discards all the old classes and loads the new ones (i.e. why does the old classloader need to be completely discarded?).  It seems better if the classloader just loads the modfied EJB class only and there is no discarding of anything other than the older version of the EJB class that changed...


                • 5. Re: Proper hot deployment, a proposal
                  Nicklas Karlsson Master

                  I must have confess I have a POJO project that was created that way because of



                  1. No EJB hot deployment

                  2. No EJB 3.1 local homeless interfaces

                  • 6. Re: Proper hot deployment, a proposal
                    Stuart Douglas Master
                    This is because a class that is loaded by two different classloaders is considered two distinct classes, even if they are exaclty the same in all other respects. Also, if you try and load a class with the same name twice in the same classloader the JVM throws a LinkageError.

                    So for the sake of the argument we want to add a business method to a EJB. This means we need to reload the ejb implementation, which is ok as nothing exept for the proxy holds a direct reference to in and we can regenerate the proxy. Unfortunatly we also need to reload the local interface, which means that all classes that reference the local interface need to be reloaded as well, and then all classes that refence those classes need to be reloaded and so on until you have more or less reloaded the majority of your app. Which is the same thing that happens in application server hot deploy does anyway.

                    The easy way to do this is to simply redefine the classes the way java rebel does. The only problem is that this is impossible. The instrumentation interface lets you modify classes before they are loaded, or change the method bodies of classes after they are loaded. It does not allow you to add or remove methods or fields to a class.

                    I think that java rebel fakes it using the method I outlined above. Basically if you add a method with a generic signiture (Object method(int, Object[])) to every class before it is loaded, then transform all classes that are loaded from that point on to point to your generic method (and replace all calls the the reflection api to a transformation aware wrapper), you can fake adding methods to classes by redefineing the method that you added.
                    • 7. Re: Proper hot deployment, a proposal
                      Jevgeni Kabanov Newbie

                      Man, it's nowhere near that simple. Plus AFAIK you're getting on a very shaky legal ground by taking traceable insights from JavaRebel to build your open source stuff on.

                      • 8. Re: Proper hot deployment, a proposal
                        Arbi Sookazian Master

                        Stuart Douglas wrote on Apr 09, 2009 15:37:


                        Unfortunatly we also need to reload the local interface, which means that all classes that reference the local interface need to be reloaded as well, and then all classes that refence those classes need to be reloaded and so on until you have more or less reloaded the majority of your app. Which is the same thing that happens in application server hot deploy does anyway.



                        Local interface is no longer required in EJB3.1.  So perhaps this problem will become irrelevant in EJB3.1...

                        • 9. Re: Proper hot deployment, a proposal
                          Arbi Sookazian Master

                          I emailed the JSR318 group regarding this topic (Kenneth responded to me once recently regarding CMT semantics question) so when I get a reply, I'll regurgitate it here...

                          • 10. Re: Proper hot deployment, a proposal
                            Arbi Sookazian Master

                            You should ask this on the EJB3 (dev) forum.
                            They are already using MC to some extent,
                            but I have no idea how much that helps wrt hot deploy.

                            I know JavaRebel has a support for the new MC ClassLoading layer,
                            perhaps that could already help. And afaik writing a plugin is not that difficult, I know Spring has one,
                            hence it would probably be easy to add one for Seam as well.

                            source: My Link


                            They probably haven't added one for Seam b/c Seam has so little market share in the JEE space...

                            • 11. Re: Proper hot deployment, a proposal
                              Clint Popetz Apprentice

                              Nicklas Karlsson wrote on Apr 09, 2009 07:26:

                              I must have confess I have a POJO project that was created that way because of


                              1. No EJB hot deployment

                              2. No EJB 3.1 local homeless interfaces





                              I'll give a big ditto to that.  I'd add two more to that list:



                              1. If you aren't using EJB or JMS, i.e. you're really just using Seam and JPA, you can run your integration tests without bootstraping jboss-embedded, which makes testing much quicker.

                              2. I've encountered several bugs related to replication in a cluster when using EJBs which went away when using POJOs.



                              I've yet to meet the nail that needs the EJB hammer.

                              • 12. Re: Proper hot deployment, a proposal
                                Stuart Douglas Master
                                There are a few reasons why I use them:

                                - Proper transaction control (I find myself using REQUIRES_NEW a fair bit)
                                - Transaction scoped entity managers
                                - instance pooling and passivation (although if you are not careful instance pooling can cause bugs with outjection)

                                With that said the majority of my components are POJO's.
                                • 13. Re: Proper hot deployment, a proposal
                                  Clint Popetz Apprentice

                                  - Proper transaction control (I find myself using REQUIRES_NEW a fair bit)



                                  I've yet to see the use case for REQUIRES_NEW that can't be recoded in a cleaner fashion to not need it, but that doesn't mean I don't believe it exists :)



                                  - Transaction scoped entity managers


                                  I'm not sure what @PersistenceContext gives you that the SMPC for POJOs (@In EntityManager) doesn't.  Would you care to elaborate?



                                  - instance pooling and passivation (although if you are not careful instance pooling can cause bugs with outjection)


                                  By the time one of my servers could benefit from instance pooling I just add another machine to the cluster, so maybe that's why I've never found it to be a compelling feature.


                                  I'm not trying to play devil's advocate...I really am genuinely interested in hearing about use-cases where using EJBs is worth the effort. 


                                  • 14. Re: Proper hot deployment, a proposal
                                    Francisco Jose Peredo Noguez Master

                                    Clint Popetz wrote on Apr 12, 2009 00:23:



                                    I'm not trying to play devil's advocate...I really am genuinely interested in hearing about use-cases where using EJBs is worth the effort. 


                                    There seems to be a transactional behavior difference, but I just do not really get it (I prefer POJOs anyway, but I would also like know if I am losing something that could help me make my work better).




                                    1 2 Previous Next