1 2 3 4 Previous Next 132 Replies Latest reply on Apr 18, 2011 3:13 PM by brian.stansberry Go to original post
      • 15. Re: Thoughts on hot deployment
        brian.stansberry

        True!

         

        Argh; I tried to rename this thread to "Thoughts on filesystem action based hot deployment" since that's a big enough topic by itself. But that just changed the title of the first post.

        • 16. Re: Thoughts on hot deployment
          dimitris

          Jason Greene wrote:

           

          jaikiran pai wrote:

           

          Nice!

           

          Here's some questions I have:

          Brian Stansberry wrote:

           

          For zipped archives:

           

          Scanner will detect an archive in the hot-deploy dir, e.g. hot-deploy/foo.war. Scanner will use the same deployment API that an external client would use to register the content with the server and deploy it. Once the content is registered the server will maintain it's own copy in an internal directory (e.g. server/data/deployments). Scanner will replace the hot-deploy/foo.war file with a simple marker file hot-deploy/foo.war.deployed.

          Would it be a "rename" or would the foo.war.deployed be a completely different file? From what I understand, I believe  it would be a completely different file without any .war semantics, but just wanted to be sure. Furthermore, where would be original foo.war be moved to? Are users expected to know where it's moved to? If not, then they probably will have to maintain a copy of their original deployment somewhere (I know most of them already do that, but again I just wanted to be clear about the expectations).

           

          Brian meant a different file, and they shouldn't necessarily be poking in the data area, which I admit is not intuitive. That's a very good point. It's also  inconsistent with the exploded dir, which preserves the original. We could alternatively leave the file there and just add the .deployed as an indicator that it was deployed. The only side effect is that it would require a copy vs a move, but IMO not a huge deal for zips.

          If you leave the original archive in place, I think there is little point having a marker file there hanging. Logs/console/cli should tell you if a deployment was successfull or not and for index purposes or otherwise you can always keep state somewhere else, outside the users view.

           

          Unless you want to have archives in ./deploy and not wanting them to deploy? (but then why have them there in first place).

           

          Now, regarding exploded deployments, why not using the containing directory to be the marker itself? "Touching" the upper level directory conceptually is similar to touching a zipped archive. Although it's not so straightforward to do on Windoze.

          • 17. Re: Thoughts on hot deployment
            jason.greene

            Brian meant a different file, and they shouldn't necessarily be poking in the data area, which I admit is not intuitive. That's a very good point. It's also  inconsistent with the exploded dir, which preserves the original. We could alternatively leave the file there and just add the .deployed as an indicator that it was deployed. The only side effect is that it would require a copy vs a move, but IMO not a huge deal for zips.

            If you leave the original archive in place, I think there is little point having a marker file there hanging. Logs/console/cli should tell you if a deployment was successfull or not and for index purposes or otherwise you can always keep state somewhere else, outside the users view.

             

            Unless you want to have archives in ./deploy and not wanting them to deploy? (but then why have them there in first place)

            Yes the state could be stored elsewhere, basically a timestamp and file size should be sufficient to determine if the file has indeed changed. It's only value with the copy approach is to provide a quick indication that something is currently deployed.

            Now, regarding exploded deployments, why not using the containing directory to be the marker itself? "Touching" the upper level directory conceptually is similar to touching a zipped archive. Although it's not so straightforward to do on Windoze.

            That still doesnt solve the undeploy problem, where the content will disappear out from under the classloader and cause frameworks like seam to fail. It also doesn't help the problem where atomic moves aren't possible for one reason or the other.

            • 18. Re: Thoughts on hot deployment
              brian.stansberry

              Is it easier just to copy the exploded deployment and have the scanner keep the copy in sync. I've done that for farming; it's not such a big deal. I'm starting to feel like not doing that is leading to a lot of internal complication (e.g. needing to keep track of multiple locations where content is stored.)

               

              That doesn't solve the atomic move problem, but maybe if people can't do atomic moves they should use the filesystem as their deployment API.

              • 19. Re: Thoughts on hot deployment
                brian.stansberry

                I mean they *should not* use the filesystem as their deployment API.

                • 20. Re: Thoughts on hot deployment
                  dmlloyd

                  Brian Stansberry wrote:

                   

                  Is it easier just to copy the exploded deployment and have the scanner keep the copy in sync. I've done that for farming; it's not such a big deal. I'm starting to feel like not doing that is leading to a lot of internal complication (e.g. needing to keep track of multiple locations where content is stored.)

                   

                  That doesn't solve the atomic move problem, but maybe if people can't do atomic moves they should [not] use the filesystem as their deployment API.

                   

                  For a single JAR it's a much simpler proposition to avoid partial copies than it is for exploded deployments.  One merely has to ignore files which were modified in the past, say, 500 ms.  This very greatly reduces the likelihood of a partial copy without really sacrificing latency significantly.

                   

                  This doesn't work for directories as most filesystems won't update the directory timestamp when one of its children are updated, and in no cases I can think of offhand for deep children.

                  • 21. Re: Thoughts on hot deployment
                    jason.greene

                    David Lloyd wrote:

                     

                    Brian Stansberry wrote:

                     

                    Is it easier just to copy the exploded deployment and have the scanner keep the copy in sync. I've done that for farming; it's not such a big deal. I'm starting to feel like not doing that is leading to a lot of internal complication (e.g. needing to keep track of multiple locations where content is stored.)

                     

                    That doesn't solve the atomic move problem, but maybe if people can't do atomic moves they should [not] use the filesystem as their deployment API.

                     

                    For a single JAR it's a much simpler proposition to avoid partial copies than it is for exploded deployments.  One merely has to ignore files which were modified in the past, say, 500 ms.  This very greatly reduces the likelihood of a partial copy without really sacrificing latency significantly.

                     

                    This doesn't work for directories as most filesystems won't update the directory timestamp when one of its children are updated, and in no cases I can think of offhand for deep children.

                     

                    You know if you look at AS4 behavior from the user perspective the proposal it not really THAT different.

                     

                    Instead of touching web.xml for a redeploy they touch "jboss.redeploy" or whatever. The only real significant difference is where jboss.redeploy does not exist until they want it to start. But hey they can still do it the old with atomic moves, just leave a jboss.redeploy file in there and do the atomic move.

                    • 22. Re: Thoughts on hot deployment
                      dimitris

                      Brian Stansberry wrote:

                       

                      Is it easier just to copy the exploded deployment and have the scanner keep the copy in sync. I've done that for farming; it's not such a big deal. I'm starting to feel like not doing that is leading to a lot of internal complication (e.g. needing to keep track of multiple locations where content is stored.)

                       

                      That doesn't solve the atomic move problem, but maybe if people can't do atomic moves they should use the filesystem as their deployment API.

                      If the cost of deep copying and keeping in synch the exploded archive is not that high, that would solve the undeploy problems. Although I would personally leave the option to the user whether to disable deep copy of deployments, either globally (system property) or per deployment (e.g. META-INF/jboss-no-deep-copy).

                       

                      On the same topic, I'm not sure if you are aware or if this still holds true. In 4.x we essentially copyied over to tmp/deploy after prefixing some random name all archives from deploy on which we had to point a classloader, because the cl would lock the file and couldn't be deleted until the next restart. Same for any nested archive discovered through the expansion process (e.g. an ejb.jar found inside a zipped or an exploded .ear) So removing those archives from the copy location may not be possible.

                      • 23. Re: Thoughts on hot deployment
                        dmlloyd

                        Dimitris Andreadis wrote:

                         

                        Brian Stansberry wrote:

                         

                        Is it easier just to copy the exploded deployment and have the scanner keep the copy in sync. I've done that for farming; it's not such a big deal. I'm starting to feel like not doing that is leading to a lot of internal complication (e.g. needing to keep track of multiple locations where content is stored.)

                         

                        That doesn't solve the atomic move problem, but maybe if people can't do atomic moves they should use the filesystem as their deployment API.

                        If the cost of deep copying and keeping in synch the exploded archive is not that high, that would solve the undeploy problems. Although I would personally leave the option to the user whether to disable deep copy of deployments, either globally (system property) or per deployment (e.g. META-INF/jboss-no-deep-copy).

                         

                        Actually this doesn't really solve the problem at all.  Say you start deleting your deployment.  At what point does the scanner decide that you're not updating your deployment by removing files, you're actually deleting it?  The same problem basically exists except now you have the additional complication of having to detect and sync up changes.  My opinion remains: using delete to signal undeploy should not be done, especially for exploded deployments.

                         

                        Thus, I don't see any reason to copy exploded deployments.  I just don't think it buys us anything.

                        • 24. Re: Thoughts on hot deployment
                          dimitris

                          David Lloyd wrote:

                           

                          Dimitris Andreadis wrote:

                           

                          Brian Stansberry wrote:

                           

                          Is it easier just to copy the exploded deployment and have the scanner keep the copy in sync. I've done that for farming; it's not such a big deal. I'm starting to feel like not doing that is leading to a lot of internal complication (e.g. needing to keep track of multiple locations where content is stored.)

                           

                          That doesn't solve the atomic move problem, but maybe if people can't do atomic moves they should use the filesystem as their deployment API.

                          If the cost of deep copying and keeping in synch the exploded archive is not that high, that would solve the undeploy problems. Although I would personally leave the option to the user whether to disable deep copy of deployments, either globally (system property) or per deployment (e.g. META-INF/jboss-no-deep-copy).

                           

                          Actually this doesn't really solve the problem at all.  Say you start deleting your deployment.  At what point does the scanner decide that you're not updating your deployment by removing files, you're actually deleting it?  The same problem basically exists except now you have the additional complication of having to detect and sync up changes.  My opinion remains: using delete to signal undeploy should not be done, especially for exploded deployments.

                           

                          Thus, I don't see any reason to copy exploded deployments.  I just don't think it buys us anything.

                          I meant it would solve the problem of missing classes/resources during undeployment.

                           

                          In any case, I don't think you should worry too much about the timing related problems of deploying/undeploying exploded deployments. We have this problem for ever and yet very few complaint because they understand the limitations of the mechanism and how to deal with it. So IMO simplicity (copy/delete) wins over correctness since the main use case for altering exploded deployments is development mode.

                           

                          Production mode users know that to be 100% safe they just need to make the directory based deployment changes off-line. Or use a proper API that can offer all sorts of guarantees (e.g. tx behaviour).

                           

                          So offering the appropriate underying API/functionality to support the more advanced scenarios while keeping a simple file-based hot-delployment mechanism that people are familiar with, I think is more important than creating the perfect file-based API abstraction.

                           

                          BTW, in some older AS version we added the ability to tell the scanner to ignore/re-enable a particular deployment, to allow the IDE to make file updates without the fear of the server picking up incomplete deployments.

                          • 25. Re: Thoughts on hot deployment
                            jason.greene

                            Dimitris Andreadis wrote:

                             

                            David Lloyd wrote:

                             

                            Dimitris Andreadis wrote:

                             

                            Brian Stansberry wrote:

                             

                            Is it easier just to copy the exploded deployment and have the scanner keep the copy in sync. I've done that for farming; it's not such a big deal. I'm starting to feel like not doing that is leading to a lot of internal complication (e.g. needing to keep track of multiple locations where content is stored.)

                             

                            That doesn't solve the atomic move problem, but maybe if people can't do atomic moves they should use the filesystem as their deployment API.

                            If the cost of deep copying and keeping in synch the exploded archive is not that high, that would solve the undeploy problems. Although I would personally leave the option to the user whether to disable deep copy of deployments, either globally (system property) or per deployment (e.g. META-INF/jboss-no-deep-copy).

                             

                            Actually this doesn't really solve the problem at all.  Say you start deleting your deployment.  At what point does the scanner decide that you're not updating your deployment by removing files, you're actually deleting it?  The same problem basically exists except now you have the additional complication of having to detect and sync up changes.  My opinion remains: using delete to signal undeploy should not be done, especially for exploded deployments.

                             

                            Thus, I don't see any reason to copy exploded deployments.  I just don't think it buys us anything.

                            I meant it would solve the problem of missing classes/resources during undeployment.

                             

                            In any case, I don't think you should worry too much about the timing related problems of deploying/undeploying exploded deployments. We have this problem for ever and yet very few complaint because they understand the limitations of the mechanism and how to deal with it. So IMO simplicity (copy/delete) wins over correctness since the main use case for altering exploded deployments is development mode.

                             

                            The whole problem is the past approach doesnt work AT ALL. How do you trigger a redeploy of an exploded directory in a servlet 3.0 application that has no web.xml? So no matter what we do it will be different.

                            • 26. Re: Thoughts on hot deployment
                              dimitris

                              Jason Greene wrote:

                               

                               

                              The whole problem is the past approach doesnt work AT ALL. How do you trigger a redeploy of an exploded directory in a servlet 3.0 application that has no web.xml? So no matter what we do it will be different.

                              You obviously need to base the decision on the fact that something changed (file added/removed/edited). The smarter way would require somekind of knowledge per deployment type about where to look for changes. The dumb way would blindly assume the unit needs redeployment if anything changed (and in the case that Brian pointed out that you copy everything over, you know what's changed on every pass).

                              • 27. Re: Thoughts on hot deployment
                                dmlloyd

                                Dimitris Andreadis wrote:

                                 

                                David Lloyd wrote:

                                 

                                Actually this doesn't really solve the problem at all.  Say you start deleting your deployment.  At what point does the scanner decide that you're not updating your deployment by removing files, you're actually deleting it?  The same problem basically exists except now you have the additional complication of having to detect and sync up changes.  My opinion remains: using delete to signal undeploy should not be done, especially for exploded deployments.

                                 

                                Thus, I don't see any reason to copy exploded deployments.  I just don't think it buys us anything.

                                I meant it would solve the problem of missing classes/resources during undeployment.

                                 

                                No, it wouldn't.  The scanner might not even realize I'm undeploying until I've deleted half of my class files.  If we start adding hacks to detect file types then where has our "simplicity" gone?

                                 

                                Dimitris Andreadis wrote:

                                 

                                In any case, I don't think you should worry too much about the timing related problems of deploying/undeploying exploded deployments. We have this problem for ever and yet very few complaint because they understand the limitations of the mechanism and how to deal with it. So IMO simplicity (copy/delete) wins over correctness since the main use case for altering exploded deployments is development mode.

                                 

                                A single marker file delete is much simpler than deleting the whole directory for undeploy, so that point doesn't seem valid to me.  Also, simplicity never wins over correctness, period.

                                 

                                Dimitris Andreadis wrote:

                                 

                                Production mode users know that to be 100% safe they just need to make the directory based deployment changes off-line. Or use a proper API that can offer all sorts of guarantees (e.g. tx behaviour).

                                There's really no need to make this concession.  I mean basically you're arguing that something that is clumsy to implement and incorrect is better than something clean & simple, because we've always done it that way.  I don't really buy this argument.

                                 

                                Dimitris Andreadis wrote:

                                 

                                BTW, in some older AS version we added the ability to tell the scanner to ignore/re-enable a particular deployment, to allow the IDE to make file updates without the fear of the server picking up incomplete deployments.

                                 

                                Another example why the solution you present which you call "simple" is more complex than the solution presented here which you call "complex", but is actually simpler

                                • 28. Re: Thoughts on hot deployment
                                  jason.greene

                                  Dimitris Andreadis wrote:

                                   

                                  Jason Greene wrote:

                                   

                                   

                                  The whole problem is the past approach doesnt work AT ALL. How do you trigger a redeploy of an exploded directory in a servlet 3.0 application that has no web.xml? So no matter what we do it will be different.

                                  You obviously need to base the decision on the fact that something changed (file added/removed/edited). The smarter way would require somekind of knowledge per deployment type about where to look for changes. The dumb way would blindly assume the unit needs redeployment if anything changed (and in the case that Brian pointed out that you copy everything over, you know what's changed on every pass).

                                  The latter approach was attempted in 5 and we had to rollback to 4 behavior because of the number of complaints issues around spurious errors it would cause. The former approach requires some kind of identifier, but we can't use deployment descriptors, since they might not be there. We can't use class files because they aren't required either (they might be in WEB-INF/lib).  We also can't use the directory for the problems you mentioned earlier. So using a special deploy file is pretty much the only option, and IMO that's more intuitive than touching the TLD, since it's at least consistent across deployments.

                                  • 29. Re: Thoughts on filesystem action driven hot deployment
                                    toddiuszho

                                    I've been experiencing a problem with atomic moves in AS5 under Windows. About twice a day, AS5's JVM refuses to release a file handle to META-INF/application.xml or an EJB descriptor file. I cannot swap out the old exploded archive when this happens. This is perplexing since it's not like AS5 is hung. All my apps are still working fine.

                                     

                                    In light of this, I would much rather have to touch/copy/delete a marker file than rely on atomic moves.

                                     

                                    Also, while I'm sure the HDScanner is performant, the amount of juggling it has to do seems like a convoluted solution for what shouldn't be such a big problem. But then again, if it were that easy, this thread wouldn't even exist.

                                    1 2 3 4 Previous Next