1 5 6 7 8 9 Previous Next 132 Replies Latest reply on Apr 18, 2011 3:13 PM by Brian Stansberry Go to original post
      • 105. Thoughts on filesystem action driven hot deployment
        Jason Greene Master

        The old mechanism had its problems but it cannot be outrightly dismissed saying it was broken. It's been in use for a decade and it was one of the key features of JBoss (drag&drop an archive to hotdeploy it, remove to undeploy), a feature that everybody loved and all our competitors copied. Hundrends of thousands of users are already familiar with it. Remember also farming, a mechanism that had even more problems than hotdeployment? We removed it at one point and then we had to add it back because customers liked it no matter how many problems it had.

        Just to address these two points:

         

        Vendors did not carbon copy this feature due to the problems. None of them support hot exploded directories the way jboss does. They either out right disallow them, or support the usage only at startup. They either provide a way for you to drop in a marker to force redeploy, or provide some kind of tool/api to trigger it.

         

        Farming was resurrected because we had no reasonable alternative.

        • 106. Thoughts on filesystem action driven hot deployment
          Jason Greene Master

          Brian Stansberry wrote:

           

          Here's what I'm doing. Following uses the term "auto-deploy" to refer to the scanner deciding to deploy without being told to via a .dodeploy marker.

           

          -snip-

           

          Excellent. Now this is an approach I can get behind.

          • 107. Thoughts on filesystem action driven hot deployment
            Jason Greene Master

            2) Reliable auto-deploy of exploded content is not possible.

             

            We may in the future be able to support it somewhat reliably using FS events, but not without native code (or NIO2 in JDK7):

            http://jpathwatch.wordpress.com/

             

            It would require a well tested algorithm, and it would not have uniform portability.

            • 108. Re: Thoughts on filesystem action driven hot deployment
              jaikiran pai Master

              Brian Stansberry wrote:

               

              Here's what I'm doing. Following uses the term "auto-deploy" to refer to the scanner deciding to deploy without being told to via a .dodeploy marker.

               

              1) I'm adding the zip scanning Jason mentioned to detect whether the zip is complete. This is the basis for reliable auto-deploy of zipped content.

               

              2) Reliable auto-deploy of exploded content is not possible. As a safety aid I will use the scanning code from 1) to check any nested jars, but that doesn't make it reliable.

               

              3) If auto-deploy is available for a type of content, a .skipdeploy marker will prevent auto-deploy of a particular file.

               

              3) I'm going to add an "auto-deploy" flag to the deployment-scanner element, with legal values "ALL", "ZIPPED", "NONE". Flag controls what types of content auto-deploy behavior is enabled for. Per Jason's statement above, default is "ZIPPED", as "ALL" cannot be made reliable. Configuring "ALL" will result in a WARN message in the server log.

               

              4) For auto-deploy of exploded content, there will be no deployment descriptor file timestamp check as the means of detecting the need to redeploy, as in EE 6 there is no universal requirement for deployment descriptors. And checking the directory timestamp is not valid as it's easy to change content without triggering a change in the directory timestamp. So, with auto-deploy if any file in the deployment changes, the deployment gets redeployed. People who want to change content without triggering redeploy are going to have to use marker files; either leave auto-deploy disabled and use .dodeploy, or enable auto-deploy and use .skipdeploy.

               

              5) If a scan tries to use auto-deploy for a file and detects incomplete content, the scan will be aborted (i.e. it will be as if no changes were detected.) The next time the scanner runs, it will check again.

               

               

              Sounds good!

               

              One final question:

               

              2) Reliable auto-deploy of exploded content is not possible. As a safety aid I will use the scanning code from 1) to check any nested jars, but that doesn't make it reliable.

              So triggering a deployment of exploded content (with the auto-deploy flag set to ZIPPED) would require a .dodeploy marker right? I have no issues with using a marker for this specific case - just want to understand the expected behaviour.

              • 109. Re: Thoughts on filesystem action driven hot deployment
                Dimitris Andreadis Master

                jaikiran pai wrote:

                 

                One final question:

                 

                2) Reliable auto-deploy of exploded content is not possible. As a safety aid I will use the scanning code from 1) to check any nested jars, but that doesn't make it reliable.

                So triggering a deployment of exploded content (with the auto-deploy flag set to ZIPPED) would require a .dodeploy marker right? I have no issues with using a marker for this specific case - just want to understand the expected behaviour.

                I think Brian meant a .dodeploy marker will be needed if the setting is NONE. If the setting is ZIPPED it will look for a change in any nested zipped archive.

                 

                BTW, thanks guys for looking into this.

                • 110. Thoughts on filesystem action driven hot deployment
                  Max Rydahl Andersen Master

                  Jason Greene wrote:

                   

                  The old mechanism had its problems but it cannot be outrightly dismissed saying it was broken. It's been in use for a decade and it was one of the key features of JBoss (drag&drop an archive to hotdeploy it, remove to undeploy), a feature that everybody loved and all our competitors copied. Hundrends of thousands of users are already familiar with it. Remember also farming, a mechanism that had even more problems than hotdeployment? We removed it at one point and then we had to add it back because customers liked it no matter how many problems it had.

                  Just to address these two points:

                   

                  Vendors did not carbon copy this feature due to the problems. None of them support hot exploded directories the way jboss does. They either out right disallow them, or support the usage only at startup. They either provide a way for you to drop in a marker to force redeploy, or provide some kind of tool/api to trigger it.

                   

                  Farming was resurrected because we had no reasonable alternative.

                   

                  Last time I tried tomcat it supported both zipped and exploded hot deployment - and Glassfish likewise. What competition do you refer to ?

                  • 111. Thoughts on filesystem action driven hot deployment
                    Max Rydahl Andersen Master

                    Also since I am replying again, I just want to mention that windows hasn't defaulted to diaplaying 3 char extensions in a long long time. In fact I would argue that java 6 probably doesn't even support these versions.

                    Just to be clear, the issue about having the first characters be "unique" were not at all about Windows/DOS 3 char extension display.

                     

                    It was Brian and me going through the state changes on a machine at Devoxx and viewing it from a file browser (i.e. Explorer and Finder) and noticing that the UI's by default were showing ".dep.." and ".dep..." when we were using .deploy and .deployed as the markers.

                     

                    That's why it was suggested - not because of ancient OS reasons.

                    • 112. Thoughts on filesystem action driven hot deployment
                      Jason Greene Master

                      Max Rydahl Andersen wrote:

                       

                      Last time I tried tomcat it supported both zipped and exploded hot deployment - and Glassfish likewise. What competition do you refer to ?

                       

                      Glassfish uses a .redeploy marker. It is true that Tomcat is the closest to JBoss, but it also doesn't implement non-war deployments.

                      • 113. Re: Thoughts on filesystem action driven hot deployment
                        Max Rydahl Andersen Master

                        I go on a small trip and this thread goes wild - sorry for being late to the "party".

                         

                         

                        1) I'm adding the zip scanning Jason mentioned to detect whether the zip is complete. This is the basis for reliable auto-deploy of zipped content.

                         

                         

                        Great - makes perfect sense.

                         

                        2) Reliable auto-deploy of exploded content is not possible. As a safety aid I will use the scanning code from 1) to check any nested jars, but that doesn't make it reliable.

                         

                        Okey...not sure if it will worth the hassle/CPU-cycles but worth experimenting with.

                         

                         

                        3) If auto-deploy is available for a type of content, a .skipdeploy marker will prevent auto-deploy of a particular file.

                         

                        ..particular file or directory, right ?

                        btw. I really like this "veto" option Much easier to implement from user/tooling perspective.

                         

                        3) I'm going to add an "auto-deploy" flag to the deployment-scanner element, with legal values "ALL", "ZIPPED", "NONE". Flag controls what types of content auto-deploy behavior is enabled for. Per Jason's statement above, default is "ZIPPED", as "ALL" cannot be made reliable. Configuring "ALL" will result in a WARN message in the server log.

                         

                        Just to be sure I understand this correctly - this means by default AS7

                         

                        cp -r foo.war AS7/deployments

                         

                        will trigger an deployment of foo.war when it is an archive when the scanner sees the war and it is complete.

                         

                        it will *not* trigger deployment of foo.war when it is a directory. Correct ?

                         

                        4) For auto-deploy of exploded content, there will be no deployment descriptor file timestamp check as the means of detecting the need to redeploy, as in EE 6 there is no universal requirement for deployment descriptors. And checking the directory timestamp is not valid as it's easy to change content without triggering a change in the directory timestamp. So, with auto-deploy if any file in the deployment changes, the deployment gets redeployed. People who want to change content without triggering redeploy are going to have to use marker files; either leave auto-deploy disabled and use .dodeploy, or enable auto-deploy and use .skipdeploy.

                        It took me a couple of reads of #3 and #4 to get what this meant but this actually makes perfect sense and gives a good balance

                        between preventing bad deployments and still have incremental updates of your exploded content.

                         

                        This basically means the *only* new thing by default is having to do a

                         

                        touch foo.war.dodeploy

                         

                        or the more portable:

                         

                        echo > foo.war.dodeploy

                         

                        for exploded directories. Everything else is just as "before" but better

                         

                        i.e. to make it clean its important that if you detect both a complete foo.war and a foo.war.dodeploy the .dodeploy still gets deleted and goes through the transitions; and that there will be a .deployed file no matter what triggered the deploy.

                         

                        5) If a scan tries to use auto-deploy for a file and detects incomplete content, the scan will be aborted (i.e. it will be as if no changes were detected.) The next time the scanner runs, it will check again.

                         

                        Perfect from first read, but i'm still curious how long that consistency check will take for large projects ?

                        And should there be a maximum of redeployment attempts to avoid the logs for filling/cpu cycles burning ?

                        Maybe have the user made aware of the .dodeploy failing and make it a .deployfailed after some time and thus require a new touch ?

                        (should be fine since if user is doing .dodeploy we should assume that the deployment is actually expected to be complete?)

                         

                        AFAICT, this is essentially completing what was discussed in Antwerp and detailed a few pages back by Max. Difference is auto-deploy of exploded content will not be default behavior.

                         

                        Yes, this works for me   I basically get all the goodies I wished for.

                        • 114. Thoughts on filesystem action driven hot deployment
                          Brian Stansberry Master

                          Apologies for not responding to the questions on this thread -- I made myself ignore the thread for a day to do some other stuff. I'll reply to all of  them later today.

                          • 115. Thoughts on filesystem action driven hot deployment
                            Brian Stansberry Master

                            jaikiran pai wrote:

                             

                            One final question:

                             

                            2) Reliable auto-deploy of exploded content is not possible. As a safety aid I will use the scanning code from 1) to check any nested jars, but that doesn't make it reliable.

                            So triggering a deployment of exploded content (with the auto-deploy flag set to ZIPPED) would require a .dodeploy marker right? I have no issues with using a marker for this specific case - just want to understand the expected behaviour.

                             

                            Correct.

                            • 116. Thoughts on filesystem action driven hot deployment
                              Brian Stansberry Master

                              Max Rydahl Andersen wrote:


                               

                              3) If auto-deploy is available for a type of content, a .skipdeploy marker will prevent auto-deploy of a particular file.

                               

                              ..particular file or directory, right ?

                              btw. I really like this "veto" option Much easier to implement from user/tooling perspective.

                               

                              Yes, zipped archive or exploded deployment.

                               

                               

                              3) I'm going to add an "auto-deploy" flag to the deployment-scanner element, with legal values "ALL", "ZIPPED", "NONE". Flag controls what types of content auto-deploy behavior is enabled for. Per Jason's statement above, default is "ZIPPED", as "ALL" cannot be made reliable. Configuring "ALL" will result in a WARN message in the server log.

                               

                              Just to be sure I understand this correctly - this means by default AS7

                               

                              cp -r foo.war AS7/deployments

                               

                              will trigger an deployment of foo.war when it is an archive when the scanner sees the war and it is complete.

                               

                              it will *not* trigger deployment of foo.war when it is a directory. Correct ?

                               

                              Yep.

                               

                               


                              4) For auto-deploy of exploded content, there will be no deployment descriptor file timestamp check as the means of detecting the need to redeploy, as in EE 6 there is no universal requirement for deployment descriptors. And checking the directory timestamp is not valid as it's easy to change content without triggering a change in the directory timestamp. So, with auto-deploy if any file in the deployment changes, the deployment gets redeployed. People who want to change content without triggering redeploy are going to have to use marker files; either leave auto-deploy disabled and use .dodeploy, or enable auto-deploy and use .skipdeploy.

                              It took me a couple of reads of #3 and #4 to get what this meant but this actually makes perfect sense and gives a good balance

                              between preventing bad deployments and still have incremental updates of your exploded content.

                               

                              This basically means the *only* new thing by default is having to do a

                               

                              touch foo.war.dodeploy

                               

                              or the more portable:

                               

                              echo > foo.war.dodeploy

                               

                              for exploded directories. Everything else is just as "before" but better

                               

                              i.e. to make it clean its important that if you detect both a complete foo.war and a foo.war.dodeploy the .dodeploy still gets deleted and goes through the transitions; and that there will be a .deployed file no matter what triggered the deploy.

                               

                              Yes. The .deployed file will get laid down either way. If you have auto-deploy enabled and put down a .dodeploy anyway, it will be removed at the end no matter which branch of the logic detects the need to deploy.

                               

                               


                               

                              5) If a scan tries to use auto-deploy for a file and detects incomplete content, the scan will be aborted (i.e. it will be as if no changes were detected.) The next time the scanner runs, it will check again.

                               

                              Perfect from first read, but i'm still curious how long that consistency check will take for large projects ?

                              And should there be a maximum of redeployment attempts to avoid the logs for filling/cpu cycles burning ?

                              Maybe have the user made aware of the .dodeploy failing and make it a .deployfailed after some time and thus require a new touch ?

                              (should be fine since if user is doing .dodeploy we should assume that the deployment is actually expected to be complete?)

                               

                              For a zip that's complete it should be fast. Open a FileChannel and read a few bytes near the very end of the file (in the end of central directory record), then jump to a copy other spots (start of central directory and first local file header) and read a few bytes to validate. If the zip is incomplete it would have to scan back through the last 65K in the file, jumping 4 bytes at a time, so that would be a bit slower.

                               

                              Re: the logs, the way it's coded now it logs once. It won't spam the logs.

                               

                              It doesn't validate the archive if you use .dodeploy. We assume the user means what they say. If someone wants that, I'll look at a patch.

                               

                              I'll look at what it would take to time out and write a .faileddeploy when I get back into this over the next day or so. I think it would be simple.

                              • 117. Thoughts on filesystem action driven hot deployment
                                Brian Stansberry Master

                                Typo in my 2nd to last paragraph above: "then jump to a copy other spots" should be "then jump to a couple other spots".

                                 

                                I'm sure there are others, but saying "copy" seemed bad.

                                • 118. Thoughts on filesystem action driven hot deployment
                                  Brian Stansberry Master

                                  Dimitris Andreadis wrote:

                                   

                                  When it comes to marker names without a uniqueness issue I'd go for:

                                   

                                  .skipdeploy

                                  .isdeploying

                                  .deployed

                                  .failed

                                  .isundeploying

                                  .undeployed

                                   

                                  Thanks for this. I joke (ok, ok, I whine) about bikeshed painting, but these are actually very nice bikeshed colors, so much appreciated.

                                   

                                  The stuff I described in this thread, including Dimitris' marker file names, is now in upstream:

                                   

                                  https://github.com/jbossas/jboss-as/commit/6a72ca965f01021d034049598451e0aca3ba38f3

                                   

                                  See also https://github.com/jbossas/jboss-as/blob/6a72ca965f01021d034049598451e0aca3ba38f3/build/src/main/resources/standalone/deployments/README.txt for a more thorough description of the FS scanner behavior.

                                  • 119. Thoughts on filesystem action driven hot deployment
                                    Wolfgang Knauf Master

                                    Sorry for coming up with a user question here, but I observe a "strange" behavior with auto deployment: I copied an EAR file to the "standalone\deployments" directory (using a JBoss picked from GIT two hours ago). Deployment fails and a "..ear.failed" file is created.

                                    So I tried to "undeploy" the app by killing the "...ear.failed" file. But this resulted in a undeploy and redeploy of the EAR (with another failure of course).

                                     

                                    Is this the expected behaviour? In case of an error, is it possible to just delete the EAR file and the "ear.failed" file to remove the app from the server?

                                    If yes, it would be worth to add this to "README.txt" ;-).

                                     

                                    Best regards

                                     

                                    Wolfgang

                                    1 5 6 7 8 9 Previous Next