1 2 Previous Next 21 Replies Latest reply on Jun 17, 2010 4:51 PM by starksm64

    Editing domain.xml

    aloubyansky

      domain.xml potentially includes everything, just about all kinds of data there can be related to the AS and its environment. From profiles and high level components to JVM args, IP addresses, ports, etc.

       

      It's potentially a large and complex file with coarse and fine grained configs mixed. I can imagine it might be difficult by openning and reading this file to get a big picture of the domain. That might be just my imagination, though. Anyway, it's still complex and large file, possibly.

       

      How is it supposed to be edited? To begin with, probably, manually. But given the complexity, in perspective, probably more with the tools. JON, admin console, command line, etc. Now ones there are tools, who will want to edit the file manually? Which means domain.xml becomes an implementation detail?

        • 1. Re: Editing domain.xml
          starksm64

          As we have talked about this so far, one can edit the domain.xml directly, but not while the server is running. Generally, some tool front end will be dealing with the model and writing the result back out as the domain.xml. It is not an implementation detail, but an authoritative view of the domain metadata model.

          • 2. Re: Editing domain.xml
            dmlloyd

            This raises an interesting problem.  If the DC is on line, and a management operation occurs which modifies the domain model, the operation can simply be stored in the change log and propagated to all the server managers in the domain as they are able to consume it, thus they can update their local view of the domain model simply by applying a difference (not a text diff per se but a semantic diff) to it.

             

            Now if the DC is shut down and the domain.xml is modified manually, unless we have some very clever algorithm to determine the difference, the DC will have to signal to all server managers to reload a clean domain.xml, which will require a restart of all servers (as there's no way to determine what has changed).

             

            Unless someone has a better idea?

            • 3. Re: Editing domain.xml
              jason.greene

              Now if the DC is shut down and the domain.xml is modified manually, unless we have some very clever algorithm to determine the difference, the DC will have to signal to all server managers to reload a clean domain.xml, which will require a restart of all servers (as there's no way to determine what has changed).

               

              If the DC has a copy of the last known state (either serialized data, or a snapshot of the file), then it can determine the differences.

              • 4. Re: Editing domain.xml
                smarlow

                Would it help if we could generate a server health report that includes validating that every server is in agreement to what the authoritative view is.  Perhaps that would help defend against bugs (e.g. update message ordering) or critical errors causing some servers to run with incorrect settings.  Corrective action could be taken to resolve any server out of sync.

                • 5. Re: Editing domain.xml
                  smarlow

                  Or we could just check the hash of the domain.xml and attempt to sync up with the newer domain.xml.  The idea being that users have control over when this checking is done to validate the correctness of the installation.

                  • 6. Re: Editing domain.xml
                    dmlloyd

                    Scott Marlow wrote:

                     

                    Would it help if we could generate a server health report that includes validating that every server is in agreement to what the authoritative view is.  Perhaps that would help defend against bugs (e.g. update message ordering) or critical errors causing some servers to run with incorrect settings.  Corrective action could be taken to resolve any server out of sync.

                     

                    The plan is to maintain a sort of "transaction log" which the remote server managers can use to play forward any changes that have occurred to the domain since they last synced up.

                    • 7. Re: Editing domain.xml
                      dmlloyd

                      Jason Greene wrote:

                       

                      Now if the DC is shut down and the domain.xml is modified manually, unless we have some very clever algorithm to determine the difference, the DC will have to signal to all server managers to reload a clean domain.xml, which will require a restart of all servers (as there's no way to determine what has changed).

                       

                      If the DC has a copy of the last known state (either serialized data, or a snapshot of the file), then it can determine the differences.

                       

                      ...in theory.  If it's an arbitrary object model though, that can be pretty complex.  If you do it via XML you'd have to detect structural differences and map them to management operations; if you do it via the object model you'd have to basically have a subtraction operation: domain A - domain B = List<ManagementOperation>.

                      • 8. Re: Editing domain.xml
                        dmlloyd

                        David Lloyd wrote:

                         

                        Jason Greene wrote:

                         

                        Now if the DC is shut down and the domain.xml is modified manually, unless we have some very clever algorithm to determine the difference, the DC will have to signal to all server managers to reload a clean domain.xml, which will require a restart of all servers (as there's no way to determine what has changed).

                         

                        If the DC has a copy of the last known state (either serialized data, or a snapshot of the file), then it can determine the differences.

                         

                        ...in theory.  If it's an arbitrary object model though, that can be pretty complex.  If you do it via XML you'd have to detect structural differences and map them to management operations; if you do it via the object model you'd have to basically have a subtraction operation: domain A - domain B = List<ManagementOperation>.

                         

                        I discussed this on the phone with Jason (to an extent) and I figure this whole thing boils down to two options, which I'll call "simple" and "complex but cool".

                        1. Simple:
                          1. Editing of domain.xml is allowed offline or possibly even online.  When domain.xml is edited and saved, the domain controller (DC) puts a marker in the transaction log like "Refresh domain model", which indicates to the server managers (SMs) that the domain model needs a complete refresh.  This differs from the normal transaction log entry which would consist of domain updates.
                          2. When a SM discovers this marker in the domain transaction log, it will indicate that further updates to its local domain view are not possible without a restart.
                          3. When a SM is restarted, before syncing with the DC, it will scan the transaction log and if there is one or more "Refresh" markers in it.  If so, it will start its sync process starting from the last such marker, which includes the newest "base" domain model from which to work (this way, updates which are not relevant will not be uselessly processed).
                          4. Pros: Easy to implement.
                          5. Cons: Not very powerful.  Your edits of the domain.xml won't take full effect without restarting every single SM.  Might be hard to undo a domain.xml replacement (we'd have to put a complete copy of the old domain model into the txn log alongside the new one in order to be able to provide a compensating transaction).
                        2. Complex (but cool):
                          1. Editing of domain.xml is allowed offline or possibly online.  When domain.xml is edited and saved, the DC builds up a secondary object model corresponding to the new domain state.  It then performs a difference operation on the two models.  The difference between two domain models is expressed as series of domain updates.  This is done by making sure that every object class in the domain model has an abstract "difference" operation which processes the equivalent structure in another domain model and calculates the updates which are required to bring one in sync with another.
                          2. The DC then forwards the list of changes as one transaction to all SMs, each of which process the changes as a regular update.
                          3. Pros: Allows a simpler domain update txn log; provides a consistent way to back out a domain.xml update; possibly more efficient handling of domain changes; may even allow us to support hot-deployment of domain.xml
                          4. Cons: The complexity of calculating a difference in the whole object model might be significant.  This can be mitigated to some extent by enforcing constraints on domain model classes (i.e. a common base class or interface which stipulates the difference operation(s)).
                        Discuss...
                        • 9. Re: Editing domain.xml
                          johnbailey

                          I guess for either option I would vote to only allow offline edits.  If you need to make changes without a restart the admin console/command-line utils should be sufficient.  It would be nice to avoid having to dedicate a task to site an monitor the file continuously.   Maybe it isn't a big deal, but it doesn't seem worth it to me.   I would assume most use cases for editing the xml file directly would be either large scale changes or some kind of automation tooling.  In either case I think a restart of the DC is acceptable. 

                          • 10. Re: Editing domain.xml
                            johnbailey

                            As for simple or complex, I think we should look into the complex but cool choice.  I think every time the DC and SM initiate contact with one another there should be a reconciliation process.  This would be true for either the DC being down during domain.xml changes or the SM being doing during object model/domain.xml changes.  Either way when the DC->SM connection is established the transaction log will be reviewed and a sync is initiated.

                            • 11. Re: Editing domain.xml
                              jason.greene

                              Just a note that IIRC, I have not seen any vendor implement something like the complex case. In most scenarios they expect you to have to recycle all servers (eventually) for the config changes to actually take effect.

                               

                              One interesting negative aspect of deltas, is that if the module that does the translation is wrong (bug), your server groups are out of sync at least until the next restart.

                              • 12. Re: Editing domain.xml
                                jason.greene

                                Jason Greene wrote:

                                 

                                One interesting negative aspect of deltas, is that if the module that does the translation is wrong (bug), your server groups are out of sync at least until the next restart.

                                 

                                Actually no I think they are out of sync forever unless we transmitted the entire domain model.

                                • 13. Re: Editing domain.xml
                                  dmlloyd

                                  Jason Greene wrote:

                                   

                                  Jason Greene wrote:

                                   

                                  One interesting negative aspect of deltas, is that if the module that does the translation is wrong (bug), your server groups are out of sync at least until the next restart.

                                   

                                  Actually no I think they are out of sync forever unless we transmitted the entire domain model.

                                  Yep.  So we have a couple strategies to deal with this.

                                  1. Don't have any bugs
                                  2. Failing that, provide a means to force a re-sync (which means a restart) at a host level.  Note that this is something we should provide regardless.  Also, this doesn't conceptually invalidate or violate the txn log in any way, so changes can still be undone.
                                  • 14. Re: Editing domain.xml
                                    dmlloyd

                                    Also as a #3: we can record a checksum of the new domain.xml so that a server can know right away if it is out of sync.

                                    1 2 Previous Next