1 2 Previous Next 17 Replies Latest reply on Jan 28, 2005 1:05 PM by Charles Crouch

    Datasource Component Design - Updating JBoss AS

    Charles Crouch Expert

      So having poked around the JCA User list I'm suggesting the following algorithm for creating a new Datasource through the Admin Console:

      1) User enters all relevant info in the Admin Console and uploads a new driver jar file.
      2) The user commits the change.
      3) The Admin Console submits the data and the driver jar file to the DeploymentService through JMX, specifing a name, e.g. mydatasource, for the datasoure and a template name which the DeploymentService knows about.
      4) The DeploymentService follows the specified template and builds a new sar module with the following structure:

      mydatasource.sar/
       META-INF/jboss-service.xml (<?xml version="1.0"?><server></server>)
       my-ds.xml (built from a velocity template, using the data provided to the DeploymentService)
       mydriver.jar (the jar file which was submitted to the DeploymentService)


      4) The DeploymentService then copies this into the ./deploy directory.

      My understanding of the DeploymentService is that it can do everything other than handle driver jar files parts? Any thoughts on the difficulty of updating it to do this.

      Thanks


        • 1. Re: Datasource Component Design - Updating JBoss AS
          Dimitris Andreadis Master

          We need to first resolve how to upload files into some directory (e.g. ./console/deploy) and have some control over that (i.e. know when uploading is succesfully completed).

          I'll ask Tom to see what the status of this WRT remoting.

          A related question I have is how those pages (e.g. JIRA attachement upload facility) work?

          The big picture is that the console WebApp will be able to manage either it's own host server, or one or more remote servers, correct?

          In the first case uploading is really something between the web-browser and the webapp, since the webapp is collocated with the server, no?

          • 2. Re: Datasource Component Design - Updating JBoss AS
            Tom Elrod Master

            Guess there are two points from my perspective. The first, which Dimitris already touched on, what is going to be the interface on the client end? If this is a web browser, then can just have a servlet for the upload of the jar (which will obviously not require remoting and should be pretty straight forward).

            If client is fat client (swing or SWT) or if have a master server that will replicate the deployments out, then can use remoting for distributing the file. My idea for doing this (is not implemented yet), is to provide a way to pass an inputstream to remoting, which will then call on the server handler and pass a proxy to it with its type being the same as what was passed on the client. Then when the server handler code asks to read it, the proxy will call across the wire and do the read. Don't know for sure that this will work, but it should. I do however expect it to be a little slow, but at least you won't have to load the whole file in memory (which would have to do otherwise if not streamed).

            • 3. Re: Datasource Component Design - Updating JBoss AS
              mazz Master

               


              but at least you won't have to load the whole file in memory (which would have to do otherwise if not streamed).


              Which is the limitation to this simple fileDrop command I am implementing - the entire file contents must be shipped to the remote service that will write the file on the remote box.

              What if you need to ship the file to several remote endpoints (as in the case when you want to remotely write out a config file or deployment sar to multiple nodes on the network) - you would need mulitple remote streams, no?

              • 4. Re: Datasource Component Design - Updating JBoss AS
                Tom Elrod Master

                 

                "mazz@jboss.com" wrote:

                What if you need to ship the file to several remote endpoints (as in the case when you want to remotely write out a config file or deployment sar to multiple nodes on the network) - you would need mulitple remote streams, no?


                Right now, would have to write something in front of remoting to have a remoting client for each target server. In the future, will be adding ability to have multiple target endpoints for a remoting call. However, this is probably a ways off (at least a few months unless gets bumped up on priority list).

                • 5. Re: Datasource Component Design - Updating JBoss AS
                  Charles Crouch Expert

                  Here are a couple of the simpler scenarios...

                  Case 1: Administering a single JBoss instance which is local to the Admin Console, i.e. the Admin Console is running inside the instance which is being administered. This the only case we truely care about for the first phase of Admin Console development:

                  0) User comes to the Admin Console.
                  1) They choose to create a datasource and browse to the driver jar on their local machine.
                  2) They submit the form to create the Datasource which causes the driver jar to be uploaded to a folder on the machine where the Admin Console is running, e.g. C:\jboss\server\default\adminconsole\uploads. This is all provided for us courtesy of Struts (http://wiki.apache.org/struts/StrutsFileUpload)
                  3) The user presses the commit button to make the Datasource creation permanent.
                  4) The Admin Console then passes the Datasource definition information, including a URL to the driver jar, e.g. file://C:/jboss/server/default/adminconsole/uploads/mydriver.jar, into a POJO service
                  5) This POJO service calls the createModule() operation on the DeploymentService MBean.
                  6) The DeploymentService works its magic to produce the appropriate SAR file in C:\jboss\server\default\deploy, for example.



                  Case 2: Administering a single JBoss instance which is remote to the Admin Console, i.e. the Admin Console is running inside JBoss on one machine and we are trying to administer a different JBoss instance which is running on a separate machine.

                  0) User comes to the Admin Console and chooses which remote server to administer.
                  1), 2), 3) as above
                  4) The Admin Console passes the Datasource definition information, including a URL to the driver jar, into the JBoss Network CommandClient on the local machine.
                  5) JBoss/Remoting magic happens. What I'm essentially saying is that how the driver jar gets from the local machine to the remote machine is an implementation detail of the JBoss Network Command Framework and JBoss/Remoting. It should not be a concern of the Admin Console or DeploymentService.
                  6) The appropriate JBoss Network CommandService gets called on the remote machine with the information which was passed to the CommandClient.
                  7) The CommandService calls the POJO serivce passing in all the data it received. This is analagous to step 4) above.
                  8) As step 5) above.
                  9) As step 6) above.


                  So overall I guess I'm saying there is a generic problem to be solved which is:
                  Pass a reference to some binary data on machine A --> get a reference to that binary data on machine B
                  [Another distinct problem is obviously where there are multiple machine B's]

                  If this is solved then the Admin Console and the DeploymentService can proceed in the same general manner whether talking to each other locally or remotely.

                  Thanks

                  • 6. Re: Datasource Component Design - Updating JBoss AS
                    Dimitris Andreadis Master

                    Ok, that was a great description of the steps.

                    How about this?

                    A file is transfered to a directory local to the AdminConsole as described. Then the Console prepares the input properties for the template to be executed. Part of the input properties is the URL to the local file. When the template runs, it copies the file over to the deployment.

                    In the remote scenario, the only thing that changes is the intermediate layer that takes care of transfering the file over (locally to the remote machine). The other steps are the same.

                    In the template config we would have something like:

                    <template-config>
                     ... bla bla properties
                     <property name="JarURL" type="java.net.URL"/>
                     ...
                     <file-list>
                     <file src="files"/> <!-- deep copy this from template dir -->
                     <file src="${JarURL}"/> <!-- copy the pointed resource -->
                     <file-list/>
                    


                    The only "problem" I see, is whenever we need to rerun the template the caller needs to make sure the URL is valid, i.e. the file exists so it can be copied.

                    • 7. Re: Datasource Component Design - Updating JBoss AS
                      Tom Elrod Master

                       

                      "charles.crouch@jboss.com" wrote:

                      4) The Admin Console passes the Datasource definition information, including a URL to the driver jar, into the JBoss Network CommandClient on the local machine.
                      5) JBoss/Remoting magic happens. What I'm essentially saying is that how the driver jar gets from the local machine to the remote machine is an implementation detail of the JBoss Network Command Framework and JBoss/Remoting. It should not be a concern of the Admin Console or DeploymentService.


                      Hmm, seems to me (and think Dimitris is saying same thing) that since the driver jar is available via URL, couldn't the remote machine just download the jar? Would assume that want the jar on remote machine's disk anyways. Might simplify things.


                      • 8. Re: Datasource Component Design - Updating JBoss AS
                        Charles Crouch Expert

                        Apologies guys. I was only using a URL to the file as an example of a type of reference to a file. It might as well have been a String containing the path relative to JBOSS_HOME, or a java.io.File object.

                        For the remote case we simply can't rely on the driver jar being available to the remote machine via the filesystem. Especially on Windows this is way too onerous a pre-requisite for admins to have to setup.
                        However for the local case I think this could work great. I think Dimitris' template example would work fine when we could guarantee access through the filesystem.

                        I think to solve the remote case we are going to have tackle the generic problem I outlined:

                        Pass in a reference to some binary data on machine A --> get a reference to that binary data on machine B

                        I think Mazz's "fileDrop" facility should be able to provide this, when we just have one machine B. Is this correct?

                        If so then maybe further down the line JBoss/Remoting could support this operation itself, and for multiple remote machines.

                        Thanks

                        • 9. Re: Datasource Component Design - Updating JBoss AS
                          mazz Master

                          depends on the size of the files. Tom E's idea is a good one for large files - that is, have a remote file input stream that a server can read to slurp down a file from the remote client.

                          My fileDrop command will take the contents of any file (or any String or any Object to be serialized), ship it over to the server and have the server write it somewhere. The command itself will not only ship over the file contents but also a suggestion to the server of where to put the file. I think this would be ideal for things like configuration files. But again, this is only useful for small-to-medium sized files.

                          We will need a streaming capability for larger files like ears and such (for when we want to remotely deploy apps).

                          • 10. Re: Datasource Component Design - Updating JBoss AS
                            Tom Elrod Master

                            Ok. I'm going to move streaming files lower on my priority list then if you don't think you would need it right away.

                            • 11. Re: Datasource Component Design - Updating JBoss AS
                              Charles Crouch Expert

                               

                              "mazz@jboss.com" wrote:
                              depends on the size of the files. Tom E's idea is a good one for large files - that is, have a remote file input stream that a server can read to slurp down a file from the remote client.


                              So database driver .jar files may typically be around 1-2Mb, with some larger ones, e.g. AS400, being around 4Mb. Will your fileDrop command be suitable, or do we need something more sophisticated from JBoss/Remoting?

                              Thanks

                              • 12. Re: Datasource Component Design - Updating JBoss AS
                                mazz Master

                                Wow. Well, this fileDrop is very simple - it slurps in the file contents in a single byte[] and transmits that byte array over the network (via JBoss/Remoting transporting and marshalling) to the server. No chunking or anything.

                                So, as you can see, the JVM must have enough memory to store the contents of the file in memory (both the client JVM and the server JVM). 2MB should be doable. Getting into the 10s of megabytes will be problematic. That why I think it is ideal for small-to-medium size files like configs and the like. Large jars/wars/ears - you have to be careful. I like the idea of a remote input stream for this kind of thing. WinAmp for JBoss :-)

                                As for performance - dunno. How fast does JBoss/Remoting transmit a several megabyte payload :-)

                                --
                                John Mazz

                                • 13. Re: Datasource Component Design - Updating JBoss AS
                                  Dimitris Andreadis Master

                                  I mean't deployment service should always be invoked with a local url (file:/....../bla.jar). A remote would work too (http://...), as long as someone can serve it :)

                                  I think we'll be ok, for the first phaze...

                                  • 14. Re: Datasource Component Design - Updating JBoss AS
                                    Tom Elrod Master

                                     

                                    "mazz@jboss.com" wrote:
                                    As for performance - dunno. How fast does JBoss/Remoting transmit a several megabyte payload :-)


                                    Don't have numbers to post, but did want to mention that the current marshallers used within remoting will actually stream the payload. This means that it does not load it all into memory (byte[]) then ship the byte array. So will really be putting the bottleneck on the network in regards to speed of sending the file. However, once on server side, it does load the while thing into memory upon the client making the call.

                                    1 2 Previous Next