6 Replies Latest reply on May 2, 2014 6:08 AM by dcdh1983

    Modeshapes multiples upload, sequencers and transactions

    dcdh1983

      Hello,

       

      I've got a question about sequencers and EJB transactions when uploading mutliples images files into ModeShape.

       

      I've got a first EJB called by my view. This EJB is responsible to call other EJBs and one of them is in charge of uploading image files to ModeShape. This EJB is called EJB Repository for this explaination.It is a @Stateless EJB and it have a Repository injected object and it uses the JcrTools to upload file easily into ModeShape. The EJB Repository is called as many time as I have a file to upload.

       

      I use an image sequencer to retrieve image file metadata.

      My image sequencer is defined as this:

      <sequencers>

           <sequencer name="image-sequencer" classname="image" module="org.modeshape.sequencer.image" path-expression="/fichiers(//*.(png|jpg|gif)[*])/jcr:content[@jcr:data] => /metadata/$1"/>

      </sequencers>

       

      I should obtain these results after uploading two images files:

       

      Fileimage metadata from sequencer
      Node path/fichiers/one.jpg/metadata/one
      Node path/fichiers/two.jpg/metadata/two

       

      For your information, fichiers and metadata nodes are not presents each time the test is run. I manually delete them, then they are dynamically created by each test. My exemple is simpler. I could create these nodes and it will work. But in my real application I may have more than one node level to dynamically create.

       

      First case : Require Transaction on the EJB Repository

       

      The transaction defined by default on the EJB Repository is Required.

       

      I obtain these results:

       

      Fileimage metadata from sequencer
      Node path/fichiers/one.jpg/metadata/one
      Node path/fichiers[2]/two.jpg/metadata/two

       

      It is ok for the metadata part but it fails in file paths nodes. Files one.jpg and two.jpg are not presents on the same parent node. We've got a sibling.

       

      Second case : Transaction set to Support on the EJB Repository

       

      In this case the transaction of the EJB Repository is bound to the parent EJB. The transaction is the same each time the EJB Repository is called for uploading an image file.

       

      I obtain these results:

       

      Fileimage metadata from sequencer
      Node path/fichiers/one.jpg/metadata/one
      Node path/fichiers/two.jpg/metadata[2]/two

       

      The result is the opposite compared with the first case. We've got a sibling on the metadata parent nodes paths.

       

      My point of view.

       

      Here is an explaination. Takes it with caution, I may have wrong.

       

      In the first case a new transaction is created each time the EJB Repository is called. Each transaction can't know what the other is or has doing. So the second call to the EJB Repository will not find the node previously created by the first transaction. The second call will create a new "fichiers" node resulting to a sibling node.

       

      I think it is the same for the second case. Each sequencer has his own transaction and can't know if common parent node was created by a previous running transaction.

       

      Are these explanations corrects ? Is there a workaround to avoid sibling nodes for common dynamic created path ?

       

      I think that this problem could be linked with this thread: https://community.jboss.org/thread/233260

       

      Thanks by advance.

       

      Regards,

      Damien

       

      My environnement:
      - ModeShape 4.0 Alpha 2

      - Wildfly 8.0.0

        • 1. Re: Modeshapes multiples upload, sequencers and transactions
          rhauch

          First of all, when you run them non-concurrently, do you get the correct results? If not, then there may be some problem that's not necessarily related to transactions. If you do get the correct results, then it very well might be related to transactions; it sounds like this is the case.

           

          With what I know about the problem, your explanation seems to be spot on. The second transaction is started and does not see anything from the first transaction until the first is committed; at that point, any "cached" nodes (e.g., such as the root node in your above examples) that were changed by the first transaction are flushed from the cache, and all sessions immediately see the updated representations. If the second transaction had already determined that the "/fichiers" node did not exist under the root, then that transaction will attempt to created it, and when it commits the new "/fichiers" node results in a same-name-sibiling node.

           

          There are several things you might fix the problem, depending upon your app and architecture:

           

          Firstly, the easiest and least-impactful solution is to simply pre-allocate the intermediate nodes, either all at once or if time-based a short period before you know they'll be used (e.g., the day before). This might not be possible with your application. (An alternative might be to preallocate immediately before being needed by using a separate transaction, either in a separate component or after suspending the transaction of your upload. Of course, you'd have to verify that you didn't end up with same-name-siblings; if you did, you'd want to remove what you created and try again, at which point you'd see the nodes created by the other concurrent transaction.)

           

          Secondly, you might consider calling the sequencer directly within your transaction, rather than as an event-based asynchronous process that is kicked off by the saving of your files. Manually invoking a sequencer was added in 3.6.0.Final. This would keep the output synchronized with the input (you'd never have the second case above), and coupled with one of the other suggestions to fix the first case could very effectively help solve the problem.

           

          Thirdly, by far the best and most correct way to address this is to use node types that do not allow same name siblings. You can't modify the node type of the root node, but you can create a child node under which you place all of your content. And you can use a custom node type for this node and all nodes below it. The result will be that the second transaction would fail, at which point your application code could simply retry. I suspect that you are using 'nt:unstructured' or something like it, and this should be relatively easy to change. (A node type can even prescribe the node type used in its children, if none is specified when creating the children.)

           

          Fourthly, you could use locks to synchronize the uploading of files. You've not mentioned how much throughput you need, and using locks would definitely slow things down a bit as it would prevent any concurrent uploads. (You'd want to manually invoking the sequencer if you use this option, as that'd be the only way to synchronize the generation of the sequencer output.) This is not an ideal solution, but it would work.

          • 2. Re: Modeshapes multiples upload, sequencers and transactions
            dcdh1983

            Thanks for your response.

            In fact my application is generating random node name. From all of your tips, I have tried the second one and it is working. Each time I upload a file I call the right sequencer. No sibling nodes are created because it is in the same transaction.

            I call the right sequencer using the mimeType content of the uploaded file.

            For information, I determine the mimeType of the jcr:content by getting the Binary and using the getMimeType() method on it because at this moment, the node is not saved on the repository and the jcr:mimeType property is not defined for the node jcr:content.

             

            However I have got an other problem.

            During the last two nights (I am working the day on other technologies) I have tried to create a custom mixin without success.

            My objective is to add a mixin on an nt:file node to allow the creation of a child node with a jcr:primaryType defined to image:metadata.

             

            I have made a test by uploading the aircraft.cnd file on my repository. A CND sequencer has been defined and has created a derived/cnd/aircraft.cnd/air:Aircraft node (jcr:primaryType = cnd:nodeType) with a lot of cnd:propertyDefinition children. The cndNodeTypeName is defined to air:Aircraft

             

            It should have create a new node type with the value air:Aircraft however this node type is not available.

             

            Should I have missing somethings ?

             

            Thanks by advance.

            • 3. Re: Modeshapes multiples upload, sequencers and transactions
              rhauch

              Can you provide a CND file with the mixin that you want to add to 'nt:file' nodes? Are you trying to add the mixin with Java code, or only have a sequencer do it for you?

              • 4. Re: Modeshapes multiples upload, sequencers and transactions
                dcdh1983

                Before creating my custom mixin I made some tests using the aircraft example provided by ModeShape. I will go further when I will be able to get an example working ^^

                 

                I am using the aircraft.cnd file available in github at this url: https://github.com/ModeShape/modeshape-tools/blob/master/tests/org.jboss.tools.modeshape.jcr.test/testdata/aircraft.cnd

                 

                I have got this sequencer defined in my standalone.xml :

                <sequencer name="cnd-sequencer" classname="cnd" module="org.modeshape" path-expression="(//*.cnd[*])/jcr:content[@jcr:data] => /derived/cnd/$1"/>

                aircraft.png

                The stacktrace when defining a new node as air:Aircraft jcr:primaryType is:

                 

                Caused by: javax.faces.el.EvaluationException: java.lang.Exception: Unable to create child "/{}toto/{}titi" in workspace "default" because the node type "{http://www.modeshape.org/examples/aircraft/1.0}Aircraft" does not exist

                 

                I have reloaded wildfly before testing.

                • 5. Re: Modeshapes multiples upload, sequencers and transactions
                  rhauch

                  Ah, I think I understand what you are doing.

                   

                  In order for ModeShape to start using the node type definitions, you must register the node types in the CND using the NodeTypeManager interface (via the Session's Workspace interface).

                   

                  You cannot simply just add the CND file to the repository, and have it be sequenced. The CND sequencer does nothing with registration, and is there purely for those scenarios when you actually want to manage/govern CND artifacts. For example, an organization might have 3 different JCR repositories, all with different content and node type definitions. They may also want a 4th completely separate repository that they use merely to manage/govern the CND files that they're using throughout their organization. In fact, this 4th repository might contain multiple versions of the same CND files, and say which ones are used in production, which ones are staged, and which ones are under development. This is why adding a CND as content to a repository does not register the node types; doing so would mean you cannot store multiple versions of the same CND file.

                   

                  And the CND sequencer merely extracts as regular content the node types defined in those files, so they can be queried, related, tagged, etc. It also has nothing to do with registration.

                   

                  So most repositories actually do not need to use the CND sequencer. If all you want to do is register node types, then just do that using the NodeTypeManager interface. Then you can create content that uses those node types.

                   

                  Hope this helps!

                  1 of 1 people found this helpful
                  • 6. Re: Modeshapes multiples upload, sequencers and transactions
                    dcdh1983

                    Thanks a lot for your reply.

                     

                    I am able to register the air:Aircraft node type. I have got now a working test case ^^

                     

                    Here is my final scenario:

                    - The workspace is defined in the standalone.xml and I am accessing it under the application and modeshape-explorer by using jndi.

                    - I have defined a Singleton EJB in charge of initilizing some stuffs like picketlink and now ModeShape. It is in this EJB that I define the nodeType by calling the registerNodeTypes method from the cnd previously uploaded under ModeShape.

                    Now this nodeType is present under the modeshape-explorer and can be used under it and the application too.

                    Now I will have fun using ModeShape.

                     

                     

                    Thank you for your help.


                    Thank to all your team and contributors for making ModeShape, it is a great product.

                     

                     

                    Regards,

                     

                    Damien