5 Replies Latest reply on Jan 4, 2012 9:33 AM by clau_babau

    Modeshape 2.6 export does not work when having large binary data ...

    clau_babau

      Hi,

       

        We're struggling to use the import/export repository capabilities, and we strongly need to export also binary content. Our problem is that when the binary content is large we get OutOfMemory error. Although we use 4G as heap space it seems not sufficient. This is because Modeshape takes the binary content and puts it in memory and does not stream it to the exporting xml . I know that 3.0 version will contain better support for binary content but can you please tell us if it's possible to resolve this issue for Modeshape 2.7 release?

       

      Regards,

      Claudiu

        • 1. Re: Modeshape 2.6 export does not work when having large binary data ...
          rhauch

          Unfortunately, I don't think the way binary values are handled can be easily fixed in the 2.x line without significant work and significant changes that will almost certainly break existing stores or behavior. With the 2.x design, there's no reliable way to lazily read the binary value content from the connectors while ensuring that same content isn't modified or removed by another operation before the binary value can be read. I know that's not helpful or nice to hear.

           

          One thing we could try is to periodically refresh the session during the XML export to allow the already-read-and-exported nodes to be garbage collected. It might be possible to base this on the size of the binary values (the more total binary bytes read, the more frequently we call refresh). The downside of this approach is that it will slow down the export (because some of the higher-level nodes might have to be re-read, though these very likely don't contain Binary values), but in the end a slow export is better than no export. This won't help with individual binary values that are too large by themselves. I'll log a critical issue to cover this; hopefully it will work.

           

          In the mean time, you could try exporting parts of the repository's content at a time. Again, this is not ideal, but it's something you could try with 2.7. It would essentially simulate what we'd do with the refresh by having only a subset of content in-memory at any one time, albeit requiring you to deal with multiple exported files. If you do this, be sure to refresh the session in between exports.

          • 2. Re: Modeshape 2.6 export does not work when having large binary data ...
            rhauch

            Logged as MODE-1348. Hopefully it'll be possible to do without losing our place during export.

            • 3. Re: Modeshape 2.6 export does not work when having large binary data ...
              clau_babau

              Thanks.

               

              One more issue. Regarding the UUIDBehavior flags it seems that they don't have any effect.

              Is there a way to export also the UUID of the nodes?

              Or how can we use UUIDBehavior flags taking into consideration that the exported xml does not contain the UUID?

               

              I exported one node [only metadata] (e.g. Subject/Samsung) and then I imported the file under the same parent, and I came with 2 identical nodes under the parent instead of updating or replacing the old node. The repository is configured to accept multiple nodes with the same name, but still the UUIDBehavior refers to the UUID and not the name.

               

              I also have a property of a node which value refers to the UUID of another node. When I export the node, the property contains the UUID of the node, but I import the node the UUID is reffering to a node that does not exist in the new repository. Even if the node is previously imported in the new repository it will have another UUID. How can we handle this kind of situations? I think its related with the export of the nodes with UUID.

               

              Regards.

              • 4. Re: Modeshape 2.6 export does not work when having large binary data ...
                rhauch

                One more issue. Regarding the UUIDBehavior flags it seems that they don't have any effect.

                Is there a way to export also the UUID of the nodes?

                Or how can we use UUIDBehavior flags taking into consideration that the exported xml does not contain the UUID?

                 

                We have several tests that check the behavior of the UUID flags, so I'd be surprised if that doesn't work. (See this test, for example.) Might it be that the "jcr:uuid" property is not being exported?

                 

                Or might it be that the nodes are not "mix:referenceable"? In this case, ModeShape does still use a UUID internally, but it is not exposed as a "jcr:uuid" property and it is not exported. (Note that the internal UUID is always exposed as the node's identifier.) Only when a node is made to be "mix:referenceable" will the UUID be exposed as a "jcr:uuid" property, be accessible via the 'getUUID()' method, and be exported.

                I exported one node [only metadata] (e.g. Subject/Samsung) and then I imported the file under the same parent, and I came with 2 identical nodes under the parent instead of updating or replacing the old node. The repository is configured to accept multiple nodes with the same name, but still the UUIDBehavior refers to the UUID and not the name.

                 

                Check that the exported XML contains the UUIDs, and which UUIDBehavior you're using upon import.

                I also have a property of a node which value refers to the UUID of another node. When I export the node, the property contains the UUID of the node, but I import the node the UUID is reffering to a node that does not exist in the new repository. Even if the node is previously imported in the new repository it will have another UUID. How can we handle this kind of situations? I think its related with the export of the nodes with UUID.

                 

                What property type is this property? Only if it is a REFERENCE will ModeShape treat the value as a UUID; otherwise, it's simply an opaque value.

                 

                Best regards,

                 

                Randall

                • 5. Re: Modeshape 2.6 export does not work when having large binary data ...
                  clau_babau

                  Hi Randall,

                   

                  Thank you very much for the detailed response. Indeed we were not using mix:referenceable for the nodes.

                   

                  Regards,

                    Claudiu