3 Replies Latest reply on Dec 10, 2014 11:08 AM by Horia Chiorean

    4.0 and 4.1 a bit hung up on external resource nodes

    Mike Conway Newbie

      A while back I mentioned that we were looking at in iRODS connector, and we had successfully implemented and demonstrated this!  That was with 3.8.1. Now we're upgrading to 4.0 and just now 4.1, and refactoring to be more 'production'.  We're hoping to do a sprint with the Fedora Commons folks on having iRODS as a backing store for that repo, and working as well with the Harvard DataVerse folks who are considering Modeshape as their data abstraction.

       

      Soooo..

       

      I have this vexing issue when I create content nodes associated with an iRODS file that was working before, and now seems to return an EmptyBinaryValue somewhere in the process.

       

      To be more specific, our connector (in dev at: jargon-modeshape/IrodsWriteableConnector.java at development · DICE-UNC/jargon-modeshape · GitHub) has node creation code for Files - jargon-modeshape/FileNodeCreator.java at development · DICE-UNC/jargon-modeshape · GitHub that does something like

       

      DocumentWriter writer = this.newDocument(id);
        writer.setPrimaryType(PathUtilities.NT_FILE);
        writer.addMixinType(PathUtilities.JCR_IRODS_IRODSOBJECT);
      
      
        writer.addProperty(PathUtilities.JCR_CREATED, factories()
        .getDateFactory().create(file.lastModified()));
        writer.addProperty(PathUtilities.JCR_CREATED_BY, null); // ignored
      
      
        String childId = PathUtilities.formatChildIdForDocument(id);
        writer.addChild(childId, PathUtilities.JCR_CONTENT);
      

       

       

       

      And this yields a nt:file node and child jcr:content node sort of like this:

       

      33882 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - ----name:  path:/ id:eeba412eaa9871/FilesCreatedInTestsRoot nodeType:nt:folder   <--parent collection

      33897 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:file0.txt  path:/file0.txt id:eeba412eaa9871/col1/subcol1/file0.txt nodeType:nt:file <--- iRODS file

      33897 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - --name:jcr:content  path:/file0.txt/jcr:content id:eeba412eaa9871/col1/subcol1/file0.txt/jcr:content nodeType:nt:resource <--- binary content is an ExternalBinaryValue subclass

       

       

      our IrodsBinaryValue is here: https://github.com/DICE-UNC/jargon-modeshape/blob/development/jargon-modeshape/irods-connector/src/main/java/org/irods/jargon/modeshape/connector/nodetypes/IrodsBinaryValue.java

       

      I can get the 'nt:file' node, and then get the child jcr:content node and it looks like this

       

      709571 [main] DEBUG org.irods.jargon.modeshape.connector.IrodsWriteableConnector  - returning document:{ "key" : "/col1/subcol1/file0.txt/jcr:content" , "properties" : { "http://www.jcp.org/jcr/1.0" : { "primaryType" : { "$name" : "nt:resource" } , "data" : { "$externalBinaryId" : "/fedZone1/home/test1/jargon-scratch/ModeshapeConnectorRoot/ModeshapeConnectorRoot/col1/subcol1/file0.txt" , "$sourceName" : "irods-modeshape" } , "lastModified" : { "$date" : "2014-12-03T14:48:02.000-05:00" } } } , "$queryable" : false , "parent" : "/col1/subcol1/file0.txt" }

       

       

      Which seems fine, but when I say:

       

      javax.jcr.Binary binary = node1Content.getProperty("jcr:data")
        .getBinary();
      

       

       

      During the calls up in JcrSession to get the node (around line 535), it ends up returning an EmptyBinaryValue.

       

      binary (0B, SHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709)

       

      And doesn't seem to access my IrodsBinaryValue methods.  I find that a bit confusing and am trying to chase this up into JcrNode in the debugger, and am a bit confused as to what could have changed between 3.8 and 4.1?  Any tips are appreciated.

       

      MC

        • 1. Re: 4.0 and 4.1 a bit hung up on external resource nodes
          Horia Chiorean Master

          From the snippet of code, you don't seem to be setting the jcr:data property on your external node. You're creating the correct hierarchy (nt:file/jcr:content) but you seem to be missing jcr:data (you can look at how the FileSystemConnector sets binary values here modeshape/FileSystemConnector.java at master · ModeShape/modeshape · GitHub)

           

          When the "$externalBinaryId"  field is present in a document, the code should basically call this method: modeshape/ExternalDocumentStore.java at master · ModeShape/modeshape · GitHub which should delegate to your connector's public ExternalBinaryValue getBinaryValue( String id )  method. So you should debug and check if that call gets delegated correctly to your connector and if your connector returns a correct ExternalBinaryValue instance. The only time when you can get an EmptyBinaryValue is when the connector implementation returns "null" from this method - which is the default implementation in the Connector base class.

          • 2. Re: Re: 4.0 and 4.1 a bit hung up on external resource nodes
            Mike Conway Newbie

            Thanks so much for the reply, it is helpful to just get a toe-hold on the issue.

             

            I did do a side-by-side between the File system connector and mine.  The original (in the master branch) was in fact a 'copy' of the file system connector.  If I read your comment correctly, I think I am indeed properly formatting the jcr:content node?  I did a side-by-side comparison which I will present

             

            File Connector - create nt:file  see line 16

             

            private static final String JCR_CONTENT = "jcr:content";
            private static final String JCR_CONTENT_SUFFIX = DELIMITER + JCR_CONTENT;
            
            ...
            
              // https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/main/java/org/modeshape/connector/filesystem/FileSystemConnector.java#L513
            
            } else if (file.isFile()) {
                        writer = newDocument(id);
                        writer.setPrimaryType(NT_FILE);
            
            
                        writer.addProperty(JCR_CREATED, createdTimeFor(file));
                        writer.addProperty(JCR_CREATED_BY, ownerFor(file));
                        String childId = contentChildId(id, isRoot);
                        writer.addChild(childId, JCR_CONTENT);
                        if (!isQueryable()) writer.setNotQueryable();
            
              ...
            
            
            protected String contentChildId( String fileId,
                                                boolean isRoot ) {
                    return isRoot ? JCR_CONTENT_SUFFIX : fileId + JCR_CONTENT_SUFFIX;
            }
            

             

             

            iRODS Connector - create nt:file  see line 21

             

            
            public static final String JCR_CONTENT = "jcr:content";
              public static final String JCR_CONTENT_SUFFIX = DELIMITER + JCR_CONTENT;
            
            
            //https://github.com/DICE-UNC/jargon-modeshape/blob/development/jargon-modeshape/irods-connector/src/main/java/org/irods/jargon/modeshape/connector/nodetypes/FileNodeCreator.java#L148
            
            
            private Document instanceForIdAsFile(String id, IRODSFile file) {
              log.info("instanceFrIdAsFile()");
              DocumentWriter writer = this.newDocument(id);
              writer.setPrimaryType(PathUtilities.NT_FILE);
              writer.addMixinType(PathUtilities.JCR_IRODS_IRODSOBJECT);
            
            
              writer.addProperty(PathUtilities.JCR_CREATED, factories()
              .getDateFactory().create(file.lastModified()));
              writer.addProperty(PathUtilities.JCR_CREATED_BY, null); // ignored
            
            
              String childId = PathUtilities.formatChildIdForDocument(id);
              writer.addChild(childId, PathUtilities.JCR_CONTENT);
            
            ...
            
            
            public static String formatChildIdForDocument(final String id) {
              if (id == null) {
              throw new IllegalArgumentException("null id");
              }
            
            
              return isRoot(id) ? JCR_CONTENT_SUFFIX : id + JCR_CONTENT_SUFFIX;
              }
            

             

             

            In the above I see that both File and iRODS add a node with an id with /jcr:content appended to the nt:file id.  That seems OK, and was ommitted from the original snippet?

             

            The actual binary in the nt:resource node is compared

             

            File - create nt:resource - see Line 09

             

            
            // https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/main/java/org/modeshape/connector/filesystem/FileSystemConnector.java#L492
            
            
             if (isResource) {
                        writer = newDocument(id);
                        BinaryValue binaryValue = binaryFor(file);
                        writer.setPrimaryType(NT_RESOURCE);
                        writer.addProperty(JCR_DATA, binaryValue);
                        if (addMimeTypeMixin) {
                            String mimeType = null;
                            try {
                                mimeType = binaryValue.getMimeType();
                            } catch (Throwable e) {
                                getLogger().error(e, JcrI18n.couldNotGetMimeType, getSourceName(), id, e.getMessage());
                            }
                            writer.addProperty(JCR_MIME_TYPE, mimeType);
                        }
                        writer.addProperty(JCR_LAST_MODIFIED, lastModifiedTimeFor(file));
                        writer.addProperty(JCR_LAST_MODIFIED_BY, ownerFor(file));
            
            
              ...
            
            
             protected ExternalBinaryValue binaryFor( File file ) {
                    try {
                        return createBinaryValue(file);
                    } catch (RuntimeException e) {
                        throw e;
                    } catch (Throwable e) {
                        throw new RuntimeException(e);
                    }
                }
            
            
                /**
                 * Utility method to create a {@link BinaryValue} object for the given file. Subclasses should rarely override this method,
                 * since the {@link UrlBinaryValue} will be applicable in most situations.
                 * 
                 * @param file the file for which the {@link BinaryValue} is to be created; never null
                 * @return the binary value; never null
                 * @throws IOException if there is an error creating the value
                 */
                protected ExternalBinaryValue createBinaryValue( File file ) throws IOException {
                    URL content = createUrlForFile(file);
                    return new UrlBinaryValue(sha1(file), getSourceName(), content, file.length(), file.getName(), getMimeTypeDetector());
                }
            

             

             

            iRODS  - create nt:resource - see Line 09

             

            // https://github.com/DICE-UNC/jargon-modeshape/blob/development/jargon-modeshape/irods-connector/src/main/java/org/irods/jargon/modeshape/connector/nodetypes/ContentNodeCreator.java#L83
            
            DocumentWriter writer = newDocument(id);
              BinaryValue binaryValue = createBinaryValue(file, id);
              writer.setPrimaryType(PathUtilities.NT_RESOURCE);
              writer.addProperty(PathUtilities.JCR_DATA, binaryValue);
              
            ....
            
            protected ExternalBinaryValue createBinaryValue(final IRODSFile file,
              final String id) {
            
            
              log.info("createBinaryFile()");
              assert file != null;
              log.info("file:{}", file);
            
            
              return new IrodsBinaryValue(this.getPathUtilities().sha1(file), this
              .getConnector().getSourceName(), file.getAbsolutePath(),
              file.length(), file.getName(), this.getConnector()
              .getMimeTypeDetector(),
              this.getIrodsAccessObjectFactory(), this.getIrodsAccount());
              }
            

             

             

            My IrodsBinaryValue at jargon-modeshape/IrodsBinaryValue.java at development · DICE-UNC/jargon-modeshape · GitHub extends ExternalBinaryValue, it had been tested with URLBinaryValue with the same effect and I had been playing with alternatives there

             

             

            The upshot is that, in this test snippet...with logs interposed, I see the following...

             

            @Test
              public void testFileInProjectionBinaryContent() throws Exception {
              File rootFile = (File) connectorIrodsSetupUtilities
              .getIrodsFileSystem()
              .getIRODSFileFactory(
              connectorIrodsSetupUtilities.getIrodsAccount())
              .instanceIRODSFile(
              connectorIrodsSetupUtilities
              .absolutePathForProjectionRoot()
              + "/col1/subcol1/file0.txt");
              Node actual = session.getNode("/irodsGrid/col1/subcol1/file0.txt");
            
            
              assertFile(actual, rootFile);
            
            
              dumpNodes(actual, 0);
            

             

             

            dumpnodes just recursively uses node iterators to print some stuff out

            29040 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:file0.txt  path:/file0.txt id:eeba412eaa9871/col1/subcol1/file0.txt nodeType:nt:file

            29040 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - --name:jcr:content  path:/file0.txt/jcr:content id:eeba412eaa9871/col1/subcol1/file0.txt/jcr:content nodeType:nt:resource

             

             

            Node node1Content = actual.getNode("jcr:content");
            
            
              assertThat(node1Content.getName(), is("jcr:content"));
              assertThat(node1Content.getPrimaryNodeType().getName(),
              is("nt:resource"));
            
            
              dumpProperties(node1Content);
            

             

             

             

            dump properties on the jcr:content node (node1Content) shows

             

            175016 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - property

            175016 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:jcr:primaryType

            175017 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - path:/file0.txt/jcr:content/jcr:primaryType

            175017 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - type:7

            175017 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - property

            175017 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:jcr:data

            175018 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - path:/file0.txt/jcr:content/jcr:data

            175018 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - type:2

            175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - is binary!....

            175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - class:class org.modeshape.jcr.JcrSingleValueProperty

            175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - size:0

            175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - stream:java.io.ByteArrayInputStream@6e0c25e2

            175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - property

            175020 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:jcr:lastModified

            175020 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - path:/file0.txt/jcr:content/jcr:lastModified

            175021 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - type:5

             

             

              javax.jcr.Binary binary = node1Content.getProperty("jcr:data")
              .getBinary();
            

             

             

            the getBinary() calls JcrSingleValueProperty.getBinary()

             

             

            without calling ExternalDocumentStore it returns

             

             

            binary (0B, SHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709)

             

             

             

             

             

             

             

             

            So it does not appear to call into any connector at all.

            • 3. Re: Re: 4.0 and 4.1 a bit hung up on external resource nodes
              Horia Chiorean Master

              This modeshape/DocumentTranslator.java at master · ModeShape/modeshape · GitHub is the only place where the EmptyBinaryValue can come from and is actually the same place where normally you're connector should be called to get the external binary.

              Does the ContentNodeCreator class you pasted above actually extend the Connector class ? Remember, the Connector  class (implementation) is the one on which this method should exist.

               

              The only other option is if your external sources are not configured correctly somehow and when ModeShape tries to resolve an external source (via its name in the repo config) it cannot find the corresponding Connector implementation. That can happen if, for example, something fails at startup in your connector implementation. To find out if this is the case, you can turn on DEBUG logging or/and look at the org.modeshape.jcr.Connectors class.