4.0 and 4.1 a bit hung up on external resource nodes

[mike.conway]

A while back I mentioned that we were looking at in iRODS connector, and we had successfully implemented and demonstrated this!  That was with 3.8.1. Now we're upgrading to 4.0 and just now 4.1, and refactoring to be more 'production'.  We're hoping to do a sprint with the Fedora Commons folks on having iRODS as a backing store for that repo, and working as well with the Harvard DataVerse folks who are considering Modeshape as their data abstraction.

Soooo..

I have this vexing issue when I create content nodes associated with an iRODS file that was working before, and now seems to return an EmptyBinaryValue somewhere in the process.

To be more specific, our connector (in dev at: jargon-modeshape/IrodsWriteableConnector.java at development · DICE-UNC/jargon-modeshape · GitHub) has node creation code for Files - jargon-modeshape/FileNodeCreator.java at development · DICE-UNC/jargon-modeshape · GitHub that does something like

DocumentWriter writer = this.newDocument(id);
  writer.setPrimaryType(PathUtilities.NT_FILE);
  writer.addMixinType(PathUtilities.JCR_IRODS_IRODSOBJECT);


  writer.addProperty(PathUtilities.JCR_CREATED, factories()
  .getDateFactory().create(file.lastModified()));
  writer.addProperty(PathUtilities.JCR_CREATED_BY, null); // ignored


  String childId = PathUtilities.formatChildIdForDocument(id);
  writer.addChild(childId, PathUtilities.JCR_CONTENT);

And this yields a nt:file node and child jcr:content node sort of like this:

33882 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - ----name:  path:/ id:eeba412eaa9871/FilesCreatedInTestsRoot nodeType:nt:folder   <--parent collection

33897 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:file0.txt  path:/file0.txt id:eeba412eaa9871/col1/subcol1/file0.txt nodeType:nt:file <--- iRODS file

33897 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - --name:jcr:content  path:/file0.txt/jcr:content id:eeba412eaa9871/col1/subcol1/file0.txt/jcr:content nodeType:nt:resource <--- binary content is an ExternalBinaryValue subclass

our IrodsBinaryValue is here: https://github.com/DICE-UNC/jargon-modeshape/blob/development/jargon-modeshape/irods-connector/src/main/java/org/irods/jargon/modeshape/connector/nodetypes/IrodsBinaryValue.java

I can get the 'nt:file' node, and then get the child jcr:content node and it looks like this

709571 [main] DEBUG org.irods.jargon.modeshape.connector.IrodsWriteableConnector  - returning document:{ "key" : "/col1/subcol1/file0.txt/jcr:content" , "properties" : { "http://www.jcp.org/jcr/1.0" : { "primaryType" : { "$name" : "nt:resource" } , "data" : { "$externalBinaryId" : "/fedZone1/home/test1/jargon-scratch/ModeshapeConnectorRoot/ModeshapeConnectorRoot/col1/subcol1/file0.txt" , "$sourceName" : "irods-modeshape" } , "lastModified" : { "$date" : "2014-12-03T14:48:02.000-05:00" } } } , "$queryable" : false , "parent" : "/col1/subcol1/file0.txt" }

Which seems fine, but when I say:

javax.jcr.Binary binary = node1Content.getProperty("jcr:data")
  .getBinary();

During the calls up in JcrSession to get the node (around line 535), it ends up returning an EmptyBinaryValue.

binary (0B, SHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709)

And doesn't seem to access my IrodsBinaryValue methods.  I find that a bit confusing and am trying to chase this up into JcrNode in the debugger, and am a bit confused as to what could have changed between 3.8 and 4.1?  Any tips are appreciated.

MC

[hchiorean]

From the snippet of code, you don't seem to be setting the jcr:data property on your external node. You're creating the correct hierarchy (nt:file/jcr:content) but you seem to be missing jcr:data (you can look at how the FileSystemConnector sets binary values here modeshape/FileSystemConnector.java at master · ModeShape/modeshape · GitHub)

When the "$externalBinaryId"  field is present in a document, the code should basically call this method: modeshape/ExternalDocumentStore.java at master · ModeShape/modeshape · GitHub which should delegate to your connector's public ExternalBinaryValue getBinaryValue( String id )  method. So you should debug and check if that call gets delegated correctly to your connector and if your connector returns a correct ExternalBinaryValue instance. The only time when you can get an EmptyBinaryValue is when the connector implementation returns "null" from this method - which is the default implementation in the Connector base class.

[mike.conway]

Thanks so much for the reply, it is helpful to just get a toe-hold on the issue.

I did do a side-by-side between the File system connector and mine.  The original (in the master branch) was in fact a 'copy' of the file system connector.  If I read your comment correctly, I think I am indeed properly formatting the jcr:content node?  I did a side-by-side comparison which I will present

File Connector - create nt:file  see line 16

private static final String JCR_CONTENT = "jcr:content";
private static final String JCR_CONTENT_SUFFIX = DELIMITER + JCR_CONTENT;

...

  // https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/main/java/org/modeshape/connector/filesystem/FileSystemConnector.java#L513

} else if (file.isFile()) {
            writer = newDocument(id);
            writer.setPrimaryType(NT_FILE);


            writer.addProperty(JCR_CREATED, createdTimeFor(file));
            writer.addProperty(JCR_CREATED_BY, ownerFor(file));
            String childId = contentChildId(id, isRoot);
            writer.addChild(childId, JCR_CONTENT);
            if (!isQueryable()) writer.setNotQueryable();

  ...


protected String contentChildId( String fileId,
                                    boolean isRoot ) {
        return isRoot ? JCR_CONTENT_SUFFIX : fileId + JCR_CONTENT_SUFFIX;
}

iRODS Connector - create nt:file  see line 21

public static final String JCR_CONTENT = "jcr:content";
  public static final String JCR_CONTENT_SUFFIX = DELIMITER + JCR_CONTENT;


//https://github.com/DICE-UNC/jargon-modeshape/blob/development/jargon-modeshape/irods-connector/src/main/java/org/irods/jargon/modeshape/connector/nodetypes/FileNodeCreator.java#L148


private Document instanceForIdAsFile(String id, IRODSFile file) {
  log.info("instanceFrIdAsFile()");
  DocumentWriter writer = this.newDocument(id);
  writer.setPrimaryType(PathUtilities.NT_FILE);
  writer.addMixinType(PathUtilities.JCR_IRODS_IRODSOBJECT);


  writer.addProperty(PathUtilities.JCR_CREATED, factories()
  .getDateFactory().create(file.lastModified()));
  writer.addProperty(PathUtilities.JCR_CREATED_BY, null); // ignored


  String childId = PathUtilities.formatChildIdForDocument(id);
  writer.addChild(childId, PathUtilities.JCR_CONTENT);

...


public static String formatChildIdForDocument(final String id) {
  if (id == null) {
  throw new IllegalArgumentException("null id");
  }


  return isRoot(id) ? JCR_CONTENT_SUFFIX : id + JCR_CONTENT_SUFFIX;
  }

In the above I see that both File and iRODS add a node with an id with /jcr:content appended to the nt:file id.  That seems OK, and was ommitted from the original snippet?

The actual binary in the nt:resource node is compared

File - create nt:resource - see Line 09

// https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/main/java/org/modeshape/connector/filesystem/FileSystemConnector.java#L492


 if (isResource) {
            writer = newDocument(id);
            BinaryValue binaryValue = binaryFor(file);
            writer.setPrimaryType(NT_RESOURCE);
            writer.addProperty(JCR_DATA, binaryValue);
            if (addMimeTypeMixin) {
                String mimeType = null;
                try {
                    mimeType = binaryValue.getMimeType();
                } catch (Throwable e) {
                    getLogger().error(e, JcrI18n.couldNotGetMimeType, getSourceName(), id, e.getMessage());
                }
                writer.addProperty(JCR_MIME_TYPE, mimeType);
            }
            writer.addProperty(JCR_LAST_MODIFIED, lastModifiedTimeFor(file));
            writer.addProperty(JCR_LAST_MODIFIED_BY, ownerFor(file));


  ...


 protected ExternalBinaryValue binaryFor( File file ) {
        try {
            return createBinaryValue(file);
        } catch (RuntimeException e) {
            throw e;
        } catch (Throwable e) {
            throw new RuntimeException(e);
        }
    }


    /**
     * Utility method to create a {@link BinaryValue} object for the given file. Subclasses should rarely override this method,
     * since the {@link UrlBinaryValue} will be applicable in most situations.
     * 
     * @param file the file for which the {@link BinaryValue} is to be created; never null
     * @return the binary value; never null
     * @throws IOException if there is an error creating the value
     */
    protected ExternalBinaryValue createBinaryValue( File file ) throws IOException {
        URL content = createUrlForFile(file);
        return new UrlBinaryValue(sha1(file), getSourceName(), content, file.length(), file.getName(), getMimeTypeDetector());
    }

iRODS  - create nt:resource - see Line 09

// https://github.com/DICE-UNC/jargon-modeshape/blob/development/jargon-modeshape/irods-connector/src/main/java/org/irods/jargon/modeshape/connector/nodetypes/ContentNodeCreator.java#L83

DocumentWriter writer = newDocument(id);
  BinaryValue binaryValue = createBinaryValue(file, id);
  writer.setPrimaryType(PathUtilities.NT_RESOURCE);
  writer.addProperty(PathUtilities.JCR_DATA, binaryValue);
  
....

protected ExternalBinaryValue createBinaryValue(final IRODSFile file,
  final String id) {


  log.info("createBinaryFile()");
  assert file != null;
  log.info("file:{}", file);


  return new IrodsBinaryValue(this.getPathUtilities().sha1(file), this
  .getConnector().getSourceName(), file.getAbsolutePath(),
  file.length(), file.getName(), this.getConnector()
  .getMimeTypeDetector(),
  this.getIrodsAccessObjectFactory(), this.getIrodsAccount());
  }

My IrodsBinaryValue at jargon-modeshape/IrodsBinaryValue.java at development · DICE-UNC/jargon-modeshape · GitHub extends ExternalBinaryValue, it had been tested with URLBinaryValue with the same effect and I had been playing with alternatives there

The upshot is that, in this test snippet...with logs interposed, I see the following...

@Test
  public void testFileInProjectionBinaryContent() throws Exception {
  File rootFile = (File) connectorIrodsSetupUtilities
  .getIrodsFileSystem()
  .getIRODSFileFactory(
  connectorIrodsSetupUtilities.getIrodsAccount())
  .instanceIRODSFile(
  connectorIrodsSetupUtilities
  .absolutePathForProjectionRoot()
  + "/col1/subcol1/file0.txt");
  Node actual = session.getNode("/irodsGrid/col1/subcol1/file0.txt");


  assertFile(actual, rootFile);


  dumpNodes(actual, 0);

dumpnodes just recursively uses node iterators to print some stuff out

29040 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:file0.txt  path:/file0.txt id:eeba412eaa9871/col1/subcol1/file0.txt nodeType:nt:file

29040 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - --name:jcr:content  path:/file0.txt/jcr:content id:eeba412eaa9871/col1/subcol1/file0.txt/jcr:content nodeType:nt:resource

Node node1Content = actual.getNode("jcr:content");


  assertThat(node1Content.getName(), is("jcr:content"));
  assertThat(node1Content.getPrimaryNodeType().getName(),
  is("nt:resource"));


  dumpProperties(node1Content);

dump properties on the jcr:content node (node1Content) shows

175016 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - property

175016 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:jcr:primaryType

175017 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - path:/file0.txt/jcr:content/jcr:primaryType

175017 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - type:7

175017 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - property

175017 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:jcr:data

175018 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - path:/file0.txt/jcr:content/jcr:data

175018 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - type:2

175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - is binary!....

175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - class:class org.modeshape.jcr.JcrSingleValueProperty

175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - size:0

175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - stream:java.io.ByteArrayInputStream@6e0c25e2

175019 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - property

175020 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - name:jcr:lastModified

175020 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - path:/file0.txt/jcr:content/jcr:lastModified

175021 [main] INFO  org.irods.jargon.modeshape.connector.IrodsConnectorTest  - type:5

  javax.jcr.Binary binary = node1Content.getProperty("jcr:data")
  .getBinary();

the getBinary() calls JcrSingleValueProperty.getBinary()

without calling ExternalDocumentStore it returns

binary (0B, SHA1=da39a3ee5e6b4b0d3255bfef95601890afd80709)

So it does not appear to call into any connector at all.

[hchiorean]

This modeshape/DocumentTranslator.java at master · ModeShape/modeshape · GitHub is the only place where the EmptyBinaryValue can come from and is actually the same place where normally you're connector should be called to get the external binary.

Does the ContentNodeCreator class you pasted above actually extend the Connector class ? Remember, the Connector  class (implementation) is the one on which this method should exist.

The only other option is if your external sources are not configured correctly somehow and when ModeShape tries to resolve an external source (via its name in the repo config) it cannot find the corresponding Connector implementation. That can happen if, for example, something fails at startup in your connector implementation. To find out if this is the case, you can turn on DEBUG logging or/and look at the org.modeshape.jcr.Connectors class.