[hchiorean]

Question: Can we get around this issue by manually removing all references from the trash folder? if so how often do we need to check the existence of the trash folder and binary data marked for deletion.

Are you sure your nodes & properties correctly reference the binary values and also you're saving your sessions ? What you're describing here *should never happen*. Can you provide a sample test case/project where we can see this behavior ?

We have extensive binary store tests (e.g. modeshape/FileSystemBinaryStoreTest.java at master · ModeShape/modeshape · GitHub) and we've never seen this behavior so far.

[mlateef]

Thanks Horia,

all the properties are still there and jcr:content is created under the appropriate node for correct/valid referencing.

And for session management, we have multiple repositories being managed by a single modeshape engine, so we are adding nodes through appropriate sessions. And we do save the session on each transaction (either the transaction succeed or failed).

I am pasting a code snippet here, for your expert opinion pls:

ModeShapeEngine msEngine = ModeShapeEngine.geModeShapetEngine();

Session session = msEngine.getSessionByRepository(sessionRepository);

try {

  ValueFactory valueFactory = session.getValueFactory();

  root = session.getRootNode();

  Node parent = root.addNode(folderName, NodeType.NT_FOLDER);

  Node file = parent.addNode(fileName, NodeType.NT_FILE);

  Node resource = file.addNode(Property.JCR_CONTENT, CustomNodeType.VC_FS_RESOURCE);

  resource.addMixin(NodeType.MIX_SHAREABLE);

  //istream is the InputStream object of the File

  Binary binaryValue = valueFactory.createBinary(istream);

  resource.setProperty(Property.JCR_DATA, binaryValue);

  // Here we set the values for few custom properties....

  resource.setProperty(Property.JCR_MIMETYPE, mimeType);

  resource.setProperty(Property.JCR_MIMETYPE, mimeType);

  resource.setProperty(Property.JCR_MIMETYPE, mimeType);

  resource.setProperty(Property.JCR_MIMETYPE, mimeType);

} catch (Exception e) {

  ...

} finally {

  if(session != null && session.hasPendingChanges()) {

  session.save()

  }

}

//Here is the implementation of ModeShapeEngine

public class ModeShapeEngine {

  private static ModeShapeEngine engineInstance;

  private org.modeshape.jcr.ModeShapeEngine engine;

  private Map<String, Repository> repositoryMap;

  private Map<String, Session> sessionMap;

  private ModeShapeEngine() {

  if(engine==null)

  engine = new org.modeshape.jcr.ModeShapeEngine();

  System.out.println(engine.getState()+"------------------------------------");

  engine.start();

  }

  public static ModeShapeEngine geModeShapetEngine(){

  if(engineInstance == null)

  engineInstance = new ModeShapeEngine();

  return engineInstance;

  }

  public org.modeshape.jcr.ModeShapeEngine getEngine(){

  return engine;

  }

  public void setEngine(org.modeshape.jcr.ModeShapeEngine engine) {

  this.engine = engine;

  }

  public javax.jcr.Repository getRepository(String repositoryName){

  return repositoryMap.get(repositoryName);

  }

  public javax.jcr.Session getSessionByRepository(String repositoryName){

  Session session = sessionMap.get(repositoryName);

  try {

  if(!session.isLive()) {

  Repository repo = repositoryMap.get(repositoryName);

  session = repo.login("default");

  }

  } catch (Exception e) {

  System.out.println("Error getting repository: " + repositoryName);

  e.printStackTrace();

  }

  return session;

  }

}

Much Thanks for your cooperation.

[wesssel]

Can you also post the configuration of your modeshape repository + infinispan configuration?

[mlateef]

Yes sure, Wessel.

Here it is:

DataRepository.json

{

    "name" : "DataRepository",

    "jndiName" : "",

    "monitoring" : {

        "enabled" : true

    },

    "workspaces" : {

        "default" : "default",

        "predefined" : ["otherWorkspace"],

        "allowCreation" : true

    },

    "storage" : {

        "cacheConfiguration" : "/opt/MS_REPOSITORY/DataRepository/cache/configuration.xml",

        "cacheName" : "DataRepository",

        "binaryStorage" : {

  "type" : "composite",

  "minimumBinarySizeInBytes" : 4096,

  "minimumStringSize" : 4096,

  "namedStores" : {

  "default" : {

  "directory" : "/opt/MS_REPOSITORY/DataRepository/first/storage",

  "type" : "file"

  }

  }

  }

    },

    "security" : {

        "anonymous" : {

            "roles" : ["readonly","readwrite","admin"],

            "username" : "<anonymous>",

            "useOnFailedLogin" : false

        }

    },

    "indexProviders" : {

        "local" : {

            "classname" : "org.modeshape.jcr.index.local.LocalIndexProvider",

            "directory" : "target/local_index_test_repository/1"

        }

    },

    "indexes" : {

        "index" : {

            "kind" : "value",

            "nodeType" : "mix:title",

            "columns" : "jcr:title(STRING)",

            "provider" : "local",

            "synchronous" : true,

            "workspaces" : "*"

        }

    },

   "textExtraction": {

        "extractors" : {

            "tikaExtractor":{

                "name" : "Tika content-based extractor",

                "classname" : "tika"

            }

        }

    },

    "sequencing" : {

    }

}

And infini:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

  xsi:schemaLocation="urn:infinispan:config:6.0 http://www.infinispan.org/schemas/infinispan-config-6.0.xsd"

  xmlns="urn:infinispan:config:6.0">

  <global>

  <!-- Defines the global settings shared by all caches -->

  <globalJmxStatistics enabled="false" allowDuplicateDomains="true"/>

  <transport clusterName="DataRepository-cluster">

  <!-- Use the property below to point to a specific JGroups configuration file on your classpath -->

  <properties>

  <property name="configurationFile" value="/opt/MS_REPOSITORY/DataRepository/config/jgroups-tcp.xml" />

  </properties>

  </transport>

  </global>

  <namedCache name="DataRepository">

  <persistence passivation="false">

  <singleFile

  shared="false"

  preload="false"

  fetchPersistentState="false"

  purgeOnStartup="false"

  location="/opt/MS_REPOSITORY/DataRepository/content">

  </singleFile>

  </persistence>

  <clustering mode="replication">

  <stateTransfer fetchInMemoryState="true" timeout="2000" />

  <sync />

  </clustering>

  <locking isolationLevel="READ_COMMITTED" writeSkewCheck="false" lockAcquisitionTimeout="1000"/>

  <transaction

  transactionManagerLookupClass="org.infinispan.transaction.lookup.DummyTransactionManagerLookup"

  transactionMode="TRANSACTIONAL" lockingMode="PESSIMISTIC" />

  </namedCache>

</infinispan>

[wesssel]

How many Modeshape instances are you running, and are they always on at the same time?

[mlateef]

Just one instance in this case, we were running a single repository instance, and only a limited amount of data around 500 files

Thanks

[hchiorean]

  Node resource = file.addNode(Property.JCR_CONTENT, CustomNodeType.VC_FS_RESOURCE);

What does CustomNodeType.VC_FS_RESOURCE look like ?

resource.addMixin(NodeType.MIX_SHAREABLE);

Why are you using the MIX_SHAREABLE here ? Are you attempting to use shareable nodes ? If yes, this doesn't really make sense for JCR_CONTENT nodes and also you'd have to call workspace.clone somewhere in order to actually create a shared node. The reason why using/attempting to use shareable nodes doesn't make sense here is that, as discussed previously, ModeShape already uses reference counting for binary values so if you have multiple nodes referencing the same binary stream, ModeShape will not duplicate the actual binary values

finally {

  if(session != null && session.hasPendingChanges()) {

  session.save()

  }

Never save a session in a finally block, since it's a sure way of corrupting your data. JCR (as a spec) is "heavily stateless" so if there's any exception triggered somewhere in the "normal" part of the code, it means that some data/state is not consistent so you should handle the exception and abort or retry the entire operation. Don't attempt to save "parts" of a session after catching Exception. Also, there is no need to call hasPendingChanges. You can just call session.save and if there aren't any changes ModeShape will not update the persistent data.

[mlateef]

Thanks alot Horia, Your reply is much helpful for us.

CustomNodeType.VC_FS_RESOURCE is an extension of file node containing approx 20 additional properties of various data types of the node. And we do initialize/define this custom type on repository initialization, as follows:

            NodeTypeManager ntManager = session.getWorkspace().getNodeTypeManager();

            NodeTypeTemplate fsResTemplate = ntManager.createNodeTypeTemplate();

            fsResTemplate.setName(CustomNodeType.VC_FS_RESOURCE);

            fsResTemplate.setDeclaredSuperTypeNames(new String[]{CustomNodeType.VC_RESOURCE_BASE});

            PropertyDefinitionTemplate p1 = ntManager.createPropertyDefinitionTemplate();

            p1.setName(CustomProperty.VC_ACCESSED_ON);

            p1.setRequiredType(PropertyType.DATE);

            p1.setAutoCreated(false);

            p1.setMandatory(false);

            p1.setMultiple(false);

........//similarly approx 20 properties are declared here

            List<PropertyDefinition> defs = fsResTemplate.getPropertyDefinitionTemplates();

            defs.add(p1);

............//all of the above properties are added here...

            ntManager.registerNodeType(fsResTemplate, true);

Whereas the CustomNodeType.VC_RESOURCE_BASE is also a sub-type of NodeType.NT_RESOURCE having few properties.

In reference to resource.addMixin(NodeType.MIX_SHAREABLE); Yes, you are right that we shouldn't use addMixin(NodeType.MIX_SHAREABLE) in modeshape. Previously we we were using jackrabbit and it was required to manage duplicate content by our-self but now we'll stop using this in mode-shape. Much Thanks.

In ref to session.save() in finally block, we were using it in finally block to keep each addition/change on nodes persistent, to avoid any data loss.

But as per your suggestion, now we'll commit/save the session after confirmation of successful transaction. Can you pls guide us to get confirmed that the transaction is successful and the data written in folder will not be sent to Trash folder (for scheduled deletion)??

Much Thanks for your time and cooperation to resolve this issue by helping in our code/design review.

[hchiorean]

Can you pls guide us to get confirmed that the transaction is successful and the data written in folder will not be sent to Trash folder (for scheduled deletion)??

As mentioned in previous posts, I have no idea why this is happening and it's not something that we've seen before (although we're testing a lot of different scenarios). If you can provide a test-case/sample code which I can run locally, I can investigate some more. Otherwise there's nothing else that comes to mind. This test case should be a simplification of your use case, which shows that binary values are sent to trash and properties which reference these values are not found.