5 Replies Latest reply on May 17, 2017 4:10 PM by ameenajmeri

    ModeShape 5.4.1 - on restart Unable to find binary value within binary store - "default" payload row deleted in Content_Store

    dharrison
    dharrison May 2, 2017 7:08 PM

      I’ve run into a blocker today.  I’ll try to reliably duplicate the issue in the minimum number of steps and reduce the number of variables for purposes of getting a solution, quickly.  I’m posting here to get any feedback the community might have...

       

      The gist is upon initial startup, everything works fine. I’m able to add nodes of various types and search on them, etc.  However, occasionally (not all the time) when I restart ModeShape,  it fails to come back up and instead gives an error about “Unable to find binary value with key”. It stops there. Once this problem occurs, I do not know how to recover the repository. This occurs even when the only type of content I’ve added to the repository is just a series of eleven “nt:unstructured” nodes to serve as a hierarchy / folder structure for other content.

       

      Example Error: Unable to find binary value with key "7505d64a54e061b7acd54ccd58b49dc43500b635" within binary store at "CONTENT_STORE"

       

      Database State (before and after)

       

      Poking around in a brand-new, freshly created ModeShape database, what I see is that there is a content node referenced in ModeShape_Repository, and it is has Content in Content_Store.

      That particular key value appears to relate to be referenced by two entries in the ModeShape-Repository table and one of those has the key-value as the first part of the ID.  The CID on this ID refers to some payload called "Default".

      I think that must be related to the "default Workspace".

       

      After the system gets into an error state, the two rows in the ModeShape_Repository table are basically the same as before (with updated last_changed),   but the  Content_Store now is missing the row for the above mentioned Key "7505d64a54e061b7acd54ccd58b49dc43500b635". 

       

      When restarting the system, it will fail to start because that Content_Store record is missing. Is this a bug or am I doing something wrong?

       

      Is the binary garbage collection mistakenly removing an essential part of the default workspace?

       

      Database View - Before

       

      ModeShape-Repository

      Content_Store

       

      Database State - After Error

      @

      ModeShape-Repository

       

      Content_Store

      No Rows! It appears the Content_Store row with the CID 7505d64a54e061b7acd54ccd58b49dc43500b635 was deleted.

       

      Possible workaround ?

      Modify the binary_store_sqlserver_database.properties file  modifications to the following two statements

      mark_unused = UPDATE {0} SET usage_flag=0, usage_time = ? WHERE cid = ? AND cid<>'7505d64a54e061b7acd54ccd58b49dc43500b635'

      remove_expired = DELETE FROM {0} WHERE usage_time < ? AND usage_flag=0 AND cid<>'7505d64a54e061b7acd54ccd58b49dc43500b635'

      ------------------

       

       

      Queries

      SELECT mr.*,convert(varchar(max),convert(varbinary(max),content)) varchar_content FROM  MODESHAPE_REPOSITORY mr 
WHERE 
  CONVERT(VARCHAR(max),CONVERT(varbinary(max),content)) LIKE '%7505d64a54e061b7acd54ccd58b49dc43500b635%'
  OR id LIKE '%7505d64a54e061b7acd54ccd58b49dc43500b635%';
  
select cs.*,convert(varchar(max),convert(varbinary(MAX),payload)) varchar_payload FROM content_store cs 
where 
  CONVERT(VARCHAR(max),CONVERT(varbinary(MAX),payload))  LIKE '%7505d64a54e061b7acd54ccd58b49dc43500b635%'
  OR cid LIKE '%7505d64a54e061b7acd54ccd58b49dc43500b635%'
ORDER BY usage_time DESC ;

       

      Current Configuration ModeShape 5.4.0 final  - running in Docker container in DC/OS.

      • JDBC Persistence w SQL Server  
        • regular and binary storage are going to SQL Server, in the same database.  
        • BinaryStorage minimumBinarySizeInBytes is set to 0
      • CNDs are defined in the repository configuration file  
        • Versioning is enabled on CNDs
        • Indexes (Mostly Lucene and one LocalIndexProvider) on CNDs are defined in repository config file
      • Clustering enabled via JGroups
      • Hikari connection pool (using defaults initially)

       

       

       

      Error in logs, upon restart

      Unable to find binary value with key "7505d64a54e061b7acd54ccd58b49dc43500b635" within binary store at "CONTENT_STORE"

      at org.modeshape.jcr.JcrRepositoriesContainer$UrlRepositoryLookup.repository(JcrRepositoriesContainer.java:228) ~[modeshape-jcr-5.4.0.Final.jar:5.4.0.Final]

      at org.modeshape.jcr.JcrRepositoriesContainer.getRepositoryNames(JcrRepositoriesContainer.java:108) ~[modeshape-jcr-5.4.0.Final.jar:5.4.0.Final]

      at org.modeshape.web.jcr.RepositoryManager.getJcrRepositoryNames(RepositoryManager.java:145) ~[modeshape-web-jcr-5.4.0.Final.jar:5.4.0.Final]

      at org.modeshape.web.jcr.rest.handler.RestServerHandler.getRepositories(RestServerHandler.java:43) [modeshape-web-jcr-rest-5.4.0.Final.jar:5.4.0.Final]

      at org.modeshape.web.jcr.rest.ModeShapeRestService.getRepositories(ModeShapeRestService.java:157) [modeshape-web-jcr-rest-5.4.0.Final.jar:5.4.0.Final]

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111]

      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111]

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]

      at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]

      at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final]

       

       

      Message was edited by: David Harrison