11 Replies Latest reply on Oct 20, 2010 12:13 PM by galder.zamarreno

    NPE during preload

    meetoblivion

      Hi

       

      I'm trying to upgrade my application from modeshape 1.2 to modeshape 2.2.  One issue I keep running into is the following stack trace:

       

      Caused by: org.infinispan.loaders.CacheLoaderException: Error while reading from file: /src/dump/jcr/staging/-1458095452
              at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:185)
              at org.infinispan.loaders.file.FileCacheStore.loopOverBuckets(FileCacheStore.java:51)
              at org.infinispan.loaders.bucket.BucketBasedCacheStore.loadAllLockSafe(BucketBasedCacheStore.java:141)
              at org.infinispan.loaders.LockSupportCacheStore.loadAll(LockSupportCacheStore.java:109)
              at org.infinispan.loaders.CacheLoaderManagerImpl.loadState(CacheLoaderManagerImpl.java:149)
              at org.infinispan.loaders.CacheLoaderManagerImpl.preload(CacheLoaderManagerImpl.java:127)
              ... 30 more
      Caused by: java.lang.NullPointerException
              at org.infinispan.marshall.jboss.ConstantObjectTable.readObject(ConstantObjectTable.java:253)
              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:357)
              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
              at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
              at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:109)
              at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.readObject(ConstantObjectTable.java:266)
              at org.infinispan.marshall.jboss.ConstantObjectTable.readObject(ConstantObjectTable.java:253)
              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:357)
              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
              at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
              at org.infinispan.loaders.bucket.Bucket$Externalizer.readObject(Bucket.java:115)
              at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.readObject(ConstantObjectTable.java:266)
              at org.infinispan.marshall.jboss.ConstantObjectTable.readObject(ConstantObjectTable.java:253)
              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:357)
              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
              at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
              at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectFromObjectStream(GenericJBossMarshaller.java:162)
              at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:179)
              at org.infinispan.loaders.file.FileCacheStore.objectFromInputStreamInReentrantMode(FileCacheStore.java:262)
              at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:181)
              ... 35 more

       

      What would cause a NPE to happen at this point in the code?

        • 1. Re: NPE during preload
          galder.zamarreno

          John, the file that you're reading, /src/dump/jcr/staging/-1458095452, what modeshape version was it created by? 1.2

           

          Do you know the Infinispan version used by that modeshape version?

           

          And what version does modeshape 2.2 use?

          • 2. Re: NPE during preload
            meetoblivion

            Galder,

             

            Thanks for looking at this.  The version doesn't seem to be relevant.  It is a modeshape 1.2 "custom brew" that I came up with that is running with a special remote cache manager connector against BETA1 of 4.1.  I am able to reproduce this without changing any of those versions, but switching to an embedded mode.  It specifically came up when I started working on MODE-758.  The files themselves were created by MS 1.2.

            • 3. Re: NPE during preload
              galder.zamarreno

              Hmmm, it's not clear to me how to replicate this. Can you be more specific in the steps I need to follow so that I can replicate it locally and get to the bottom of it?

               

              What do you mean by "remote cache manager connector"? Is this something to do with any of our Infinispan Servers? Hot Rod? Memcached? Or what the heck is it? What API is it?

               

              Did try with Infinispan 4.1.0.Final? Or 4.2.0.Alpha2?

               

              Do the embedded cache and the cache when accessed remotely use the same configuration file?

              • 4. Re: NPE during preload
                meetoblivion

                Galder, I attached a test case to the jira that demonstrates this outside of modeshape.  It appears that cache.get(key) returns null when it runs in a simpler test case (I assume that the modeshape based example above adds a few other complexities since all of the keys are UUID's).

                • 5. Re: NPE during preload
                  galder.zamarreno

                  Cheers John! That should help clarify this a lot quicker

                  • 6. Re: NPE during preload
                    galder.zamarreno

                    For interest of everyone, the JIRA is: https://jira.jboss.org/browse/ISPN-695

                    • 7. Re: NPE during preload
                      galder.zamarreno

                      John, a few comments about the test attached to the JIRA:

                       

                      First of all, I've run it a couple of times and can't get the NullPointerException being thrown from ConstantObjectTable. I'm particularly interested in that aspect of your JIRA. Did you manage to replicate that particular NPE again? I'd like to know more about it.

                       

                      Re: System.out.println("Reading directly off the server cache - these are all null.");

                       

                      The reason the cache returns null for this is cos key/value are stored as byte[] in the cache. So, if you stored a Long via HotRodClient, this will actually be stored as byte[] in the underlying cache. So, trying to retrieve it passing a Long will just return null. To actually retrieve what you stored, you need to take the Long that you're after and serialize into a byte[].

                       

                              System.out.println("Reading directly off the server cache - these are all null.");
                      new GenericJBossMarshaller().objectToByteBuffer(new Long(...))

                       

                      There's a few people that have been wanting to do this and so I've created https://jira.jboss.org/browse/ISPN-706 so that people don't have to do this in the future. If you want to store anything via the local cache and access it via remote cache, you'd have to make sure the local cache is populated with byte[] for both key and value.

                       

                      Finally, the test that you passed me throws an NPE in line 60 (System.out.println(sdo.getName());), and that looks to me that is right cos you have stored data in the cache store in byte[] format, so trying to retrieve it via Long(x) will obviously return null.

                       

                      So far, I don't see anything wrong with Infinispan except for the fact that you currently need to do the hard work of marshalling/demarshalling in this case. So, I'm interested in finding out more about the NPE in ConstantObjectTable.

                       

                      Cheers and thanks for attaching the test case

                      • 8. Re: NPE during preload
                        meetoblivion

                        Galder, more likely than not ISPN-706 will fix the issues I am seeing.  If I wanted to convert early (a repo from remote to local or vice versa) I would essentially need to demarshall all of the keys found.    

                        • 9. Re: NPE during preload
                          sannegrinovero

                          Hi John,

                          I understand you have two issues, one being solved/improved by ISPN-706, the other is relating to nullpointers in ConstantObjectTable.

                          I'm not affected by the ISPN-706 issue but I'm experiencing NPEs in ConstantObjectTable under high load, but I was unable to create a good unit test to hand over to Galder; I would really appreciate if you could provide more information about how to reproduce this specific issue.

                          thanks,

                          Sanne

                          • 10. Re: NPE during preload
                            galder.zamarreno

                            A few things to mention here:

                             

                            - Based on some investigation I've done for https://jira.jboss.org/browse/ISPN-699 and the stacktrace shown at the top, maybe the CacheLoaderManagerImpl is starting before the marshaller somehow. Although, this shouldn't be the case since marshaller has start priority 9 and CacheLoaderManagerImpl has priority 11 and preload has prio 56. Anyway, work done for 699 will help clarify this further. John, I don't think 706 will really solve that NPE shown at the beginning on the forum post.

                             

                            - Wrt to 706, I have created a wiki explaining what code a client needs to write to access data in Hot Rod server directly. A wrapper for this is what 706 should deliver, but for those that cannot wait, I hope the wiki gives enough pointers to get going: http://community.jboss.org/docs/DOC-15985

                            • 11. Re: NPE during preload
                              galder.zamarreno

                              John, that's fine that you'll need to marshall/demarshall. The wiki should help.

                               

                              As Sanne said, at this point we're particularly interested in the NPE coming from ConstantObjectTable (COT), so anything you can do to help us replicate it would be of great value