14 Replies Latest reply on Aug 11, 2009 2:58 AM by galder.zamarreno

    Preload problem

    john.prince

      Hi, I'm trying to get persistence working, without much success. There seems to be a problem with the serialization of large objects. Here is a test to show the problem:


      import java.util.Arrays;
      
      import org.infinispan.Cache;
      import org.infinispan.config.CacheLoaderManagerConfig;
      import org.infinispan.config.Configuration;
      import org.infinispan.loaders.file.FileCacheStoreConfig;
      import org.infinispan.manager.CacheManager;
      import org.infinispan.manager.DefaultCacheManager;
      
      /**
       * @author John Prince
       * Date: Aug 5, 2009
       * Time: 1:15:48 PM
       */
      public class InfinispanTest
      {
       private static void runTest(int count)
       {
       try
       {
       Configuration cacheConfig = new Configuration();
       CacheLoaderManagerConfig cacheLoaders = new CacheLoaderManagerConfig();
       cacheLoaders.setPreload(true);
       FileCacheStoreConfig fileStoreConfig = new FileCacheStoreConfig();
       fileStoreConfig.setLocation("/work/temp/cache_store_" + count);
       cacheLoaders.addCacheLoaderConfig(fileStoreConfig);
       cacheConfig.setCacheLoaderManagerConfig(cacheLoaders);
       CacheManager manager = new DefaultCacheManager(true);
       manager.defineCache("test", cacheConfig);
       Cache<String, byte[]> cache = manager.getCache("test");
       cache.start();
       byte[] bytes = new byte[count];
       Arrays.fill(bytes, (byte) 1);
       cache.put("test_object", bytes);
       int cacheSize = cache.size();
       manager.stop();
       assert 1 == cacheSize;
       }
       catch (Throwable e)
       {
       System.out.println("FAILURE::: Test with " + count + " failed. ");
       e.printStackTrace();
       }
       }
      
       public static void main(String[] args)
       {
       runTest(60000);
       runTest(60000);
       runTest(70000);
       runTest(70000);
       }
      }
      


      With a 60000 byte array, the preload works. With 70000 it doesn't. Environment is Windows Vista, JDK 1.6.0_u14, yesterday's SVN version (before you just broke it today :>))

      Is this a known problem? JBoss Serialization seems to have problems with large objects too - I assume that the origin of the problem is there, but I took a look at the code and then looked away again :>)

      I realize this is alpha software, and I am otherwise very excited about the possibilities, but this is a bit of a killer for me. Any hints on what I might be doing wrong, or what the problem might be, would be greatly appreciated.

      FWIW, you don't seem to have any tests for this - you have a couple of preload tests, but they are on very small objects.

      Best wishes

      John

        • 1. Re: Preload problem
          manik

          Good point re: adding more preload tests. See https://jira.jboss.org/jira/browse/ISPN-146.


          (before you just broke it today :>))


          What do you mean by 'broke'? Just tried a fresh checkout off trunk and it built fine.


          • 2. Re: Preload problem
            john.prince

             

            "manik.surtani@jboss.com" wrote:
            Good point re: adding more preload tests. See https://jira.jboss.org/jira/browse/ISPN-146.


            (before you just broke it today :>))


            What do you mean by 'broke'? Just tried a fresh checkout off trunk and it built fine.


            Fair enough. svn update led to failures. A clean checkout works for me too, apart from 2 failing tests

            Failed tests:
            testNotifier(org.infinispan.util.concurrent.WatchableValueTest)
            testBasicPropagation(org.infinispan.api.mvcc.PutForExternalReadTest)

            skipTests is my friend :>)

            Best wishes

            John

            • 3. Re: Preload problem
              manik

              Weird, both of these tests pass in trunk.

              See http://pastie.org/572438 for details.

              What environment are you running the tests on? What command-line params, etc?

              Also, at the end of the test suite run, you should see some environment info printed out, like:

              ~~~~~~~~~~~~~~~~~~~~~~~~~ ENVIRONMENT INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~
              bind.address = 127.0.0.1
              java.runtime.version = 1.5.0_15-b04
              java.runtime.name =Java(TM) 2 Runtime Environment, Standard Edition
              java.vm.version = 1.5.0_15-b04
              java.vm.vendor = Sun Microsystems Inc.
              os.name = Linux
              os.version = 2.6.9-42.0.10.ELsmp
              sun.arch.data.model = 32
              sun.cpu.endian = little
              protocol.stack = tcp
              infinispan.marshaller.class = ${marshaller.class}
              java.net.preferIpv4Stack = true
              java.net.preferIpv6Stack = null
              ~~~~~~~~~~~~~~~~~~~~~~~~~ ENVIRONMENT INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~
              


              Could you tell us what you see there?



              • 4. Re: Preload problem
                john.prince

                 

                "manik.surtani@jboss.com" wrote:
                Weird, both of these tests pass in trunk.

                See http://pastie.org/572438 for details.

                What environment are you running the tests on? What command-line params, etc?

                Also, at the end of the test suite run, you should see some environment info printed out, like:

                ~~~~~~~~~~~~~~~~~~~~~~~~~ ENVIRONMENT INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~
                bind.address = 127.0.0.1
                java.runtime.version = 1.5.0_15-b04
                java.runtime.name =Java(TM) 2 Runtime Environment, Standard Edition
                java.vm.version = 1.5.0_15-b04
                java.vm.vendor = Sun Microsystems Inc.
                os.name = Linux
                os.version = 2.6.9-42.0.10.ELsmp
                sun.arch.data.model = 32
                sun.cpu.endian = little
                protocol.stack = tcp
                infinispan.marshaller.class = ${marshaller.class}
                java.net.preferIpv4Stack = true
                java.net.preferIpv6Stack = null
                ~~~~~~~~~~~~~~~~~~~~~~~~~ ENVIRONMENT INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~
                


                Could you tell us what you see there?



                ~~~~~~~~~~~~~~~~~~~~~~~~~ ENVIRONMENT INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~
                bind.address = 127.0.0.1
                java.runtime.version = 1.6.0_14-b08
                java.runtime.name =Java(TM) SE Runtime Environment
                java.vm.version = 14.0-b16
                java.vm.vendor = Sun Microsystems Inc.
                os.name = Windows Vista
                os.version = 6.0
                sun.arch.data.model = 32
                sun.cpu.endian = little
                protocol.stack = tcp
                infinispan.marshaller.class = ${marshaller.class}
                java.net.preferIpv4Stack = true
                java.net.preferIpv6Stack = null
                ~~~~~~~~~~~~~~~~~~~~~~~~~ ENVIRONMENT INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~

                MAVEN_OPTS=-Xmx1024m -Xms512m -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m

                Maven 2.2.0

                Best wishes

                John

                • 5. Re: Preload problem
                  john.prince

                   

                  "manik.surtani@jboss.com" wrote:
                  Weird, both of these tests pass in trunk.


                  WatchableValueTest seems to have some sort of timing problem - if I increase the sleep time to 5000 ms, it generally (although not always) works. If I debug and stop, it always works.

                  testBasicPropagation seems to be a state (or possibly also timing) problem. When I run the single test, it works fine. When I run the class, the key is already in the cache, so the assertion fails.

                  Hope this helps

                  John

                  • 6. Re: Preload problem
                    galder.zamarreno

                    I'm a bit intrigued here cos running the code you provided with latest from trunk, even 60000 does not work. There's an NPE being thrown from VersionAwareMarshaller's @Start method not being called in spite of its @Inject method is called. The NPE exception looks something like this:

                    java.lang.NullPointerException
                     at org.infinispan.marshall.jboss.JBossMarshaller$1.initialValue(JBossMarshaller.java:70)
                     at org.infinispan.marshall.jboss.JBossMarshaller$1.initialValue(JBossMarshaller.java:1)


                    Is this the same exception you're encountering, or is it a different one?

                    • 7. Re: Preload problem
                      john.prince

                       

                      "galder.zamarreno@jboss.com" wrote:
                      I'm a bit intrigued here cos running the code you provided with latest from trunk, even 60000 does not work. There's an NPE being thrown from VersionAwareMarshaller's @Start method not being called in spite of its @Inject method is called. The NPE exception looks something like this:
                      java.lang.NullPointerException
                       at org.infinispan.marshall.jboss.JBossMarshaller$1.initialValue(JBossMarshaller.java:70)
                       at org.infinispan.marshall.jboss.JBossMarshaller$1.initialValue(JBossMarshaller.java:1)


                      Is this the same exception you're encountering, or is it a different one?


                      No, this is a different one. I fixed that by adding a call to start on the VersionAwareMarshaller in the FileCacheStore (this starts the JBossMarshaller too, which initializes the factory...)

                      • 8. Re: Preload problem
                        galder.zamarreno

                        We've fixed the NPE now. With regards to your error, I get the following Exception the 2nd time runTest(60000) is called:

                        Caused by: java.io.EOFException: Read past end of file
                         at org.jboss.marshalling.AbstractUnmarshaller.eofOnRead(AbstractUnmarshaller.java:180)
                         at org.jboss.marshalling.AbstractUnmarshaller.readFully(AbstractUnmarshaller.java:204)
                         at org.jboss.marshalling.river.RiverUnmarshaller.doReadByteArray(RiverUnmarshaller.java:1290)


                        Is this the same error you're getting?

                        • 9. Re: Preload problem
                          john.prince

                           

                          "galder.zamarreno@jboss.com" wrote:
                          We've fixed the NPE now. With regards to your error, I get the following Exception the 2nd time runTest(60000) is called:
                          Caused by: java.io.EOFException: Read past end of file
                           at org.jboss.marshalling.AbstractUnmarshaller.eofOnRead(AbstractUnmarshaller.java:180)
                           at org.jboss.marshalling.AbstractUnmarshaller.readFully(AbstractUnmarshaller.java:204)
                           at org.jboss.marshalling.river.RiverUnmarshaller.doReadByteArray(RiverUnmarshaller.java:1290)


                          Is this the same error you're getting?


                          Ah, no, sorry. I tried very hard to get this working before I posted the problem, and I found what seems to be a bug in the jboss marshalling AbstractUnmarshaller. The class seems to me a little, er, liberal in its mix of local and class variables, so I had trouble working out exactly what the problem was, but it seemed to be fixed by adding another condition ( && len > remaining) to the while loop in this method:

                          public void readFully(final byte[] b, int off, int len) throws IOException {
                           if (limit == -1) {
                           throw eofOnRead();
                           }
                           int remaining = limit - position;
                           if (len > remaining) {
                           if (remaining > 0) {
                           System.arraycopy(buffer, position, b, off, remaining);
                           limit = position = 0;
                           }
                           do {
                           off += remaining;
                           len -= remaining;
                           remaining = byteInput.read(b, off, len);
                           if (remaining == -1) {
                           throw eofOnRead();
                           }
                           } while (len != 0 && len > remaining);
                           } else try {
                           System.arraycopy(buffer, position, b, off, len);
                           position += len;
                           } catch (NullPointerException e) {
                           throw eofOnRead();
                           }
                           }
                          


                          I'm pretty sure this was the last hack I had to make to get it running :>)

                          The error I'm getting is

                          Caused by: java.lang.NullPointerException
                           at java.lang.reflect.Array.newArray(Native Method)
                           at java.lang.reflect.Array.newInstance(Array.java:52)
                           at org.jboss.marshalling.river.RiverUnmarshaller.doReadObjectArray(RiverUnmarshaller.java:1369)
                          


                          though now I look at this, it is not the same error I was getting in my application. Anyway, it is not working... I'll check the other error later

                          • 10. Re: Preload problem
                            galder.zamarreno

                            I've patched JBMAR and managed to replicate that NPE. It appears that the type passed to RiverUnmarshaller.doReadObjectArray(RiverUnmarshaller.java:1369)
                            is null when that NPE is thrown. I'm looking into it.

                            Btw, thanks very much for your help uncovering these issues :)

                            • 11. Re: Preload problem
                              galder.zamarreno

                              I've created a JIRA for the latest NPE: https://jira.jboss.org/jira/browse/JBMAR-62

                              I've also created a JIRA for the previous JBMAR bug: https://jira.jboss.org/jira/browse/JBMAR-63

                              • 12. Re: Preload problem
                                galder.zamarreno

                                @john, I've found a fix for the latest issue. In RiverUnmarshaller.java, find the following piece of code:

                                case ID_ARRAY_LARGE_UNSHARED: {
                                 if (unshared != (leadByte == ID_ARRAY_LARGE_UNSHARED)) {
                                 throw sharedMismatch();
                                 }
                                 final int len = readUnsignedShort();
                                 return doReadArray(len, unshared);
                                }

                                And change readUnsignedShort(); for readInt(); and that should work

                                • 13. Re: Preload problem
                                  john.prince

                                   

                                  "galder.zamarreno@jboss.com" wrote:
                                  @john, I've found a fix for the latest issue. In RiverUnmarshaller.java, find the following piece of code:
                                  case ID_ARRAY_LARGE_UNSHARED: {
                                   if (unshared != (leadByte == ID_ARRAY_LARGE_UNSHARED)) {
                                   throw sharedMismatch();
                                   }
                                   final int len = readUnsignedShort();
                                   return doReadArray(len, unshared);
                                  }

                                  And change readUnsignedShort(); for readInt(); and that should work


                                  Excellent. That works in my app now too. Many thanks :>)

                                  • 14. Re: Preload problem
                                    galder.zamarreno

                                    John, fyi and the rest of Infinispan users, trunk has now been updated to JBoss Marshalling 1.2.0.CR3 that contains fixes for both the issues you're encountering. Just update trunk to latest and it should work fine :)