1 2 Previous Next 20 Replies Latest reply on Nov 8, 2012 6:45 PM by Jaroslaw Palka

    EOFException with clustered Modeshape3

    Jaroslaw Palka Newbie

      All,

       

      I am playing with clustered Modeshape3 and have found weird problem. I have attached test Java file with configuration. From my poor understanding of Infinispan looks like a problem with unmarshling of org.infinispan.schematic.document.Path objects.

       

      20:59:08,233  WARN CommandAwareRpcDispatcher:203 - Problems unmarshalling remote command from byte buffer

      java.io.EOFException

                at org.jboss.marshalling.UTFUtils.readUTFBytes(UTFUtils.java:157)

                at org.jboss.marshalling.river.BlockUnmarshaller.readUTF(BlockUnmarshaller.java:334)

                at org.infinispan.schematic.internal.document.Paths$Externalizer.readObject(Paths.java:477)

                at org.infinispan.schematic.internal.document.Paths$Externalizer.readObject(Paths.java:453)

                at org.infinispan.marshall.jboss.JBossExternalizerAdapter.createExternal(JBossExternalizerAdapter.java:48)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1273)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

                at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:153)

                at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:139)

                at org.infinispan.schematic.internal.delta.AddValueOperation$Externalizer.readObject(AddValueOperation.java:121)

                at org.infinispan.schematic.internal.delta.AddValueOperation$Externalizer.readObject(AddValueOperation.java:108)

                at org.infinispan.marshall.jboss.JBossExternalizerAdapter.createExternal(JBossExternalizerAdapter.java:48)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1273)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.exts.LinkedListExternalizer.readObject(LinkedListExternalizer.java:57)

                at org.infinispan.marshall.exts.LinkedListExternalizer.readObject(LinkedListExternalizer.java:45)

                at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:401)

                at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:290)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

                at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:153)

                at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:139)

                at org.infinispan.schematic.internal.SchematicEntryDelta$Externalizer.readObject(SchematicEntryDelta.java:101)

                at org.infinispan.schematic.internal.SchematicEntryDelta$Externalizer.readObject(SchematicEntryDelta.java:86)

                at org.infinispan.marshall.jboss.JBossExternalizerAdapter.createExternal(JBossExternalizerAdapter.java:48)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1273)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:118)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:106)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:57)

                at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:401)

                at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:290)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:118)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:164)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:68)

                at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:401)

                at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:290)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:148)

                at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:110)

                at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:84)

                at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:196)

                at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)

                at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)

                at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)

                at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)

                at org.jgroups.JChannel.up(JChannel.java:716)

                at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)

                at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:178)

                at org.jgroups.protocols.SIZE.up(SIZE.java:73)

                at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)

                at org.jgroups.protocols.FRAG.up(FRAG.java:165)

                at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

                at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:656)

                at org.jgroups.protocols.UNICAST.up(UNICAST.java:317)

                at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:595)

                at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140)

                at org.jgroups.protocols.FD.up(FD.java:273)

                at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

                at org.jgroups.protocols.Discovery.up(Discovery.java:355)

                at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)

                at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)

                at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                at java.lang.Thread.run(Unknown Source)

      Caused by: an exception which occurred:

                in object of type org.infinispan.schematic.internal.document.Paths$MultiSegmentPath

       

      I will continue to investigate it, but it becomes pretty urgent as we are planning release of new code based on Modeshape in mid December. Let me know if there is any way I can help with it.

       

      Regards,

      Jarek

        • 1. Re: EOFException with clustered Modeshape3
          Randall Hauch Master

          Thanks, Jarek. Can you log an issue in our JIRA? And any chance you can get more of the exception (in particular, the "Caused by" exception)?

          • 2. Re: EOFException with clustered Modeshape3
            Jaroslaw Palka Newbie

            Sure, I will include full exception log in JIRA issue.

            • 3. Re: EOFException with clustered Modeshape3
              Jaroslaw Palka Newbie

              Randall,

               

              I was debugging this case and have found that in Paths.Externalizer.readObject there is some weird behavior. When object is serialized we write paths.size() as a first value. And in my case this is 2. When this is being deserialized in readObject method I get size equal to 33554432 . After this, things get worse, because first we create ArrayList of this size (that's the reason I sometimes get OutOfMemoryError) and later on we start reading strings which then later is used to compose path object,but you easily reach the end of stream. At the moment I think that this is not a problem with Paths object serialization but rather serialization of object which contains path. Can you point me to it?

              • 4. Re: EOFException with clustered Modeshape3
                Randall Hauch Master

                Jaroslaw -- nice work tracking this down. I do think that there's a problem in the Externalizer code, rather than something else.

                 

                Looking at the Paths.Externalizer code, I think maybe the problem is that the writeObject method is using "output.write(...)" rather than "output.writeInt(...)". Note that the readObject method is already using "readInt()". Is there any way you can build the code, make that change, and re-try? We probably won't get to this issue until later in the week or early next, so we'd really appreciate any help you can provide.

                • 6. Re: EOFException with clustered Modeshape3
                  Jaroslaw Palka Newbie

                  Randall,

                   

                  I have made changes in two places (same problem found in AddValueOperation), I have commited it to my fork (https://github.com/kcrimson/modeshape/commit/55c828f57506d8ce3294b657fa59468355138f6a). It makes serialization work but this time I get NullPointer in ArrayOperation.mutableParent. I probably know how to fix it, but I think it is better to ask. Do you exepct this method to return root of the hierarchy? I see that you iterate over and you get null on which you try to getDocument(fieldName). Should you return latest non null value?

                  • 7. Re: EOFException with clustered Modeshape3
                    Randall Hauch Master

                    ArrayOperation.mutableParent should never get a null value for a field name in the parent's path. The Path object returned by getParentPath() should be iterable. Is that Path object null, or does it contain a null segment? I suspect there's something wrong with the serialization/deserialization logic for such Path objects.

                    • 8. Re: EOFException with clustered Modeshape3
                      Jaroslaw Palka Newbie

                      It happens in this loop:

                       

                      {code}

                              for (String fieldName : getParentPath()) {

                                  parent = parent.getDocument(fieldName);

                              }

                      {code}

                       

                      getDocument(fieldName) return null so next invocation ends with NullPointerException.

                      • 9. Re: EOFException with clustered Modeshape3
                        Jaroslaw Palka Newbie

                        Randall,

                         

                        There is definitely problem with serialization in ArrayEditor class. I have removed Infinispan storage directory and run my test with clean directory. This time I got different exception (full log attched to this post):

                         

                        java.lang.IllegalArgumentException: Can not set final org.infinispan.schematic.internal.document.MutableArray field org.infinispan.schematic.internal.document.ArrayEditor.array to org.infinispan.schematic.internal.document.BasicDocument

                         

                        I have found that ArrayEditor has two fields that are serialized:

                         

                            private final MutableArray array;

                            private final DocumentValueFactory factory;

                         

                        I will remove final from 'array' field, but I am not sure if DocumentValueFactory should be declared as transient and set to DefaultDocumentValueFactory.INSTANCE in non arg constructor. I see that there is only one implementation which doesn't have any state. Is this a good assumption?

                         

                        Jarek

                        • 10. Re: EOFException with clustered Modeshape3
                          Jaroslaw Palka Newbie

                          My bad, it is not problem with final, but rather with incorrect type

                          • 11. Re: EOFException with clustered Modeshape3
                            Randall Hauch Master

                            My bad, it is not problem with final, but rather with incorrect type

                             

                            Can you explain? My guess is that the problem with the BSON reading and writing may be due entirely to the corruption of the data resulting from the incorrect externalized format (sent from one process to another in the Infinispan cluster).

                             

                            The ArrayEditor class is serializable because the Array interface it implements is expected to be serializable, but there really never should be a case where such an object is indeed serialized and sent over the wire. That's likely the problem we need to track down - how is that ArrayEditor object being reached during externalization/serialization.

                            • 12. Re: EOFException with clustered Modeshape3
                              Jaroslaw Palka Newbie

                              BsonReader.read always returns BasicDocument instance, if look at this piece of code:

                               

                                      protected void startDocument() throws IOException {

                                          object = readDocument(false);

                                      }

                               

                                     

                                      protected MutableDocument readDocument( boolean array ) throws IOException {

                                          // Read the size int32, but we don't care about the value 'cuz it's in bytes ...

                                          int length = data.readInt();

                                          int startingIndex = data.getTotalBytesRead();

                                          int endingIndex = startingIndex + length;

                                          MutableDocument doc = array ? new BasicArray() : new BasicDocument();

                                          // Read the elements ...

                                          while (data.getTotalBytesRead() < endingIndex) {

                                              byte type = data.readByte();

                                              if (type == Bson.END_OF_DOCUMENT) break;

                                              readElement(type, doc);

                                          }

                                          return doc;

                                      }

                              As you can see array argument is false so it will always return BasicDocument which later we try to set to MutableArray array field in during ArrayEditor deserialization. At the moment I trying to do dirty hack to overwrite read/writeExternal in ArrayEditor and convert BasicDocument into BasicArray.

                              • 13. Re: EOFException with clustered Modeshape3
                                Randall Hauch Master

                                We have test code that verifies that this works, and I strongly urge you to look at the tests (like BsonReadingAndWritingTest) before jumping to too many conclusions. I'm not saying there are no bugs, but rather that this code has been pretty well tested and used for quite some time. Regardless, try find (or can create) a test case that fails.

                                • 14. Re: EOFException with clustered Modeshape3
                                  Randall Hauch Master

                                  Jaroslaw Palka wrote:

                                   

                                  BsonReader.read always returns BasicDocument instance, if look at this piece of code:

                                   

                                          protected void startDocument() throws IOException {

                                              object = readDocument(false);

                                          }

                                   

                                   

                                          protected MutableDocument readDocument( boolean array ) throws IOException {

                                              // Read the size int32, but we don't care about the value 'cuz it's in bytes ...

                                              int length = data.readInt();

                                              int startingIndex = data.getTotalBytesRead();

                                              int endingIndex = startingIndex + length;

                                              MutableDocument doc = array ? new BasicArray() : new BasicDocument();

                                              // Read the elements ...

                                              while (data.getTotalBytesRead() < endingIndex) {

                                                  byte type = data.readByte();

                                                  if (type == Bson.END_OF_DOCUMENT) break;

                                                  readElement(type, doc);

                                              }

                                              return doc;

                                          }

                                  As you can see array argument is false so it will always return BasicDocument which later we try to set to MutableArray array field in during ArrayEditor deserialization. At the moment I trying to do dirty hack to overwrite read/writeExternal in ArrayEditor and convert BasicDocument into BasicArray.

                                   

                                  A BSON file always starts with a document, and so the BsonReader class starts by reading a Document (and not an array). However, "readElement()" is the method that checks the type of value and calls the appropriate "readDocument(true)" in the case of an array.

                                   

                                  Again, the ArrayEditor should not be serialized, so we need to track down why it is being serialized.

                                  1 2 Previous Next