1 2 Previous Next 20 Replies Latest reply on Nov 8, 2012 6:45 PM by jpalka

    EOFException with clustered Modeshape3

    jpalka

      All,

       

      I am playing with clustered Modeshape3 and have found weird problem. I have attached test Java file with configuration. From my poor understanding of Infinispan looks like a problem with unmarshling of org.infinispan.schematic.document.Path objects.

       

      20:59:08,233  WARN CommandAwareRpcDispatcher:203 - Problems unmarshalling remote command from byte buffer

      java.io.EOFException

                at org.jboss.marshalling.UTFUtils.readUTFBytes(UTFUtils.java:157)

                at org.jboss.marshalling.river.BlockUnmarshaller.readUTF(BlockUnmarshaller.java:334)

                at org.infinispan.schematic.internal.document.Paths$Externalizer.readObject(Paths.java:477)

                at org.infinispan.schematic.internal.document.Paths$Externalizer.readObject(Paths.java:453)

                at org.infinispan.marshall.jboss.JBossExternalizerAdapter.createExternal(JBossExternalizerAdapter.java:48)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1273)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

                at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:153)

                at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:139)

                at org.infinispan.schematic.internal.delta.AddValueOperation$Externalizer.readObject(AddValueOperation.java:121)

                at org.infinispan.schematic.internal.delta.AddValueOperation$Externalizer.readObject(AddValueOperation.java:108)

                at org.infinispan.marshall.jboss.JBossExternalizerAdapter.createExternal(JBossExternalizerAdapter.java:48)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1273)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.exts.LinkedListExternalizer.readObject(LinkedListExternalizer.java:57)

                at org.infinispan.marshall.exts.LinkedListExternalizer.readObject(LinkedListExternalizer.java:45)

                at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:401)

                at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:290)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

                at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:153)

                at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:139)

                at org.infinispan.schematic.internal.SchematicEntryDelta$Externalizer.readObject(SchematicEntryDelta.java:101)

                at org.infinispan.schematic.internal.SchematicEntryDelta$Externalizer.readObject(SchematicEntryDelta.java:86)

                at org.infinispan.marshall.jboss.JBossExternalizerAdapter.createExternal(JBossExternalizerAdapter.java:48)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1273)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:118)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:106)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:57)

                at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:401)

                at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:290)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:118)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:164)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:68)

                at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:401)

                at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:290)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

                at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

                at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:148)

                at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:110)

                at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:84)

                at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:196)

                at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)

                at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)

                at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)

                at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)

                at org.jgroups.JChannel.up(JChannel.java:716)

                at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)

                at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:178)

                at org.jgroups.protocols.SIZE.up(SIZE.java:73)

                at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881)

                at org.jgroups.protocols.FRAG.up(FRAG.java:165)

                at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

                at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:656)

                at org.jgroups.protocols.UNICAST.up(UNICAST.java:317)

                at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:595)

                at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140)

                at org.jgroups.protocols.FD.up(FD.java:273)

                at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

                at org.jgroups.protocols.Discovery.up(Discovery.java:355)

                at org.jgroups.protocols.TP.passMessageUp(TP.java:1174)

                at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722)

                at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                at java.lang.Thread.run(Unknown Source)

      Caused by: an exception which occurred:

                in object of type org.infinispan.schematic.internal.document.Paths$MultiSegmentPath

       

      I will continue to investigate it, but it becomes pretty urgent as we are planning release of new code based on Modeshape in mid December. Let me know if there is any way I can help with it.

       

      Regards,

      Jarek

        • 1. Re: EOFException with clustered Modeshape3
          rhauch

          Thanks, Jarek. Can you log an issue in our JIRA? And any chance you can get more of the exception (in particular, the "Caused by" exception)?

          • 2. Re: EOFException with clustered Modeshape3
            jpalka

            Sure, I will include full exception log in JIRA issue.

            • 3. Re: EOFException with clustered Modeshape3
              jpalka

              Randall,

               

              I was debugging this case and have found that in Paths.Externalizer.readObject there is some weird behavior. When object is serialized we write paths.size() as a first value. And in my case this is 2. When this is being deserialized in readObject method I get size equal to 33554432 . After this, things get worse, because first we create ArrayList of this size (that's the reason I sometimes get OutOfMemoryError) and later on we start reading strings which then later is used to compose path object,but you easily reach the end of stream. At the moment I think that this is not a problem with Paths object serialization but rather serialization of object which contains path. Can you point me to it?

              • 4. Re: EOFException with clustered Modeshape3
                rhauch

                Jaroslaw -- nice work tracking this down. I do think that there's a problem in the Externalizer code, rather than something else.

                 

                Looking at the Paths.Externalizer code, I think maybe the problem is that the writeObject method is using "output.write(...)" rather than "output.writeInt(...)". Note that the readObject method is already using "readInt()". Is there any way you can build the code, make that change, and re-try? We probably won't get to this issue until later in the week or early next, so we'd really appreciate any help you can provide.

                • 5. Re: EOFException with clustered Modeshape3
                  jpalka

                  Rebuilding code and checking.

                  • 6. Re: EOFException with clustered Modeshape3
                    jpalka

                    Randall,

                     

                    I have made changes in two places (same problem found in AddValueOperation), I have commited it to my fork (https://github.com/kcrimson/modeshape/commit/55c828f57506d8ce3294b657fa59468355138f6a). It makes serialization work but this time I get NullPointer in ArrayOperation.mutableParent. I probably know how to fix it, but I think it is better to ask. Do you exepct this method to return root of the hierarchy? I see that you iterate over and you get null on which you try to getDocument(fieldName). Should you return latest non null value?

                    • 7. Re: EOFException with clustered Modeshape3
                      rhauch

                      ArrayOperation.mutableParent should never get a null value for a field name in the parent's path. The Path object returned by getParentPath() should be iterable. Is that Path object null, or does it contain a null segment? I suspect there's something wrong with the serialization/deserialization logic for such Path objects.

                      • 8. Re: EOFException with clustered Modeshape3
                        jpalka

                        It happens in this loop:

                         

                        {code}

                                for (String fieldName : getParentPath()) {

                                    parent = parent.getDocument(fieldName);

                                }

                        {code}

                         

                        getDocument(fieldName) return null so next invocation ends with NullPointerException.

                        • 9. Re: EOFException with clustered Modeshape3
                          jpalka

                          Randall,

                           

                          There is definitely problem with serialization in ArrayEditor class. I have removed Infinispan storage directory and run my test with clean directory. This time I got different exception (full log attched to this post):

                           

                          java.lang.IllegalArgumentException: Can not set final org.infinispan.schematic.internal.document.MutableArray field org.infinispan.schematic.internal.document.ArrayEditor.array to org.infinispan.schematic.internal.document.BasicDocument

                           

                          I have found that ArrayEditor has two fields that are serialized:

                           

                              private final MutableArray array;

                              private final DocumentValueFactory factory;

                           

                          I will remove final from 'array' field, but I am not sure if DocumentValueFactory should be declared as transient and set to DefaultDocumentValueFactory.INSTANCE in non arg constructor. I see that there is only one implementation which doesn't have any state. Is this a good assumption?

                           

                          Jarek

                          • 10. Re: EOFException with clustered Modeshape3
                            jpalka

                            My bad, it is not problem with final, but rather with incorrect type

                            • 11. Re: EOFException with clustered Modeshape3
                              rhauch

                              My bad, it is not problem with final, but rather with incorrect type

                               

                              Can you explain? My guess is that the problem with the BSON reading and writing may be due entirely to the corruption of the data resulting from the incorrect externalized format (sent from one process to another in the Infinispan cluster).

                               

                              The ArrayEditor class is serializable because the Array interface it implements is expected to be serializable, but there really never should be a case where such an object is indeed serialized and sent over the wire. That's likely the problem we need to track down - how is that ArrayEditor object being reached during externalization/serialization.

                              • 12. Re: EOFException with clustered Modeshape3
                                jpalka

                                BsonReader.read always returns BasicDocument instance, if look at this piece of code:

                                 

                                        protected void startDocument() throws IOException {

                                            object = readDocument(false);

                                        }

                                 

                                       

                                        protected MutableDocument readDocument( boolean array ) throws IOException {

                                            // Read the size int32, but we don't care about the value 'cuz it's in bytes ...

                                            int length = data.readInt();

                                            int startingIndex = data.getTotalBytesRead();

                                            int endingIndex = startingIndex + length;

                                            MutableDocument doc = array ? new BasicArray() : new BasicDocument();

                                            // Read the elements ...

                                            while (data.getTotalBytesRead() < endingIndex) {

                                                byte type = data.readByte();

                                                if (type == Bson.END_OF_DOCUMENT) break;

                                                readElement(type, doc);

                                            }

                                            return doc;

                                        }

                                As you can see array argument is false so it will always return BasicDocument which later we try to set to MutableArray array field in during ArrayEditor deserialization. At the moment I trying to do dirty hack to overwrite read/writeExternal in ArrayEditor and convert BasicDocument into BasicArray.

                                • 13. Re: EOFException with clustered Modeshape3
                                  rhauch

                                  We have test code that verifies that this works, and I strongly urge you to look at the tests (like BsonReadingAndWritingTest) before jumping to too many conclusions. I'm not saying there are no bugs, but rather that this code has been pretty well tested and used for quite some time. Regardless, try find (or can create) a test case that fails.

                                  • 14. Re: EOFException with clustered Modeshape3
                                    rhauch

                                    Jaroslaw Palka wrote:

                                     

                                    BsonReader.read always returns BasicDocument instance, if look at this piece of code:

                                     

                                            protected void startDocument() throws IOException {

                                                object = readDocument(false);

                                            }

                                     

                                     

                                            protected MutableDocument readDocument( boolean array ) throws IOException {

                                                // Read the size int32, but we don't care about the value 'cuz it's in bytes ...

                                                int length = data.readInt();

                                                int startingIndex = data.getTotalBytesRead();

                                                int endingIndex = startingIndex + length;

                                                MutableDocument doc = array ? new BasicArray() : new BasicDocument();

                                                // Read the elements ...

                                                while (data.getTotalBytesRead() < endingIndex) {

                                                    byte type = data.readByte();

                                                    if (type == Bson.END_OF_DOCUMENT) break;

                                                    readElement(type, doc);

                                                }

                                                return doc;

                                            }

                                    As you can see array argument is false so it will always return BasicDocument which later we try to set to MutableArray array field in during ArrayEditor deserialization. At the moment I trying to do dirty hack to overwrite read/writeExternal in ArrayEditor and convert BasicDocument into BasicArray.

                                     

                                    A BSON file always starts with a document, and so the BsonReader class starts by reading a Document (and not an array). However, "readElement()" is the method that checks the type of value and calls the appropriate "readDocument(true)" in the case of an array.

                                     

                                    Again, the ArrayEditor should not be serialized, so we need to track down why it is being serialized.

                                    1 2 Previous Next