3 Replies Latest reply on Oct 18, 2016 1:18 PM by jfisherdev

    WildFly 9 Remote EJB Invocation Marshalling Issues

    jfisherdev

      My organization is currently migrating from JBoss AS 4.2.2 to WildFly 9.0.2 final.

       

      We have been seeing some issues with remote EJB invocations from standalone Swing client applications that seem to be with marshalling. The EJB client approach for remote EJB access is being used in this case.

       

      We have seen a significant decrease in performance in remote EJB invocations that return certain objects used to represent large data sets in some cases where this was not the case on JBoss AS 4.2.2. One common element with the objects where this has been observed is that they typically contain Object type fields or collections typed to Object. Note that all objects contained in these fields or collections are serializable.

       

      Examples:

       

      public class PojoModel implements Serializable {
           
           private Map<String, Object> attributes;
           ...
      }
      

       

      public class TableDataModel implements Serializable {
      
           private Object[][] tableData;
           ...
      }
      

       

      These are the most common types of objects returned by remote EJB invocations in our application suite and we have not really tested this with other types of objects used for this purpose, so it may not be the case, but is there possibly an issue in JBoss Marshalling with serializing Object type fields or instances of generic classes typed to Object? If it were a case of an object not being serializable, an exception would be thrown to indicate this; however, that is not the case here.

       

      Here is an example of another object called PojoModelList that was sometimes causing issues when it was the return type for a remote EJB invocation:

       

      public class PojoModelList implements Serializable {
          private static final long serialVersionUID = 1L;
      
      
          private List<String> properties;
          private Object[][] data;
      
      
          private transient List<PojoModel> pojos;
      
      
          public PojoModelList(List<? extends PojoModel> pojos) {
              if (pojos != null && !pojos.isEmpty()) {
                  properties = pojos.get(0).getProperties();
                  int colCount = properties.size();
      
      
                  data = new Object[pojos.size()][colCount];
                  int row = 0;
                  for (PojoModel pojo : pojos) {
                      int col = 0;
                      for (String p : properties) {
                          data[row][col++] = pojo.getProperty(p);
                      }
                      row++;
                  }
              }
          }
      
      
      
          public synchronized List<PojoModels> getPojos() {
              if (pojos== null) {
                  int rowCount = (data == null) ? 0 : data.length;
                  pojos = new ArrayList<PojoModel>(rowCount);
      
      
                  for (int row=0; row<rowCount; row++) {
                      Object[] rowData = data[row];
                      pojos.add(new PojoModel(rowData, properties));
                  }
              }
              return pojos;
          }
      
      
          private void writeObject(ObjectOutputStream oos) throws IOException {
              GZIPOutputStream gzip = new GZIPOutputStream(oos);
              ObjectOutputStream oos2 = new ObjectOutputStream(gzip);
      
      
              oos2.writeUnshared(properties);
              oos2.writeUnshared(data);
              oos2.flush();
              gzip.finish();
          }
      
      
          @SuppressWarnings({"unchecked"})
          private void readObject(ObjectInputStream ois) throws IOException, ClassNotFoundException {
              GZIPInputStream zip = new GZIPInputStream(ois);
              ObjectInputStream ois2 = new ObjectInputStream(zip);
      
      
              properties = (List) ois2.readUnshared();
              data = (Object[][]) ois2.readUnshared();
          }
      }
      

       

      While using PojoModelList, we were seeing rather cryptic client errors like this:

       

      java.io.EOFException: Read past end of file
       at org.jboss.marshalling.SimpleDataInput.eofOnRead(SimpleDataInput.java:151)
       at org.jboss.marshalling.SimpleDataInput.readByte(SimpleDataInput.java:232)
       at org.jboss.ejb.client.remoting.ProtocolMessageHandler.readAttachments(ProtocolMessageHandler.java:48)
       at org.jboss.ejb.client.remoting.MethodInvocationResponseHandler$MethodInvocationResultProducer.getResult(MethodInvocationResponseHandler.java:106)
       at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:276)
       at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocationResult(EJBObjectInterceptor.java:64)
       at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
       at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocationResult(EJBHomeInterceptor.java:88)
       at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
       at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:46)
       at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:290)
       at org.jboss.ejb.client.ReceiverInterceptor.handleInvocationResult(ReceiverInterceptor.java:129)
       at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:265)
       at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:453)
       at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:204)
       at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
       at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
      

       

      We eventually switched to using Lists of PojoModel objects, as opposed to PojoModelList objects and have not seen the issue since.

       

      I am also curious if any considerations need to be made for NIO/XNIO in either the applications or WildFly configuration. This is not something we are especially familiar with, and it was not something that was a consideration in JBoss AS 4.2.2.

       

      One thing we did do that was helpful was set the jboss.remoting.pooled-buffers=false system property to work around the issue described here [REM3-200] Memory leak in org.xnio.ByteBufferSlicePool - JBoss Issue Tracker.

       

      Any information about this would be very much appreciated.

        • 1. Re: WildFly 9 Remote EJB Invocation Marshalling Issues
          jaikiran

          By what amount do you see the performance slowdown and how exactly is it measured. Given that you admit you aren't sure it's related to fields of type Object, it's hard to say what the issue is. If you can attach a simple application and explain how to run it and see the issue, I guess someone will be able to narrow it down.

          • 2. Re: WildFly 9 Remote EJB Invocation Marshalling Issues
            jaikiran

            I also see that you have write/readObject methods implemented in those classes which use a Gzip wrapping on those objects. Does this slowdown show up only in objects of such classes?

             

            Edit: So looking more at this code:

            private void writeObject(ObjectOutputStream oos) throws IOException {  
                    GZIPOutputStream gzip = new GZIPOutputStream(oos);  
                    ObjectOutputStream oos2 = new ObjectOutputStream(gzip);  
                    oos2.writeUnshared(properties);  
                    oos2.writeUnshared(data);  
                    oos2.flush();  
                    gzip.finish();  
                }  
            
            

             

             

            I don't think this will work - i.e. Gzipping an existing outputstream to which data would already have been written out.

            • 3. Re: WildFly 9 Remote EJB Invocation Marshalling Issues
              jfisherdev

              Thank you for your response.

               

              To your first question:

               

              We unfortunately do not have a formal measure of performance at this time. What we do know is that objects that contain fields of type Object or collections typed to Object are the types of objects most commonly sent between the clients and the server. Upon investigating high server load average/CPU usage, we noticed in thread dumps that marshalling EJB client responses appeared to be causing this.

               

              We started using a serialization proxy approach as described here 5 things you didn't know about ... Java Object Serialization in section 1.4 to pre-serialize some of these types of objects. Basically, we use an ObjectOutputStream to convert the object to an array of bytes stored in a field in the proxy class. This has noticeably improved client request/response times and decreased server load averages.

               

              An example thread dump prior to using the serialization proxy is shown below. One thing to note is that we have only observed this when running WildFly on Linux VM servers, but not on Windows, which is only used for local developer deployments and probably not subject to the same load.

               

              To your second question:

               

              The output stream zipping was exclusive the PojoModelList as far as I know and is not something we use anymore. The reason I asked about this is that it worked on JBoss AS 4.2.2 without any issues and more importantly we would get cryptic marshalling errors on the client when making a remote EJB invocation. There was no clear indication of a server-side exception, which seems like it would be the cause in this case.

               

              "EJB default - 162" #520 prio=5 os_prio=0 tid=0x00007fc8a8342000 nid=0x7082 runnable [0x00007fc7cceee000]

              java.lang.Thread.State: RUNNABLE

              at java.lang.Throwable.getStackTraceElement(Native Method)

              at java.lang.Throwable.getOurStackTrace(Throwable.java:827)

              - locked <0x000000077bd89c08> (a java.lang.Throwable)

              at java.lang.Throwable.getStackTrace(Throwable.java:816)

              at org.jboss.remoting3.remote.DebuggingBufferPool$1.getResource(DebuggingBufferPool.java:67)

              - locked <0x000000077bccbf88> (a org.jboss.remoting3.remote.DebuggingBufferPool$1)

              at org.jboss.remoting3.remote.DebuggingBufferPool$1.getResource(DebuggingBufferPool.java:44)

              at org.xnio.streams.BufferPipeOutputStream.getBuffer(BufferPipeOutputStream.java:71)

              at org.xnio.streams.BufferPipeOutputStream.write(BufferPipeOutputStream.java:92)

              - locked <0x00000007b02977c8> (a org.xnio.streams.BufferPipeOutputStream)

              at org.jboss.remoting3.remote.OutboundMessage.write(OutboundMessage.java:272)

              at java.io.DataOutputStream.write(DataOutputStream.java:107)

              - locked <0x00000007b0c1c718> (a java.io.DataOutputStream)

              at org.jboss.as.ejb3.remote.protocol.AbstractMessageHandler$1.write(AbstractMessageHandler.java:207)

              at org.jboss.marshalling.OutputStreamByteOutput.write(OutputStreamByteOutput.java:52)

              at org.jboss.marshalling.UTFUtils.writeUTFBytes(UTFUtils.java:131)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:778)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:786)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)

              at org.jboss.marshalling.river.RiverObjectOutputStream.defaultWriteObject(RiverObjectOutputStream.java:165)

              at TableDataModel.writeObject(TableDataModel.java:2529)

              at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.lang.reflect.Method.invoke(Method.java:498)

              at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)

              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

              at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

              at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

              at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.writeMethodInvocationResponse(MethodInvocationMessageHandler.java:374)

              at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$600(MethodInvocationMessageHandler.java:69)

              at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:245)

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

              at org.jboss.threads.JBossThread.run(JBossThread.java:320)

               

                 Locked ownable synchronizers:

              - <0x00000006c813b888> (a java.util.concurrent.ThreadPoolExecutor$Worker)