11 Replies Latest reply on Apr 18, 2014 8:03 AM by max_kuffs

    Performance gap between JBoss 7.1 and Wildfly 8.0

    max_kuffs

      Hello,

       

      I migrated a project from JavaEE6 to JavaEE7 and switched my development server from JBoss 7.1 to Wildfly.

      This was quite straight forward, but then I ran into a timeout in one of my integration tests. That was quite odd, because I never tweaked JBoss in any way. I used both products out of the box (just added datasources and realm) and I had never performance issues during development.

       

      First I analyzed which part of my code caused this behaviour. I extracted the code parts into a smaller Rest Service and started the profiler. The same code ran on JBoss and Wildfly without any changes. In Wildfly there is a lot of noise from WELD and logging. The business logic comes way down on the chart. In JBoss 7 the business logic comes right on the second place and looks like I would expect it.

       

      My test szenario:

      // Entity loading happens before. No lazy loading or any other database access involved.
      for(int i = 0; i< 1000;i++) {
           log.info("Run {}", i);
           SightContainer sightBefore = sightService.getGlobalSight(game);
           SightContainer sightAfter = sightService.getGlobalSight(game);
               
           sightService.createCreateRemoveEvents(game, sightBefore, sightAfter);
      }

       

      getGameObjectsCached(Game) retrieves the list of objects once from the database and then pins it with the TransactionSynchronizationRegistry onto the transaction. That worked very good so far.

      public SightContainer getGlobalSight(Game game) {

           ...

          List<GameObject> objects = gameObjectService.getGameObjectsCached(game);

          int objectsWithSightCount = 0;

          for (GameObject objToSightTest : objects) {

              ...

              objectsWithSightCount++;

              Set<GameObject> objectsInSight = getObjectsInSightOfObject(objToSightTest); // iterating over objects and checking the lines. plain java so far.

              ...

          }

           log.info("Sight calculation of {} objects took {}ms", objectsWithSightCount, System.currentTimeMillis() - start );

           ...

      }

       

       

      On Wildfly:

      09:19:53,072 INFO  [net.hq.test.TestWithinContainer] (default task-1) Run 1

      09:19:54,508 INFO  [net.hq.service.SightService] (default task-1) Sight calculation of 6 objects took 1436ms

      09:19:55,882 INFO  [net.hq.service.SightService] (default task-1) Sight calculation of 6 objects took 1374ms

       

      On Jboss:

      09:24:02,386 INFO  [net.hq.test.TestWithinContainer] (http-localhost-127.0.0.1-8080-1) Run 1

      09:24:02,448 INFO  [net.hq.service.SightService] (http-localhost-127.0.0.1-8080-1) Sight calculation of 6 objects took 62ms

      09:24:02,509 INFO  [net.hq.service.SightService] (http-localhost-127.0.0.1-8080-1) Sight calculation of 6 objects took 60ms

       

      In my application I have to calculate sight lines between objects which are described with coordinates. So I have to iterate a lot and use the DDA Algorithm to draw a line and look if it is blocked by anything. There is no Hibernate involved in this point so the calculation happens without any database access. I optimized out a lot yesterday. Before I had a factor 10 difference.

       

      I guess I have to change my wildfly configuration in any way. My question is ...which one? At this stage of development I run both servers in eclipse. So I did not expect the full performance. What bothers me is the performance gap between the two servers.

      greetings,

      m

        • 1. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
          jaikiran

          Which version of Weld is this? Perhaps these are related:

           

          https://issues.jboss.org/browse/AS7-6583

          https://github.com/weld/core/pull/276/files

           

          I can't think of anything else, but in WildFly 8 CDI is enabled by default because the spec says so. But I wouldn't expect it to degrade the performance to such an extent.

          • 2. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
            max_kuffs

            11:05:03,099 INFO  [org.jboss.weld.Version] (MSC service thread 1-10) WELD-000900: 2.1.2 (Final)

             

            Well it is quite strange. :-/

            • 3. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
              ctomc

              As per EE7 requirements, CDI is now enabled for all deployments.

               

              what you can do is

               

              change global configuration: (this will put it to more AS7 like behavior)

               

              /subsystem=weld:write-attribute(name=require-bean-descriptor,value=true)

               

              per-deployment configuration - add the following content to META-INF/jboss-all.xml of the application

               

              <jboss xmlns="urn:jboss:1.0"> <weld xmlns="urn:jboss:weld:1.0" require-bean-descriptor="true"/> </jboss>

              or exclude weld subsystem from being applied to your application by having following content in META-INF/jboss-all.xml

               

              <jboss xmlns="urn:jboss:1.0">

                <deployment>

                   <exclude-subsystems>

                      <subsystem name="weld" />

                  </exclude-subsystems>

              </jboss>

              • 4. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
                jaikiran

                In addition to what Tomaz noted, you can even attach that same application to this forum thread (the advanced editor has that option) so that someone can take a more closer look.

                • 5. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
                  tremes

                  Well I think it would be useful to profile your application more deeply. I mean org.jboss.threads.JBossThread.run() is too general and can hide a lot of things. In other words - there is need to exactly recognize class/package, where this gap potentially starts. It's also interesting that net.hq.model.embedded.Coordinate.contains()  takes much longer on AS7 than on WF. 

                  • 6. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
                    max_kuffs

                    In nearly the same time JBoss 7 reaches more runs. So the contains adds up more, I think. The contains does not much. Just searches for a coordinate in a collection. It is just used a lot when checking sight lines.

                     

                    public static boolean contains(Collection<? extends ICoordinate> coords, int x, int y) {

                      for(ICoordinate c : coords) {

                           int cx = c.getX();

                           int cy = c.getY();

                     

                           if(cx == x && cy == y) {

                                return true;

                           }

                      }

                     

                      return false;

                    }

                     

                    I can't post the full application. Tear out just the sight calculation parts would take its time and I am not sure how well that compares to the real application.

                    How can I profile the server in more detail with visualvm? Any settings or start parameters for the server?

                     

                    If you want I can offer a remote connection to my system (eg. via Teamviewer, Skype, VNC, ...) and we analyze it together. So we have test data and everything we need for tests. This evening would be possible for me if that is an option.

                    Turing off the Weld subsystem completely would disable some central parts in the application. I use CDI for Events after the commit phase. I will try out your deployment settings and report back asap.

                     

                    Thanks a lot for the help!

                    • 7. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
                      ctomc

                      Before you do more profiling, make sure you do it on latest version which is 8.1.0.CR1 as we did some perf improvements there.

                      Upgrade should be strait forward as it is mostly bugfix release

                      • 8. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
                        tremes

                        Well I don't know jvisualvm much, I've always used it only for some basic measurement and overall information. I use JProfiler (version 7 is enough), where you can track almost everything. If you can at least create some simple reproducer, then I can take a look. But try your app with 8.1.0.CR1 at first as Tomaz suggested.

                        • 9. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
                          max_kuffs

                          Ok then I try the new version and profile again.

                          • 10. Re: Performance gap between JBoss 7.1 and Wildfly 8.0
                            m.koitz

                            hi,

                            i am working on the same project as max. i upgraded to the new wildfly 8.1.0CR1. the calculation is still as slow as on wildfly 8.0.0Final. i also added the

                            /subsystem=weld:write-attribute(name=require-bean-descriptor,value=true)

                            it made it a little faster but still way slower than on JBoss AS 7.1

                            • 11. Re: Re: Performance gap between JBoss 7.1 and Wildfly 8.0
                              max_kuffs

                              I downloaded JProfiler and upgraded to CR1 with the Wildfly config I posted above. I connected to the VM after the 20th run so there is no hibernate involved and messing up any time statistics.

                               

                              As mike said it did not get any faster. I don't know JProfiler very well, but I tried my best to capture a session. If I need any other settings let me know.

                               

                              The file was to large to upload correctly. I re-uploaded it to google drive: https://drive.google.com/file/d/0B5sKVb2kb4BpWXJ6Tm9RUUVkUGs/edit?usp=sharing

                               

                              Greetings,

                              m

                               

                              Update:

                              After I removed every EJB lookup from our sight calculation and put it into a pojo I get 20ms instead of the 2000ms on CR1.

                               

                              I also observed that the TransactionSynchronizationRegistry is slower in Wildfly than on JBoss 7.1. I use this feature quite often the get rid of any threadlocals. I testet 1000 Object retrievals. 150ms difference is quite a lot.

                               

                              JBoss 7.1

                              13:29:40,913 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Object retrival: 675

                              13:29:42,380 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Object retrival: 664

                              13:29:43,803 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Object retrival: 658

                              13:29:47,891 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) Object retrival: 666

                               

                               

                              Wildfly:

                               

                              13:31:53,432 INFO  [stdout] (default task-63) Object retrival: 837

                              13:31:55,093 INFO  [stdout] (default task-64) Object retrival: 838

                              13:31:56,440 INFO  [stdout] (default task-1) Object retrival: 850

                              13:31:57,856 INFO  [stdout] (default task-2) Object retrival: 831

                              13:31:59,314 INFO  [stdout] (default task-3) Object retrival: 819

                               

                              @Resource

                              private TransactionSynchronizationRegistry registry;

                               

                              public Object get(String key){

                                  return registry.getResource(key);

                              }

                               

                              The Injects are faster as far as I can see:

                               

                              Beanlocator lookup:

                              1000 manual jndi Lookup with the Beanlocator pattern.

                               

                              JBoss 7.1:

                              13:43:56,651 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) JNDI Lookup: 16

                              13:43:56,843 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) JNDI Lookup: 16

                              13:43:56,994 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) JNDI Lookup: 8

                              13:43:57,210 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) JNDI Lookup: 20

                              13:43:57,401 INFO  [stdout] (http-localhost-127.0.0.1-8080-1) JNDI Lookup: 19

                               

                              Wildfly:

                               

                              13:41:53,546 INFO  [stdout] (default task-39) JNDI Lookup: 9

                              13:41:53,784 INFO  [stdout] (default task-40) JNDI Lookup: 6

                              13:41:53,967 INFO  [stdout] (default task-41) JNDI Lookup: 10

                              13:41:54,163 INFO  [stdout] (default task-42) JNDI Lookup: 6

                              13:41:54,309 INFO  [stdout] (default task-43) JNDI Lookup: 7

                               

                              So I guess that was the Problem. The TransactionSynchronisationRegistry got way slower. In a production env. the numbers will be much smaller so I think the Lookups and Injects are perfectly fine.