1 2 Previous Next 15 Replies Latest reply on Nov 19, 2016 7:39 AM by pablor.py Branched to a new discussion.

    Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6

    abl-dam

      We have a large Application running on AS7/EAP6, heavily using CDI (among other JEE stuff). On a local workstation application deploys and starts on EAP6.0.1 in about 15 sec.

      Now - after deploying exactly the same application to Wildfly 8.1.0 - deployment and startup time increases up to 120 sec.

       

      from TRACE/DEBUG output it _seems_ that most of the time is used in WELD scanning and deployment

       

      I already found this thread Performance gap between JBoss 7.1 and Wildfly 8.0 and tried creating a jboss-all.xml with

      <jboss xmlns="urn:jboss:1.0">
         <weld xmlns="urn:jboss:weld:1.0" require-bean-descriptor="true"/>
      </jboss>
      
      

      in META-INF, but with no success. Startup time is still about 120 sec. If the server is running, performance of the application is ok, so I'm not sure if above discussion covers exactly our problem.

       

      Are there any other solutions to switch back WELD to AS7 behaviour (no implicit bean archives)? Any Settings to check or improve? Anyone with similar experience?

      Any help appreciated - thanks!

       

       

      additional note: on most startups (not always) I get this error 1 or 2 times:

      20:34:32,826 ERROR [connection] (XNIO-1 I/O-2) JBREM000200: Remote connection failed: java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen

      this is also only on wildfly. I'm not sure if it's related to the slow startup or a separate issue

        • 1. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
          jaikiran

          Can you attach the server.log which shows this 2 minute deployment logs? Also, have you been able to profile (or at least take thread dumps at regular intervals) during this 2 minute long deployment? Do you have any jboss-ejb-client.xml in your deployment, if yes, what does it look like?

          • 2. Re: Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
            abl-dam

            thanks for your reply!

            attached is the server.log from startup, using the out-of-the-box log-config. if you need one with more logging details, please let me know

            also I took some thread dumps during the startup, and one shortly after

             

            no, no jboss-ejb-client.xml.

            a jboss-deplyment-structure.xml is the only jboss specific DD

            • 3. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
              jblinick

              we are just starting out with jbpm, but can confirm that it seems very slow ( >1.5 mins) in Wildfly (haven't tried AS7).

              • 4. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                ctomc

                Only thing that kinda stands out from your thread dumps is that there is some remote RMI calls going on which could be a cause of the slowness.

                 

                Is you app doing some RMI lookups during deployment? That could make it slow, especially if remote host to which RMI connection is done is slow to respond.

                 

                That would also explain this entry in server.log

                 

                2014-08-21 09:43:54,318 ERROR [org.jboss.remoting.remote.connection] (XNIO-1 I/O-1) JBREM000200: Remote connection failed: java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen

                • 5. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                  abl-dam

                  @Tomaz, Josh ... thanks for your replies

                  no, we make no RMI lookups at all in out application (at least not explicit in our code). maybe something in our code is causing RMI lookups ... can you see the origin for this lookups in the threaddump?

                  as stated in the original post: the identical application deploys fast in AS7/EAP6 - without any errors on deployment

                  • 6. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                    abl-dam

                    I came back to this and did some more tests. I tracked it down by comparing logs and putting this in log config:

                    <logger category="org.jboss.weld">
                       <level name="TRACE"/>
                    </logger>
                    

                     

                    the time difference results mostly from WELD deployment:

                     

                    WILDFLY 8.1.0

                    13:08:23,603 TRACE [org.jboss.weld.Bean] (weld-worker-1) WELD-001541: Adding method to proxy: ... (first entry)
                    ...
                    13:09:28,192 TRACE [org.jboss.weld.Bean] (weld-worker-4) WELD-001541: Adding method to proxy: ... (last entry)
                    -> 64,5 sec.

                     

                    JBoss EAP 6.0.1

                    13:30:40,049 TRACE [org.jboss.weld.Bean] (MSC service) Adding method ... (first entry)
                    ...
                    13:30:44,629 TRACE [org.jboss.weld.Bean] (MSC service) Adding method ... (last entry)
                    -> 4,5 sec.

                     

                    although WFLY starts 4 worker threads for WELD scanning and deployment (weld-worker-1 to 4), it needs about 1 min. longer for the job.

                     

                    Any ideas how to avoid or improve that?

                    We are still not able to move on to Wildfly (or Java8) and stick with old JB7/EAP6 Version!

                    • 7. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                      ctomc

                      Any chance you can try with 9.0.0.Alpha1 which has new Weld 2.2 in it which has lots of fixes for boot time performance.

                       

                      Same weld is also in 8.x branch and will be part of 8.2 release we are working on.

                       

                      9.0.0.Alpha1 shouldn't be much different when it comes to deployment then 8.x stuff, just build / assembly / code organization is what changed the most.

                      1 of 1 people found this helpful
                      • 8. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                        abl-dam

                        ok, I will give it a try. I'll give you feedback asap

                        do you know when beta 1 comes out? (should be today according to wlfy roadmap)

                        • 9. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                          abl-dam

                          MUCH better with 9.0.0.Alpha1 !

                          Startup / deployment in about 28,5 sec. (comparing to 120 sec. in 8.1.0.Final):

                           

                          12:20:37,789 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 9.0.0.Alpha1 (WildFly Core 1.0.0.Alpha8) started in 28482ms - Started 1141 of 1244 services (176 services are lazy, passive or on-demand)

                           

                           

                          12:20:28,468 TRACE [org.jboss.weld.Bean] (MSC service thread 1-5) WELD-001541: Adding method to Proxy ... (first entry)

                          ...

                           

                          12:20:37,384 TRACE [org.jboss.weld.Bean] (MSC service thread 1-3) WELD-001541: Adding method to Proxy ... (last entry)

                          -> 9 sec.

                          but still about double time compared to EAP6

                           

                           

                          the new version also writes out the following lines:

                           

                          12:20:26,017 TRACE [org.jboss.weld.deployer] (weld-worker-1) Falling back to reflection for org.apache.wicket.markup.html.form.Form

                          ...

                           

                          12:20:28,312 TRACE [org.jboss.weld.deployer] (weld-worker-4) Falling back to reflection for org.apache.wicket.markup.html.panel.Panel

                          ...

                          nearly all for wicket classes. we use wicket and have it's libs in the EAR, but also a lot of other libraries. any idea why?

                           

                          again, it makes no difference if I have this in jboss-all.xml. not in matters of startup time or trace output:

                          <jboss xmlns="urn:jboss:1.0">

                             <weld xmlns="urn:jboss:weld:1.0" require-bean-descriptor="true"/>

                          </jboss>



                          • 10. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                            ctomc

                            maybe jharting can help.

                            • 11. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                              jharting

                              abl-dam glad to hear you observe improvement with Weld 2.2 (WildFly 9). Until WildFly 8.2 is released, you can use Weld 2.2 with WildFly 8.1.0.Final by applying a patch that can be found here: http://sourceforge.net/projects/jboss/files/Weld/2.2.5.Final/

                               

                              Anyway, thanks for attaching stack dumps. These are very useful for us. Any chance you could do the same with WildFly 9 Alpha for us (or 8.1.0 with Weld 2.2)? You can also add a text file named WEB-INF/classes/org.jboss.weld.executor.properties

                              containing:

                              threadPoolDebug=true

                              to your application. This will cause more performance-related information to be printed into the server log.

                              • 12. Re: Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                                abl-dam

                                attaching server.log and thread dumps taken from 2 startups for Wildfly9/Weld 2.2

                                file org.jboss.weld.executor.properties is added to WEB-INF/classes

                                 

                                you can see also the TRACE outputs in there regarding wicket classes, although they should be completely ignored by Weld (IMO)

                                if I can give you any further information or tracing output, please let me know!

                                 

                                thanks for the provided link to patch Wildfly 8.1 to Weld 2.2 - I will try that also and give you the results

                                • 13. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                                  abl-dam

                                  applied the Weld 2.2.5 patch above to Wildfly 8.1.0 and tested again

                                  result is as expected: 28-29 sec. for Startup ... same as with Wildfly 9 Alpha

                                  • 14. Re: Deployment and Startup very slow on Wildfly 8.1.0 compared to AS7/EAP6
                                    karthik-vasishta.ramesh

                                    Thanks a lot Jozef. I applied the weld 2.2.5 patch that you mentioned on Wildfly8.1.0.Final and the performance is so much better!

                                    1 2 Previous Next