5 Replies Latest reply on Jan 19, 2016 8:07 AM by pavan4488

    Application deployment in wildfly-9.0.2 is taking long time

    pavan4488

      Hi,

      I'm new to wildfly AS, am trying to deploy a war file but it is taking really long time to get deployed, i also increased the transaction timeout which was 300 by default.

      The same application is getting deployed in tomcat but when i try to deploy the same war in wildfly am facing a problem.

      Please do the needful. Thanks in advance.

        • 1. Re: Application deployment in wildfly-9.0.2 is taking long time
          jaysensharma

          0. Are you doing a remote deployment or local deployment ?

          1. How are you trying to deploy the application ? using management utilities like  Console/ jboss-cli.sh?

          2. It is just taking longer time? or the deployment itself is failing after the timeout period?

          3. After increasing the timeout do you see the application is getting deployed or is it continuously hanging?

          4. Can you please collect 4-5 thread dumps next time of the WildFly server when you are trying to deploy the application so that by looking at the thread dumps we can see what might be causing the issue?

          5. What is the Size of your application?

          6. During deployment did you see any drastic spike in the PermGen Memory or the Heap memory ? (may be you can use the jconsole.sh script to monitor the memory during deployment)

           

          Regards

          Jay SenSharma

          1 of 1 people found this helpful
          • 2. Re: Application deployment in wildfly-9.0.2 is taking long time
            pavan4488

            Hi Jay, Thanks for the response,

            0. Are you doing a remote deployment or local deployment ?

            Ans. I am deploying the application in local but communication to the database which is remotely located.

            1. How are you trying to deploy the application ? using management utilities like  Console/ jboss-cli.sh?

            Ans. directly placing the .war file into deployments folder of AS.

            2. It is just taking longer time? or the deployment itself is failing after the timeout period?

            Ans. i could able to deploy the application using the DB which is in my local machine, but when i try to make use of the remote DB it is deploying slowly and throwing TimeoutException.

            3. After increasing the timeout do you see the application is getting deployed or is it continuously hanging?

            Using Admin console i increased the Transaction timeout but it still throwing below error,

              WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[

                ("core-service" => "management"),

                ("management-interface" => "http-interface")

              ]'

            4. Can you please collect 4-5 thread dumps next time of the WildFly server when you are trying to deploy the application so that by looking at the thread dumps we can see what might be causing the issue?

            5. What is the Size of your application?

            Ans. 60MB

            6. During deployment did you see any drastic spike in the PermGen Memory or the Heap memory ? (may be you can use the jconsole.sh script to monitor the memory during deployment)

            Ans. No PermGen occured.

             

            Currently am running this in windows.

             

            Thanks & Regards,

            Pavan Kumar.

            • 3. Re: Application deployment in wildfly-9.0.2 is taking long time
              jaysensharma

              Hello Paven,

               

                  As you mentioned

                     2. It is just taking longer time? or the deployment itself is failing after the timeout period?

                     Ans. i could able to deploy the application using the DB which is in my local machine, but when i try to make use of the remote DB it is deploying slowly and throwing TimeoutException.

               

                 So i am suspecting that it might be the JBoss to Remote communication issue. Is the remote DB health correct?

                 Are you able to run some sample DB SQL queries from the Box where the JBoss is installed  to the remote DB, Just to verify of it is responding fine or not?

               

                 Collecting Thread dumps will definitely help us much here to find out what kind of activities the threads are performing when JBoss is getting hanged.  You can use the following command to generate the Thread dumps when you see that JBoss is responding very slow:

               

                  $JAVA_HOME/bin/jstack  -l   $JBOSS_PID
              
              Suppose if JBoss PID is 1234  then the command should be:
              
                  $JAVA_HOME/bin/jstack  -l   1234
              

               

               

              Regards

              Jay SenSharma

              • 4. Re: Application deployment in wildfly-9.0.2 is taking long time
                pavan4488

                Hello Jay,

                What you've suspected was right, since i could able to communicate within my local and internal access, but not with the one which is located remotely.

                DB is fine because i could able to connect it through tomcat and function properly.

                 

                I am not able to attach the complete log file, but below logs could help you in solving my problem,

                 

                2016-01-13 18:54:42,778 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000261: Table found: YMS_QA20150413.MASTER_HOL_EMPLOYEE

                2016-01-13 18:54:42,779 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000037: Columns: [code, employee_id, id, master_ref_id]

                2016-01-13 18:54:42,781 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000108: Foreign keys: [fk_i1ra4by7jpvjph00nt9e210kt]

                2016-01-13 18:54:42,782 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000126: Indexes: [sys_c0070585]

                2016-01-13 18:54:43,418 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[

                    ("core-service" => "management"),

                    ("management-interface" => "http-interface")

                ]'

                2016-01-13 18:54:43,433 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = ywc-1.0.war_oracle.jdbc.OracleDriver_11_2

                2016-01-13 18:54:43,434 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0019: Stopped Driver service with driver-name = ywc-1.0.war_com.mysql.jdbc.Driver_5_1

                2016-01-13 18:54:44,120 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000261: Table found: YMS_QA20150413.MASTER_HOL_EMP_GROUP

                2016-01-13 18:54:44,121 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000037: Columns: [emp_grp_id, code, id, master_ref_id]

                2016-01-13 18:54:44,122 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000108: Foreign keys: [fk_nk3qyrswrenwrp0ovfdqguxo1]

                2016-01-13 18:54:44,124 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000126: Indexes: [sys_c0070589]

                2016-01-13 18:54:45,628 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000261: Table found: YMS_QA20150413.MASTER_HOL_LANGUAGE

                2016-01-13 18:54:45,629 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000037: Columns: [description, language, id, language_id, master_ref_id]

                2016-01-13 18:54:45,631 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000108: Foreign keys: [fk_5f8oyy8fg7dlj83xe1ljxk9h5]

                2016-01-13 18:54:45,632 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000126: Indexes: [sys_c0070594]

                2016-01-13 18:54:47,005 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000261: Table found: YMS_QA20150413.MASTER_HOL_LOCATION

                2016-01-13 18:54:47,007 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000037: Columns: [code, id, location_id, master_ref_id]

                2016-01-13 18:54:47,008 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000108: Foreign keys: [fk_951d7cmqrc28bl6qp2cbkuida]

                2016-01-13 18:54:47,009 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000126: Indexes: [sys_c0070598]

                2016-01-13 18:54:48,352 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000261: Table found: YMS_QA20150413.MASTER_HOL_SHIPMENT_MODE

                2016-01-13 18:54:48,353 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000037: Columns: [code, ship_mode_id, id, master_ref_id]

                2016-01-13 18:54:48,354 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000108: Foreign keys: [fk_r5hjgau5vnryrnrm9sytdwy9j]

                2016-01-13 18:54:48,355 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000126: Indexes: [sys_c0070602]

                2016-01-13 18:54:48,424 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@df7ee2 for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

                  at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:396)

                  at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1384)

                  at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1332)

                  at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1292)

                  at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1180)

                  at org.jboss.as.controller.AbstractOperationContext.handleContainerStabilityFailure(AbstractOperationContext.java:964)

                  at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:590)

                  at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:354)

                  at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:330)

                  at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1183)

                  at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:453)

                  at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:327)

                  at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:313)

                  at org.jboss.as.server.ServerService.boot(ServerService.java:384)

                  at org.jboss.as.server.ServerService.boot(ServerService.java:359)

                  at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:271)

                  at java.lang.Thread.run(Thread.java:745)

                 

                 

                2016-01-13 18:54:48,433 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@df7ee2 for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException

                2016-01-13 18:54:50,138 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000261: Table found: YMS_QA20150413.MASTER_LOCALE_DATA

                2016-01-13 18:54:50,139 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000037: Columns: [centralized_location_id, short_description, description, id, haz_master_id, locale, master_ref_id]

                2016-01-13 18:54:50,140 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000108: Foreign keys: [fk_p7hj46t7ts8rklg5c0e7xbltt, fk_t01lmiqup09tnfpleucfcdmuu]

                2016-01-13 18:54:50,142 INFO  [org.hibernate.tool.hbm2ddl.TableMetadata] (ServerService Thread Pool -- 59) HHH000126: Indexes: [sys_c0070072]

                2016-01-13 18:54:53,437 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@e595d4 for operation {"address" => [("core-service" => "management"),("security-realm" => "ApplicationRealm")],"operation" => "add","map-groups-to-roles" => undefined} at address [

                    ("core-service" => "management"),

                    ("security-realm" => "ApplicationRealm")

                ] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException

                  at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:396)

                  at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1384)

                  at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1332)

                  at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1307)

                  at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1180)

                  at org.jboss.as.controller.AbstractOperationContext.handleContainerStabilityFailure(AbstractOperationContext.java:964)

                  at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:590)

                  at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:354)

                  at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:330)

                  at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1183)

                  at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:453)

                  at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:327)

                  at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:313)

                  at org.jboss.as.server.ServerService.boot(ServerService.java:384)

                  at org.jboss.as.server.ServerService.boot(ServerService.java:359)

                  at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:271)

                  at java.lang.Thread.run(Thread.java:745)

                 

                 

                Please do the needful.

                 

                Thanks & Regards,

                Pavan kumar.

                • 5. Re: Application deployment in wildfly-9.0.2 is taking long time
                  pavan4488

                  Hi,

                  The issue has been resolved, it was the network issue where it is taking much time to deploy.