10 Replies Latest reply on Aug 29, 2012 10:28 AM by markorocko

    Where do you look when deployment fails?

    markorocko

      Folks so I'm trying to deploy a new ear file.  It's saying the ear deployment failed but not giving me any clues or reason why.

      What can I do to determine the true error and resolve when it's just speaking generalities?

      This is a large app I'm converting from JBoss 4.

       

      Error in log:

      13:58:30,980 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.1.Final "Brontes" started in 9953ms - Started 169 of 248 services

      (78 services are passive or on-demand)

      13:59:20,886 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "hello.ear"

      14:00:01,636 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "power.war"

      14:00:01,636 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "libs.jar"

      14:00:20,870 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 1) JBAS015870: Deploy of deployment "hello.ear" was rolled back with failure mes

      sage Operation cancelled

      14:00:20,870 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) JBAS015052: Did not receive a response to the deployment o

      peration within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the

      deployment.

      14:00:21,183 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015877: Stopped deployment libs.jar in 304ms

      14:00:23,651 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry avalon-framework-4.2.0.jar in "/C:/jboss-as-7.1.1.Final/

      bin/content/hello.ear/power.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

      14:00:23,667 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry batik-all-1.7.jar in "/C:/jboss-as-7.1.1.Final/bin/conte

      nt/hello.ear/power.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

      14:00:23,683 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry commons-io-1.3.1.jar in "/C:/jboss-as-7.1.1.Final/bin/co

      ntent/hello.ear/power.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

      14:00:23,683 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry serializer-2.7.0.jar in "/C:/jboss-as-7.1.1.Final/bin/co

      ntent/hello.ear/power.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

      14:01:05,448 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment power.war in 44581ms

      14:01:05,620 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment hello.ear in 44750ms

       

      server.log

      13:59:20,886 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "hello.ear"

      14:00:01,636 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "power.war"

      14:00:01,636 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "libs.jar"

      14:00:20,870 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 1) JBAS015870: Deploy of deployment "hello.ear" was rolled back with failure message Operation cancelled

      14:00:20,870 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.

      14:00:21,183 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015877: Stopped deployment libs.jar in 304ms

      14:00:23,651 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry avalon-framework-4.2.0.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/power.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

      14:00:23,667 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry batik-all-1.7.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/power.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

      14:00:23,683 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry commons-io-1.3.1.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/power.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

      14:01:05,448 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment power.war in 44581ms

      14:01:05,620 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment hello.ear in 44750ms

        • 1. Re: Where do you look when deployment fails?
          nickarls

          Hmm. How are you deploying this? Dropping into the deployments folder?

           

          You might want to crank up the log level to see if there is any suspicous hangs while processing the archive. Or if the application is really big you might want to increase the deployment timeout in standalone.xml

          • 2. Re: Where do you look when deployment fails?
            markorocko

            Nice!  Turned some INFOs to DEBUG, I now see a missing ClassNotFoundError.  Let me see if I can fix and resolve now.

            • 3. Re: Where do you look when deployment fails?
              markorocko

              Yes, just dropping EAR file into deploy directory.

              I fixed the ClassNotFoundException but still stuck, it's showing deployment failed with no explanation.

              I bumped up timeouts from 3000 to 30000.
              Any Suggestions?

               

              15:23:11,870 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.

              15:23:11,870 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 1) JBAS015870: Deploy of deployment "hello.ear" was rolled back with failure message Operation cancelled

              15:23:14,464 DEBUG [org.jboss.modules] (MSC service thread 1-4) Module deployment.hello.ear.power.war:main defined by Service Module Loader

              15:23:14,636 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment libs.jar in 2762ms

              • 4. Re: Where do you look when deployment fails?
                nickarls

                Crank up the JBoss Modules log level to trace. You'll probably find some transistive dependency from a 3rd party lib that's missing. Then you'll go "Why can't this be shown as an error!?"

                1 of 1 people found this helpful
                • 5. Re: Where do you look when deployment fails?
                  markorocko

                  Very good advice.  Unfortunately, nothing in particular leaped out of the logs to indicate why AS7 could not deploy my ear file.

                  I am even considering deploying it on another app server like Glassfish in the hope that it might provide clues.

                  These are the only blatant errors I see but nothing pointing to a problem with the ear.

                   

                   

                  07:59:30,984 TRACE [org.jboss.modules] (ServerService Thread Pool -- 37) Attempting to define class org.jboss.marshalling.reflect.ConcurrentReferenceHashMap$ReferenceType in Module "org.jboss.marshalling:main" from local module loader @13c468a (roots: C:\jboss-as-7.1.1.Final\modules)

                  07:59:31,000 FINE  [org.hornetq.core.asyncio.impl.AsynchronousFileImpl] (MSC service thread 1-4) HornetQAIO_ia64 -> error loading the native library: java.lang.UnsatisfiedLinkError: no HornetQAIO_ia64 in java.library.path

                      at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738) [rt.jar:1.6.0_34]

                      at java.lang.Runtime.loadLibrary0(Runtime.java:823) [rt.jar:1.6.0_34]

                      at java.lang.System.loadLibrary(System.java:1028) [rt.jar:1.6.0_34]

                      at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.loadLibrary(AsynchronousFileImpl.java:92) [hornetq-core-2.2.13.Final.jar:]

                      at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.<clinit>(AsynchronousFileImpl.java:119) [hornetq-core-2.2.13.Final.jar:]

                      at org.hornetq.core.journal.impl.AIOSequentialFileFactory.isSupported(AIOSequentialFileFactory.java:113) [hornetq-core-2.2.13.Final.jar:]

                      at org.jboss.as.messaging.HornetQService.start(HornetQService.java:89) [jboss-as-messaging-7.1.1.Final.jar:7.1.1.Final]

                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_34]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_34]

                      at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_34]

                   

                   

                  08:00:41,312 TRACE [org.hornetq.core.client.impl.Topology] (MSC service thread 1-4) Topology@8b4796 CREATE: java.lang.Exception: trace

                      at org.hornetq.core.client.impl.Topology.<init>(Topology.java:70) [hornetq-core-2.2.13.Final.jar:]

                      at org.hornetq.core.client.impl.ServerLocatorImpl.<init>(ServerLocatorImpl.java:484) [hornetq-core-2.2.13.Final.jar:]

                      at org.hornetq.api.core.client.HornetQClient.createServerLocatorWithoutHA(HornetQClient.java:112) [hornetq-core-2.2.13.Final.jar:]

                      at org.hornetq.jms.client.HornetQConnectionFactory.<init>(HornetQConnectionFactory.java:104) [hornetq-jms-2.2.13.Final.jar:]

                      at org.hornetq.jms.client.HornetQXAConnectionFactory.<init>(HornetQXAConnectionFactory.java:71) [hornetq-jms-2.2.13.Final.jar:]

                      at org.hornetq.api.jms.HornetQJMSClient.createConnectionFactoryWithoutHA(HornetQJMSClient.java:184) [hornetq-jms-2.2.13.Final.jar:]

                      at org.hornetq.ra.HornetQResourceAdapter.createHornetQConnectionFactory(HornetQResourceAdapter.java:1587) [hornetq-ra-2.2.13.Final.jar:]

                      at org.hornetq.ra.HornetQRAManagedConnectionFactory.createConnectionFactory(HornetQRAManagedConnectionFactory.java:149) [hornetq-ra-2.2.13.Final.jar:]

                      at org.jboss.jca.deployers.common.AbstractResourceAdapterDeployer.createObjectsAndInjectValue(AbstractResourceAdapterDeployer.java:1929) [ironjacamar-deployers-common-1.0.9.Final.jar:1.0.9.Final]

                      at  org.jboss.jca.deployers.common.AbstractResourceAdapterDeployer.createObjectsAndInjectValue(AbstractResourceAdapterDeployer.java:1034) [ironjacamar-deployers-common-1.0.9.Final.jar:1.0.9.Final]

                      at  org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator.doDeploy(ResourceAdapterActivatorService.java:155) [jboss-as-connector-7.1.1.Final.jar:7.1.1.Final]

                      at org.jboss.as.connector.services.ResourceAdapterActivatorService.start(ResourceAdapterActivatorService.java:94) [jboss-as-connector-7.1.1.Final.jar:7.1.1.Final]

                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_34]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_34]

                      at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_34]

                  • 6. Re: Where do you look when deployment fails?
                    markorocko

                    09:17:29,812 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.

                     

                    Does anyone know how to increase the allowed timeout period [60 seconds]?

                    I changed some numbers in standalone-full.xml but didn't appear to have an effect.

                    • 7. Re: Where do you look when deployment fails?
                      nickarls

                      Well, the deployer has a deployment-timeout but that defaults to 600 seconds so I'm not sure where the 60 comes from in this place.

                      Can you give a jar -tf output if the deployment and and manifest-defines dependencies? Can you get a jstack dump when it hangs?

                      • 8. Re: Where do you look when deployment fails?
                        markorocko

                        Not sure how to do the "jar -tf output".

                        Is this helpful?  Thanks Nicklas.

                         

                        08:03:04,140 TRACE [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) Scan complete

                        08:03:04,140 TRACE [org.jboss.vfs.VFSUtils] (MSC service thread 1-3) Can't find manifest for /C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/classes

                        08:03:04,156 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry avalon-framework-4.2.0.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,156 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry batik-all-1.7.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,156 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry commons-io-1.3.1.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,156 DEBUG [org.jboss.as.server.deployment] (MSC service thread 1-3) Class-Path entry "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/commons-logging-1.0.4.jar" in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar" ignored, as target is already accessible

                        08:03:04,171 DEBUG [org.jboss.as.server.deployment] (MSC service thread 1-3) Class-Path entry "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop-hyph.jar" in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar" ignored, as target is already accessible

                        08:03:04,171 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry serializer-2.7.0.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,171 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry servlet-2.2.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,187 DEBUG [org.jboss.as.server.deployment] (MSC service thread 1-3) Class-Path entry "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/xalan-2.7.0.jar" in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar" ignored, as target is already accessible

                        08:03:04,187 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry xercesImpl-2.7.1.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,203 DEBUG [org.jboss.as.server.deployment] (MSC service thread 1-3) Class-Path entry "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/xml-apis-1.3.04.jar" in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar" ignored, as target is already accessible

                        08:03:04,203 DEBUG [org.jboss.as.server.deployment] (MSC service thread 1-3) Class-Path entry "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/xml-apis-ext-1.3.04.jar" in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar" ignored, as target is already accessible

                        08:03:04,203 DEBUG [org.jboss.as.server.deployment] (MSC service thread 1-3) Class-Path entry "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/xmlgraphics-commons-1.4.jar" in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/fop.jar" ignored, as target is already accessible

                        08:03:04,218 TRACE [org.jboss.vfs.VFSUtils] (MSC service thread 1-3) Can't find manifest for /C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/jimi-1.0.jar

                        08:03:04,218 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry commons-beanutils.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/struts.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,218 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry commons-collections.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/struts.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,234 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry commons-digester.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/struts.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,234 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry commons-logging.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/struts.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,234 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry commons-validator.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/struts.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,250 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry jakarta-oro.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/struts.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,250 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry struts-legacy.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/struts.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,250 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry xercesImpl.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/xalan-2.7.0.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,265 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry xml-apis.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/xalan-2.7.0.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,265 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-3) Class Path entry serializer.jar in "/C:/jboss-as-7.1.1.Final/bin/content/hello.ear/app.war/WEB-INF/lib/xalan-2.7.0.jar"  does not point to a valid jar for a Class-Path reference.

                        08:03:04,281 TRACE [org.jboss.vfs.zip] (MSC service thread 1-2) Closing zip filesystem org.jboss.vfs.spi.JavaZipFileSystem@1293709

                        • 9. Re: Where do you look when deployment fails?
                          nickarls

                          Just a "jar tf hello.ear" should give you the layout

                          • 10. Re: Where do you look when deployment fails?
                            markorocko

                            jar tf hello.ear

                             

                            META-INF/

                            META-INF/MANIFEST.MF

                            META-INF/application.xml

                            libs.jar

                            app.war