6 Replies Latest reply on Jul 7, 2015 5:45 AM by ctomc

    HotDeployment confusion

    epeters

      Looking today at a fresh JVM crash file, from Widlfly 8.2-Final, I'm finding myself scratching my head...

       

      #

      # A fatal error has been detected by the Java Runtime Environment:

      #

      #  SIGSEGV (0xb) at pc=0x00002b43e6f73c19, pid=29437, tid=47571906619712

      #

      # JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build 1.7.0_79-mockbuild_2015_04_11_03_39-b00)

      # Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)

      # Derivative: IcedTea 2.5.5

      # Distribution: Built on Red Hat Enterprise Linux Server release 5.11 (Tikanga) (Sat Apr 11 03:39:43 EDT 2015)

      # Problematic frame:

      # C  [libzip.so+0x4c19]  ZIP_GetEntry+0x79

      ...

       

       

      From the JVM frames, it's clear it was attempting a hot-deployment:

      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)

      J 172  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00002b43e7241e38 [0x00002b43e7241de0+0x58]

      J 1126 C2 org.jboss.modules.JarFileResourceLoader.getJarEntry(Ljava/lang/String;)Ljava/util/jar/JarEntry; (52 bytes) @ 0x00002b43e7539184 [0x00002b43e75390a0+0xe4]

      J 2264 C2 org.jboss.modules.JarFileResourceLoader.getResource(Ljava/lang/String;)Lorg/jboss/modules/Resource; (207 bytes) @ 0x00002b43e787e228 [0x00002b43e787e180+0xa8]

      J 6399 C2 org.jboss.modules.ModuleClassLoader.loadResourceLocal(Ljava/lang/String;)Ljava/util/List; (128 bytes) @ 0x00002b43e84a51e8 [0x00002b43e84a4b20+0x6c8]

      J 6118 C2 org.jboss.modules.Module.getResources(Ljava/lang/String;)Ljava/util/Enumeration; (298 bytes) @ 0x00002b43e83ed12c [0x00002b43e83ecb00+0x62c]

      j  org.jboss.modules.ModuleClassLoader.findResources(Ljava/lang/String;Z)Ljava/util/Enumeration;+5

      J 6321 C2 org.jboss.modules.ConcurrentClassLoader.getResources(Ljava/lang/String;)Ljava/util/Enumeration; (65 bytes) @ 0x00002b43e7da9a74 [0x00002b43e7da9a00+0x74]

      j  java.lang.ClassLoader.getResources(Ljava/lang/String;)Ljava/util/Enumeration;+19

      J 6860 C2 org.jboss.seam.deployment.SeamDeploymentProperties.getResources()Ljava/util/Enumeration; (63 bytes) @ 0x00002b43e86551cc [0x00002b43e8654fc0+0x20c]

      J 7037 C2 org.jboss.seam.deployment.SeamDeploymentProperties.addPropertiesFromResourceBundle(Ljava/lang/String;Ljava/util/List;)V (95 bytes) @ 0x00002b43e86fce40 [0x00002b43e86fce00+0x40]

      j  org.jboss.seam.deployment.SeamDeploymentProperties.getPropertyValues(Ljava/lang/String;)Ljava/util/List;+17

      j  org.jboss.seam.deployment.DeploymentStrategy.initDeploymentHandlers()V+26

      j  org.jboss.seam.deployment.DeploymentStrategy.getDeploymentHandlers()Ljava/util/Map;+8

      j  org.jboss.seam.deployment.HotDeploymentStrategy.<init>(Ljava/lang/ClassLoader;Ljava/io/File;Ljavax/servlet/ServletContext;Z)V+53

      j  org.jboss.seam.init.Initialization.createHotDeployment(Ljava/lang/ClassLoader;Z)Lorg/jboss/seam/deployment/HotDeploymentStrategy;+60

      j  org.jboss.seam.init.Initialization.redeploy(Ljavax/servlet/http/HttpServletRequest;Lorg/jboss/seam/core/Init;)V+22

      j  org.jboss.seam.web.HotDeployFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+442

      ...

       

      From that context, the error makes some level of sense -- digging into into libzip.so related crashes, I find that libzip.so will trigger a crash if the zip file is being modified while the library tries to read from it.

      If a hot-deploy gets its timing wrong, I could see it reading from a zip still being written, stepping into this error.

       

      Except, our deployment scanner is set to only scan at startup, which happened thirteen hours before the crash - this leaves open question of "What was being hot-deployed when the JVM crashed?"  I don't believe it was our application.  The management console might, but we make minimal use of the management console (nearly zero), and I'm seeing no indication of deployments triggered there.  If I stretch my imagination, I could see New Relic playing a role in this, but the redeploy request was triggered through Undertow.  I'd be a little surprised if New Relic used that interface any.

       

      Does anyone have suggestions on where to go looking for root cause in this?

      //EricP

        • 1. Re: HotDeployment confusion
          jaikiran

          Can you please paste the entire call stacktrace? I'm surprised Seam is involved with any kind of hot deployments in WildFly. Do you use Seam in your applications?

          • 2. Re: HotDeployment confusion
            epeters

            We do use seam in our applications.

             

            Stack: [0x00002b4432886000,0x00002b4432987000],  sp=0x00002b4432983d90,  free space=1015k
            Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
            C  [libzip.so+0x4c19]  ZIP_GetEntry+0x79
            C  [libzip.so+0x3bba]  Java_java_util_zip_ZipFile_getEntry+0xfa
            J 172  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00002b43e7241eae [0x00002b43e7241de0+0xce]
            
            Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
            J 172  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00002b43e7241e38 [0x00002b43e7241de0+0x58]
            J 1126 C2 org.jboss.modules.JarFileResourceLoader.getJarEntry(Ljava/lang/String;)Ljava/util/jar/JarEntry; (52 bytes) @ 0x00002b43e7539184 [0x00002b43e75390a0+0xe4]
            J 2264 C2 org.jboss.modules.JarFileResourceLoader.getResource(Ljava/lang/String;)Lorg/jboss/modules/Resource; (207 bytes) @ 0x00002b43e787e228 [0x00002b43e787e180+0xa8]
            J 6399 C2 org.jboss.modules.ModuleClassLoader.loadResourceLocal(Ljava/lang/String;)Ljava/util/List; (128 bytes) @ 0x00002b43e84a51e8 [0x00002b43e84a4b20+0x6c8]
            J 6118 C2 org.jboss.modules.Module.getResources(Ljava/lang/String;)Ljava/util/Enumeration; (298 bytes) @ 0x00002b43e83ed12c [0x00002b43e83ecb00+0x62c]
            j  org.jboss.modules.ModuleClassLoader.findResources(Ljava/lang/String;Z)Ljava/util/Enumeration;+5
            J 6321 C2 org.jboss.modules.ConcurrentClassLoader.getResources(Ljava/lang/String;)Ljava/util/Enumeration; (65 bytes) @ 0x00002b43e7da9a74 [0x00002b43e7da9a00+0x74]
            j  java.lang.ClassLoader.getResources(Ljava/lang/String;)Ljava/util/Enumeration;+19
            J 6860 C2 org.jboss.seam.deployment.SeamDeploymentProperties.getResources()Ljava/util/Enumeration; (63 bytes) @ 0x00002b43e86551cc [0x00002b43e8654fc0+0x20c]
            J 7037 C2 org.jboss.seam.deployment.SeamDeploymentProperties.addPropertiesFromResourceBundle(Ljava/lang/String;Ljava/util/List;)V (95 bytes) @ 0x00002b43e86fce40 [0x00002b43e86fce00+0x40]
            j  org.jboss.seam.deployment.SeamDeploymentProperties.getPropertyValues(Ljava/lang/String;)Ljava/util/List;+17
            j  org.jboss.seam.deployment.DeploymentStrategy.initDeploymentHandlers()V+26
            j  org.jboss.seam.deployment.DeploymentStrategy.getDeploymentHandlers()Ljava/util/Map;+8
            j  org.jboss.seam.deployment.HotDeploymentStrategy.(Ljava/lang/ClassLoader;Ljava/io/File;Ljavax/servlet/ServletContext;Z)V+53
            j  org.jboss.seam.init.Initialization.createHotDeployment(Ljava/lang/ClassLoader;Z)Lorg/jboss/seam/deployment/HotDeploymentStrategy;+60
            j  org.jboss.seam.init.Initialization.redeploy(Ljavax/servlet/http/HttpServletRequest;Lorg/jboss/seam/core/Init;)V+22
            j  org.jboss.seam.web.HotDeployFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+442
            j  org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+72
            j  org.jboss.seam.web.IdentityFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+428
            j  org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+72
            j  org.jboss.seam.servlet.SeamFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+415
            j  io.undertow.servlet.core.ManagedFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+60
            j  io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+149
            j  io.undertow.servlet.handlers.FilterHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+129
            j  io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+104
            j  io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+19
            j  org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+118
            j  io.undertow.server.handlers.PredicateHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+27
            j  io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+101
            j  io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+40
            j  io.undertow.server.handlers.PredicateHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+27
            j  io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+24
            j  io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+21
            j  io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+99
            j  io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+113
            J 8815 C2 io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V (61 bytes) @ 0x00002b43e89cfae4 [0x00002b43e89cf960+0x184]
            j  io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+51
            j  io.undertow.security.handlers.SecurityInitialHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+33
            j  io.undertow.server.handlers.PredicateHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+27
            j  org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+16
            j  io.undertow.server.handlers.PredicateHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+27
            j  io.undertow.server.handlers.PredicateHandler.handleRequest(Lio/undertow/server/HttpServerExchange;)V+27
            j  io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(Lio/undertow/server/HttpServerExchange;Lio/undertow/servlet/handlers/ServletChain;Lio/undertow/servlet/handlers/ServletRequestContext;Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+33
            j  io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(Lio/undertow/server/HttpServerExchange;Lio/undertow/servlet/handlers/ServletRequestContext;Lio/undertow/servlet/handlers/ServletChain;Ljavax/servlet/DispatcherType;)V+39
            j  io.undertow.servlet.handlers.ServletInitialHandler.access$000(Lio/undertow/servlet/handlers/ServletInitialHandler;Lio/undertow/server/HttpServerExchange;Lio/undertow/servlet/handlers/ServletRequestContext;Lio/undertow/servlet/handlers/ServletChain;Ljavax/servlet/DispatcherType;)V+6
            j  io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(Lio/undertow/server/HttpServerExchange;)V+25
            j  io.undertow.server.Connectors.executeRootHandler(Lio/undertow/server/HttpHandler;Lio/undertow/server/HttpServerExchange;)V+8
            j  io.undertow.server.HttpServerExchange$1.run()V+8
            J 2831 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00002b43e7a5c77c [0x00002b43e7a5c620+0x15c]
            j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
            j  java.lang.Thread.run()V+11
            v  ~StubRoutines::call_stub
            
            • 3. Re: HotDeployment confusion
              ctomc

              do you have debug enabled for seam?

               

              Chapter 2. Getting started with Seam, using seam-gen

              • 4. Re: HotDeployment confusion
                epeters

                As it happens, yes, we do.

                 

                Is this a question of wanting us to check information on the debug page, or is it an expectation that the incremental hot deployments are at fault?

                //EricP

                • 5. Re: HotDeployment confusion
                  jaikiran

                  I'm guessing that Tomaz was hinting that you should be disabling debug mode in Seam.

                  • 6. Re: HotDeployment confusion
                    ctomc

                    Yes, as seam has some redeployment logic in it in case you have debug mode enabled.

                    this is documented in doc i linked.

                     

                    So in short, disable seam debug and it should be fine. In any case it is recommended to have it disabled in production.