5 Replies Latest reply on Jan 8, 2018 1:06 AM by jaikiran

    Error On Jboss AS 5.1

    nandhujoy18

      Hi Friends,

       

      We can see below error on jboss AS 5.1 sever.log once or twice in a day. We dont see any application issue, but still we would like to know what is this and why we are getting this error in our logs,

       

      2018-01-03 09:11:56,383 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/invoker].[default]] (ajp-x.x.x.x-8009-66) Servlet.service() for servlet default threw exception

      org.apache.commons.collections.FunctorException: InvokerTransformer: The method 'exec' on 'class java.lang.Runtime' threw an exception

              at org.apache.commons.collections.functors.InvokerTransformer.transform(InvokerTransformer.java:133)

              at org.apache.commons.collections.functors.ChainedTransformer.transform(ChainedTransformer.java:123)

              at org.apache.commons.collections.map.LazyMap.get(LazyMap.java:158)

              at sun.reflect.annotation.AnnotationInvocationHandler.invoke(AnnotationInvocationHandler.java:50)

              at $Proxy286.entrySet(Unknown Source)

              at sun.reflect.annotation.AnnotationInvocationHandler.readObject(AnnotationInvocationHandler.java:327)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)

              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)

              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)

              at org.jboss.invocation.http.servlet.ReadOnlyAccessFilter.doFilter(ReadOnlyAccessFilter.java:106)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

              at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

              at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)

              at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)

              at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)

              at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

              at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

              at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.internalProcess(ActiveRequestResponseCacheValve.java:74)

              at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:47)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

              at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)

              at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:385)

              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)

              at java.lang.Thread.run(Thread.java:662)

      Caused by: java.lang.reflect.InvocationTargetException

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.commons.collections.functors.InvokerTransformer.transform(InvokerTransformer.java:126)

              ... 38 more

      Caused by: java.io.IOException: Cannot run program "powershell.exe": java.io.IOException: error=2, No such file or directory

              at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

              at java.lang.Runtime.exec(Runtime.java:593)

              at java.lang.Runtime.exec(Runtime.java:431)

              at java.lang.Runtime.exec(Runtime.java:328)

              ... 43 more

      Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)

              at java.lang.ProcessImpl.start(ProcessImpl.java:65)

              at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)

              ... 46 more

       

      We checked with application team whether they execute any program  from application level. They are sure that they dont execute any program in application

       

      2018-01-02 03:26:37,825 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/invoker].[default]] (ajp-x.x.x.x@-8009-79) Servlet.service() for servlet default threw exception

      java.lang.ClassCastException: sun.reflect.annotation.AnnotationInvocationHandler cannot be cast to org.jboss.invocation.MarshalledInvocation

              at org.jboss.invocation.http.servlet.ReadOnlyAccessFilter.doFilter(ReadOnlyAccessFilter.java:106)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

              at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

              at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)

              at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)

              at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)

              at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

              at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

              at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.internalProcess(ActiveRequestResponseCacheValve.java:74)

              at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:47)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

              at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)

              at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:385)

              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)

              at java.lang.Thread.run(Thread.java:662)

      2018-01-02 03:26:38,759 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/invoker].[default]] (ajp-x.x.x.x-8009-15) Servlet.service() for servlet default threw exception

      org.apache.commons.collections.FunctorException: InvokerTransformer: The method 'exec' on 'class java.lang.Runtime' threw an exception

              at org.apache.commons.collections.functors.InvokerTransformer.transform(InvokerTransformer.java:133)

              at org.apache.commons.collections.functors.ChainedTransformer.transform(ChainedTransformer.java:123)

              at org.apache.commons.collections.map.TransformedMap.checkSetValue(TransformedMap.java:204)

              at org.apache.commons.collections.map.AbstractInputCheckedMapDecorator$MapEntry.setValue(AbstractInputCheckedMapDecorator.java:192)

              at sun.reflect.annotation.AnnotationInvocationHandler.readObject(AnnotationInvocationHandler.java:334)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)

              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)

              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)

              at org.jboss.invocation.http.servlet.ReadOnlyAccessFilter.doFilter(ReadOnlyAccessFilter.java:106)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

              at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

              at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)

              at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)

              at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)

              at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

              at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

              at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.internalProcess(ActiveRequestResponseCacheValve.java:74)

              at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:47)

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

              at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)

              at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:385)

              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)

              at java.lang.Thread.run(Thread.java:662)

      Caused by: java.lang.reflect.InvocationTargetException

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.commons.collections.functors.InvokerTransformer.transform(InvokerTransformer.java:126)

              ... 37 more

      Caused by: java.io.IOException: Cannot run program "cmd.exe": java.io.IOException: error=2, No such file or directory

              at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

              at java.lang.Runtime.exec(Runtime.java:593)

              at java.lang.Runtime.exec(Runtime.java:466)

              ... 42 more

      Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)

              at java.lang.ProcessImpl.start(ProcessImpl.java:65)

              at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)

              ... 44 more

       

      Message was edited by: Nantha Kumar

        • 1. Re: Error On Jboss AS 5.1
          andey

          Ensure the machine has a JDK with proper path. Check your system environment variables, whether powershell.exe path is correct or not?

          If you tried to use the plugin or to execute an external file try to re-check the file path

          • 2. Re: Error On Jboss AS 5.1
            jaikiran

            nandhujoy18  wrote:

             

            Hi Friends,

             

            We can see below error on jboss AS 5.1 sever.log once or twice in a day. We dont see any application issue, but still we would like to know what is this and why we are getting this error in our logs,

             

             

            Caused by: java.io.IOException: Cannot run program "powershell.exe": java.io.IOException: error=2, No such file or directory

                    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)

                    at java.lang.Runtime.exec(Runtime.java:593)

                    at java.lang.Runtime.exec(Runtime.java:431)

                    at java.lang.Runtime.exec(Runtime.java:328)

                    ... 43 more

            Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory

                    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)

                    at java.lang.ProcessImpl.start(ProcessImpl.java:65)

                    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)

                    ... 46 more

            ...

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)

                    at org.jboss.invocation.http.servlet.ReadOnlyAccessFilter.doFilter(ReadOnlyAccessFilter.java:106)

                  

             

            This appears to be an attack/attempt to hack your server and is related to this CVE CVE-2017-12149 - Red Hat Customer Portal . I highly recommend you either upgrade your server to a version which isn't affected by this or do necessary steps to isolate and quarantine the server until you can patch this issue either yourself, or by using JBoss EAP with support. Read through that CVE and you'll even find the steps to mitigate this.

            • 3. Re: Error On Jboss AS 5.1
              nandhujoy18

              thanks for your reply. Anywhere CVE-2015-7501  and CVE-2017-12149 is related to each other ? Can we remove commons-collections.jar from our war files to avoid this exploit ?

              • 4. Re: Error On Jboss AS 5.1
                nandhujoy18

                To add more information, this application is running in Linux server. cmd.exe and poweshell.exe are not been invoked from the web applications which is running on the server. Even in jboss eap layer we havent configured any such properties. Still wonder where it is  coming from ??

                • 5. Re: Error On Jboss AS 5.1
                  jaikiran

                  As noted in my previous reply, this isn't something that your application is doing. Instead, this appears to be an attempt (through a well crafted HTTP request) to hack your server due to a security bug in one of the default applications that is shipped in this version of JBoss AS. The linked CVE, in my previous reply, has more details on what it is and how to mitigate it (one of which is removing that default application, http-invoker.sar, from your installation if you aren't using it).