1 2 Previous Next 24 Replies Latest reply on Feb 17, 2015 5:02 AM by Stian Lund

    Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?

    Stian Lund Expert

      Hi,

      since migrating to 4.13.1 and the new plugin for AS7, servers seem to pop up and down a lot, and being marked as Unavailable even though they are running fine.

       

      The following error is logged in the Agent logs:

       

      2015-02-12 11:46:44,913 WARN  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- An error occurred while checking for an updated Resource configuration for Resource[id=12530, uuid=94c5d7e5-b0e7-4642-bbc8-59f0a078c80e, type={JBossAS7}JMX, key=subsystem=jmx, name=jmx, parent=Xmlstilling (9943)].
      org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
          at org.rhq.modules.plugins.jbossas7.ASConnection.executeRaw(ASConnection.java:304)
          at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:512)
          at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:453)
          at org.rhq.modules.plugins.jbossas7.BaseComponent.loadResourceConfiguration(BaseComponent.java:374)
          at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:759)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:662)
      
      
      

       

      After that lots of similar WARNings are logged:

      2015-02-12 11:52:17,071 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13764, uuid=79296408-c6aa-40b9-abf1-5cb8ff6c8081, type={JBossAS7}Entity, key=deployment=nav-sbl-xmlstilling-ear.ear,subdeployment=nav-sbl-xmlstilling-web.war,subsystem=jpa,hibernate-persistence-unit=nav-sbl-xmlstilling-ear.ear/nav-sbl-xmlstilling-web.war#xmlstilling-web-persistence,entity=no.nav.sbl.entity.six.StillingLoggEntity, name=no.nav.sbl.entity.six.StillingLoggEntity, parent=nav-sbl-xmlstilling-ear.ear/nav-sbl-xmlstilling-web.war#xmlstilling-web-persistence], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      2015-02-12 11:52:40,332 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13170, uuid=20d17c2e-fbea-4489-b101-1d05242dd179, type={JBossAS7}Connection Definitions, key=subsystem=resource-adapters,resource-adapter=wmq.jmsra-7.0.1.7.rar,connection-definitions=QueueConnection, name=QueueConnection, parent=wmq.jmsra-7.0.1.7.rar], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      2015-02-12 11:59:35,545 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13064, uuid=247e7840-08a0-44ee-ab14-01689a3da1a6, type={JBossAS7}Strict Max Bean Instance Pool, key=subsystem=ejb3,strict-max-bean-instance-pool=slsb-strict-max-pool, name=slsb-strict-max-pool, parent=ejb3], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      2015-02-12 11:59:36,022 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13066, uuid=71d130c5-864a-460e-acbe-7feb7fecd48b, type={JBossAS7}SFSB Cache, key=subsystem=ejb3,cache=passivating, name=passivating, parent=ejb3], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      2015-02-12 11:59:36,490 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13069, uuid=aa3aef83-76a9-4b75-819a-4f8d386a9b26, type={JBossAS7}IIOP Service, key=subsystem=ejb3,service=iiop, name=iiop, parent=ejb3], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      2015-02-12 11:59:36,974 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=12555, uuid=0599c7ac-67f5-4f32-a144-9488c3ebf8fd, type={JBossAS7}JMX, key=subsystem=jmx, name=jmx, parent=Arbeid (9444)], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      2015-02-12 11:59:37,215 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=12556, uuid=08ef9fac-305e-4e4e-980a-7ff3c250d0fb, type={JBossAS7}Resource Adapters, key=subsystem=resource-adapters, name=resource-adapters, parent=Arbeid (9444)], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      2015-02-12 11:59:38,385 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=12597, uuid=6e8ad507-0253-4d44-97f1-1fe46e6ca6c2, type={JBossAS7}Operating System, key=core-service=platform-mbean,type=operating-system, name=operating-system, parent=platform-mbean], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      2015-02-12 11:59:38,862 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=12599, uuid=161bb996-f760-4723-a8c4-e82e79eb2f2f, type={JBossAS7}Runtime, key=core-service=platform-mbean,type=runtime, name=runtime, parent=platform-mbean], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
      
      

       

      I can't say I noticed the same with 4.13.0 but then again I've had problems with the AS7 plugin failing to upgrade properly in our environments (Differences in resource hierarchy for AS7 servers in RHQ 4.13?) so it might have been there a long time, I've just not actually ran the latest plugin.

       

      These solutions for JON seem to be related:

      https://access.redhat.com/solutions/305633

      https://access.redhat.com/solutions/269473

       

      But they don't really solve anything for me, as I know they mangement credentials are valid, and the AS7 server comes back as UP after a small interval (10-15 minutes).

      Obviously this would lead to a lot of false positives if alerts are created to check for As7 availability.

       

      Stian

        • 1. Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
          Thomas Segismont Expert

          Hi Stian,

           

          Could you re-configure one of the agents involved so that we get more context? You'd need to:

          * start the agent with the as7plugin.verbose=true system property

          * set log level to DEBUG for the org.rhq.modules.plugins.jbossas7 and org.apache.http categories

           

          Beware that such settings will produce a lot of logs.

           

          Also, if I understand correctly, availability checking works fine until the ConfigurationManager runs?

           

          Thanks,

          Thomas

          • 2. Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
            Stian Lund Expert

            Thanks for the reply Thomas.

             

            Hope I understand correctly; I set

            RHQ_AGENT_DEBUG=true

            RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Das7plugin.verbose=true"

             

            In bin/rhq-agent-env.sh, and

             

               <category name="org.apache.http">

                 <priority value="DEBUG"/>

               </category>

             

               <category name="org.rhq.modules.plugins.jbossas7">

                 <priority value="DEBUG"/>

               </category>

             

            in conf/log4j.xml ?

             

            > Also, if I understand correctly, availability checking works fine until the ConfigurationManager runs?

             

            Yes, well, it's intermittent. So resources in AS7 plugin goes up and down a lot.

             

            I also receive a lot of Error messages in RHQ UI saying "resource with id=x is unavailable" when I go to resources of that type, and then it goes away.

             

            Especially when I arrive in the morning, I check and several AS7 are reported as Down, and then after a while they are Ok. Almost as if it's related to my logging into the UI...

             

            Edit: **** that's a lot of logging. I will have to let it run overnight, so I guess increase log files to 100x10MB or something... I have about 5GB free space on this server. Or, I guess not set RHQ_AGENT_DEBUG, you didn't actually say I should

            • 3. Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
              Stian Lund Expert

              Already getting a lot of these:

               

              2015-02-12 14:36:18,247 DEBUG [ResourceContainer.invoker.daemon-1] (apache.http.impl.conn.DefaultClientConnection)- Sending request: POST /management HTTP/1.1

              2015-02-12 14:36:18,248 DEBUG [ResourceContainer.invoker.daemon-1] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized

               

              Which is strange, because the servers are UP on this host.

               

              This host is running 8 AS7 instances and I will keep watching it.

               

              Attaching some logs (edit:no)

              • 4. Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                Thomas Segismont Expert

                Do not set the agent in debug mode (well, not yet).

                 

                As for the Receiving response: HTTP/1.1 401 Unauthorized messages, they are not necessarily problematic. The as7plugin does not do preemptive authentication. So when a new connection is established, an unauthenticated request is sent to the server, then the server replies with the 401 status and announces the authentication challenge. Then the client re-sends the request with the correct authorization headers.

                • 5. Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                  Stian Lund Expert

                  Yeah, that makes sense Thomas - that's how the protocol should work I guess.

                   

                  I've tried finding instances of "InvalidPluginConfigurationException" in the debug logs. Here are a couple of examples.

                   

                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized
                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS
                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication required
                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9744 requested authentication
                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed
                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication failed
                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 shut down
                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 closed
                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 1205][route: {s}->https://0.0.0.0:9744][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 10]
                  2015-02-12 15:31:18,305 WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=13234, uuid=8ee12c61-2b3c-4fb4-91f7-c5e44ef927b2, type={JBossAS7}Stateless Session Bean Runtime, key=deployment=nav-sbl-stilling-j2ee.ear,subdeployment=nav-sbl-stilling-ejb.jar,subsystem=ejb3,stateless-session-bean=SequenceFacade, name=SequenceFacade, parent=ejb3] - cause: org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException:Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
                  
                  

                   

                  2015-02-12 15:21:00,360 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 200 OK
                  2015-02-12 15:21:00,360 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS
                  2015-02-12 15:21:00,360 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.client.DefaultHttpClient)- Authentication succeeded
                  2015-02-12 15:21:00,360 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.client.TargetAuthenticationStrategy)- Caching 'basic' auth scheme for https://0.0.0.0:9444
                  2015-02-12 15:21:00,399 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection [id: 959][route: {s}->https://0.0.0.0:9444] can be kept alive for 5000 MILLISECONDS
                  2015-02-12 15:21:00,399 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 959][route: {s}->https://0.0.0.0:9444][total kept alive: 1; route allocated: 2 of 10; total allocated: 2 of 10]
                  2015-02-12 15:21:00,400 INFO  [ResourceContainer.invoker.daemon-21] (rhq.modules.plugins.jbossas7.ASConnection)- {
                    "outcome" : "success",
                    "result" : 0
                  }
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- Authentication required
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9444 requested authentication
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- Authentication failed
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:39871<->10.49.67.52:9444 shut down
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:39871<->10.49.67.52:9444 closed
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 955][route: {s}->https://0.0.0.0:9444][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 10]
                  2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (rhq.modules.plugins.jbossas7.BaseServerComponent)- ResourceType[id=0, name=JBossAS7 Standalone Server, plugin=JBossAS7, category=Server] [hostConfig: /opt/jboss/server/arbeid/configuration/standalone.xml]: exception while checking availability
                  org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
                      at org.rhq.modules.plugins.jbossas7.ASConnection.executeRaw(ASConnection.java:304)
                      at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:512)
                      at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:465)
                      at org.rhq.modules.plugins.jbossas7.BaseComponent.readAttribute(BaseComponent.java:869)
                      at org.rhq.modules.plugins.jbossas7.BaseComponent.readAttribute(BaseComponent.java:860)
                      at org.rhq.modules.plugins.jbossas7.BaseServerComponent.getAvailability(BaseServerComponent.java:133)
                      at org.rhq.core.pc.inventory.AvailabilityProxy.call(AvailabilityProxy.java:143)
                      at org.rhq.core.pc.inventory.AvailabilityProxy.call(AvailabilityProxy.java:52)
                      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                      at java.lang.Thread.run(Thread.java:662)
                  
                  

                   

                  To give an example of how this looks in monitoring, every resource has an availability graph that looks like this:

                   

                   

                  The full logs contain a lot of sensitive data (something I found out after the fact) so I can't send the full logs but tell me what I should look for and I will.

                   

                  Stian

                  • 6. Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                    Thomas Segismont Expert

                    Le 13/02/2015 09:25, Stian Lund a écrit :

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authentication required

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9744 requested authentication

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authentication failed

                     

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 shut down

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 closed

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: id: 1205[route: ->https://0.0.0.0:9744][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 10] > 2015-02-12 15:31:18,305 WARN  (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=13234, uuid=8ee12c61-2b3c-4fb4-91f7-c5e44ef927b2, type=Stateless Session Bean Runtime, key=deployment=nav-sbl-stilling-j2ee.ear,subdeployment=nav-sbl-stilling-ejb.jar,subsystem=ejb3,stateless-session-bean=SequenceFacade, name=SequenceFacade, parent=ejb3] - cause: org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException:Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials. > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authentication required > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9444 requested authentication > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authentication failed > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:39871<->10.49.67.52:9444 shut down > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:39871<->10.49.67.52:9444 closed > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: id: 955[route: ->https://0.0.0.0:9444][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 10]

                    2015-02-12 15:21:00,400 DEBUG (rhq.modules.plugins.jbossas7.BaseServerComponent)- ResourceType[id=0, name=JBossAS7 Standalone Server, plugin=JBossAS7, category=Server] hostConfig: /opt/jboss/server/arbeid/configuration/standalone.xml: exception while checking availability

                    org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.

                     

                     

                    These two samples are interesting enough.

                     

                    So the as7plugin is trying to start a new connection and, for some

                    reason, authentication challenge processing fails. Can you attach what's

                    going through the wire at that time?

                     

                    How are the AS7 servers configured with respect to authentication?

                    Simple properties file? LDAP backend?

                     

                    I noticed you're using SSL protected management connections. Is this a

                    test environment? In this case, could you try to use raw http on one host?

                    • 7. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                      Stian Lund Expert

                      > Can you attach what's going through the wire at that time?

                       

                      Sure, do you mean by using TCPdump? Is there another way apart from the logs? There's no more output from the specific threads.

                       

                      > How are the AS7 servers configured with respect to authentication?

                       

                      They are connected to Active Directory  using ldaps:// and a filter that ensures our service-account is authenticated.

                       

                      The SSL is configured:

                      <ssl>
                              <keystore path="etc/keystore.jks" relative-to="jboss.home.dir" keystore-password=""/>
                      </ssl>

                       

                      Interfaces are just the defaults, we have management-http and management-https configured.

                       

                      Initally we are just using the management-http which is fine for us, but since RHQ started detecting mangement-https and using it then it's too much hassle to change it to just http for all the servers.

                       

                      I could try changing the AS7 connection to go through management-http instead but since the problem is intermittent it would be hard to catch raw data.

                      Maybe to see if problem goes away using management-http?

                       

                      Edit; I am tracing port 9990 using tcpdump but really not sure what to look for...

                       

                      I am getting a lot of 500 Internal Server Errors though.

                       

                      POST /management HTTP/1.1
                      Accept: application/json
                      Content-Length: 210
                      Content-Type: application/json; charset=UTF-8
                      Host: 0.0.0.0:9990
                      Connection: Keep-Alive
                      User-Agent: Apache-HttpClient/4.2.5 (java 1.5)
                      
                      {"operation":"read-attribute","address":[{"deployment":"nav-sbl-arbeid-j2ee-4.2.14.2.ear"},{"subdeployment":"nav-sbl-arbeid-ejb.jar"},{"subsystem":"ejb3"},{"entity-bean":"BrukerHendelseLogg"}],"name":"methods"}
                      11:25:41.233719 IP 10.49.67.52.9990 > 10.49.67.52.51180: Flags [P.], seq 1:136, ack 425, win 1023, options [nop,nop,TS val 705038964 ecr 705038963], length 135
                      E.....@.@...
                      1C4
                      1C4'...;.v=p..'.....w.....
                      *.
                      t*.
                      sHTTP/1.1 500 Internal Server Error
                      Transfer-encoding: chunked
                      Content-type: application/json
                      Date: Fri, 13 Feb 2015 10:25:41 GMT
                      
                      

                       

                      Stian

                      • 8. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                        Thomas Segismont Expert

                        Le 13/02/2015 11:11, Stian Lund a écrit :

                        > Can you attach what's going through the wire at that time?

                         

                        Sure, do you mean by using TCPdump? Is there another way apart from the

                        logs? There's no more output from the specific threads.

                         

                        Do httpclient wire logs show nothing? Strange.

                         

                         

                        > How are the AS7 servers configured with respect to authentication?

                         

                        They are connected to Active Directory  using ldaps:// and a filter that

                        ensures our service-account is authenticated.

                         

                        The SSL is configured:

                         

                        <ssl>

                                 <keystore path="etc/keystore.jks" relative-to="jboss.home.dir"

                        keystore-password=""/>

                        </ssl>

                         

                        Interfaces are just the defaults, we have management-http and

                        management-https configured.

                         

                        Initally we are just using the management-http which is fine for us, but

                        since RHQ started detecting mangement-https and using it then it's too

                        much hassle to change it to just http for all the servers.

                         

                        CLI script to the rescue?

                         

                         

                        I could try changing the AS7 connection to go through management-http

                        instead but since the problem is intermittent it would be hard to catch

                        raw data.

                         

                        Maybe to see if problem goes away using management-http?

                         

                         

                        Yest that's the purpose. Configure just one of the AS7 resources to be

                        monitored with management-http and see if the intermittent failures go away.

                        • 9. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                          Thomas Segismont Expert

                          Le 13/02/2015 11:11, Stian Lund a écrit :

                          > How are the AS7 servers configured with respect to authentication?

                           

                          They are connected to Active Directory  using ldaps:// and a filter that

                          ensures our service-account is authenticated.

                           

                          The SSL is configured:

                           

                          <ssl>

                                   <keystore path="etc/keystore.jks" relative-to="jboss.home.dir"

                          keystore-password=""/>

                          </ssl>

                           

                           

                          Are there any errors in the AS7 server logs?

                           

                          Can you try on another server to keep SSL but switch to properties file

                          backend?

                          • 10. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                            Stian Lund Expert

                            Could the 500 internal errors be a clue?

                             

                            Not seeing anything in AS7 server.log apart from the many exceptions our applications throw... but from experience I know the AS7 management interface logs almost nothing unless you increase loglevel for security.

                             

                            I could change to properties-based for one of the servers, and keep SSL on.

                            • 11. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                              Thomas Segismont Expert

                               

                              1. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized 
                              2. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS 
                              3. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication required 
                              4. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9744 requested authentication 
                              5. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed 
                              6. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication failed 
                              7. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 shut down 
                              8. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 closed 
                              9. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 1205][route: {s}->https://0.0.0.0:9744][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 10] 
                              10. 2015-02-12 15:31:18,305 WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=13234, uuid=8ee12c61-2b3c-4fb4-91f7-c5e44ef927b2, type={JBossAS7}Stateless Session Bean Runtime, key=deployment=nav-sbl-stilling-j2ee.ear,subdeployment=nav-sbl-stilling-ejb.jar,subsystem=ejb3,stateless-session-bean=SequenceFacade, name=SequenceFacade, parent=ejb3] - cause: org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException:Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials. 

                               

                              Is there any message logged by ResourceContainer.invoker.daemon-27 just before 2015-02-12 15:31:18,305 ?

                              • 12. Re: Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                                Stian Lund Expert

                                Here's what I find for that thread just before that time:

                                 

                                2015-02-12 15:31:08,465 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection [id: 1205][route: {s}->https://0.0.0.0:9744] can be kept alive for 5000 MILLISECONDS
                                2015-02-12 15:31:08,465 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 1205][route: {s}->https://0.0.0.0:9744][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 10]
                                2015-02-12 15:31:08,466 INFO  [ResourceContainer.invoker.daemon-27] (rhq.modules.plugins.jbossas7.ASConnection)- {
                                2015-02-12 15:31:08,466 INFO  [ResourceContainer.invoker.daemon-27] (rhq.modules.plugins.jbossas7.ASConnection)- JSON to send: {"operation":"read-attribute","address":[{"deployment":"nav-sbl-stilling-j2ee-3.0.27.ear"},{"subdeployment":"nav-sbl-stilling-ejb.jar"},{"subsystem":"ejb3"},{"stateless-session-bean":"SequenceFacade"}],"name":"methods"}
                                2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection request: [route: {s}->https://0.0.0.0:9744][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 10]
                                2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection leased: [id: 1205][route: {s}->https://0.0.0.0:9744][total kept alive: 0; route allocated: 1 of 10; total allocated: 1 of 10]
                                2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestAddCookies)- CookieSpec selected: best-match
                                2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestAuthCache)- Auth cache not set in the context
                                2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestTargetAuthentication)- Target auth state: UNCHALLENGED
                                2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestProxyAuthentication)- Proxy auth state: UNCHALLENGED
                                2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Attempt 1 to execute request
                                2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Sending request: POST /management HTTP/1.1
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication required
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9744 requested authentication
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Authentication schemes in the order of preference: [negotiate, Kerberos, NTLM, Digest, Basic]
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Challenge for negotiate authentication scheme not available
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Challenge for Kerberos authentication scheme not available
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Challenge for NTLM authentication scheme not available
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Challenge for Digest authentication scheme not available
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Selected authentication options: [BASIC]
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestAddCookies)- CookieSpec selected: best-match
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestAuthCache)- Auth cache not set in the context
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestTargetAuthentication)- Target auth state: CHALLENGED
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestTargetAuthentication)- Generating response to an authentication challenge using basic scheme
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestProxyAuthentication)- Proxy auth state: UNCHALLENGED
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Attempt 2 to execute request
                                2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Sending request: POST /management HTTP/1.1
                                2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized
                                
                                
                                

                                 

                                Attaching some logs where I've parsed out the more sensitive stuff.

                                • 13. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                                  Thomas Segismont Expert

                                  Thanks. So the logs show that:

                                  1. the as7plugin sends an unauthenticated request
                                  2. gets a 401 response
                                  3. replies to the challenge
                                  4. gets another 401 response

                                  That is the problem. If you look at the other http exchanges in your logs, at step 4 as7plugin gets a 200 response.

                                   

                                  I really need to see what went on the wire at this time to determine if there's a bug on the client side. No progess with the org.apache.http.wire logging?

                                  See https://hc.apache.org/httpcomponents-client-ga/logging.html

                                  • 14. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                                    Stian Lund Expert

                                    Oh sorry Thomas, when you said "wire" I thought you meant the *ethernet* interface aka the wire - and not the class org.apache.http.wire - sometimes I'm just daft...

                                     

                                    I had already set org.apache.http to debug though, do I need to add org.apache.http.* or just org.apache.http.wire will be enough?


                                    It's a bit late now but come monday I will turn on DEBUG on class org.apache.http.wire in log4j.xml

                                     

                                    Have a good weekend!

                                    Stian

                                    1 2 Previous Next