1 2 Previous Next 24 Replies Latest reply on Feb 17, 2015 5:02 AM by pathduck

    Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?

    pathduck
    pathduck Feb 12, 2015 6:11 AM

      Hi,

      since migrating to 4.13.1 and the new plugin for AS7, servers seem to pop up and down a lot, and being marked as Unavailable even though they are running fine.

       

      The following error is logged in the Agent logs:

       

      2015-02-12 11:46:44,913 WARN  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- An error occurred while checking for an updated Resource configuration for Resource[id=12530, uuid=94c5d7e5-b0e7-4642-bbc8-59f0a078c80e, type={JBossAS7}JMX, key=subsystem=jmx, name=jmx, parent=Xmlstilling (9943)].
org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
    at org.rhq.modules.plugins.jbossas7.ASConnection.executeRaw(ASConnection.java:304)
    at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:512)
    at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:453)
    at org.rhq.modules.plugins.jbossas7.BaseComponent.loadResourceConfiguration(BaseComponent.java:374)
    at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:759)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

       

      After that lots of similar WARNings are logged:

      2015-02-12 11:52:17,071 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13764, uuid=79296408-c6aa-40b9-abf1-5cb8ff6c8081, type={JBossAS7}Entity, key=deployment=nav-sbl-xmlstilling-ear.ear,subdeployment=nav-sbl-xmlstilling-web.war,subsystem=jpa,hibernate-persistence-unit=nav-sbl-xmlstilling-ear.ear/nav-sbl-xmlstilling-web.war#xmlstilling-web-persistence,entity=no.nav.sbl.entity.six.StillingLoggEntity, name=no.nav.sbl.entity.six.StillingLoggEntity, parent=nav-sbl-xmlstilling-ear.ear/nav-sbl-xmlstilling-web.war#xmlstilling-web-persistence], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
2015-02-12 11:52:40,332 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13170, uuid=20d17c2e-fbea-4489-b101-1d05242dd179, type={JBossAS7}Connection Definitions, key=subsystem=resource-adapters,resource-adapter=wmq.jmsra-7.0.1.7.rar,connection-definitions=QueueConnection, name=QueueConnection, parent=wmq.jmsra-7.0.1.7.rar], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
2015-02-12 11:59:35,545 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13064, uuid=247e7840-08a0-44ee-ab14-01689a3da1a6, type={JBossAS7}Strict Max Bean Instance Pool, key=subsystem=ejb3,strict-max-bean-instance-pool=slsb-strict-max-pool, name=slsb-strict-max-pool, parent=ejb3], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
2015-02-12 11:59:36,022 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13066, uuid=71d130c5-864a-460e-acbe-7feb7fecd48b, type={JBossAS7}SFSB Cache, key=subsystem=ejb3,cache=passivating, name=passivating, parent=ejb3], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
2015-02-12 11:59:36,490 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=13069, uuid=aa3aef83-76a9-4b75-819a-4f8d386a9b26, type={JBossAS7}IIOP Service, key=subsystem=ejb3,service=iiop, name=iiop, parent=ejb3], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
2015-02-12 11:59:36,974 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=12555, uuid=0599c7ac-67f5-4f32-a144-9488c3ebf8fd, type={JBossAS7}JMX, key=subsystem=jmx, name=jmx, parent=Arbeid (9444)], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
2015-02-12 11:59:37,215 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=12556, uuid=08ef9fac-305e-4e4e-980a-7ff3c250d0fb, type={JBossAS7}Resource Adapters, key=subsystem=resource-adapters, name=resource-adapters, parent=Arbeid (9444)], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
2015-02-12 11:59:38,385 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=12597, uuid=6e8ad507-0253-4d44-97f1-1fe46e6ca6c2, type={JBossAS7}Operating System, key=core-service=platform-mbean,type=operating-system, name=operating-system, parent=platform-mbean], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
2015-02-12 11:59:38,862 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=12599, uuid=161bb996-f760-4723-a8c4-e82e79eb2f2f, type={JBossAS7}Runtime, key=core-service=platform-mbean,type=runtime, name=runtime, parent=platform-mbean], availability will be reported as DOWN, reason=Availability check failed : Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.

       

      I can't say I noticed the same with 4.13.0 but then again I've had problems with the AS7 plugin failing to upgrade properly in our environments (Differences in resource hierarchy for AS7 servers in RHQ 4.13?) so it might have been there a long time, I've just not actually ran the latest plugin.

       

      These solutions for JON seem to be related:

      https://access.redhat.com/solutions/305633

      https://access.redhat.com/solutions/269473

       

      But they don't really solve anything for me, as I know they mangement credentials are valid, and the AS7 server comes back as UP after a small interval (10-15 minutes).

      Obviously this would lead to a lot of false positives if alerts are created to check for As7 availability.

       

      Stian

      • 1501 Views
      • Tags:
        • 1. Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
          tsegismont
          tsegismont Feb 12, 2015 8:12 AM (in response to pathduck)

          Hi Stian,

           

          Could you re-configure one of the agents involved so that we get more context? You'd need to:

          * start the agent with the as7plugin.verbose=true system property

          * set log level to DEBUG for the org.rhq.modules.plugins.jbossas7 and org.apache.http categories

           

          Beware that such settings will produce a lot of logs.

           

          Also, if I understand correctly, availability checking works fine until the ConfigurationManager runs?

           

          Thanks,

          Thomas

            • Actions
          • 2. Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
            pathduck
            pathduck Feb 12, 2015 9:14 AM (in response to tsegismont)

            Thanks for the reply Thomas.

             

            Hope I understand correctly; I set

            RHQ_AGENT_DEBUG=true

            RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Das7plugin.verbose=true"

             

            In bin/rhq-agent-env.sh, and

             

               <category name="org.apache.http">

                 <priority value="DEBUG"/>

               </category>

             

               <category name="org.rhq.modules.plugins.jbossas7">

                 <priority value="DEBUG"/>

               </category>

             

            in conf/log4j.xml ?

             

            > Also, if I understand correctly, availability checking works fine until the ConfigurationManager runs?

             

            Yes, well, it's intermittent. So resources in AS7 plugin goes up and down a lot.

             

            I also receive a lot of Error messages in RHQ UI saying "resource with id=x is unavailable" when I go to resources of that type, and then it goes away.

             

            Especially when I arrive in the morning, I check and several AS7 are reported as Down, and then after a while they are Ok. Almost as if it's related to my logging into the UI...

             

            Edit: **** that's a lot of logging. I will have to let it run overnight, so I guess increase log files to 100x10MB or something... I have about 5GB free space on this server. Or, I guess not set RHQ_AGENT_DEBUG, you didn't actually say I should

              • Actions
            • 3. Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
              pathduck
              pathduck Feb 12, 2015 9:48 AM (in response to pathduck)

              Already getting a lot of these:

               

              2015-02-12 14:36:18,247 DEBUG [ResourceContainer.invoker.daemon-1] (apache.http.impl.conn.DefaultClientConnection)- Sending request: POST /management HTTP/1.1

              2015-02-12 14:36:18,248 DEBUG [ResourceContainer.invoker.daemon-1] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized

               

              Which is strange, because the servers are UP on this host.

               

              This host is running 8 AS7 instances and I will keep watching it.

               

              Attaching some logs (edit:no)

                • Actions
              • 4. Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                tsegismont
                tsegismont Feb 12, 2015 10:05 AM (in response to pathduck)

                Do not set the agent in debug mode (well, not yet).

                 

                As for the Receiving response: HTTP/1.1 401 Unauthorized messages, they are not necessarily problematic. The as7plugin does not do preemptive authentication. So when a new connection is established, an unauthenticated request is sent to the server, then the server replies with the 401 status and announces the authentication challenge. Then the client re-sends the request with the correct authorization headers.

                  • Actions
                • 5. Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                  pathduck
                  pathduck Feb 13, 2015 3:24 AM (in response to tsegismont)

                  Yeah, that makes sense Thomas - that's how the protocol should work I guess.

                   

                  I've tried finding instances of "InvalidPluginConfigurationException" in the debug logs. Here are a couple of examples.

                   

                  2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication required
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9744 requested authentication
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication failed
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 shut down
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 closed
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 1205][route: {s}->https://0.0.0.0:9744][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 10]
2015-02-12 15:31:18,305 WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=13234, uuid=8ee12c61-2b3c-4fb4-91f7-c5e44ef927b2, type={JBossAS7}Stateless Session Bean Runtime, key=deployment=nav-sbl-stilling-j2ee.ear,subdeployment=nav-sbl-stilling-ejb.jar,subsystem=ejb3,stateless-session-bean=SequenceFacade, name=SequenceFacade, parent=ejb3] - cause: org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException:Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.

                   

                  2015-02-12 15:21:00,360 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 200 OK
2015-02-12 15:21:00,360 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS
2015-02-12 15:21:00,360 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.client.DefaultHttpClient)- Authentication succeeded
2015-02-12 15:21:00,360 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.client.TargetAuthenticationStrategy)- Caching 'basic' auth scheme for https://0.0.0.0:9444
2015-02-12 15:21:00,399 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection [id: 959][route: {s}->https://0.0.0.0:9444] can be kept alive for 5000 MILLISECONDS
2015-02-12 15:21:00,399 DEBUG [ResourceContainer.invoker.daemon-21] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 959][route: {s}->https://0.0.0.0:9444][total kept alive: 1; route allocated: 2 of 10; total allocated: 2 of 10]
2015-02-12 15:21:00,400 INFO  [ResourceContainer.invoker.daemon-21] (rhq.modules.plugins.jbossas7.ASConnection)- {
  "outcome" : "success",
  "result" : 0
}
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- Authentication required
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9444 requested authentication
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.client.DefaultHttpClient)- Authentication failed
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:39871<->10.49.67.52:9444 shut down
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:39871<->10.49.67.52:9444 closed
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 955][route: {s}->https://0.0.0.0:9444][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 10]
2015-02-12 15:21:00,400 DEBUG [ResourceContainer.invoker.availCheck.daemon-46] (rhq.modules.plugins.jbossas7.BaseServerComponent)- ResourceType[id=0, name=JBossAS7 Standalone Server, plugin=JBossAS7, category=Server] [hostConfig: /opt/jboss/server/arbeid/configuration/standalone.xml]: exception while checking availability
org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
    at org.rhq.modules.plugins.jbossas7.ASConnection.executeRaw(ASConnection.java:304)
    at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:512)
    at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:465)
    at org.rhq.modules.plugins.jbossas7.BaseComponent.readAttribute(BaseComponent.java:869)
    at org.rhq.modules.plugins.jbossas7.BaseComponent.readAttribute(BaseComponent.java:860)
    at org.rhq.modules.plugins.jbossas7.BaseServerComponent.getAvailability(BaseServerComponent.java:133)
    at org.rhq.core.pc.inventory.AvailabilityProxy.call(AvailabilityProxy.java:143)
    at org.rhq.core.pc.inventory.AvailabilityProxy.call(AvailabilityProxy.java:52)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

                   

                  To give an example of how this looks in monitoring, every resource has an availability graph that looks like this:

                   

                   

                  The full logs contain a lot of sensitive data (something I found out after the fact) so I can't send the full logs but tell me what I should look for and I will.

                   

                  Stian

                    • Actions
                  • 6. Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                    tsegismont
                    tsegismont Feb 13, 2015 4:40 AM (in response to pathduck)

                    Le 13/02/2015 09:25, Stian Lund a écrit :

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authentication required

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9744 requested authentication

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authentication failed

                     

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 shut down

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 closed

                    2015-02-12 15:31:18,305 DEBUG (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: id: 1205[route: ->https://0.0.0.0:9744][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 10] > 2015-02-12 15:31:18,305 WARN  (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=13234, uuid=8ee12c61-2b3c-4fb4-91f7-c5e44ef927b2, type=Stateless Session Bean Runtime, key=deployment=nav-sbl-stilling-j2ee.ear,subdeployment=nav-sbl-stilling-ejb.jar,subsystem=ejb3,stateless-session-bean=SequenceFacade, name=SequenceFacade, parent=ejb3] - cause: org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException:Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials. > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authentication required > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9444 requested authentication > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.client.DefaultHttpClient)- Authentication failed > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:39871<->10.49.67.52:9444 shut down > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:39871<->10.49.67.52:9444 closed > 2015-02-12 15:21:00,400 DEBUG (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: id: 955[route: ->https://0.0.0.0:9444][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 10]

                    2015-02-12 15:21:00,400 DEBUG (rhq.modules.plugins.jbossas7.BaseServerComponent)- ResourceType[id=0, name=JBossAS7 Standalone Server, plugin=JBossAS7, category=Server] hostConfig: /opt/jboss/server/arbeid/configuration/standalone.xml: exception while checking availability

                    org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.

                     

                     

                    These two samples are interesting enough.

                     

                    So the as7plugin is trying to start a new connection and, for some

                    reason, authentication challenge processing fails. Can you attach what's

                    going through the wire at that time?

                     

                    How are the AS7 servers configured with respect to authentication?

                    Simple properties file? LDAP backend?

                     

                    I noticed you're using SSL protected management connections. Is this a

                    test environment? In this case, could you try to use raw http on one host?

                      • Actions
                    • 7. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                      pathduck
                      pathduck Feb 13, 2015 5:31 AM (in response to tsegismont)

                      > Can you attach what's going through the wire at that time?

                       

                      Sure, do you mean by using TCPdump? Is there another way apart from the logs? There's no more output from the specific threads.

                       

                      > How are the AS7 servers configured with respect to authentication?

                       

                      They are connected to Active Directory  using ldaps:// and a filter that ensures our service-account is authenticated.

                       

                      The SSL is configured:

                      <ssl>
                              <keystore path="etc/keystore.jks" relative-to="jboss.home.dir" keystore-password=""/>
                      </ssl>

                       

                      Interfaces are just the defaults, we have management-http and management-https configured.

                       

                      Initally we are just using the management-http which is fine for us, but since RHQ started detecting mangement-https and using it then it's too much hassle to change it to just http for all the servers.

                       

                      I could try changing the AS7 connection to go through management-http instead but since the problem is intermittent it would be hard to catch raw data.

                      Maybe to see if problem goes away using management-http?

                       

                      Edit; I am tracing port 9990 using tcpdump but really not sure what to look for...

                       

                      I am getting a lot of 500 Internal Server Errors though.

                       

                      POST /management HTTP/1.1
Accept: application/json
Content-Length: 210
Content-Type: application/json; charset=UTF-8
Host: 0.0.0.0:9990
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.2.5 (java 1.5)

{"operation":"read-attribute","address":[{"deployment":"nav-sbl-arbeid-j2ee-4.2.14.2.ear"},{"subdeployment":"nav-sbl-arbeid-ejb.jar"},{"subsystem":"ejb3"},{"entity-bean":"BrukerHendelseLogg"}],"name":"methods"}
11:25:41.233719 IP 10.49.67.52.9990 > 10.49.67.52.51180: Flags [P.], seq 1:136, ack 425, win 1023, options [nop,nop,TS val 705038964 ecr 705038963], length 135
E.....@.@...
1C4
1C4'...;.v=p..'.....w.....
*.
t*.
sHTTP/1.1 500 Internal Server Error
Transfer-encoding: chunked
Content-type: application/json
Date: Fri, 13 Feb 2015 10:25:41 GMT

                       

                      Stian

                        • Actions
                      • 8. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                        tsegismont
                        tsegismont Feb 13, 2015 5:31 AM (in response to pathduck)

                        Le 13/02/2015 11:11, Stian Lund a écrit :

                        > Can you attach what's going through the wire at that time?

                         

                        Sure, do you mean by using TCPdump? Is there another way apart from the

                        logs? There's no more output from the specific threads.

                         

                        Do httpclient wire logs show nothing? Strange.

                         

                         

                        > How are the AS7 servers configured with respect to authentication?

                         

                        They are connected to Active Directory  using ldaps:// and a filter that

                        ensures our service-account is authenticated.

                         

                        The SSL is configured:

                         

                        <ssl>

                                 <keystore path="etc/keystore.jks" relative-to="jboss.home.dir"

                        keystore-password=""/>

                        </ssl>

                         

                        Interfaces are just the defaults, we have management-http and

                        management-https configured.

                         

                        Initally we are just using the management-http which is fine for us, but

                        since RHQ started detecting mangement-https and using it then it's too

                        much hassle to change it to just http for all the servers.

                         

                        CLI script to the rescue?

                         

                         

                        I could try changing the AS7 connection to go through management-http

                        instead but since the problem is intermittent it would be hard to catch

                        raw data.

                         

                        Maybe to see if problem goes away using management-http?

                         

                         

                        Yest that's the purpose. Configure just one of the AS7 resources to be

                        monitored with management-http and see if the intermittent failures go away.

                          • Actions
                        • 9. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                          tsegismont
                          tsegismont Feb 13, 2015 5:58 AM (in response to pathduck)

                          Le 13/02/2015 11:11, Stian Lund a écrit :

                          > How are the AS7 servers configured with respect to authentication?

                           

                          They are connected to Active Directory  using ldaps:// and a filter that

                          ensures our service-account is authenticated.

                           

                          The SSL is configured:

                           

                          <ssl>

                                   <keystore path="etc/keystore.jks" relative-to="jboss.home.dir"

                          keystore-password=""/>

                          </ssl>

                           

                           

                          Are there any errors in the AS7 server logs?

                           

                          Can you try on another server to keep SSL but switch to properties file

                          backend?

                            • Actions
                          • 10. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                            pathduck
                            pathduck Feb 13, 2015 6:08 AM (in response to tsegismont)

                            Could the 500 internal errors be a clue?

                             

                            Not seeing anything in AS7 server.log apart from the many exceptions our applications throw... but from experience I know the AS7 management interface logs almost nothing unless you increase loglevel for security.

                             

                            I could change to properties-based for one of the servers, and keep SSL on.

                              • Actions
                            • 11. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                              tsegismont
                              tsegismont Feb 13, 2015 6:43 AM (in response to pathduck)

                               

                              1. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized 
                              2. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS 
                              3. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication required 
                              4. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9744 requested authentication 
                              5. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authorization challenge processed 
                              6. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication failed 
                              7. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 shut down 
                              8. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Connection 0.0.0.0:52598<->10.49.67.52:9744 closed 
                              9. 2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 1205][route: {s}->https://0.0.0.0:9744][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 10] 
                              10. 2015-02-12 15:31:18,305 WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=13234, uuid=8ee12c61-2b3c-4fb4-91f7-c5e44ef927b2, type={JBossAS7}Stateless Session Bean Runtime, key=deployment=nav-sbl-stilling-j2ee.ear,subdeployment=nav-sbl-stilling-ejb.jar,subsystem=ejb3,stateless-session-bean=SequenceFacade, name=SequenceFacade, parent=ejb3] - cause: org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException:Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials. 

                               

                              Is there any message logged by ResourceContainer.invoker.daemon-27 just before 2015-02-12 15:31:18,305 ?

                                • Actions
                              • 12. Re: Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                                pathduck
                                pathduck Feb 13, 2015 7:17 AM (in response to tsegismont)

                                Here's what I find for that thread just before that time:

                                 

                                2015-02-12 15:31:08,465 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection [id: 1205][route: {s}->https://0.0.0.0:9744] can be kept alive for 5000 MILLISECONDS
2015-02-12 15:31:08,465 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection released: [id: 1205][route: {s}->https://0.0.0.0:9744][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 10]
2015-02-12 15:31:08,466 INFO  [ResourceContainer.invoker.daemon-27] (rhq.modules.plugins.jbossas7.ASConnection)- {
2015-02-12 15:31:08,466 INFO  [ResourceContainer.invoker.daemon-27] (rhq.modules.plugins.jbossas7.ASConnection)- JSON to send: {"operation":"read-attribute","address":[{"deployment":"nav-sbl-stilling-j2ee-3.0.27.ear"},{"subdeployment":"nav-sbl-stilling-ejb.jar"},{"subsystem":"ejb3"},{"stateless-session-bean":"SequenceFacade"}],"name":"methods"}
2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection request: [route: {s}->https://0.0.0.0:9744][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 10]
2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.PoolingClientConnectionManager)- Connection leased: [id: 1205][route: {s}->https://0.0.0.0:9744][total kept alive: 0; route allocated: 1 of 10; total allocated: 1 of 10]
2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestAddCookies)- CookieSpec selected: best-match
2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestAuthCache)- Auth cache not set in the context
2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestTargetAuthentication)- Target auth state: UNCHALLENGED
2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestProxyAuthentication)- Proxy auth state: UNCHALLENGED
2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Attempt 1 to execute request
2015-02-12 15:31:08,466 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Sending request: POST /management HTTP/1.1
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Connection can be kept alive for 5000 MILLISECONDS
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Authentication required
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- 0.0.0.0:9744 requested authentication
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Authentication schemes in the order of preference: [negotiate, Kerberos, NTLM, Digest, Basic]
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Challenge for negotiate authentication scheme not available
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Challenge for Kerberos authentication scheme not available
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Challenge for NTLM authentication scheme not available
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.TargetAuthenticationStrategy)- Challenge for Digest authentication scheme not available
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Selected authentication options: [BASIC]
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestAddCookies)- CookieSpec selected: best-match
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestAuthCache)- Auth cache not set in the context
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestTargetAuthentication)- Target auth state: CHALLENGED
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestTargetAuthentication)- Generating response to an authentication challenge using basic scheme
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.client.protocol.RequestProxyAuthentication)- Proxy auth state: UNCHALLENGED
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.client.DefaultHttpClient)- Attempt 2 to execute request
2015-02-12 15:31:08,467 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Sending request: POST /management HTTP/1.1
2015-02-12 15:31:18,305 DEBUG [ResourceContainer.invoker.daemon-27] (apache.http.impl.conn.DefaultClientConnection)- Receiving response: HTTP/1.1 401 Unauthorized

                                 

                                Attaching some logs where I've parsed out the more sensitive stuff.

                                  • Actions
                                • 13. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                                  tsegismont
                                  tsegismont Feb 13, 2015 10:44 AM (in response to pathduck)

                                  Thanks. So the logs show that:

                                  1. the as7plugin sends an unauthenticated request
                                  2. gets a 401 response
                                  3. replies to the challenge
                                  4. gets another 401 response

                                  That is the problem. If you look at the other http exchanges in your logs, at step 4 as7plugin gets a 200 response.

                                   

                                  I really need to see what went on the wire at this time to determine if there's a bug on the client side. No progess with the org.apache.http.wire logging?

                                  See https://hc.apache.org/httpcomponents-client-ga/logging.html

                                    • Actions
                                  • 14. Re: Re: Re: Lots of InvalidPluginConfigurationException - AS7 plugin, RHQ 4.13.1?
                                    pathduck
                                    pathduck Feb 13, 2015 11:40 AM (in response to tsegismont)

                                    Oh sorry Thomas, when you said "wire" I thought you meant the *ethernet* interface aka the wire - and not the class org.apache.http.wire - sometimes I'm just daft...

                                     

                                    I had already set org.apache.http to debug though, do I need to add org.apache.http.* or just org.apache.http.wire will be enough?


                                    It's a bit late now but come monday I will turn on DEBUG on class org.apache.http.wire in log4j.xml

                                     

                                    Have a good weekend!

                                    Stian

                                      • Actions
                                    1 2 Previous Next