Drift Management in RHQ 4.1

[hounslg]

Hopefully a simple question does it work or is it still really a beta feature?

We've been looking at RHQ and quite like the drift management though it doesn't seem to work for us as though there are no errors when we set-up a change set no files seem to be show up.

I've been trying to set it up to monitor the conf directory for our servers and have tried the various options but the files in /server/default/conf just don't show up,  I just don't get my version 0 baseline.

The servers are running on Windows 2003 server currently using java wrapper to run as a windows service. So far everything else seems to work ok.

If I do a detect now I can see it being triggered in the agent log

2011-10-14 15:27:03,237 INFO  [WorkerThread#0[10.196.129.208:1160]] (rhq.core.pc.drift.DriftManager)- Received request to schedule drift detection immediately for [resourceId: 10734, driftConfigurationId: 10072, driftConfigurationName: Application Conf]

and shortly afterwards a message - is this an unreleated message though?

2011-10-14 15:27:26,581 INFO  [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementSenderRunner)- Measurement collection for [431] metrics took 1266ms - sending report to Server...

Any advice or help would be much appreciated.

regards

Graham

[john.sanda]

Drift managent is stable in the RHQ 4.1 release; however, there is a lot of functionality planned for the feature that was not yet implemented for the 4.1 release. There are some some bugs that I believe were found right after after that release that you may be hitting. Have you verified that the base directory for your drift definition (formerly called drift configuration) does in fact exist on the agent machine? There is an outstanding issue where drift detection fails if the base directory does not exist. This will be fixed in the next release. You might also want to turn on debug logging for drift on the agent. You can do so by adding the following to <AGENT_HOME/conf/log4j.xml,

<category name="org.rhq.core.pc.drift">

  <priority value="DEBUG"/>

</category>

Hope that helps.

- John

[hounslg]

Hi John, thanks for your answer.

I'm not in front of the machine at the moment might seem a bit vague and/or confused. I've tried the default option for a monitored Jboss AS server, pluginconfiguration option which is set to jbosshome. I've also tried the filesystem option as well. Neither of which seemed to work. I've also tried it with back and forward slashes (c:\temp and c:/temp) just in case this caused a problem.

I also changed to log level, though I didn't see anything wrong.

I'll grab some screenshots on Monday and double check the log file settings as that should make it clearer what settings I'm trying.

cheers

Graham

[hounslg]

After restarting the agent and triggering a "Detect Now" I'm now seeing an error message in the log files.

Checking in the directory mentioned there are two empty files, changeset.txt and a dirft-changeset.txt.

Log message

2011-10-17 10:16:15,383 INFO  [WorkerThread#0[10.196.129.208:3677]] (rhq.core.pc.drift.DriftManager)- Received request to schedule drift detection immediately for [resourceId: 10743, driftConfigurationId: 10091, driftConfigurationName: Core Files]

2011-10-17 10:16:15,383 DEBUG [WorkerThread#0[10.196.129.208:3677]] (rhq.core.pc.drift.DriftManager)- Resetting DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files] for immediate detection.

2011-10-17 10:16:15,383 DEBUG [WorkerThread#0[10.196.129.208:3677]] (rhq.core.pc.drift.DriftManager)- DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files] has been added to ScheduleQueue[DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files]] for immediate detection.

2011-10-17 10:16:19,352 INFO  [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementSenderRunner)- Measurement collection for [14] metrics took 626ms - sending report to Server...

2011-10-17 10:16:48,883 DEBUG [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Processing DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files]

2011-10-17 10:16:48,883 DEBUG [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Generating drift change set for DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files]

2011-10-17 10:16:48,883 ERROR [pool-3-thread-1] (org.rhq.common.drift.ChangeSetReaderImpl)- Unable to read headers from C:\rhq\rhq-agent\data\changesets\10743\Core Files\changeset.txt: null

2011-10-17 10:16:48,883 ERROR [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- An error occurred while scanning for drift

org.rhq.common.drift.ChangeSetReaderException: Unable to read headers from C:\rhq\rhq-agent\data\changesets\10743\Core Files\changeset.txt

          at org.rhq.common.drift.ChangeSetReaderImpl.<init>(ChangeSetReaderImpl.java:70)

          at org.rhq.core.pc.drift.ChangeSetManagerImpl.getChangeSetReader(ChangeSetManagerImpl.java:96)

          at org.rhq.core.pc.drift.DriftDetector.generateDriftChangeSet(DriftDetector.java:126)

          at org.rhq.core.pc.drift.DriftDetector.run(DriftDetector.java:100)

          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

          at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:619)

Caused by: java.lang.NumberFormatException: null

          at java.lang.Integer.parseInt(Integer.java:417)

          at java.lang.Integer.parseInt(Integer.java:499)

          at org.rhq.common.drift.ChangeSetReaderImpl.readHeaders(ChangeSetReaderImpl.java:96)

          at org.rhq.common.drift.ChangeSetReaderImpl.<init>(ChangeSetReaderImpl.java:64)

          ... 12 more

2011-10-17 10:17:15,899 DEBUG [WorkerThread#0[10.196.129.208:3677]] (jboss.remoting.transport.socket.ServerThread)- WorkerThread#0[10.196.129.208:3677] closing socketWrapper: ServerSocketWrapper[Socket[addr=/10.196.129.208,port=3677,localport=16163].1bbf45e]

2011-10-17 10:17:15,899 DEBUG [WorkerThread#0[10.196.129.208:3677]] (jboss.remoting.transport.socket.ServerSocketWrapper)- wrote CLOSING

2011-10-17 10:17:15,899 DEBUG [WorkerThread#0[10.196.129.208:3677]] (jboss.remoting.transport.socket.SocketWrapper)- ServerSocketWrapper[Socket[addr=/10.196.129.208,port=3677,localport=16163].1bbf45e] closing

2011-10-17 10:17:18,805 INFO  [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementSenderRunner)- Measurement collection for [6] metrics took 0ms - sending report to Server...

2011-10-17 10:17:48,883 DEBUG [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Processing DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files]

2011-10-17 10:17:48,883 DEBUG [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Skipping DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files] because it is too early to do the next detection.

I've also uploaded some images show the steps I've taken to set-up the change set in the console.

Any ideas what I've done wrong?

[john.sanda]

Hi Graham,

Whatever error you originally hit is not being properly handled by the agent. The agent (with respect to your drift definition) is an inconsistent state. That is what you are running into with that ChangeSetReaderException. In short that should never happen. This is one of the issues we identified post 4.1 and it will be resolved in the 4.2 release. You will have to go to the drift UI, delete the definition/configuration, and then recreate it in order to get out of that inconsistent state. This does not however address the original issue that led to the inconsistent state. After you've recreated the drift definition, hopefully we can get a better idea of what is happening now that you have debug logging enabled.

- John

[hounslg]

I got the above error message(s) when I first set the change set up. This was with nothing already set-up. 

To try and resolve this;

     I've deleted the change set, and deleted the changeset.txt and dirft-changeset.txt. Then recreated the change set in the console as above.

I've not had a chance to have a good look but it looks like its only created an empty changeset.txt file. I've only had a quick look and I couldn't see any errors the log file. I'll have a chance to have a proper look tomorrow and post any updates.

Graham

[hounslg]

Gone through and deleted all drift configurations, stopped and started the agent, created a new configuration. I removed the space from the configuration name just in case that was causing a problem on Windows.

Log file

2011-10-19 11:20:02,303 INFO  [WorkerThread#0[10.196.129.208:2180]] (rhq.core.pc.drift.DriftManager)- Received request to update schedule for [resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: CoreFiles]

2011-10-19 11:20:02,303 INFO  [WorkerThread#0[10.196.129.208:2180]] (rhq.core.pc.drift.DriftManager)- Adding DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: CoreFiles] to ScheduleQueue[DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files], DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files]]

2011-10-19 11:20:02,303 DEBUG [WorkerThread#0[10.196.129.208:2180]] (rhq.core.pc.drift.DriftManager)- DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: CoreFiles] has been added to ScheduleQueue[DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: CoreFiles], DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files], DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: Core Files]]

2011-10-19 11:20:14,507 DEBUG [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Processing DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: CoreFiles]

2011-10-19 11:20:14,507 DEBUG [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Generating coverage change set for DriftDetectionSchedule[resourceId: 10743, driftConfigurationId: 0, driftConfigurationName: CoreFiles]

2011-10-19 11:20:44,444 INFO  [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementSenderRunner)- Measurement collection for [6] metrics took 0ms - sending report to Server...

2011-10-19 11:21:02,835 DEBUG [WorkerThread#0[10.196.129.208:2180]] (jboss.remoting.transport.socket.ServerThread)- WorkerThread#0[10.196.129.208:2180] closing socketWrapper: ServerSocketWrapper[Socket[addr=/10.196.129.208,port=2180,localport=16163].1a0d6c9]

2011-10-19 11:21:02,835 DEBUG [WorkerThread#0[10.196.129.208:2180]] (jboss.remoting.transport.socket.ServerSocketWrapper)- wrote CLOSING

2011-10-19 11:21:02,835 DEBUG [WorkerThread#0[10.196.129.208:2180]] (jboss.remoting.transport.socket.SocketWrapper)- ServerSocketWrapper[Socket[addr=/10.196.129.208,port=2180,localport=16163].1a0d6c9] closing

It creates an empty changeset.txt file, no drift-changeset.txt file is created.

Stopped and started the agent and an empty drift-changeset.txt file is created - though no errors in the log file.

[john.sanda]

Hi Graham,

It is difficult for me to pinpoint the issue since there has been some many changes (including bug fixes) since the 4.1 release. Based on the information that you have already provided, I am inclined to think that the problem might be with parsing a path name. Can you post a directory listing where the drift detection is running?

[hounslg]

The directories are

Jboss install - e:\apps\healthcare\jboss423\

This is where jboss_home is set to and also the connection settings in RHQ point here as well.

The agent is installed in c:\rhq\rhq-agent

It's by default trying to scan bin, client and lib these are listed below - I can't see anything to obviously wrong.

I've also tried setting up a change set using the file system option with the base directory being as above, but this doesn't work either - or works the same as the plugin configuration option depending on how you look at it.

Directory of E:\apps\healthcare\jboss423\bin

Bin Directory

19/10/2011  16:32    <DIR>          .

19/10/2011  16:32    <DIR>          ..

18/07/2008  14:40             3,535 classpath.sh

12/01/2011  10:39             1,273 InstallApp-NT-node-01.bat

13/08/2009  10:05             1,265 InstallApp-NT.bat

19/10/2011  09:56                 6 jboss-wrapper.pid

19/10/2011  09:56                 6 jboss.pid

18/07/2008  14:41             8,808 jboss_init_hpux.sh

18/07/2008  14:41             3,639 jboss_init_redhat.sh

18/07/2008  14:41             4,671 jboss_init_suse.sh

18/07/2008  14:40               535 probe.bat

18/07/2008  14:40               918 probe.sh

18/07/2008  14:40             3,527 run.bat

18/07/2008  14:40             1,519 run.conf

18/07/2008  14:44            43,560 run.jar

18/07/2008  14:40             7,107 run.sh

18/07/2008  14:41             1,665 shutdown.bat

18/07/2008  14:44            21,710 shutdown.jar

18/07/2008  14:41             1,903 shutdown.sh

09/06/2011  15:43         1,447,793 stacktrace.log

07/10/2011  15:58               110 startdef.bat

18/07/2008  14:42             1,919 twiddle.bat

18/07/2008  14:44            47,369 twiddle.jar

18/07/2008  14:42             2,348 twiddle.sh

12/01/2011  10:56             1,277 UninstallApp-NT-node-01.bat

13/08/2009  10:05             1,269 UninstallApp-NT.bat

13/08/2009  10:05           167,936 wrapper.exe

13/08/2009  10:05             2,065 wrapper.log

13/08/2009  10:05             2,773 wsconsume.bat

13/08/2009  10:05             3,803 wsconsume.sh

13/08/2009  10:05             2,859 wsprovide.bat

13/08/2009  10:05             3,885 wsprovide.sh

13/08/2009  10:05             3,009 wsrunclient.bat

13/08/2009  10:05             3,972 wsrunclient.sh

13/08/2009  10:05             2,378 wstools.bat

13/08/2009  10:05             3,380 wstools.sh

Lib directory

Directory of E:\apps\healthcare\jboss423\lib

13/08/2009  10:05    <DIR>          .

13/08/2009  10:05    <DIR>          ..

18/07/2008  14:44            47,469 commons-codec.jar

18/07/2008  14:44           214,798 commons-httpclient.jar

18/07/2008  14:44            55,423 commons-logging.jar

18/07/2008  14:44           255,512 concurrent.jar

13/08/2009  10:05    <DIR>          endorsed

18/07/2008  14:44            15,857 getopt.jar

18/07/2008  14:44           406,562 jboss-common.jar

18/07/2008  14:44           880,190 jboss-jmx.jar

18/07/2008  14:44           267,175 jboss-system.jar

18/07/2008  14:44           376,342 jboss-xml-binding.jar

18/07/2008  14:44           128,400 log4j-boot.jar

              10 File(s)      2,647,728 bytes

Directory of E:\apps\healthcare\jboss423\lib\endorsed

13/08/2009  10:05    <DIR>          .

13/08/2009  10:05    <DIR>          ..

13/08/2009  10:05            89,967 jaxb-api.jar

18/07/2008  14:44            29,383 jboss-jaxrpc.jar

18/07/2008  14:44            33,293 jboss-jaxws-ext.jar

18/07/2008  14:44            34,715 jboss-jaxws.jar

18/07/2008  14:44            26,981 jboss-saaj.jar

13/08/2009  10:05            29,165 jbossws-native-jaxrpc.jar

13/08/2009  10:05            20,141 jbossws-native-jaxws-ext.jar

13/08/2009  10:05            47,411 jbossws-native-jaxws.jar

13/08/2009  10:05            27,251 jbossws-native-saaj.jar

18/07/2008  14:44           188,992 serializer.jar

18/07/2008  14:44         1,713,709 xalan.jar

18/07/2008  14:44         1,200,040 xercesImpl.jar

Client Directory

Directory of E:\apps\healthcare\jboss423\client

13/08/2009  10:05    <DIR>          .

13/08/2009  10:05    <DIR>          ..

18/07/2008  14:44            58,223 activation.jar

18/07/2008  14:44           355,300 antlr.jar

18/07/2008  14:41               596 auth.conf

18/07/2008  14:44            73,334 avalon-framework.jar

18/07/2008  14:44            47,469 commons-codec.jar

18/07/2008  14:44           214,798 commons-httpclient.jar

18/07/2008  14:44            55,423 commons-logging.jar

18/07/2008  14:44           255,512 concurrent.jar

18/07/2008  14:44            52,219 ejb3-persistence.jar

13/08/2009  10:05           291,779 FastInfoset.jar

18/07/2008  14:44            15,857 getopt.jar

18/07/2008  14:44           444,510 hibernate-annotations.jar

18/07/2008  14:44           246,649 hibernate-client.jar

18/07/2008  14:44         4,717,899 jacorb.jar

18/07/2008  14:44           599,742 javassist.jar

13/08/2009  10:05            89,967 jaxb-api.jar

13/08/2009  10:05           849,481 jaxb-impl.jar

13/08/2009  10:05         3,126,060 jaxb-xjc.jar

13/08/2009  10:05         1,187,868 jaxws-rt.jar

13/08/2009  10:05           509,137 jaxws-tools.jar

18/07/2008  14:44            41,485 jboss-annotations-ejb3.jar

18/07/2008  14:44           106,590 jboss-aop-jdk50-client.jar

18/07/2008  14:44            46,156 jboss-aspect-jdk50-client.jar

18/07/2008  14:44           193,844 jboss-client.jar

18/07/2008  14:44           406,562 jboss-common-client.jar

18/07/2008  14:44            59,446 jboss-deployment.jar

18/07/2008  14:44           300,322 jboss-ejb3-client.jar

18/07/2008  14:44            20,389 jboss-ejb3x.jar

18/07/2008  14:44           152,441 jboss-iiop-client.jar

18/07/2008  14:44           423,725 jboss-j2ee.jar

18/07/2008  14:44            28,411 jboss-jsr77-client.jar

18/07/2008  14:44           907,987 jboss-remoting.jar

18/07/2008  14:44           125,850 jboss-serialization.jar

18/07/2008  14:44            22,227 jboss-srp-client.jar

18/07/2008  14:44            26,313 jboss-system-client.jar

18/07/2008  14:44            53,275 jboss-transaction-client.jar

18/07/2008  14:44           376,342 jboss-xml-binding.jar

18/07/2008  14:44         5,000,359 jbossall-client.jar

18/07/2008  14:44            22,226 jbosscx-client.jar

18/07/2008  14:44            58,339 jbossha-client.jar

18/07/2008  14:44           166,785 jbossjmx-ant.jar

18/07/2008  14:44           333,776 jbossmq-client.jar

18/07/2008  14:44           151,240 jbosssx-client.jar

13/08/2009  10:05           117,113 jbossws-common.jar

13/08/2009  10:05            95,527 jbossws-framework.jar

13/08/2009  10:05            79,729 jbossws-jboss423.jar

13/08/2009  10:05             4,826 jbossws-native-client.jar

13/08/2009  10:05         1,662,592 jbossws-native-core.jar

13/08/2009  10:05            29,165 jbossws-native-jaxrpc.jar

13/08/2009  10:05            20,141 jbossws-native-jaxws-ext.jar

13/08/2009  10:05            47,411 jbossws-native-jaxws.jar

13/08/2009  10:05            27,251 jbossws-native-saaj.jar

13/08/2009  10:05           107,896 jbossws-spi.jar

13/08/2009  10:05            54,187 jettison.jar

18/07/2008  14:44           454,151 jmx-client.jar

18/07/2008  14:44             4,809 jmx-invoker-adaptor-client.jar

18/07/2008  14:44            33,147 jnp-client.jar

18/07/2008  14:44           355,751 log4j.jar

18/07/2008  14:44            86,380 logkit.jar

18/07/2008  14:44           387,689 mail.jar

13/08/2009  10:05            39,688 policy.jar

18/07/2008  14:44            82,773 scout.jar

18/07/2008  14:44            89,677 servlet-api.jar

13/08/2009  10:05            23,346 stax-api.jar

13/08/2009  10:05            11,074 stax-ex.jar

13/08/2009  10:05            57,984 streambuffer.jar

18/07/2008  14:44           385,226 trove.jar

13/08/2009  10:05           148,522 wsdl4j.jar

13/08/2009  10:05           525,813 wstx.jar

18/07/2008  14:44           289,208 xmlsec.jar

It might be something we're have to leave until 4.2 comes out maybe.

[john.sanda]

Hi Graham,

My apologies for not replying sooner. We have been heads down trying to get things wrapped up for a 4.2 release. Unfortunately I was not able to determine the source of your problem, though I wasn't able to spend too much time looking at it. An RHQ 4.2 release is due out very soon, and for drift management, that means lots of bug fixes and a whole bunch of new features. I hope you are willing and able to give it another try with 4.2. The community feedback is invaluable. I will be sure to post a note on the forum in the coming days about changes and enhancements with respect to drift management.

[hounslg]

Hi John,

no need to apologise, I'm really grateful for the help you've been able to give.

Really looking forward to the 4.2 release, as soon as we get it on our servers I'll let you know how it goes.

We're really pleased with RHQ and it's great to monitor our environments. Still trying to get my head round the bundle and provisioning though that's next on the list.

cheers

Graham