1 Reply Latest reply on Apr 23, 2010 5:14 AM by pilhuhn

    java.lang.OutOfMemoryError: GC overhead limit exceeded

    robert.breault

      I keep getting this error on a server with 4.0 GB's of RAM running PostgreSQL with about 10 DB's and the main JOPR server.

       

      Its the agent that keeps throwing this error message. I have tried many different JAVA_OPTS settings but can't seem to nail this down. The agent always throws this error unless I add the arg ::::   -XX:-UseGCOverheadLimit

       

      Now that causes other issues of course like maxing out my CPU so I tried the following with no luck

       

      -Xms64m -Xmx128m -XX:MaxPermSize=128m -Xss512k and again it crashes.

       

      Now my DB's are about 1GB in size and such so I am not sure if that is just running the agent out.

       

      Executor)- New configuration version detected on resource: Resource[id=11598, type=Table, key=servicerecord, na
      me=servicerecord, parent=demo Database]
      2010-04-16 14:30:43,787 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11599, type=Table, key=useracct, name=us
      eracct, parent=demo Database]
      2010-04-16 14:30:43,793 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11600, type=Table, key=scenario_template
      _charge, name=scenario_template_charge, parent=demo Database]
      2010-04-16 14:30:44,427 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11601, type=Table, key=uwcatqual, name=u
      wcatqual, parent=demo Database]
      2010-04-16 14:30:45,838 WARN  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- Unable to check for updated configuration
      java.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
          at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:53
      0)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
          at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
          at java.lang.StringCoding.decode(StringCoding.java:173)
          at java.lang.String.<init>(String.java:443)
          at org.postgresql.core.Encoding.decode(Encoding.java:193)
          at org.postgresql.core.Encoding.decode(Encoding.java:205)
          at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(AbstractJdbc2ResultSet.java:1893)
          at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBoolean(AbstractJdbc2ResultSet.java:1907)
          at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBoolean(AbstractJdbc2ResultSet.java:2351)
          at org.rhq.plugins.postgres.PostgresTableComponent.loadResourceConfiguration(PostgresTableComponent.jav
      a:131)
          at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:52
      5)
          ... 5 more
      2010-04-16 14:30:45,847 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11603, type=Table, key=orignotify, name=
      orignotify, parent=demo Database]
      2010-04-16 14:30:46,385 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11604, type=Table, key=nsf_fee_full_pymt
      _offset, name=nsf_fee_full_pymt_offset, parent=demo Database]
      2010-04-16 14:30:47,425 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11605, type=Table, key=origteam_case_com
      m, name=origteam_case_comm, parent=demo Database]
      2010-04-16 14:30:48,090 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11606, type=Table, key=loan_list_config,
      name=loan_list_config, parent=demo Database]
      2010-04-16 14:30:48,098 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11607, type=Table, key=late_fee, name=la
      te_fee, parent=demo Database]
      2010-04-16 14:30:50,327 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11608, type=Table, key=tc_period, name=t
      c_period, parent=demo Database]
      2010-04-16 14:30:50,330 ERROR [ClientCommandSenderTask Thread #2] (enterprise.communications.command.client.Cli
      entCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo];
      cmd-in-response=[false]; config=[{rhq.agent-name=, rhq.externalizable-strategy=AGE
      NT, rhq.security-token=1271349632660-137927991-5698344754604900791, rhq.guaranteed-delivery=true, rhq.send-thro
      ttle=true}]; params=[{invocation=NameBasedInvocation[persistUpdatedResourceConfiguration], targetInterfaceName=
      org.rhq.core.clientapi.server.configuration.ConfigurationServerService}]]. Cause: java.lang.Exception:java.lang
      .OutOfMemoryError: GC overhead limit exceeded -> java.lang.OutOfMemoryError:GC overhead limit exceeded. Cause:
      java.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit exceeded
      2010-04-16 14:30:50,331 WARN  [ClientCommandSenderTask Thread #2] (enterprise.communications.command.client.Cli
      entCommandSenderTask)- {ClientCommandSenderTask.queuing-failed-command}The command that failed has its guarante
      ed-delivery flag set so it is being queued again
      2010-04-16 14:30:50,334 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11609, type=Table, key=person, name=pers
      on, parent=demo Database]
      2010-04-16 14:30:51,540 INFO  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheck
      Executor)- New configuration version detected on resource: Resource[id=11610, type=Table, key=party_account_tra
      nsaction_batch, name=party_account_transaction_batch, parent=demo Database]
      2010-04-16 14:30:52,683 ERROR [ClientCommandSenderTask Thread #4] (enterprise.communications.command.client.Cli

        • 1. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
          pilhuhn

          Hey,

           

          we have seen some issues with growing agent size, so this may be related.

           

          How many resources do you have in inventory -- and especially on this agent. You say "10 DBs" - how many tables etc. do they have in sum? Are they all imported into RHQ/Jopr and managed by this agent?

          What version of RHQ/Jopr are you using?

          The size of the Postgres DB (postgres directories) should not matter here - but then I probably misunderstood "ow my DB's are about 1GB in size and such so I am not sure if that is just running the agent out.". Could you please clarify?

           

          Thanks

            Heiko