Jboss keeps using up the max memory with large dataset
Hello,
Could someone help me troubleshoot the issue I've been having. My JBOSS server has been running out of memory when querying against a large table (130 million rows with 45 columns). I think the server is beefy enough to handle large data set. Below are the specs:
- Jboss EAP 6.4 with Teiid 8.7.1
- It's running on a dedicated AWS Windows Server instance with 244GB ram, 32 processors
I have the JVM arguments as below:
-server -d64 -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:+PrintGCDetails -Xms200g -Xmx200g -XX:NewSize=8g -XX:MaxNewSize=8g -XX:SurvivorRatio=8 -XX:MaxDirectMemorySize=10g -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+UseLargePages
Teiid Buffer Manager settings:
"allow-env-function" => false,
"async-thread-pool" => "teiid-async",
"authorization-validator-module" => undefined,
"buffer-service-encrypt-files" => false,
"buffer-service-inline-lobs" => true,
"buffer-service-max-buffer-space" => 500000L,
"buffer-service-max-file-size" => 150000L,
"buffer-service-max-open-files" => 64,
"buffer-service-max-processing-kb" => -1,
"buffer-service-max-reserve-kb" => -1,
"buffer-service-max-storage-object-size" => 16777216,
"buffer-service-memory-buffer-off-heap" => true,
"buffer-service-memory-buffer-space" => -1,
"buffer-service-processor-batch-size" => 30000,
"buffer-service-use-disk" => true,
"detect-change-events" => true,
"distributed-cache-jgroups-stack" => undefined,
"exception-on-max-source-rows" => true,
"lob-chunk-size-in-kb" => 100,
"max-active-plans" => 20,
"max-row-fetch-size" => 20480,
"max-source-rows-allowed" => -1,
"max-threads" => 64,
"policy-decider-module" => "org.jboss.teiid",
"preparedplan-cache-enable" => true,
"preparedplan-cache-infinispan-container" => "teiid-cache",
"preparedplan-cache-name" => "preparedplan",
"query-threshold-in-seconds" => 600,
"query-timeout" => 0,
"resultset-cache-enable" => true,
"resultset-cache-infinispan-container" => "teiid-cache",
"resultset-cache-max-staleness" => 60,
"resultset-cache-name" => "resultset",
"thread-count-for-source-concurrency" => 0,
"time-slice-in-milliseconds" => 2000,
"workmanager" => "default",
As you can see I've assigned the Heap size to 200gb but it's still maxing out the memory every time and the server just hangs up with "Full GC(Ergonmics)" error. Below is a sample log:
[GC (Allocation Failure) [PSYoungGen: 7549952K->838656K(7549952K)] 176791593K->176710678K(208876544K), 64.7327072 secs] [Times: user=1476.97 sys=9.75, real=64.73 secs]
[GC (Allocation Failure) [PSYoungGen: 7549952K->838656K(7549952K)] 183421974K->183391750K(208876544K), 68.1115075 secs] [Times: user=1554.99 sys=9.14, real=68.12 secs]
[GC (Allocation Failure) [PSYoungGen: 7549952K->838656K(7549952K)] 190103046K->190019046K(208876544K), 70.2219754 secs] [Times: user=1603.11 sys=9.98, real=70.23 secs]
[GC (Allocation Failure) [PSYoungGen: 7549952K->838656K(7549952K)] 196730342K->196631886K(208876544K), 87.7628514 secs] [Times: user=2007.72 sys=8.92, real=87.76 secs]
[Full GC (Ergonomics) [PSYoungGen: 838656K->0K(7549952K)] [ParOldGen: 195793230K->193496415K(201326592K)] 196631886K->193496415K(208876544K), [Metaspace: 130959K->130959K(139264K)], 429.0076056 secs] [Times: user=1582.83 sys=5.59, real=429.00 secs]
[GC (Allocation Failure) [PSYoungGen: 6711296K->838656K(7549952K)] 200207711K->200111591K(208876544K), 49.8578163 secs] [Times: user=1141.05 sys=3.48, real=49.86 secs]
[Full GC (Ergonomics) [PSYoungGen: 838656K->0K(7549952K)] [ParOldGen: 199272935K->200062941K(201326592K)] 200111591K->200062941K(208876544K), [Metaspace: 130999K->130999K(139264K)], 383.1849674 secs] [Times: user=570.09 sys=7.91, real=383.17 secs]
[Full GC (Ergonomics) [PSYoungGen: 6711296K->5262703K(7549952K)] [ParOldGen: 200062941K->201326365K(201326592K)] 206774237K->206589069K(208876544K), [Metaspace: 131005K->131005K(139264K)], 573.4518437 secs] [Times: user=608.89 sys=15.22, real=573.45 secs]
[Full GC (Ergonomics)
Do you know what the issue is here? Does it have something to do with not configuring JVM or Teiid buffer manager correctly? Please advice or let me know if you need additional info.
Thanks so much,
Tom
