HornetQ 2.2.5 Paging + Memory and GC Pauses
mrobson Mar 6, 2012 9:17 AMHey,
I have a remote HornetQ server with 2 queues, the primary queue has 1 message for every request which is processed by an MDB. When the MDB processes from the primary queue, it can generate up top 3 messages on the 2nd queue.
What I am finding is that during the testing of various failure scenarios, my server has the potential to die due to GC pauses and extremely high memory.
My question is regarding the settings I am using and if they could be the problem.
256MB max size and 100MB page files for each queue.
<max-size-bytes>268435456</max-size-bytes> | |
<page-size-bytes>104857600</page-size-bytes> |
Journal Settings are 100MB with 10 files using NIO over NFS.
<journal-type>NIO</journal-type>
<journal-file-size>104857600</journal-file-size>
<journal-min-files>10</journal-min-files>
Here are my JVM settings:
-Xms2048m -Xmx2048m -XX:MaxPermSize=256m -XX:MaxDirectMemorySize=512m -XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:NewRatio=4 -XX:+UseLargePages -XX:LargePageSizeInBytes=4m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Dprogram.name=run.sh
Where I start up my MDB consumers with around 1.1million messages in the primary queue, this is what the GC looks like:
(5 sec interval below)
100.00 99.89 100.00 24.28 65.85 | 68 48.923 | 9 146.470 195.392 |
100.00 99.89 100.00 24.78 65.85 | 68 48.923 | 9 146.470 195.392 |
100.00 99.89 100.00 25.41 65.85 | 68 48.923 | 9 146.470 195.392 |
100.00 99.89 100.00 25.97 65.85 | 68 48.923 | 9 146.470 195.392 |
100.00 99.89 100.00 26.74 65.85 | 68 48.923 | 9 146.470 195.392 |
100.00 0.00 1.76 27.42 65.85 | 68 78.527 | 9 146.470 224.997 |
100.00 0.00 9.55 27.42 65.85 | 68 78.527 | 9 146.470 224.997 |
100.00 0.00 23.87 27.42 65.85 | 68 78.527 | 9 146.470 224.997 |
100.00 0.00 38.14 27.42 65.85 | 68 78.527 | 9 146.470 224.997 |
100.00 0.00 52.18 27.42 65.85 | 68 78.527 | 9 146.470 224.997 |
100.00 0.00 67.06 27.42 65.85 | 68 78.527 | 9 146.470 224.997 |
100.00 0.00 81.11 27.42 65.85 | 68 78.527 | 9 146.470 224.997 |
100.00 0.00 95.98 27.42 65.85 | 68 78.527 | 9 146.470 224.997 |
100.00 19.88 100.00 27.78 65.85 | 69 78.527 | 9 146.470 224.997 |
0.00 100.00 4.31 28.00 65.85 | 69 85.117 | 9 146.470 231.586 |
0.00 100.00 16.72 28.00 65.85 | 69 85.117 | 9 146.470 231.586 |
0.00 100.00 31.60 28.00 65.85 | 69 85.117 | 9 146.470 231.586 |
0.00 100.00 45.39 28.00 65.85 | 69 85.117 | 9 146.470 231.586 |
0.00 100.00 59.74 28.00 65.85 | 69 85.117 | 9 146.470 231.586 |
0.00 100.00 73.59 28.00 65.85 | 69 85.117 | 9 146.470 231.586 |
0.00 100.00 87.84 28.00 65.85 | 69 85.117 | 9 146.470 231.586 |
10.01 100.00 100.00 28.09 65.85 | 70 85.117 | 9 146.470 231.586 |
85.95 100.00 100.00 28.28 65.85 | 70 85.117 | 9 146.470 231.586 |
100.00 0.00 10.37 28.51 65.85 | 70 92.374 | 9 146.470 238.843 |
100.00 0.00 25.67 28.51 65.85 | 70 92.374 | 9 146.470 238.843 |
100.00 0.00 42.62 28.51 65.85 | 70 92.374 | 9 146.470 238.843 |
100.00 0.00 59.51 28.51 65.85 | 70 92.374 | 9 146.470 238.843 |
100.00 0.00 76.43 28.51 65.85 | 70 92.374 | 9 146.470 238.843 |
100.00 0.00 94.56 28.51 65.85 | 70 92.374 | 9 146.470 238.843 |
0.00 100.00 6.37 28.88 65.85 | 71 94.194 | 9 146.470 240.664 |
0.00 100.00 22.75 28.88 65.85 | 71 94.194 | 9 146.470 240.664 |
0.00 100.00 40.94 28.88 65.85 | 71 94.194 | 9 146.470 240.664 |
0.00 100.00 58.56 28.88 65.85 | 71 94.194 | 9 146.470 240.664 |
0.00 100.00 76.78 28.88 65.85 | 71 94.194 | 9 146.470 240.664 |
0.00 100.00 95.14 28.88 65.85 | 71 94.194 | 9 146.470 240.664 |
64.60 0.00 6.78 29.24 65.85 | 72 95.847 | 9 146.470 242.317 |
64.60 0.00 23.46 29.24 65.85 | 72 95.847 | 9 146.470 242.317 |
64.60 0.00 42.86 29.24 65.85 | 72 95.847 | 9 146.470 242.317 |
64.60 0.00 61.43 29.24 65.85 | 72 95.847 | 9 146.470 242.317 |
64.60 0.00 81.24 29.24 65.85 | 72 95.847 | 9 146.470 242.317 |
64.60 0.00 99.96 29.24 65.85 | 72 95.847 | 9 146.470 242.317 |
0.00 100.00 12.08 29.62 65.85 | 73 97.601 | 9 146.470 244.071 |
0.00 100.00 28.95 29.62 65.85 | 73 97.601 | 9 146.470 244.071 |
0.00 100.00 47.52 29.62 65.85 | 73 97.601 | 9 146.470 244.071 |
0.00 100.00 65.17 29.62 65.85 | 73 97.601 | 9 146.470 244.071 |
0.00 100.00 82.96 29.62 65.85 | 73 97.601 | 9 146.470 244.071 |
12.04 100.00 100.00 29.67 65.85 | 74 97.601 | 9 146.470 244.071 |
57.51 0.00 15.84 29.97 65.85 | 74 99.084 | 9 146.470 245.553 |
57.51 0.00 32.87 29.97 65.85 | 74 99.084 | 9 146.470 245.553 |
57.51 0.00 51.20 29.97 65.85 | 74 99.084 | 9 146.470 245.553 |
57.51 0.00 69.74 29.97 65.85 | 74 99.084 | 9 146.470 245.553 |
57.51 0.00 89.46 29.97 65.85 | 74 99.084 | 9 146.470 245.553 |
0.00 57.27 4.03 30.31 65.85 | 75 100.332 | 9 146.470 246.802 |
0.00 57.27 20.87 30.31 65.85 | 75 100.332 | 9 146.470 246.802 |
0.00 57.27 41.08 30.31 65.85 | 75 100.332 | 9 146.470 246.802 |
0.00 57.27 59.68 30.31 65.85 | 75 100.332 | 9 146.470 246.802 |
0.00 57.27 78.56 30.31 65.85 | 75 100.332 | 9 146.470 246.802 |
0.00 57.27 97.67 30.31 65.85 | 75 100.332 | 9 146.470 246.802 |
58.28 0.00 11.76 30.67 65.85 | 76 101.879 | 9 146.470 248.349 |
58.28 0.00 28.74 30.67 65.85 | 76 101.879 | 9 146.470 248.349 |
58.28 0.00 48.51 30.67 65.85 | 76 101.879 | 9 146.470 248.349 |
58.28 0.00 68.69 30.67 65.85 | 76 101.879 | 9 146.470 248.349 |
58.28 0.00 88.68 30.67 65.85 | 76 101.879 | 9 146.470 248.349 |
0.00 58.22 3.98 31.01 65.85 | 77 103.336 | 9 146.470 249.806 |
0.00 58.22 21.00 31.01 65.85 | 77 103.336 | 9 146.470 249.806 |
0.00 58.22 40.87 31.01 65.85 | 77 103.336 | 9 146.470 249.806 |
0.00 58.22 60.68 31.01 65.85 | 77 103.336 | 9 146.470 249.806 |
0.00 58.22 79.82 31.01 65.85 | 77 103.336 | 9 146.470 249.806 |
0.00 58.22 99.32 31.01 65.85 | 77 103.336 | 9 146.470 249.806 |
59.14 0.00 13.88 31.37 65.85 | 78 104.890 | 9 146.470 251.360 |
59.14 0.00 29.44 31.37 65.85 | 78 104.890 | 9 146.470 251.360 |
59.14 0.00 47.57 31.37 65.85 | 78 104.890 | 9 146.470 251.360 |
59.14 0.00 65.64 31.37 65.85 | 78 104.890 | 9 146.470 251.360 |
59.14 0.00 84.31 31.37 65.85 | 78 104.890 | 9 146.470 251.360 |
59.14 12.69 100.00 31.60 65.85 | 79 104.890 | 9 146.470 251.360 |
0.00 59.51 17.57 31.72 65.85 | 79 106.482 | 9 146.470 252.951 |
0.00 59.51 35.99 31.72 65.85 | 79 106.482 | 9 146.470 252.951 |
0.00 59.51 55.55 31.72 65.85 | 79 106.482 | 9 146.470 252.951 |
0.00 59.51 74.99 31.72 65.85 | 79 106.482 | 9 146.470 252.951 |
0.00 59.51 93.93 31.72 65.85 | 79 106.482 | 9 146.470 252.951 |
62.70 0.00 4.44 32.09 65.85 | 80 108.941 | 9 146.470 255.411 |
62.70 0.00 21.28 32.09 65.85 | 80 108.941 | 9 146.470 255.411 |
62.70 0.00 41.02 32.09 65.85 | 80 108.941 | 9 146.470 255.411 |
62.70 0.00 60.49 32.09 65.85 | 80 108.941 | 9 146.470 255.411 |
62.70 0.00 78.48 32.09 65.85 | 80 108.941 | 9 146.470 255.411 |
62.70 0.00 95.25 32.09 65.85 | 80 108.941 | 9 146.470 255.411 |
0.00 61.25 7.51 32.42 65.85 | 81 110.999 | 9 146.470 257.469 |
0.00 61.25 23.91 32.42 65.85 | 81 110.999 | 9 146.470 257.469 |
0.00 61.25 44.14 32.42 65.85 | 81 110.999 | 9 146.470 257.469 |
0.00 61.25 63.11 32.42 65.85 | 81 110.999 | 9 146.470 257.469 |
0.00 61.25 82.35 32.42 65.85 | 81 110.999 | 9 146.470 257.469 |
13.11 61.25 100.00 32.62 65.85 | 82 110.999 | 9 146.470 257.469 |
63.14 0.00 15.83 32.76 65.85 | 82 112.500 | 9 146.470 258.969 |
63.14 0.00 35.23 32.76 65.85 | 82 112.500 | 9 146.470 258.969 |
63.14 0.00 53.67 32.76 65.85 | 82 112.500 | 9 146.470 258.969 |
63.14 0.00 72.22 32.76 65.85 | 82 112.500 | 9 146.470 258.969 |
63.14 0.00 90.79 32.76 65.85 | 82 112.500 | 9 146.470 258.969 |
0.00 66.16 4.12 33.12 65.85 | 83 114.118 | 9 146.470 260.587 |
0.00 66.16 20.68 33.12 65.85 | 83 114.118 | 9 146.470 260.587 |
0.00 66.16 38.63 33.12 65.85 | 83 114.118 | 9 146.470 260.587 |
0.00 66.16 55.53 33.12 65.85 | 83 114.118 | 9 146.470 260.587 |
0.00 66.16 71.74 33.12 65.85 | 83 114.118 | 9 146.470 260.587 |
0.00 66.16 88.34 33.12 65.85 | 83 114.118 | 9 146.470 260.587 |
16.78 66.16 100.00 33.39 65.85 | 84 114.118 | 9 146.470 260.587 |
64.90 0.00 15.79 33.46 65.85 | 84 116.343 | 9 146.470 262.813 |
64.90 0.00 34.04 33.46 65.85 | 84 116.343 | 9 146.470 262.813 |
64.90 0.00 52.64 33.46 65.85 | 84 116.343 | 9 146.470 262.813 |
64.90 0.00 71.00 33.46 65.85 | 84 116.343 | 9 146.470 262.813 |
64.90 0.00 89.39 33.46 65.85 | 84 116.343 | 9 146.470 262.813 |
64.90 2.99 100.00 33.53 65.86 | 85 116.343 | 9 146.470 262.813 |
64.90 59.92 100.00 33.83 65.86 | 85 116.343 | 9 146.470 262.813 |
0.00 65.54 68.16 33.83 65.87 | 85 123.095 | 9 146.470 269.565 |
0.00 65.54 91.72 33.83 65.89 | 85 123.095 | 9 146.470 269.565 |
100.00 65.54 100.00 36.22 65.89 | 86 123.095 | 9 146.470 269.565 |
100.00 0.00 10.89 37.98 65.89 | 86 128.197 | 9 146.470 274.667 |
100.00 0.00 11.26 37.98 65.89 | 86 128.197 | 9 146.470 274.667 |
100.00 0.00 24.93 37.98 65.89 | 86 128.197 | 9 146.470 274.667 |
100.00 0.00 46.45 37.98 65.89 | 86 128.197 | 9 146.470 274.667 |
100.00 0.00 70.35 37.98 65.90 | 86 128.197 | 9 146.470 274.667 |
100.00 0.00 94.66 37.98 65.90 | 86 128.197 | 9 146.470 274.667 |
100.00 6.07 100.00 40.39 65.90 | 87 128.197 | 9 146.470 274.667 |
100.00 6.07 100.00 43.06 65.90 | 87 128.197 | 9 146.470 274.667 |
0.00 27.56 1.51 43.08 65.90 | 87 142.577 | 9 146.470 289.046 |
0.00 27.56 2.37 43.08 65.90 | 87 142.577 | 9 146.470 289.046 |
0.00 27.56 12.58 43.08 65.90 | 87 142.577 | 9 146.470 289.046 |
0.00 27.56 32.65 43.08 65.90 | 87 142.577 | 9 146.470 289.046 |
0.00 27.56 57.61 43.08 65.90 | 87 142.577 | 9 146.470 289.046 |
0.00 27.56 80.33 43.08 65.90 | 87 142.577 | 9 146.470 289.046 |
0.20 27.56 100.00 43.09 65.90 | 88 142.577 | 9 146.470 289.046 |
11.83 27.56 100.00 43.17 65.90 | 88 142.577 | 9 146.470 289.046 |
27.29 27.56 100.00 43.18 65.90 | 88 142.577 | 9 146.470 289.046 |
28.47 0.00 9.04 43.18 65.90 | 88 153.810 | 9 146.470 300.279 |
28.47 0.00 26.42 43.18 65.90 | 88 153.810 | 9 146.470 300.279 |
28.47 0.00 50.58 43.18 65.90 | 88 153.810 | 9 146.470 300.279 |
28.47 0.00 74.49 43.18 65.90 | 88 153.810 | 9 146.470 300.279 |
28.47 0.00 97.60 43.18 65.90 | 88 153.810 | 9 146.470 300.279 |
28.47 28.86 100.00 43.33 65.90 | 89 153.810 | 9 146.470 300.279 |
0.00 32.47 16.69 43.33 65.90 | 89 158.665 | 9 146.470 305.134 |
0.00 32.47 35.45 43.33 65.90 | 89 158.665 | 9 146.470 305.134 |
0.00 32.47 56.90 43.33 65.90 | 89 158.665 | 9 146.470 305.134 |
0.00 32.47 81.63 43.33 65.90 | 89 158.665 | 9 146.470 305.134 |
15.08 32.47 100.00 43.60 65.90 | 90 158.665 | 9 146.470 305.134 |
31.93 0.00 16.29 43.61 65.90 | 90 161.093 | 9 146.470 307.562 |
31.93 0.00 37.62 43.61 65.90 | 90 161.093 | 9 146.470 307.562 |
31.93 0.00 59.77 43.61 65.90 | 90 161.093 | 9 146.470 307.562 |
31.93 0.00 83.55 43.61 65.90 | 90 161.093 | 9 146.470 307.562 |
31.93 26.08 100.00 43.90 65.90 | 91 161.093 | 9 146.470 307.562 |
0.00 37.88 5.57 43.90 65.90 | 91 165.813 | 9 146.470 312.282 |
0.00 37.88 24.88 43.90 65.90 | 91 165.813 | 9 146.470 312.282 |
0.00 37.88 47.73 43.90 65.90 | 91 165.813 | 9 146.470 312.282 |
0.00 37.88 68.35 43.90 65.90 | 91 165.813 | 9 146.470 312.282 |
0.00 37.88 91.61 43.90 65.90 | 91 165.813 | 9 146.470 312.282 |
36.93 0.00 2.44 44.17 65.90 | 92 168.375 | 9 146.470 314.845 |
36.93 0.00 21.44 44.17 65.90 | 92 168.375 | 9 146.470 314.845 |
36.93 0.00 44.17 44.17 65.90 | 92 168.375 | 9 146.470 314.845 |
36.93 0.00 66.08 44.17 65.90 | 92 168.375 | 9 146.470 314.845 |
36.93 0.00 87.26 44.17 65.90 | 92 168.375 | 9 146.470 314.845 |
36.93 31.53 100.00 44.49 65.90 | 93 168.375 | 9 146.470 314.845 |
0.00 44.95 6.60 44.49 65.90 | 93 173.766 | 9 146.470 320.236 |
0.00 44.95 23.67 44.49 65.90 | 93 173.766 | 9 146.470 320.236 |
0.00 44.95 44.94 44.49 65.90 | 93 173.766 | 9 146.470 320.236 |
0.00 44.95 66.33 44.49 65.90 | 93 173.766 | 9 146.470 320.236 |
0.00 44.95 86.42 44.49 65.90 | 93 173.766 | 9 146.470 320.236 |
30.82 44.95 100.00 44.76 65.90 | 94 173.766 | 9 146.470 320.236 |
43.74 0.00 1.75 44.76 65.90 | 94 179.859 | 9 146.470 326.329 |
43.74 0.00 15.81 44.76 65.90 | 94 179.859 | 9 146.470 326.329 |
43.74 0.00 34.48 44.76 65.90 | 94 179.859 | 9 146.470 326.329 |
43.74 0.00 54.55 44.76 65.90 | 94 179.859 | 9 146.470 326.329 |
43.74 0.00 73.73 44.76 65.90 | 94 179.859 | 9 146.470 326.329 |
43.74 0.00 93.40 44.76 65.90 | 94 179.859 | 9 146.470 326.329 |
43.74 40.26 100.00 45.04 65.90 | 95 179.859 | 9 146.470 326.329 |
0.00 48.50 10.15 45.04 65.90 | 95 184.872 | 9 146.470 331.342 |
0.00 48.50 26.85 45.04 65.90 | 95 184.872 | 9 146.470 331.342 |
0.00 48.50 47.64 45.04 65.90 | 95 184.872 | 9 146.470 331.342 |
0.00 48.50 66.33 45.04 65.90 | 95 184.872 | 9 146.470 331.342 |
0.00 48.50 86.19 45.04 65.90 | 95 184.872 | 9 146.470 331.342 |
42.46 48.50 100.00 45.35 65.90 | 96 184.872 | 9 146.470 331.342 |
52.38 0.00 9.08 45.36 65.90 | 96 188.802 | 9 146.470 335.272 |
52.38 0.00 25.45 45.36 65.90 | 96 188.802 | 9 146.470 335.272 |
52.38 0.00 43.85 45.36 65.90 | 96 188.802 | 9 146.470 335.272 |
52.38 0.00 61.32 45.36 65.90 | 96 188.802 | 9 146.470 335.272 |
52.38 0.00 79.80 45.36 65.90 | 96 188.802 | 9 146.470 335.272 |
52.38 0.00 98.71 45.36 65.90 | 96 188.802 | 9 146.470 335.272 |
52.38 52.75 100.00 45.70 65.90 | 97 188.802 | 9 146.470 335.272 |
0.00 59.32 10.97 45.70 65.90 | 97 194.821 | 9 146.470 341.290 |
0.00 59.32 27.34 45.70 65.90 | 97 194.821 | 9 146.470 341.290 |
0.00 59.32 45.71 45.70 65.90 | 97 194.821 | 9 146.470 341.290 |
0.00 59.32 64.45 45.70 65.90 | 97 194.821 | 9 146.470 341.290 |
0.00 59.32 83.43 45.70 65.90 | 97 194.821 | 9 146.470 341.290 |
49.45 59.32 100.00 46.04 65.90 | 98 194.821 | 9 146.470 341.290 |
61.24 0.00 8.65 46.05 65.90 | 98 197.942 | 9 146.470 344.411 |
61.24 0.00 24.37 46.05 65.90 | 98 197.942 | 9 146.470 344.411 |
61.24 0.00 41.42 46.05 65.90 | 98 197.942 | 9 146.470 344.411 |
61.24 0.00 59.77 46.05 65.90 | 98 197.942 | 9 146.470 344.411 |
61.24 0.00 76.74 46.05 65.90 | 98 197.942 | 9 146.470 344.411 |
61.24 0.00 93.93 46.05 65.90 | 98 197.942 | 9 146.470 344.411 |
61.24 57.66 100.00 46.43 65.90 | 99 197.942 | 9 146.470 344.411 |
0.00 68.58 6.75 46.43 65.90 | 99 203.920 | 9 146.470 350.390 |
0.00 68.58 20.73 46.43 65.90 | 99 203.920 | 9 146.470 350.390 |
0.00 68.58 39.03 46.43 65.90 | 99 203.920 | 9 146.470 350.390 |
0.00 68.58 56.10 46.43 65.90 | 99 203.920 | 9 146.470 350.390 |
0.00 68.58 74.40 46.43 65.90 | 99 203.920 | 9 146.470 350.390 |
0.00 68.58 92.16 46.43 65.90 | 99 203.920 | 9 146.470 350.390 |
67.38 0.00 3.71 46.78 65.90 | 100 205.734 | 9 146.470 352.203 |
67.38 0.00 18.99 46.78 65.90 | 100 205.734 | 9 146.470 352.203 |
67.38 0.00 35.91 46.78 65.90 | 100 205.734 | 9 146.470 352.203 |
67.38 0.00 47.32 46.78 65.91 | 100 205.734 | 9 146.470 352.203 |
67.38 0.00 63.85 46.78 65.92 | 100 205.734 | 9 146.470 352.203 |
Thanks for any recommendations you may have!