-
1. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
ctomc Jun 12, 2017 10:17 AM (in response to caro82)Well solaris JVM & NIO are notoriously buggy, so first step would be to upgrade JDK to latest (u131 atm)
and than we can look in other options.
Also upgrading WildFly to 10.1 would not hurt
-
2. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
caro82 Jun 14, 2017 8:56 AM (in response to ctomc)Besides the upgrade, do you have any recommendations on how to tackle/investigate this? The issue appeared over night, without any modifications on the application itself.
-
3. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
ptyagi_redhat.com Jun 19, 2017 12:48 PM (in response to caro82)You may try increasing direct buffer memory using -XX:MaxDirectMemorySize=512m JVM option. However, consider to upgrade to JDK to latest update and Wildfly too.
-
4. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
caro82 Jun 26, 2017 5:43 AM (in response to ptyagi_redhat.com)MaxDirectMemorySize is not set at this point. How can I determine its default value - the value the my app is using right now?
-
5. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
claudio4j Jun 26, 2017 8:07 AM (in response to caro82)You may use jinfo as in the example:
jinfo -flag MaxDirectMemorySize <PID>
-
6. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
caro82 Jun 26, 2017 8:19 AM (in response to claudio4j)Given that the flag is not set, the jinfo comand has the following output:
myzone# jinfo -flag MaxDirectMemorySize PID
-XX:MaxDirectMemorySize=0
What is the maximum size of direct memory in this case?
-
7. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
claudio4j Jun 26, 2017 8:59 AM (in response to caro82)Accordingly to this dzone article, you can use a java api to print the max direct memory, the value is in bytes.
You can use the following to display the value as MB
System.out.println(sun.misc.VM.maxDirectMemory()/1024/1024);
-
8. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
caro82 Jun 26, 2017 9:21 AM (in response to claudio4j)This is the value I get printing sun.misc.VM.maxDirectMemory(): 24811929600 in bytes. This is aprox 23 Gb. This means setting it to 512Mb would make no sense. I am right? Are there any mbeans or commands that would indicate the usage of this memory? Is this process specific?
-
9. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
caro82 Jul 3, 2017 10:23 AM (in response to caro82)We manage to print the usage of Direct Memory by using the Java.nio BufferPool mbean attributes. The usage reaches around 1.5Gb when the OutOfMemory errors occur. We reproduced the issue even after a jdk upgrade to 1.8 update 131 and Wildfly to10.1.0. Any hints on how we should continue to tackle this?
Note: the issue does not occur in a similar environment where the number of processors is 64 (instead of 256). This difference in no of processors triggers a difference in number of threads (io workers). We will run a test with a decreased number of threads (io workers).
-
10. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
ctomc Jul 3, 2017 1:24 PM (in response to caro82)yeah, tuning IO subsystem worker pools would be a proper approach I think.
As default config calculates number of threads and buffers based on available CPUs.
And in case of such huge number of CPUs default formula would not be appropriate anymore.
-
11. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
caro82 Sep 4, 2017 8:44 AM (in response to ctomc)We decreased the number of IO workers from 4k to 1k and the issue did not reoccurred.
-
12. Re: Wildfly 9.0.1, Java 8, solaris sparc java.lang.OutOfMemoryError: Direct buffer memory errors
ctomc Sep 4, 2017 1:12 PM (in response to caro82)You could probably go even much lower, unless you have huge load on the server (more than 500K req/sec) you should be fine with 256 worker threads.