getting OOM error while executing rules drools 6.5 version

[sudhishvnair]

I am getting OOm error during rule execution. Following are the configuration and software details.

1. Tomcat application server with 48 GB RAM running on centos
2. Drools version 6.5.0 final
3. Java 1.8

I am population data to WM on a quarter wise and one type of fact per quarter 200000 avg and another fact type is around 12 million per quarter.

I am processing the big data set one million at a time. I read one million record from file, insert to working memory, execute the rules then retract one million records. This process will continue until all the records are processed. The first fact type remains in the working memory.

I was able to process 40 quarters of data using a 24GB RAM VM using drools 5.x. After moving to drools 6.x with the same set of rules I am not able to run the rules for more than 4 quarters. I increased the RAM to 48 GB still getting OOM.

I am not sure what went wrong. I think I am missing some configuration or something like that.

Looking forward to some help from you guys.

[nramos95]

Hi,

Ok for RAM upgrading but what about your JVM configuration ? What about heap size of your tomcat ? Did you configured for example MaxMetaSpaceSize, gc type ... ? Is there a heapdump generated ? What about gc.log ?

[sudhishvnair]

Hi Nicolas,

         Thanks for your reply. Please find the details below.

Xms 48000 and Xmx 48000

I have not configured te MaxMetaSpaceSize and gc type. Would you please explain how can I add these flags?

There are no heap dump created as the tomcat service is getting terminated by the "kill" command sometime with flag 3 and sometime with 9

[nramos95]

Hi,

So 48GB is your heap size ? So huge one !!! I think you should not increase the heap until you know how your JVM is working : you have differents types of gc (ConcurrentMarkSweep, ParallelGC, G1GC etc ..), differents flags for tuning JVM. All these parameters should increase or decrease your JVM performance : you need to choose the more pertinents.

Do you have gc.log ? it could help you ! Enabling garbage collection logging with Tomcat on a Linux/UNIX platform (2009621) – All Help & Support

Perhaps you have to link JConsole, VisualVM, JProfiler or an APM (Introscope, Appd, Dynatrace) on your JVM to understand why it fails in OOM (Leak Memory).

You say you process about 1 million records from file : what about fetching this data ?

[sudhishvnair]

Hi Nicolas,

    I used G1GC garbage collector and it is working fine now from the memory point of view. Thanks a ton for your suggestions it helped me a lot. I used VisualVM to check the memory usage and it helped me a lot.

Regarding processing file, we are using a batch approach and reading files in chunks of one million. This then parsed and inserted to the working memory. This works fine and have a better performance than compared to 5.x.

I have one more question for you. I have a simple rule which is given below.

rule "Add to learning category from Loan Performance facts"

salience 1590

    when

        lp : LoanPerformance(lid : loanIdentifier)

        la : LoanAcquisitionRecord(loanIdentifier == lid, src : source)

then

        String status = getLoanStatus(lp);

        String [] data = src.split("\\|",src.length());

        String [] sample = new String[]{

            data[1],data[2], data[3],

            data[4],data[5],data[6],

            data[8],data[9], data[10],

            data[11],getRangeValue(data[12]),data[13],

            data[14],data[15],data[16],

            data[17],data[18],data[19],

            data[20],data[21],data[22],

            data[23],data[24],           

            lp.getServiceName(),lp.getCurrentInterestRate().toString(),

            lp.getCurrentActualUPB().toString(),lp.getLoanAge().toString(),

            String.valueOf(lp.getAdjusterRemainingMonthToMaturity()),

            lp.getMetropolitanStatArea(),status};

            BayesianService.addSample("loan_performance",sample,false);

This rule was taking roughly one minute to execute one million times when we were using 5.x and now with 6.x it is taking 3 minutes and 30 seconds. Will you be able to help me out in this?

Thanks a lot for your help and prompt reply.

[nramos95]

Hi,

I can't help you anymore on rules : but have you identify which code is taking too much time ? Is it the creation time of the Array String "sample" or is it the BayesianService addSample method ? Can you use Timestamp flag in this code ? Try to work with dichotomy ...

[sudhishvnair]

Hi Nicolas,

      I think I found out the issue. During my earlier run I noticed that the time taken for GC was very high. I am now fine tuning the JVM performance and now the time for rule firing is coming down.

Thanks for your suggestions and help. It was very useful and helpful.