Did you ever come to a conclusion on this? I am using BRMS 5.3.1 with EAP 6.0.0.GA and I have received this error twice during about a month of development. It is not as consistent or repeatable as yours, however I am not calling rules as frequent - I am doing a lot of deploy/undeploy of my war, testing other aspects of it.
I am interested in what you have learned so far.
I would recommend enabling JMX on your JVM , then start JVisualVM and monitor the permgen space while your Java process starts.
Yeah i monitor the permgen space and did some research.. After that also i got that error.
I resolved this error by doing following things:
1) First measured the response time .
2 ) The knowledge agent should be cached and reused every time it is required. In your current set up, it is compiling "source code" which in this case is the spreadsheets.
3) I created Singleton class for knowledgeagent and cached and reusing the kagent.
Analyze the Heap dump and thread dump that would help you troubleshoot whether there is large number of block threads.