Old VFS files being read.

[richard.hart]

I am using JBoss AS 7.1.1.Final. I do my development on a Windows 7 system and deploy to a Linux server. We are using Jenkins to do builds and deployments.

We are using Drools and we read drools rules files that have been included in the deployment in a war file's classes/rules directory.

On my local Windows JBoss everything works fine. In fact, I do not have to use VFS at all to read the drools rules files out of the deployment.

However, on the Linux server, VFS must be used.

The problem we are having is that on the Linux server, after a build and deploy, everything works fine. Then all of a sudden, the application starts reading old drools rules files that no longer compile. These old files were used at the very beginning of the project and have long ago been removed from the project.

The utility class file used to read and compile the drools rules files is a singleton class so that once the rules have been read and compiled, the knowledge base is kept around for reuse by every user. The application will work for a period of time, but I see from our log file that for some unkown reason, JBoss apparently reloads the singleton class and the initialization of the class begins by reading the drools rules files but this time through, the path in the tmp directory resolved by VFS has changed from the first time the class was initialized resulting in an old rule files being read.

I am at a loss as to the reason why the VFS class resolves the path to an old one. What are we doing to cause this to happen? Is it because of hot deployments using Jenkins? I get that idea from a different thread EAR deployment uses old jar file.

[jaikiran]

Richard Hart wrote:

The application will work for a period of time, but I see from our log file that for some unkown reason, JBoss apparently reloads the singleton class and the initialization of the class begins by reading the drools rules files but this time through, the path in the tmp directory resolved by VFS has changed from the first time the class was initialized resulting in an old rule files being read.

Perhaps in the context of a different classloader? Remember, singletons aren't really singletons per JVM but they are per classloader.

I am at a loss as to the reason why the VFS class resolves the path to an old one

When you say VFS, are you explicitly using any VFS API within the application? If yes, which one and what does that code look like?

[richard.hart]

Here is the code used to initialize the drools knowledgebase. This class (DroolsUtil) is a singleton.

    public void initializeKnowledgebase() {
        final KnowledgeBuilder kbuilder = KnowledgeBuilderFactory
                .newKnowledgeBuilder();
        List listOfFiles = new ArrayList();
        try {
            listOfFiles = getListOfFiles();
        } catch (final IOException | URISyntaxException e) {
            log.error("Unexpected Exception reading drools resource files.",
                    e);
        }
        for (final File file : listOfFiles) {
            final String[] tokens = file.getName().split("\\.(?=[^\\.]+$)");
            log.info("reading file " + file.toURI().getPath());
            try (Reader reader = new FileReader(file)) {
                final Resource myResource = ResourceFactory
                        .newReaderResource(reader);
                switch (tokens[1].toLowerCase()) {
                case "bpmn2":
                    kbuilder.add(myResource, ResourceType.BPMN2);
                    break;
                case "brl":
                    kbuilder.add(myResource, ResourceType.BRL);
                    break;
                case "change_set":
                    kbuilder.add(myResource, ResourceType.CHANGE_SET);
                    break;
                case "descr":
                    kbuilder.add(myResource, ResourceType.DESCR);
                    break;
                case "drf":
                    kbuilder.add(myResource, ResourceType.DRF);
                    break;
                case "drl":
                    kbuilder.add(myResource, ResourceType.DRL);
                    break;
                case "dsl":
                    kbuilder.add(myResource, ResourceType.DSL);
                    break;
                case "dslr":
                    kbuilder.add(myResource, ResourceType.DSLR);
                    break;
                case "dtable":
                    kbuilder.add(myResource, ResourceType.DTABLE);
                    break;
                case "pkg":
                    kbuilder.add(myResource, ResourceType.PKG);
                    break;
                case "pmml":
                    kbuilder.add(myResource, ResourceType.PMML);
                    break;
                case "xdrl":
                    kbuilder.add(myResource, ResourceType.XDRL);
                    break;
                case "xsd":
                    kbuilder.add(myResource, ResourceType.XSD);
                    break;
                default:
                    log.error("Invalid file extention '" + tokens[1]
                            + "' for Drools ResourceType. File "
                            + file.getAbsolutePath()
                            + " not loaded into KnowledgeBase.");
                    break;
                }
            } catch (final IOException e) {
                log.warn("Unexpected IOException.", e);
                ; // intentionally left blank. Potentially thrown on close
                  // of
                  // reader.
            }
        }
        final KnowledgeBuilderErrors errors = kbuilder.getErrors();
        if (errors.size() > 0) {
            final StringBuilder errorMsgBuilder = new StringBuilder(
                    "Could not parse knowledge.\n");
            for (final KnowledgeBuilderError error : errors) {
                log.error("KnowledgeBuilderError:\n" + error.getMessage());
                errorMsgBuilder.append(error.getMessage() + "\n");
            }
            throw new IllegalArgumentException(errorMsgBuilder.toString());
        }
        internKnowledgeBase = KnowledgeBaseFactory.newKnowledgeBase();
        internKnowledgeBase.addKnowledgePackages(kbuilder
                .getKnowledgePackages());
    }

This is the getListOfFiles method called above. This is the method that can use VFS. VFS is used only if the files cannot be found using standard Java classes. Only on the Linux system is VFS used. This method is in the FileUtil class that is NOT a singleton, just a static method in the class.

    public static List getListOfFiles(String path) throws IOException, URISyntaxException {
        final List listOfFiles = new ArrayList();
        URI classPath = null;
        if (DroolsUtil.RULES_PATH.equals(path)) {
            classPath = FileUtil.class.getResource("/" + path).toURI();
        } else {
            classPath = new URI(path);
        }
        log.info("classPath = " + classPath);
        URL url = classPath.toURL(); //Thread.currentThread().getContextClassLoader().getResource(path);
        final File file = new File(url.getFile());
        final File[] fileArray = file.listFiles();

        if (fileArray != null) {
            for (int i = 0; i < fileArray.length; i++) {
                listOfFiles.add(fileArray[i]);
            }
        } else {
            log.info("No files found at " + url.getFile()
                    + " Trying with VFS.");
            try {
                final URI uri = url.toURI();
                final VirtualFile vf = VFS.getChild(uri);
                if (vf.isDirectory()) {
                    final List vFiles = vf.getChildren();
                    for (final VirtualFile tempVf : vFiles) {
                        final File vfile = tempVf.getPhysicalFile();
                        listOfFiles.add(vfile);
                    }
                    if (listOfFiles.size() == 0) {
                        throw new IllegalArgumentException("No files found using VFS at "
                                + uri.getPath());
                    }
                    return listOfFiles;
                }
            } catch (final URISyntaxException e) {
                log.error("Unexpected URISyntaxError", e);
            }
        }

        return listOfFiles;
    }

I was previously using Thread.currentThread().getContextClassLoader().getResource(path) but I changed to using the FileUtil (non-singleton) classloader. This has not fixed the issue.

In this case DroolsUtil.RULES_PATH does equal path which is "rules" which is the directory under the deployment classes directory.

I should be using VFS3. The imports are:

import org.jboss.vfs.VFS;
import org.jboss.vfs.VirtualFile;

And here is an example from the log file: It starts good then some time later, initialization of the singleton happens again pointing to old rule files. No builds, no deployments, nothing other than using the application happened between the two initializations. I have no idea why JBoss needs to reload the DroolsUtil singleton.

2014-04-03/13:51:23 [com.pwc.fdb.utils.FileUtil] INFO  classPath = vfs:/content/voyager.war/WEB-INF/classes/rules/
2014-04-03/13:51:23 [com.pwc.fdb.utils.FileUtil] INFO  No files found at /content/voyager.war/WEB-INF/classes/rules/ Trying with VFS.
2014-04-03/13:51:23 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-9388a94e003f8800/WEB-INF/classes/rules/CoACalcsExcludingTime.drl
2014-04-03/13:51:26 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-9388a94e003f8800/WEB-INF/classes/rules/FdbRuleFlowMaster.bpmn2
2014-04-03/13:51:28 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-9388a94e003f8800/WEB-INF/classes/rules/TestRuleFlow.drl
.
.
.
2014-04-03/16:20:23 [com.pwc.fdb.utils.FileUtil] INFO  classPath = vfs:/content/voyager.war/WEB-INF/classes/rules/
2014-04-03/16:20:23 [com.pwc.fdb.utils.FileUtil] INFO  No files found at /content/voyager.war/WEB-INF/classes/rules/ Trying with VFS.
2014-04-03/16:20:23 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-81e2935cd45e3fc1/WEB-INF/classes/rules/CoACalcsExcludingTime.drl
2014-04-03/16:20:25 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-81e2935cd45e3fc1/WEB-INF/classes/rules/FdbRuleFlowMaster.bpmn2
2014-04-03/16:20:26 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-81e2935cd45e3fc1/WEB-INF/classes/rules/README.TXT
2014-04-03/16:20:26 [com.pwc.fdb.utils.DroolsUtil] ERROR Invalid file extention 'TXT' for Drools ResourceType. File /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-81e2935cd45e3fc1/WEB-INF/classes/rules/README.TXT not loaded into KnowledgeBase.
2014-04-03/16:20:26 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-81e2935cd45e3fc1/WEB-INF/classes/rules/test.drl
2014-04-03/16:20:27 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-81e2935cd45e3fc1/WEB-INF/classes/rules/Sample.drl
2014-04-03/16:20:28 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-81e2935cd45e3fc1/WEB-INF/classes/rules/sample rules.drl
2014-04-03/16:20:28 [com.pwc.fdb.utils.DroolsUtil] INFO  reading file /opt/jboss/standalone/tmp/vfs/temp2fda10626e64d1ff/voyager.war-81e2935cd45e3fc1/WEB-INF/classes/rules/TestRuleFlow.drl
2014-04-03/16:20:28 [com.pwc.fdb.utils.DroolsUtil] ERROR KnowledgeBuilderError:
Unable to create Field Extractor for 'jurisdiction'Field/method 'jurisdiction' not found for class 'com.pwc.fdb.beans.Entities'

[jaikiran]

I have no idea why JBoss needs to reload the DroolsUtil singleton.

Whatever code is using the DroolsUtil seems to be using it in a classloader in which the DroolsUtil hasn't yet been initialized. If you have access to the DroolsUtil class, just add a few debug kind statements like:

Thread.dumpstacktrace(); // to see the code path

System.out.println("Classloader of DroolsUtil is: " + this.getClass().getClassLoader());

in the initializeKnowledgebase method of that class.

[richard.hart]

Thank you for your response.

I am starting to see a pattern in the log files. As I mentioned, we use Jenkins to build and deploy. I am beginning to think the issue revolves around the use of Jenkins. After doing a manual build, everything works fine. However, after some time, old drools rules files are read. I look in the log file and find this entry:

14:50:43,767 INFO  [hudson.model.Run] (Executor #0 for master : executing vger #132) vger #132 main build action completed: UNSTABLE

14:50:47,654 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/voyager]] (MSC service thread 1-1) Cleaning up Shiro Environment

14:50:48,145 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment voyager.war in 498ms

14:50:48,149 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "voyager.war"

Thereafter old drools rules files are read until a manual build is performed.

I have added your suggested log entries. Nothing unusual is found with threads. I went so far as to make the class no longer a singleton but still have the same issue.

[richard.hart]

I have the issue solved now. It was a Jenkins configuration I had nothing to do with that was incomplete for a test project that was building a voyager.war file. It was not fetching from Git repository and so was building a war file with old data.

Thank you for your assistance.