Problem with compilation of dynamically generated JSPs on JBoss WildFly

[mortenheine]

We have this application with a bunch of JSPs. Some are static, some are generated dynamically.

The application is packed up as an ear file that includes a war file. That war file includes the static JSPs.

The ear file is dumped in deployments and JBoss deploys the ear file OK, including the war file, and in turn including the JSPs.

We see the following generated directories:

1) tmp/vfs/deployment/deploymentxxxxxxxx/<myWarFile>.war-yyyyyyyyy/

2) tmp/<myEarFile>.ear.<myWarFile>.war/org/apache/jsp/

The static JSP-files appear in directory 1).

When we access the static JSP-files via a browser, the corresponding java files are automatically generated and compiled and the result resides in directory 2).

Along the way, some dynamic JSP-files are generated and placed in directory 1). Yes, the internal directory that JBoss has generated. This is an application that was written some time ago and has apparently worked fine on an earlier JBoss AS version.

However, when we access these dynamic JSPs from a browser, JBoss returns an error that the file is not found, instead of generating the java file and compiling it:

Unhandled exception occurred whilst decorating page: java.lang.RuntimeException: javax.servlet.ServletException: JBWEB004036: File "<myJSP>.jspx" not found
       at com.opensymphony.sitemesh.webapp.decorator.BaseWebAppDecorator.render(BaseWebAppDecorator.java:40)
       at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:83)
       at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
       at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
       at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareFilter.doFilter(StrutsPrepareFilter.java:91)
       at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
       at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

It is as if JBoss internally knows that it must compile the static JSPs that were there from the beginning, but does not know it must compile the dynamic JSPs.  Or for some reason it does not see the JSP even if it is there.

Sometimes Jboss after a while (that appears to have random duration) decides to compile the dynamic JSPs when I hit them. Sometimes it seems to make a difference that I restart the server, other times not. After hours of experimenting, it still seems totally random whether Jboss compiles the dynamic JSPs or not.

Of course, we have experimented with the jsp-configuration element in the configuration, but it seems to make no difference. We assume this configuration only applies to the deployments directory, not these internal directories.

Of course, we have considered exploded deployment, but we understand that this is not intended for production, plus we are running a cluster in production and in this case, exploded deployment is not possible, as far as we understand. In our test setup, we're running a single server.

The only way out seems to be to avoid the generation of dynamic JSPs. Those JSPs are a mixture of HTML and java code. But probably the java code can be executed at generation time instead of when the JPSs are hit. However, this is still a significant rewrite of the code, and we would prefer to keep the existing code.

Any suggestions would be highly appreciated.

[ctomc]

lets get one thing right first, *ALL* JSPs are dynamic, no matter how you look at them or what their content is.

they all need to be compiled into "servlet" so they can be served.

You did quite some analysis but you don't provide us with info that is really needed to help you.

1) what is exact version of WildFly you are using

2) what is the structure of your war and where is your jsp / jsp file located

3) is there anything special in the content of the jsp? any includes

4) how do you access this jsp file? via some filters, includes directly?

5) is there some caching going on?

6) web.xml of your application, that shows servlets & filters configured.

....

If not already did you try with latest WildFly 10 release, I ask as it has much newer jsp compiler included.

[jaikiran]

In addition to what Tomaz noted, how exactly are you generating those JSP files in first place? Can you post the relevant piece of code that does that and places it in those (temp?) folders?

[mortenheine]

Answers to the questions:

1) WildFly Full 10.0.0.Final

2) The structure looks standard. The war contains these directories/files
- META-INF
- WEB-INF/web.xml
- WEB-INF/classes
- WEB-INF/lib
- all the pre-generated (“static”) JSPs located in the root of the war file plus some in a subdirectory.

3) Nothing special that I can see, no includes.

4) I hit a static page A, submit and get another static page B back. Here comes the tricky part. Page B includes yet another page. Depending on a form parameter, this last page can be either a static page C or a dynamic page D. If the choice is the static page C, everything works. Page C gets compiled and I see the class files in the place I have mentioned. If the choice is the dynamic page D, the page is generated and placed in the same temporary folder as the static Page C is located. However, the dynamic Page D is usually not compiled and I get the error message. But sometimes it works.

Both C and D are referred in the same way. Page B includes a struts tag
<s:property value="methodCall" escape="false" />
and the methodCall returns this fragment
"<meta name=\"<myName>\" content='" + <myFileName> + "' />"

The only difference between the two cases is that in one case the JSP exists already, whereas in the other the JSP is created and written to the same folder.

5) I don't think so. Besides, it is totally random when the dynamic page works or not. I cannot correlate with server start, timing, or anything else.

6) It contains some filters and filter mappings plus welcome-file-list (just index.html) and <distributable />.

Nothing that I think can have a bearing of our problem.

About version: we are using WildFly 10 release, already.

The dynamic JSPs are created this way:

File jspFile = new File(getServletContext().getRealPath(fileName));

Subsequently the contents is written in a way that should not matter. Note that the JSP is always created fine. The problems it that it is not found and compiled by JBoss.

[ctomc]

The dynamic JSPs are created this way:

File jspFile = new File(getServletContext().getRealPath(fileName));

Subsequently the contents is written in a way that should not matter. Note that the JSP is always created fine. The problems it that it is not found and compiled by JBoss.

Ok, now I see why you mean by "dynamic" jsp pages. you are generating JSPs on the fly which server than compiles and processes.

I think you are hitting jsp compiling cache. Really odd one, but still.

You should probably enable development mode for jsps in undertow subsystem, which should fix this problem.

[mortenheine]

About dynamic: yes :-).

As alluded to in the first post, we already tried replacing <jsp-configuration/> with <jsp-configuration development="true" check-interval="1" modification-test-interval="1" recompile-on-fail="true"/> in the configuration of undertow. It did not solve the problem.

I think this configuration tells JBoss when to scan the deployments directory. In contrast, our application write the dynamic JSP-files to tmp/vfs/deployment/deploymentxxxxxxxx/<myWarFile>.war-yyyyyyyyy/.

Anyway, I doubt that a cache can be the problem, because the main scenario is that we are putting a NEW JSP in the directory. The  cache cannot have an entry for a JSP it has not seen before.

[ctomc]

Morten Heine Sørensen wrote:

I think this configuration tells JBoss when to scan the deployments directory. In contrast, our application write the dynamic JSP-files to tmp/vfs/deployment/deploymentxxxxxxxx/<myWarFile>.war-yyyyyyyyy/.

File notification change listener doesn't listen in this directory at all.

Only way you will get this working properly, you would need to have exploded deployment in "deployments" folder which doesn't use vfs expanding to tmp folder.

And if you would generate JSP files in that folder it should work just fine.

[mortenheine]

By "this" I assume you mean tmp/vfs/deployment etc. I agree that the File listener does not listen to this directory (at least this is my expectation). The strange thing is, however, that it sometimes works.

As alluded to in the original post, we have considered exploded deployment.

There are two reasons we have not gone down that path, a hard one and a soft one.

1) We are running a cluster in production, and as far as we understand, exploded deployment does not work with clusters. See, e.g. 1.4.3. Farming Deployment. I'm not sure i WildFly is different from earlier versions in this respect.

2) Indeed, note that the dynamic JSPs are generated in production, not just development, and we think it may be too exotic to trigger a real deployment during production bsaed on a user action.

We have considered the possibility to explicitly call the JSP-compiler when we place a new JSP in the directory, but we would really like JBoss to handle this for us, if we continue to use this overall approach of generating dynamic JSPs.

It sounds like a more plausible way forward to change the basic approach so that we do not need to generate these dynamic JSPs.

[ctomc]

1) We are running a cluster in production, and as far as we understand, exploded deployment does not work with clusters. See, e.g. 1.4.3. Farming Deployment. I'm not sure i WildFly is different from earlier versions in this respect.

Well, it is possible but we don't really advertise it too much.

You would need to use cli's deploy command with option --unmanaged and it would work.

Why we don't really promote that approach is that in domain mode when using unmanaged deployments, you would need to take care of "distributing" deployment yourself around different servers. Either by some script or using shared drive, etc..

We have considered the possibility to explicitly call the JSP-compiler when we place a new JSP in the directory, but we would really like JBoss to handle this for us, if we continue to use this overall approach of generating dynamic JSPs.

It sounds like a more plausible way forward to change the basic approach so that we do not need to generate these dynamic JSPs.

I would recommend you to do this, basically not generating jsp files on the fly.

What could probably be better is if you really need to generate something, is to generate something like xhtml (facelets) template if you still need some more markup or go with freemarker or something similar.

with facelets you can still bind to standard jsf / cdi components and use them when rendering.

And providing custom "locator" for this files is quite simple, they could even be in-memmory.

Problem with JSPs in general is that they get "transformed" or better said server generates servlet java file which is than compiled to class file and than loaded by server and served. In short all jsps end up being just servlets (with some complex logic that maps stacktraces / errors back to original .jsp fle).

If you would go with some other templating engine, most of this steps are not required, they would just be executed and that is it.

[mortenheine]

FreeMarker looks like what we had in mind - thanks for the pointer and all the help in general .