It turns out that I had set the work-dir in a context.xml file a while back. This was causing JBoss to "clean up" my classes directory on sutdown. For some reason JBoss 3.2.6 was not doing this before.
My reason for setting the working directory to deploy/MyApp/WEB-INF/classes in the first place was to make use of some precompiled JSPs generated during my build. I got around this by changing the scratch directory in Tomcat's web.xml file.
Hope this helps someone down the road :)