Figured this out. Again, this is for JBoss 4.2.2, not sure if it applies to any other JBoss versions.
In my web.xml, I added: <distributable>true</distributable>
In my jboss-web.deployer/context.xml, I added:
<Manager pathname="SESSIONS.ser" />
(Removing the existing Manager node with an empty pathname attribute)
Now when JBoss restarts, I see the SESSIONS.ser, and sessions persist across the restart.
Obviously this means that everything that goes into the session needs to be serializable.