I haven't worked with any projects that have gone through this transition/migration, but I know both the jBPM 3 and 5 code (or at least, they end up fighting for space in my brain.. :/ ).
If I were planning an upgrade, I would do the following:
1. Do a "prepratory" migrade of your jBPM 3 process definitions to "BPMN2 friendly" process definitions. That means essentially that for all of those extended ActionHandler (or DecisionHandler, AssignmentHandler, etc. ) classes in your process definitions and PAR's, you should migrate as much of the (java-code) logic in any of those handlers to the JPDL. That's really the biggest difference between JPDL/jBPM 3 and BPMN2/JBPM 5: the first is much more developer oriented, where BPMN2/JBPM 5 forces users to be write more transparent business processes. This step will mean that you will be able to migrate more of your jBPM 3 processes to BPMN2 (jBPM 5).
2. Once you've gotten this done, I would look at Eric Schabell's jbpmmigration project: this is available in the jBPM (5) Designer. What's designer? Designer is the web-based BPMN2 editor, which you can access via Guvnor. Yes, we're working on making Designer standalone as quickly as possible, but it's not there yet. But, in short, this is what you do:
a. Download a jBoss AS (preferably AS 7, since it's SOOOOOO much faster.)
b. Download a Guvnor AS7 compatible war (like 5.4.0.Final, jboss-as-7.0.war: https://repository.jboss.org/nexus/index.html#nexus-search;gav~org.drools~guvnor-distribution-wars~5.4.0.Final~~ (click on the jboss-as-7.0.war link).
c. Download a Designer AS7 compatible war (like 2.4.0.Beta1, jboss7.war: https://repository.jboss.org/nexus/index.html#nexus-search;quick~jbpm-designer (click on jboss7.war link.. )
d. Add wars to standalone/deployment (and touch <war-name>.dodeploy)
e. Start server up
f. Open guvnor: and create a new BPMN process
g. Use jbpmmigration capability in Designer
You can also use the jbpmmigration capability outside of designer, I'll have to ping Eric about that.
3. Once you've converted your process(es), I would indeed run both in parallel, in two different ways:
a. Obviously, you have to let the jBPM 3 processes finish, and start using the JBPM 5 processes. There's no other way to do that except have both simultaneously running in production.
b. I'm assuming you have (D)TAP environments? (Devel, Test, Accept, Production). In that case, depending on the size of your system(s) and how critical the systems are, it might be worth it to setup some sort of parallel processing, where production handles the jBPM 3 processes and accept _also_ handles the same process instance (request), but with jBPM 5. Depending on the number of (Human) tasks involved, this would of course require some sort of infrastructure so that both jBPM 3 tasks and jBPM 5 (human) tasks are completed/handled by the same user interaction. (Hopefully, that makes sense? If not, please ask).
Re the jbpm migration project:
By the way, it appears that the jbpm migration integration with designer is minimal -- it looks like you're better off using the jbpmmigration project outside of designer, and then importing the bpmn2 results into Designer.
Obviously, bugs are always appreciated, if you should run into them (for the migration project): https://issues.jboss.org/browse/JBPM ("migration" component).