I wouldn't recommend 5.x series, now that AS 7.1.1 has been released. However if you still want to stick with AS5, then 5.1.0 is the latest stable in that series.
It depends on your application.
If you have an EJB3 application which is near standard it should be easy to migrate to AS7.1.1
If you use EJB2.1 it might be more complex, also if you use special features of JBoss4.
From my experience AS7.1 is easier to handle (apart from other nice things) and also smart from developer perspective hot startup times minimal roundtrips for test.
Also with AS5 you will not have bugfixes and might upgrade to AS7 very soon.