It's not necessary to fetch the entire data for the rich:datascroller component. You only need to extend an appropriate data model - for instance the ExtendedDataModel.
You may follow up with:
The solution about how to do correct pagination using datascroller (load a part of data from database) can be found on the RichFaces Users Forum
from: Richfaces Developer Guide (datascroller, 6.33.10. Relevant Resources Links)