fetching that much stuff is annoying. Everyone i know is overstrained when there are more than 50 records :-)
Anyway, the datascroller from RichFaces gets his data about list size and so on from the datatable. You could create an own proxy-component that intercepts the datascroller-calls (Some sort of MyHtmlDataTable extending by subclassing OR HtmlDataTableProxy for using HtmlDataTable as delegate). When you reach your preloaded records, you easily fire some events to your controller to get next records.
The only problem i see is, when you want to use sorting-feature from rich:dataTable, because not all data is available for sorting.