You can't search binary data per se. Instead, you configure/enable the Tika text extractor in your configuration: Tika text extractor - ModeShape 4 - Project Documentation Editor and then you issue your full text search queries against the node which contains the binary property (Full text search - ModeShape 4 - Project Documentation Editor).
e.g. select [jcr:path] from [nt:resource] as res where contains(res.*, 'ModeShape supports')
You should also be aware that in ModeShape 4.0 and the upcoming 4.1 FTS is limited & not optimized atm. (it's basically using regexes for pattern matching). This will change once we implement the Lucene Index provider which will support "real" FTS.