Can you try with 5.3.0.Final ?
Just tried with 5.3.0.Final and the result is the same.
Compatibility between hotrod and REST seems fine, but memcached seems wrong.
there was recently added a default marshaller (JavaSerializationMarshaller) used by Memcached clients: https://github.com/infinispan/infinispan/commit/39ef40bc33c58b74d0597e9f4cb948549db8aaf4. Before that, users always had to specify their own marshaller.
So what you get by calling "get hotrod" through telnet is actually a Java object. If you really want to get a String value through this call, you need to register your own marshaller, similarly to the test attached with the commit above. There's StringMarshaller that handles the marshalling the way you want and creates the resulting string for you.
How to register your own marshaller? Look into current XSD schema.