Caching for connectors and regular JCR nodes really only works at a workspace level (via the workspace cache as you noticed). However, a workspace cache is cleared each time a session.save is performed for that workspace (or when dealing with system sessions in some cases like versioning).
So regular nodes and "external nodes" (coming from connectors) are only cached for the duration of a JCR session and evicted once the session is saved. The "cacheTTL" setting is ISPN-specific and its only purpose is to prevent stale reads - i.e. to be able to evict external nodes earlier than a session.save since these external nodes are outside the control of ModeShape. You cannot use this setting in "reverse" - to enforce a caching time "past" session.save call. The workspace cache has to be evicted on each session.save to ensure correct isolation behavior (in a transactional sense) between JCR sessions.
When dealing with your own custom connector, the only way to get the caching behavior you want is to implement it at a connector level.
Just out of curiosity; would there be an major impact if I was to use ModeShape's Environment to get a cache container I defined in the infinispan.xml? In essence I could use the Environment available in my connector to get a named cache and use that. Is there any harm in doing this? Not sure what ModeShape would be doing internally to try and manage these caches or the impact of letting ModeShape know this extra cache existed.
From an SPI perspective, you can definitively use the Environment instance in your connector (this why we expose it in the base org.modeshape.jcr.spi.federation.Connector class via a protected method).
The only thing you need to be aware of is that any ISPN cache retrieved/created via the Environment will be placed under repository "management", meaning that if a repository shuts down/is restarted etc it will do the same for all of the caches that it manages. As long as the external system "behind" the connector doesn't care/know that there is an in-memory cache in front of it, it shouldn't matter. In other words, you just need to be sure that the presence/lifecycle of the connector cache is strictly tied to that of the repository.