Correct, at the moment, it is configured once at the broker level. The broker needs a context to use for outbound communications, networks and the like and it uses the same sslContext for listeners. Multiple brokers in the same JVM can have different sslContexts.
But yes, it does make sense to have a context per transport connector or network connector so that difference endpoints could use different credentials and QOS. Can you raise an enhancement request for that?