Is there a way to know that items are being evicted from the cache without attaching a listener to the cache?
You could use JMX. The jbosscache documentation tells you how enable JMX. Depending on what your needs are, you could then use jconsole to view evictions.