we have setup JBoss AS7 in our environment. our application servers are situated in different geographies (US/UK etc). splunk server/forwarders are situated in US and different geographies are connected by vpn.
when the setup is ready, we restart application servers to make connection in raw mode to splunk forwarders.
this works fine, but after an hour (or so) of sending log data from application servers to forwarders, UK servers stop sending the data to splunk forwarders.
there isn't any ERROR in splunkd.log on forwarder, and the metrics.log shows that it is still having tcpin connection from UK machine.
i created a shell script to send data from UK server on same tcp port of forwarder and it showed up fine in splunk server, so that means forwarder and splunk server are working fine, and the problem seems to be jboss specific.