You will need to use a shared file system like NFS in a clustered environment. Thats the only way the data stored on the file system will be in sync.
without the synchronization, on fail over when a new node takes over as master, it will just read data from its local file system which does not have any changes reflected
Using the db as storage for the cms seems to have correct behaviour, but is un-useable as it doesn't work correctly with oracle and concurrent uploads seem to break the cms.
This should not be the case. The cms is certified to work on oracle, so for this there can be several environmental factors at play such as. database version, database driver, network latency, transaction timeout, etc etc