After downloading the sources of JBoss TS JTA project in the exact same version (4.16.4) after debugging the com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple#enlistResource(-) it occurred that there was a silent OracleXAException without any stacktrace or error code thrown from xaRes.start(-). The Transaction Manager have some fixed number of retries in such case, so it invoked once again my XAResource methods.
After knowing that this is an Oracle related exception I suspected it's a module-related issue. I've removed the Oracle JDBC from my RAR libs, added the latest version to the server modules and added dependencies in jboss-deployment-structure.xml. After those changes, it started working.
Hope this will help someone else and prevent him from spending few hours of happy debugging.