I tried your settings (well, I set MaxFileSize to a smaller number so that I could force new log files), and got the expected results - server.log.1 and so on. There should not be a limit to MaxBackupIndex.
Try setting the log4j.debug system property (add "-Dlog4j.debug" to JAVA_OPTS in run.bat or run.conf). This will cause log4j to print out some initialization information. Perhaps that will help pinpoint the issue.