JBoss uses log4j for logging.
You can limit the filesize in two ways:
1. Turn off logging in the log4j.properties file located in a subdirectory of the jboss/conf. Look for an entry like log4j.rootCategory=DEBUG. If you replace DEBUG with FATAL the amount of logged information should be minimized.
2. Use the RollingFileAppender instead of the FileAppender. This gives you the possibility to define the maximum size of the logfile before its overwritten.
Looking at the documentation of log4j (http://jakarta.apache.org/log4j/docs/index.html) would be a good idea.