Yes, this is quite normal. log4j will only load its configuration once, using the most immediate configuration file found in the classpath. Typically, this will be the one in the webapp as that would be the first one to see any activity.
You can probably solve your problem by defining all of your appenders in the same log4j.xml file with different names and then point your loggers/categories at them as needed. Read the log4j doco http://logging.apache.org/log4j/1.2/manual.html for more information on this.