Steps to configure WildFly / EAP to send the logs using JBoss Logmanager and Log4j to Kafka:

1. Add Kafka client module:

$ cat <<EOF > $JBOSS_HOME/modules/system/layers/base/org/apache/kafka/clients/main/module.xml

<?xml version="1.0" encoding="UTF-8"?>

<module name="org.apache.kafka.clients" xmlns="urn:jboss:module:1.8">

    <resources>

        <resource-root path="kafka-clients-2.2.0.jar"/>

        <resource-root path="snappy-java-1.1.7.2.jar"/>

        <resource-root path="lz4-java-1.5.0.jar"/>

    </resources>

    <dependencies>

        <module name="javax.api"/>

        <module name="org.slf4j"/>

    </dependencies>

</module>

EOF

(See also: Chapter 6. JBoss EAP Class Loading - Red Hat Customer Portal )

2. Add Kafka log4jappender module:

$ cat <<EOF > $JBOSS_HOME/modules/system/layers/base/org/apache/kafka/log4jappender/main/module.xml

<?xml version="1.0" encoding="UTF-8"?>

<module name="org.apache.kafka.log4jappender" xmlns="urn:jboss:module:1.8">

    <resources>

        <resource-root path="kafka-log4j-appender-2.2.0.jar"/>

        <resource-root path="slf4j-log4j12-1.7.25.jar"/>

    </resources>

    <dependencies>

        <module name="org.slf4j"/>

        <module name="org.apache.kafka.clients" />

        <module name="org.jboss.log4j.logmanager" />

    </dependencies>

</module>

EOF

3. Add a log4jappender module as a dependency to the existing org.apache.log4j module:

<module name="org.apache.log4j" xmlns="urn:jboss:module:1.6">

    ...

    <dependencies>

        ...

        <module name="org.apache.kafka.log4jappender" export="true"/>

    </dependencies>

</module>

4. Configure custom log handler in standalone.xml, by adding this to the logging subsystem configuration:

<custom-handler name="kafka" class="org.apache.kafka.log4jappender.KafkaLog4jAppender" module="org.apache.log4j">

    <level name="INFO"/>

    <formatter>

        <named-formatter name="PATTERN"/>

    </formatter>

    <properties>

        <property name="brokerList" value="..."/>

        <property name="topic" value="..."/>

        <!-- other kafka.properties -->

    </properties>

</custom-handler>

(see also: Chapter 11. Logging with JBoss EAP - Red Hat Customer Portal )

5. Staring the server at this point will fail (freeze) as Kafka log4jappender will try to log messages while initializing, the workaround is to add a async logging handler which properly handles the NPE that happens.

Configure async log hadler in the logging subsystem (standalone.xml):

<async-handler name="async-kafka-wrapper">

    <level name="ALL"/> 

    <queue-length value="1024"/> 

    <overflow-action value="block"/> 

    <subhandlers> 

        <handler name="kafka"/> 

    </subhandlers> 

</async-handler>

(see also: Chapter 11. Logging with JBoss EAP - Red Hat Customer Portal )

*note that the async-kafka-wrapper must be used when the handler is referenced

6. This last step is required to make the initialisation logging work, the logging that is defined in logging.properties (the changes above are applied to the logging.properties on server start/stop, meaning the first boot will work without this and the next one will fail).

Kafka client is dynamically loading the serializer (org.apache.kafka.common.serialization.ByteArraySerializer), since this is happening with the org.jboss.as.standalone classloader a dependency must be added.

Add a Kafka client module dependency to the org.jboss.as.standalone module:

<module name="org.jboss.as.standalone" xmlns="urn:jboss:module:1.6">

    ...

    <dependencies>

        ...

        <module name="org.apache.kafka.clients" />

    </dependencies>

</module>

7. A common use case is it send the logs in json format, to configure a JSON formatter see Chapter 11. Logging with JBoss EAP - Red Hat Customer Portal