When using Log4j2 version 2.24.0, the values in MDC are ignored, and only the context from the Elastic APM agent remains in the logs.

[bright-k]

When using Log4j2 version 2.24.0, the values in MDC are ignored, and only the context from the Elastic APM agent remains in the logs.

We are using MDC to log the execution context for additional logging on our server. However, after upgrading to Log4j2 version 2.24.0, all MDC values that were directly added on the server are removed during logging, and only transaction.id and trace.id are left in the logs. This issue occurs with all versions of the Elastic APM Java agent on servers using Log4j2 version 2.24.0.

When I added the code below

MDC.put("test", "context")
println("mdc ====> ${MDC.getCopyOfContextMap()}")
log.info("mdc with log")

It will be printed as follows:

mdc ====> {test=leijflsijf, http.request.method=GET, client.ip=0:0:0:0:0:0:0:1, event.id=66f34bf7592c9908063e98cc}
2024-09-25 08:32:08.027  INFO 2054 [...] ... | mdc with log | {trace.id=c061f5b6bec4a128ca74d4f8fef25abf, transaction.id=ff649430cb802078} 

When the agent is not added, when using a version of Log4j2 prior to 2.23.1, or when using Log4j2 version 2.24.0 with the -Dlog4j2.garbagefreeThreadContextMap=true VM option, the MDC is logged correctly as expected.

mdc ====> {test=leijflsijf, http.request.method=GET, client.ip=0:0:0:0:0:0:0:1, event.id=66f34bf7592c9908063e98cc}
2024-09-25 08:32:08.027  INFO 2054 [...] ... | mdc with log | {test=context, http.request.method=GET, client.ip=0:0:0:0:0:0:0:1, event.id=66f34bf7592c9908063e98cc, trace.id=1b9b28c6a255639c5b24d768cd1cc3ff, transaction.id=f48a0a6f189902b3} 

My log4j2 config file is as follows

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Properties>
        <Property name="LOG_PATTERN">%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}} %highlight{${LOG_LEVEL_PATTERN:-%5p}}{FATAL=red blink, ERROR=red, WARN=yellow bold, INFO=green, DEBUG=blue bold, TRACE=blue} %style{${sys:PID}}{magenta} [%15.15t] %style{%-40.40c{1.}}{cyan} | %m | %clr{%X}{faint} %n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}</Property>
    </Properties>

    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="${LOG_PATTERN}"/>
        </Console>
    </Appenders>

    <Loggers>
        <AsyncRoot level="INFO">
            <AppenderRef ref="Console"/>
        </AsyncRoot>
    </Loggers>
</Configuration>