////
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements. See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
    The ASF licenses this file to You under the Apache License, Version 2.0
    (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at

        https://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
////
= Lookups
Ralph Goers <rgoers@apache.org>

Lookups provide a way to add values to the Log4j configuration at
arbitrary places. They are a particular type of Plugin that implements
the
link:../log4j-core/apidocs/org/apache/logging/log4j/core/lookup/StrLookup.html[`StrLookup`]
interface. Information on how to use Lookups in configuration files can
be found in the link:configuration.html#PropertySubstitution[Property
Substitution] section of the link:configuration.html[Configuration]
page.

[#ContextMapLookup]
== Context Map Lookup

The ContextMapLookup allows applications to store data in the Log4j
ThreadContext Map and then retrieve the values in the Log4j
configuration. In the example below, the application would store the
current user's login id in the ThreadContext Map with the key "loginId".
During initial configuration processing the first '$' will be removed.
The PatternLayout supports interpolation with Lookups and will then
resolve the variable for each event. Note that the pattern
"%X\{loginId}" would achieve the same result.

[source,xml]
----
<File name="Application" fileName="application.log">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] $${ctx:loginId} %m%n</pattern>
  </PatternLayout>
</File>
----

[#DateLookup]
== Date Lookup

The DateLookup is somewhat unusual from the other lookups as it doesn't
use the key to locate an item. Instead, the key can be used to specify a
date format string that is valid for
http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html[SimpleDateFormat].
The current date, or the date associated with the current log event will
be formatted as specified.

[source,xml]
----
<RollingFile name="Rolling-${map:type}" fileName="${filename}" filePattern="target/rolling1/test1-$${date:MM-dd-yyyy}.%i.log.gz">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] %m%n</pattern>
  </PatternLayout>
  <SizeBasedTriggeringPolicy size="500" />
</RollingFile>
----

[#DockerLookup]
== Docker Lookup

The DockerLookup can be used to lookup attributes from the Docker container the application is running in.

Log4j Docker provides access to the following container attributes:
[cols="1m,4a"]
|===
|Key |Description

|containerId
|The full id assigned to the container.

|containerName
|The name assigned to the container.

|imageId
|The id assigned to the image.

|imageName
|The name assigned to the image.

|shortContainerId
|The first 12 characters of the container id.

|shortImageId
|The first 12 characters of the image id.
|===

----
<JsonLayout properties="true" compact="true" eventEol="true">
  <KeyValuePair key="containerId" value="${docker:containerId}"/>
  <KeyValuePair key="containerName" value="${docker:containerName}"/>
  <KeyValuePair key="imageName" value="${docker:imageName}"/>
</JsonLayout>
----

This Lookup is subject to the requirements listed at link:../log4j-docker/index.html[Log4j Docker Support]

[#EnvironmentLookup]
== Environment Lookup

The EnvironmentLookup allows systems to configure environment variables,
either in global files such as /etc/profile or in the startup scripts
for applications, and then retrieve those variables from within the
logging configuration. The example below includes the name of the
currently logged in user in the application log.

[source,xml]
----
<File name="Application" fileName="application.log">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] $${env:USER} %m%n</pattern>
  </PatternLayout>
</File>
----

This lookup also supports default value syntax. In the sample below,
when the `USER` environment variable is undefined, the default value
`jdoe` is used:

[source,xml]
----
<File name="Application" fileName="application.log">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] $${env:USER:-jdoe} %m%n</pattern>
  </PatternLayout>
</File>
----
[#EventLookup]
== Event Lookup

The EventLookup provides access to fields within the log event from the configuration.

[cols="1m,4a"]
|===
|Key |Description

|Exception
|Returns the simple class name of the Exception, if one is included in the event.

|Level
|Returns the logging Level of the event.

|Logger
|Returns the name of the Logger.

|Marker
|Returns the name of the Marker associated with the log event, if one is present.

|Message
|Returns the formatted Message string.

|ThreadId
|Returns the thread id associated with the log event.

|ThreadName
|Returns the name of the thread associate with the log event.

|Timestamp
|Returns the time in milliseconds when the event occurred.

|===

In this example the RoutingAppender picks a route based on the presence of a Marker named "AUDIT" being
present in the log event.
[source,prettyprint,linenums]
----
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" name="RoutingTest">
  <Appenders>
    <Console name="STDOUT" target="SYSTEM_OUT" />
    <Flume name="AuditLogger" compress="true">
      <Agent host="192.168.10.101" port="8800"/>
      <Agent host="192.168.10.102" port="8800"/>
      <RFC5424Layout enterpriseNumber="18060" includeMDC="true" appName="MyApp"/>
    </Flume>
    <Routing name="Routing">
      <Routes>
        <Route pattern="$${event:Marker}">
          <RollingFile
              name="Rolling-${mdc:UserId}"
              fileName="${mdc:UserId}.log"
              filePattern="${mdc:UserId}.%i.log.gz">
            <PatternLayout>
              <pattern>%d %p %c{1.} [%t] %m%n</pattern>
            </PatternLayout>
            <SizeBasedTriggeringPolicy size="500" />
          </RollingFile>
        </Route>
        <Route ref="AuditLogger" key="AUDIT"/>
        <Route ref="STDOUT" key="STDOUT"/>
      </Routes>
      <IdlePurgePolicy timeToLive="15" timeUnit="minutes"/>
    </Routing>
  </Appenders>
  <Loggers>
    <Root level="error">
      <AppenderRef ref="Routing" />
    </Root>
  </Loggers>
</Configuration>
----

[#JavaLookup]
== Java Lookup

The JavaLookup allows Java environment information to be retrieved in
convenient preformatted strings using the `java:` prefix.

[cols="1m,4a"]
|===
|Key |Description

|version
|The short Java version, like:

`Java version 1.7.0_67`

|runtime
|The Java runtime version, like:

`Java(TM) SE Runtime Environment (build 1.7.0_67-b01) from Oracle Corporation`

|vm
|The Java VM version, like:

`Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)`

|os
|The OS version, like:

`Windows 7 6.1 Service Pack 1, architecture: amd64-64`

|locale
|Hardware information, like:

`default locale: en_US, platform encoding: Cp1252`

|hw
|Hardware information, like:

`processors: 4, architecture: amd64-64, instruction sets: amd64`

|===

For example:

[source,xml]
----
<File name="Application" fileName="application.log">
  <PatternLayout header="${java:runtime} - ${java:vm} - ${java:os}">
    <Pattern>%d %m%n</Pattern>
  </PatternLayout>
</File>
----

[#JndiLookup]
== JNDI Lookup

The JndiLookup allows variables to be retrieved via JNDI. By default the
key will be prefixed with java:comp/env/, however if the key contains a
":" no prefix will be added.

[source,xml]
----
<File name="Application" fileName="application.log">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] $${jndi:logging/context-name} %m%n</pattern>
  </PatternLayout>
</File>
----

*Java's JNDI module is not available on Android.*

[#JmxRuntimeInputArgumentsLookup]
== JVM Input Arguments Lookup (JMX)

Maps JVM input arguments -- but not _main_ arguments -- using JMX to
acquire the JVM arguments.

Use the prefix `jvmrunargs` to access JVM arguments.

See the Javadocs for
https://docs.oracle.com/javase/8/docs/api/java/lang/management/RuntimeMXBean.html#getInputArguments--[java.lang.management.RuntimeMXBean.getInputArguments()].

*Java's JMX module is not available on Android or on Google App Engine.*

[#KubernetesLookup]
== Kubernetes Lookup

The KubernetesLookup can be used to lookup attributes from the Kubernetes environment for the container
the application is running in.

Log4j Kubernetes provides access to the following container attributes:
[cols="1m,4a"]
|===
|Attribute |Description
|accountName|The service account name
|clusterName|The name of the cluster the application is deployed in
|containerId|>The full id assigned to the container
|containerName|The name assigned to the container
|host|The name assigned to the host operating system
|hostIp|The host's ip address
|imageId|The id assigned to the container image
|imageName|The name assigned to the container image
|labels|All labels formatted in a list
|labesl.app|The application name
|labels.podTemplateHash|The pod's template hash value
|masterUrl|The URL used to access the API server
|namespaceId|The id of the namespace the various kubernetes components are located within
|namespaceName|The namespace the various kubernetes components are located within
|podId|The pod's ip number
|podIp|The pod's ip address
|podName|The name of the pod

      <GelfLayout includeStackTrace="true" host="${hostName}" includeThreadContext="true" includeNullDelimiter="true" compressionType="OFF">
        <ThreadContextIncludes>requestId,sessionId,loginId,userId,ipAddress,callingHost</ThreadContextIncludes>
        <MessagePattern>%d [%t] %-5p %X{requestId, sessionId, loginId, userId, ipAddress} %C{1.}.%M:%L - %m%n</MessagePattern>
        <KeyValuePair key="docker.containerId" value="${docker:containerId:-}"/>
        <KeyValuePair key="application" value="$${lower:${spring:spring.application.name}}"/>
        <KeyValuePair key="kubernetes.serviceAccountName" value="${k8s:accountName:-}"/>
        <KeyValuePair key="kubernetes.clusterName" value="${k8s:clusterName:-}/>
        <KeyValuePair key="kubernetes.containerId" value="${k8s:containerId:-}"/>
        <KeyValuePair key="kubernetes.containerName" value="${k8s:containerName:-}"/>
        <KeyValuePair key="kubernetes.host" value="${k8s:host:-}"/>
        <KeyValuePair key="kubernetes.labels.app" value="${k8s:labels.app:-}"/>
        <KeyValuePair key="kubernetes.labels.pod-template-hash" value="${k8s:labels.podTemplateHash:-}"/>
        <KeyValuePair key="kubernetes.master_url" value="${k8s:masterUrl:-}"/>
        <KeyValuePair key="kubernetes.namespaceId" value="${k8s:namespaceId:-}"/>
        <KeyValuePair key="kubernetes.namespaceName" value="${k8s:namespaceName:-}"/>
        <KeyValuePair key="kubernetes.podID" value="${k8s:podId:-}"/>
        <KeyValuePair key="kubernetes.podIP" value="${k8s:podIp:-}"/>
        <KeyValuePair key="kubernetes.podName" value="${k8s:podName:-}"/>
        <KeyValuePair key="kubernetes.imageId" value="${k8s:imageId:-}"/>
        <KeyValuePair key="kubernetes.imageName" value="${k8s:imageName:-}"/>
      </GelfLayout>]]></pre>

This Lookup is subject to the configuration requirements listed at link:../log4j-kubernetes/index.html[Log4j Kubernetes Support]

[#Log4jConfigLookup]
== Log4j Configuration Location Lookup

Log4j configuration properties. The expressions
`${log4j:configLocation}` and `${log4j:configParentLocation}`
respectively provide the absolute path to the log4j configuration file
and its parent folder.

The example below uses this lookup to place log files in a directory
relative to the log4j configuration file.

[source,xml]
----
<File name="Application" fileName="${log4j:configParentLocation}/logs/application.log">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] %m%n</pattern>
  </PatternLayout>
</File>
----

[#LowerLookup]
== Lower Lookup

The LowerLookup converts the passed in argument to lower case. Presumably the value will be the
result of a nested lookup.

[source,xml]
----
<File name="Application" fileName="application.log">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] $${lower:${spring:spring.application.name}} %m%n</pattern>
  </PatternLayout>
</File>
----

[#AppMainArgsLookup]
== Main Arguments Lookup (Application)

This lookup requires that you manually provide the main arguments of the
application to Log4j:

[source,java]
----
import org.apache.logging.log4j.core.lookup.MainMapLookup;

public static void main(String args[]) {
  MainMapLookup.setMainArguments(args);
  ...
}
----

If the main arguments have been set, this lookup allows applications to
retrieve these main argument values from within the logging
configuration. The key that follows the `main:` prefix can either be a
0-based index into the argument list, or a string, where
`${main:myString}` is substituted with the value that follows `myString`
in the main argument list.

Note: Many applications use leading dashes to identify command arguments. Specifying
`${main:--file}` would result in the lookup failing because it would look for a variable
named "main" with a default value of "-file". To avoid this the ":" separating the Lookup name from the
key must be followed by a backslash as an escape character as in `${main:\--file}`.

For example, suppose the static void main String[] arguments are:

....
--file foo.txt --verbose -x bar
....

Then the following substitutions are possible:

[cols="m,m"]
|===
|Expression |Result

|${main:0}
|--file

|${main:1}
|foo.txt

|${main:2}
|--verbose

|${main:3}
|-x

|${main:4}
|bar

|${main:\--file}
|foo.txt

|${main:\-x}
|bar

|${main:bar}
|null
|===

|${main:\--quiet:-true}
|true
|===

Example usage:

[source,xml]
----
<File name="Application" fileName="application.log">
  <PatternLayout header="File: ${main:--file}">
    <Pattern>%d %m%n</Pattern>
  </PatternLayout>
</File>
----

[#MapLookup]
== Map Lookup

The MapLookup serves several purposes.

1.  Provide the base for Properties declared in the configuration file.
2.  Retrieve values from MapMessages in LogEvents.
3.  Retrieve values set with
link:../log4j-core/apidocs/org/apache/logging/log4j/core/lookup/MapLookup.html#setMainArguments%28java.lang.String%5B%5D%29[MapLookup.setMainArguments(String[])]

The first item simply means that the MapLookup is used to substitute
properties that are defined in the configuration file. These variables
are specified without a prefix - e.g. `${name}`. The second usage allows
a value from the current
link:../log4j-api/apidocs/org/apache/logging/log4j/message/MapMessage.html[`MapMessage`],
if one is part of the current log event, to be substituted. In the
example below the RoutingAppender will use a different
RollingFileAppender for each unique value of the key named "type" in the
MapMessage. Note that when used this way a value for "type" should be
declared in the properties declaration to provide a default value in
case the message is not a MapMessage or the MapMessage does not contain
the key. See the link:configuration.html#PropertySubstitution[Property
Substitution] section of the link:configuration.html[Configuration]
page for information on how to set the default values.

[source,xml]
----
<Routing name="Routing">
  <Routes pattern="$${map:type}">
    <Route>
      <RollingFile name="Rolling-${map:type}" fileName="${filename}"
                   filePattern="target/rolling1/test1-${map:type}.%i.log.gz">
        <PatternLayout>
          <pattern>%d %p %c{1.} [%t] %m%n</pattern>
        </PatternLayout>
        <SizeBasedTriggeringPolicy size="500" />
      </RollingFile>
    </Route>
  </Routes>
</Routing>
----

The marker lookup allows you to use markers in interesting
configurations like a routing appender. Consider the following YAML
configuration and code that logs to different files based on markers:

[source,yaml]
----
Configuration:
  status: debug

  Appenders:
    Console:
    RandomAccessFile:
      - name: SQL_APPENDER
        fileName: logs/sql.log
        PatternLayout:
          Pattern: "%d{ISO8601_BASIC} %-5level %logger{1} %X %msg%n"
      - name: PAYLOAD_APPENDER
        fileName: logs/payload.log
        PatternLayout:
          Pattern: "%d{ISO8601_BASIC} %-5level %logger{1} %X %msg%n"
      - name: PERFORMANCE_APPENDER
        fileName: logs/performance.log
        PatternLayout:
          Pattern: "%d{ISO8601_BASIC} %-5level %logger{1} %X %msg%n"

    Routing:
      name: ROUTING_APPENDER
      Routes:
        pattern: "$${marker:}"
        Route:
        - key: PERFORMANCE
          ref: PERFORMANCE_APPENDER
        - key: PAYLOAD
          ref: PAYLOAD_APPENDER
        - key: SQL
          ref: SQL_APPENDER

  Loggers:
    Root:
      level: trace
      AppenderRef:
        - ref: ROUTING_APPENDER
----

[source,java]
----
public static final Marker SQL = MarkerFactory.getMarker("SQL");
public static final Marker PAYLOAD = MarkerFactory.getMarker("PAYLOAD");
public static final Marker PERFORMANCE = MarkerFactory.getMarker("PERFORMANCE");

final Logger logger = LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);

logger.info(SQL, "Message in Sql.log");
logger.info(PAYLOAD, "Message in Payload.log");
logger.info(PERFORMANCE, "Message in Performance.log");
----

Note the key part of the configuration is `pattern: "$${marker:}"`. This
will produce three log files, each with a log event for a specific
marker. Log4j will route the log event with the `SQL` marker to
`sql.log`, the log event with the `PAYLOAD` marker to `payload.log`, and
so on.

You can use the notation `"${marker:name}"` and `"$${marker:name}"` to
check for the existence of a marker where `name` is the marker name. If
the marker exists, the expression returns the name, otherwise `null`.

[#StructuredDataLookup]
== Structured Data Lookup

The StructuredDataLookup is very similar to the MapLookup in that it
will retrieve values from StructuredDataMessages. In addition to the Map
values it will also return the name portion of the id (not including the
enterprise number) and the type field. The main difference between the
example below and the example for MapMessage is that the "type" is an
attribute of the
link:../log4j-api/apidocs/org/apache/logging/log4j/message/StructuredDataMessage.html[StructuredDataMessage]
while "type" would have to be an item in the Map in a MapMessage.

[source,xml]
----
<Routing name="Routing">
  <Routes pattern="$${sd:type}">
    <Route>
      <RollingFile name="Rolling-${sd:type}" fileName="${filename}"
                   filePattern="target/rolling1/test1-${sd:type}.%i.log.gz">
        <PatternLayout>
          <pattern>%d %p %c{1.} [%t] %m%n</pattern>
        </PatternLayout>
        <SizeBasedTriggeringPolicy size="500" />
      </RollingFile>
    </Route>
  </Routes>
</Routing>
----

[#SpringLookup]
== Spring Boot Lookup


The Spring Boot Lookup retrieves the values of Spring properties from the Spring configuration as well as
values of the active and default profiles. Specifying a key of `profiles.active` will reutrn the active
profiles while a key of `profiles.default` will return the default profiles. The default and active
profiles can be an array. If more than one profile is present they will be returned as a comma separated
list. To retrieve a single item from the array append `[{index}]` to the key. For example, to return the
first active profile in the list specify `profiles.active[0]`.

This Lookup will return null values until Spring Boot initializes application logging.


[source,xml]
----
<File name="Application" fileName="application-${spring:profiles.active[0]}.log">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] $${spring:spring.application.name} %m%n</pattern>
  </PatternLayout>
</File>
----

This Lookup requires log4j-spring-cloud-config-client be included in the application.

[#SystemPropertiesLookup]
== System Properties Lookup

As it is quite common to define values inside and outside the
application by using System Properties, it is only natural that they
should be accessible via a Lookup. As system properties are often
defined outside the application it would be quite common to see
something like:

[source,xml]
----
<Appenders>
  <File name="ApplicationLog" fileName="${sys:logPath}/app.log"/>
</Appenders>
----

This lookup also supports default value syntax. In the sample below,
when the `logPath` system property is undefined, the default value
`/var/logs` is used:

[source,xml]
----
<Appenders>
  <File name="ApplicationLog" fileName="${sys:logPath:-/var/logs}/app.log"/>
</Appenders>
----

[#UpperLookup]
== Upper Lookup

The LowerLookup converts the passed in argument to upper case. Presumably the value will be the
result of a nested lookup.

[source,xml]
----
<File name="Application" fileName="application.log">
  <PatternLayout>
    <pattern>%d %p %c{1.} [%t] $${upper:{${spring:spring.application.name}} %m%n</pattern>
  </PatternLayout>
</File>
----

[#WebLookup]
== Web Lookup

The WebLookup allows applications to retrieve variables that are
associated with the ServletContext. In addition to being able to
retrieve various fields in the ServletContext, WebLookup supports
looking up values stored as attributes or configured as initialization
parameters. The following table lists various keys that can be
retrieved:

[cols="1m,4"]
|===
|Key |Description

|attr._name_
|Returns the ServletContext attribute with the specified name

|request.attr._name_
|Returns the ServletRequest attribute with the specified name - requires Log4jServletFilter

|header._name_
|Returns the HttpServletRequest header with the specified name - requires Log4jServletFilter

|cookie._name_
|Returns the HttpServletRequest cookie with the specified name - requires Log4jServletFilter

|header._name_
|Returns the HttpServletRequest header with the specified name - requires Log4jServletFilter

|request._method_
|Returns the HttpServletRequest method - requires Log4jServletFilter

|request._uri_
|Returns the HttpServletRequest URI - requires Log4jServletFilter

|request._url_
|Returns the HttpServletRequest URL - requires Log4jServletFilter

|request._remoteAddress_
|Returns the HttpServletRequest remote address - requires Log4jServletFilter

|request._remoteHost_
|Returns the HttpServletRequest remote host - requires Log4jServletFilter

|request.parameter._name_
|Returns the HttpServletRequest parameter - requires Log4jServletFilter

|request.principal
|Returns the HttpServletRequest principal name - requires Log4jServletFilter

|session.id
|Returns the HttpSession id or null if none is started - requires Log4jServletFilter

|session.attr._name_
|Returns the HttpSession attribute value (using _toString()_ if not null) or null if absent - requires Log4jServletFilter

|contextPath
|The context path of the web application

|contextPathName
|The first token in the context path of the web application splitting on "/" characters.

|effectiveMajorVersion
|Gets the major version of the Servlet specification that the application
represented by this ServletContext is based on.

|effectiveMinorVersion
|Gets the minor version of the Servlet specification that the application
represented by this ServletContext is based on.

|initParam._name_
|Returns the ServletContext initialization parameter with the specified name

|majorVersion
|Returns the major version of the Servlet API that this servlet container supports.

|minorVersion
|Returns the minor version of the Servlet API that this servlet container supports.

|rootDir
|Returns the result of calling getRealPath with a value of "/".

|serverInfo
|Returns the name and version of the servlet container on which the servlet is running.

|servletContextName
|Returns the name of the web application as defined in the display-name element of the deployment descriptor
|===

Any other key names specified will first be checked to see if a
ServletContext attribute exists with that name and then will be checked
to see if an initialization parameter of that name exists. If the key is
located then the corresponding value will be returned.

[source,xml]
----
<Appenders>
  <File name="ApplicationLog" fileName="${web:rootDir}/app.log"/>
</Appenders>
----

=== Request lookups and asynchronous calls

Servlet 3.0 supports asynchronous calls, by default the request tracking - and therefore request related lookups,
will not work. To make it work you can extract the servlet context attribute `log4j.requestExecutor` which
is a `BiConsumer<ServletRequest, Runnable>` and call it passing the correct request and task to execute
synchronously. During this task execution the lookups will be set up properly:

[source,java]
----
@GET // example using JAX-RS asynchronous feature backed by servlet AsyncContext
public void get(@Suspended AsyncResponse response,
                @Context ServletContext context,
                @Context ServletRequest request) {
   final BiConsumer<ServletRequest, Runnable> log4jWrapper =
        (BiConsumer<ServletRequest, Runnable>) context.getAttribute("log4j.requestExecutor");
   myThreadPool.submit(() -> log4jWrapper.accept(request, () -> response.resume(doInternalGet()));
}
----