.. Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.

.. include:: ../../common.defs

.. configfile:: logging.yaml

logging.yaml
**************

The :file:`logging.yaml` file defines all custom log file formats, filters,
and processing options.

.. important::

   This configuration file replaces the XML based logs_xml.config, as well as
   the Lua based logging.config from past |TS| releases. If you are upgrading
   from a |TS| release which used either the XML or the Lua configuration file
   format, and you have created custom log formats, filters, and destinations,
   you will need to update those settings to this format.

.. _admin-custom-logs:

Log Definitions
===============

Custom logs are configured by the combination of three key elements: a format,
an optional filter, and a log destination.

A :ref:`format <admin-custom-logs-formats>` defines how log lines will appear
(as well as whether the logs using the format will be event logs or summary
logs).

A :ref:`filter <admin-custom-logs-filters>` defines what events do, and what
events don't, make it into the logs employing the filter.

A :ref:`log <admin-custom-logs-logs>` defines where the record of events or
summaries ends up.

.. _admin-custom-logs-formats:

Formats
-------

Custom logging formats may be provided directly to a log definition, or they
may be defined as a reusable variable in your :file:`logging.yaml` for ease
of reference, particularly when you may have more than one log using the same
format. Which approach you use is entirely up to you, though it's strongly
recommended to create an explicit format object if you intend to reuse the same
format for multiple log files.

Custom formats are defined by choosing a ``name`` to identify the given logging
format, and a ``format`` string, which defines the output format string for
every event. An optional ``interval`` attribute can be specified to define the
aggregation interval for summary logs.

.. code:: yaml

   # A one-line-per-event format that just prints event timestamps.
   formats:
   - name: myformat
     format: '%<cqtq>'

   # An aggregation/summary format that prints the last event timestamp from
   # the interval along with the total count of events in the same interval.
   # (Doing so every 30 seconds.)
   formats:
   - name: mysummaryformat
     format: '%<LAST(cqtq)> %<COUNT(*)>'
     interval: 30

You may define as many and as varied a collection of format objects as you
desire.

Format Specification
~~~~~~~~~~~~~~~~~~~~

The format specification provided as the required ``format`` attribute of the
objects listed in ``formats`` is a simple string, containing whatever mixture
of logging field variables and literal characters meet your needs. Logging
fields are discussed in great detail in the :ref:`admin-logging-fields`
section.

Flexible enough to not only emulate the logging formats of most other proxy and
HTTP servers, but also to provide even finer detail than many of them, the
logging fields are very easy to use. Within the format string, logging fields
are indicated by enclosing their name within angle brackets (``<`` and ``>``),
preceded by a percent symbol (``%``). For example, returning to the altogether
too simple format shown earlier, the following format string::

    '%<cqtq>'

Defines a format in which nothing but the value of the logging field :ref:`cqtq
<cqtq>` is interpolated for each event's entry in the log. We could include
some literal characters in the log output by updating the format specification
as so::

    'Event received at %<cqtq>'

Because the string "Event received at " (including the trailing space) is just
a bunch of characters, not enclosed in ``%<...>``, it is repeated verbatim in
the logging output.

Multiple logging fields may of course be used::

    '%<cqtq> %<chi> %<cqhm> %<cqtx>'

Each logging field is separately enclosed in its own percent-brace set.

There are a small number of logging fields which extend this simple format,
primarily those dealing with request and response headers. Instead of defining
a separate logging field name for every single possible HTTP header (an
impossible task, given that arbitrary vendor/application headers may be present
in both requests and responses), there are instead single logging fields for
each of the major stages of an event lifecycle that permit access to named
headers, such as::

    '%<{User-Agent}cqh>'

Which emits to the log the value of the client request's ``User-Agent`` HTTP
header. Other stages of the event lifecycle have similar logging fields:
``pqh`` (proxy requests), ``ssh`` (origin server responses), and ``psh``
(proxy responses).

You will find a complete listing of the available fields in
:ref:`admin-logging-fields`.

Aggregation Interval
~~~~~~~~~~~~~~~~~~~~

Every format may be given an optional ``interval`` value, specified as the
number of seconds over which events destined for a log using the format are
aggregated and summarized. Logs which use formats containing an aggregation
interval do not behave like regular logs, with a single line for every event.
Instead, they emit a single line only every *interval*\ -seconds.

These types of logs are described in more detail in
:ref:`admin-logging-type-summary`.

Formats have no interval by default, and will generate event-based logs unless
given one.

.. _admin-custom-logs-filters:

Filters
-------

Trafficserver supports different type of filters : ``accept``, ``reject`` and ``wipe_field_value``.
They may be used, optionally, to accept, reject logging or mask query param values for matching events.

Filter objects are created by assigning them a ``name`` to be used later to
refer to the filter, as well as an ``action`` (either ``accept``, ``reject`` or
``wipe_field_value``). ``Accept``, ``reject`` or ``wipe_field_value`` filters require
a ``condition`` against which to match all events. The ``condition`` fields must
be in the following format::

    <field> <operator> <value>

For example, the following snippet defines a filter that matches all POST
requests:

.. code:: yaml

   filters:
   - name: postfilter
     action: accept
     condition: cqhm MATCH POST

Filter Fields
~~~~~~~~~~~~~

The log fields have already been discussed in the `Formats`_ section above. For
a reference to the available log field names, see :ref:`admin-logging-fields`.
Unlike with the log format specification, you do not wrap the log field names
in any additional markup.

Filter Operators
~~~~~~~~~~~~~~~~

The operators describe how to perform the matching in the filter rule, and may
be any one of the following:

``MATCH``
    True if the values of ``field`` and ``value`` are identical.
    Case-sensitive.

``CASE_INSENSITIVE_MATCH``
    True if the values of ``field`` and ``value`` are identical.
    Case-insensitive.

``CONTAIN``
    True if the value of ``field`` contains ``value`` (i.e. ``value`` is a
    substring of the contents of ``field``). Case-sensitive.

``CASE_INSENSITIVE_CONTAIN``
    True if the value of ``field`` contains ``value`` (i.e. ``value`` is a
    substring of the contents of ``field``). Case-insensitive.

Filter Values
~~~~~~~~~~~~~

The final component of a filter string specifies the value against which the
name field will be compared.

For integer matches, all of the operators are effectively equivalent and require
the field to be equal to the given integer. If you wish to match multiple
integers, provide a comma separated list like this::

    <field> <operator> 4,5,6,7

String matches work similarly to integer matches. Multiple matches are also
supported via a comma separated list. For example::

    <field> <operator> e1host,host2,hostz

For IP addresses, ranges may be specified by separating the first address and
the last of the range with a single ``-`` dash, as ``10.0.0.0-10.255.255.255``
which gives the ranges for the 10/8 network. Other network notations are not
supported at this time.

.. note::

    It may be tempting to attach multiple Filters to a log object
    reject multiple log fields (in lieu of providing a single comma separated list
    to a single Filter). Avoid this temptation and use a comma separated list of reject objects
    instead. Remember that you may not have multiple accept filter objects.
    Attaching multiple filters does the opposite of what you'd
    expect. If, for example, we had 2 accept log filters, each disjoint from the other,
    nothing will ever get logged on the given log object.


.. _admin-custom-logs-logs:

Logs
----

Up to this point, we've only described what events should be logged and what
they should look like in the logging output. Now we define where those logs
should be sent.

Three options currently exist for the type of logging output: ``ascii``,
``binary``, and ``ascii_pipe``.  Which type of logging output you choose
depends largely on how you intend to process the logs with other tools, and a
discussion of the merits of each is covered elsewhere, in
:ref:`admin-logging-ascii-v-binary`.

The following subsections cover the attributes you should specify when creating
your logging object. Only ``filename`` and ``format`` are required.

====================== =========== =================================================
Name                   Type        Description
====================== =========== =================================================
filename               string      The name of the logfile relative to the default
                                   logging directory (set with
                                   :ts:cv:`proxy.config.log.logfile_dir`).
format                 string      a string with a valid named format specification.
header                 string      If present, emitted as the first line of each
                                   new log file.
rolling_enabled        *see below* Determines the type of log rolling to use (or
                                   whether to disable rolling). Overrides
                                   :ts:cv:`proxy.config.log.rolling_enabled`.
rolling_interval_sec   number      Interval in seconds between log file rolling.
                                   Overrides
                                   :ts:cv:`proxy.config.log.rolling_interval_sec`.
rolling_offset_hr      number      Specifies an hour (from 0 to 23) at which log
                                   rolling is guaranteed to align. Only has an
                                   effect if RollingIntervalSec is set to greater
                                   than one hour. Overrides
                                   :ts:cv:`proxy.config.log.rolling_offset_hr`.
rolling_size_mb        number      Size, in megabytes, at which log files are
                                   rolled.
rolling_min_count      number      Specifies the minimum number of rolled logs to
                                   keep.
filters                array of    The optional list of filter objects which
                       filters     restrict the individual events logged. The array
                                   may only contain one accept filter.
====================== =========== =================================================

Enabling log rolling may be done globally in :file:`records.config`, or on a
per-log basis by passing appropriate values for the ``rolling_enabled`` key. The
latter method may also be used to effect different rolling settings for
individual logs. The numeric values that may be passed are the same as used by
:ts:cv:`proxy.config.log.rolling_enabled`. For convenience and readability,
the following predefined variables may also be used in :file:`logging.yaml`:

log.roll.none
    Disable log rolling.

log.roll.time
    Roll at a certain time frequency, specified by RollingIntervalSec and
    RollingOffsetHr.

log.roll.size
    Roll when the size exceeds RollingSizeMb.

log.roll.both
    Roll when either the specified rolling time is reached or the specified
    file size is reached.

log.roll.any
    Roll the log file when the specified rolling time is reached if the size of
    the file equals or exceeds the specified size.

Examples
========

The following is an example of a format that collects information using three
common fields:

.. code:: yaml

   formats:
   - name: minimalfmt
     format: '%<chi> , %<cqu> , %<pssc>'

The following is an example of a format that uses aggregate operators to
produce a summary log:

.. code:: yaml

   formats:
   - name: summaryfmt
     format: '%<LAST(cqts)>:%<COUNT(*)>:%<SUM(psql)>'
     interval: 10

The following is an example of a filter that will cause only REFRESH_HIT events
to be logged:

.. code:: yaml

   filters:
   - name: refreshhitfilter
     action: accept
     condition: pssc MATCH REFRESH_HIT

The following is an example of a log specification that creates a local log
file for the minimal format defined earlier. The log filename will be
``minimal.log`` because we select the ASCII logging format.

.. code:: yaml

   logs:
   - mode: ascii
     filename: minimal
     format: minimalfmt

The following is an example of a log specification that creates a local log
file using the summary format from earlier, and only includes events that
matched the REFRESH_HIT filter we created.

.. code:: yaml

   logs:
   - mode: ascii
     filename: refreshhit_summary
     format: summaryfmt
     filters:
     - refreshhitfilter