doc/admin-guide/tools/converting-records-to-yaml.en.rst

.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license
   agreements.  See the NOTICE file distributed with this work for additional information regarding
   copyright ownership.  The ASF licenses this file to you under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with the License.  You may obtain
   a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software distributed under the License
   is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
   or implied.  See the License for the specific language governing permissions and limitations
   under the License.

.. include:: ../../common.defs

.. _rec-config-to-yaml:


Converting records.config to records.yaml
*****************************************


Since ATS 10 ``records.config`` is not longer used and instead we have migrated to
:file:`records.yaml`.
This is a guide on how to migrate from your legacy ``records.config`` style file to a
new :file:`records.yaml`.

This document will give details about the change and how to convert files from the
legacy style to the new YAML.


New YAML structure
==================


#. We are introducing a root node ``ts`` for the :file:`records.yaml`, so all existing records
will belong to the new ``ts`` node.
#. From the current ``records.config`` structure we have dropped the prefix ``proxy.config``
and ``proxy.local`` and use the rest to build a YAML structure.

The logic around this is basically to walk down the record name separated by the dots and build
a new YAML map from each name, so for instance, with the following records:


.. code-block:: bash
   :linenos:

   CONFIG proxy.config.exec_thread.autoconfig.scale FLOAT 1.0
   CONFIG proxy.config.exec_thread.limit INT 2
   CONFIG proxy.config.accept_threads INT 1
   CONFIG proxy.config.task_threads INT 2
   CONFIG proxy.config.cache.threads_per_disk INT 8
   CONFIG proxy.config.exec_thread.affinity INT 1
   CONFIG proxy.config.diags.debug.enabled INT 0
   CONFIG proxy.config.diags.debug.tags STRING http|dns


we first drop the ``proxy.config.`` and walk down each field, so we will end up with
the following:

.. code-block:: yaml
   :linenos:

      ts:
      accept_threads: 1
      cache:
         threads_per_disk: 8
      diags:
         debug:
            enabled: 0
            tags: http|dns
      exec_thread:
         affinity: 1
         autoconfig:
            scale: 1.0
         limit: 2
      task_threads: 2

There were a few things that were done on the names and types that are described next.

Records renaming
----------------

There are few changes that I had to perform in order to be able to keep the similar
structure used in the ``records.config`` into a YAML based configuration.

An example of this would be a record key holding a value which also contains children fields:

.. code-block:: bash

   proxy.config.exec_thread.autoconfig INT 1
   proxy.config.exec_thread.autoconfig.scale FLOAT 1.0 << children of autoconfig


As this cannot be done in YAML we need to rename ``proxy.config.exec_thread.autoconfig``
to ``proxy.config.exec_thread.autoconfig.enabled`` so we no longer have a map with
values other than some children's fields. So in YAML we can go from something like this:


.. code-block:: bash

   proxy.config.exec_thread.autoconfig INT 1
   proxy.config.exec_thread.autoconfig.scale FLOAT 1.0

to

.. code-block:: bash

   exec_thread:
      autoconfig:
         enabled: 1 # new field.
         scale: 1.0


Naming convention
-----------------

All this renamed records are subject to the following reasoning:

If a value is meant to be used as a toggle on/off, We have used the name ``enabled``
If a value is meant to be used a an enumeration, We have used the name ``value``

Possible issue with this is that if we scale the field meaning and we need to move
from a toggle to an enumeration then this logic will make no sense.
Should we just use one or the other? Current diags.debug.enabled is an example of
an enable field holding more than 0,1 value.




Converting files
================

We have provided a tool script that will help converting any ``records.config`` style
file into a YAML format file. The script will not only convert record by record but
it will also work around the names that were renamed.

Just run the script on your existing ``records.config`` file and that should be it.

Example 1
---------

Converting a file with a detailed output.

.. code-block:: bash
   :linenos:

   $ python3 convert2yaml.py -f records.config -o records.yaml
   [████████████████████████████████████████] 494/494

   ┌■ 8 Renamed records:
   └┬──» #1 : proxy.config.output.logfile -> proxy.config.output.logfile.name
    ├──» #2 : proxy.config.exec_thread.autoconfig -> proxy.config.exec_thread.autoconfig.enabled
    ├──» #3 : proxy.config.hostdb -> proxy.config.hostdb.enabled
    ├──» #4 : proxy.config.tunnel.prewarm -> proxy.config.tunnel.prewarm.enabled
    ├──» #5 : proxy.config.ssl.TLSv1_3 -> proxy.config.ssl.TLSv1_3.enabled
    ├──» #6 : proxy.config.ssl.client.TLSv1_3 -> proxy.config.ssl.client.TLSv1_3.enabled
    ├──» #7 : proxy.config.ssl.origin_session_cache -> proxy.config.ssl.origin_session_cache.enabled
    └──» #8 : proxy.config.ssl.session_cache -> proxy.config.ssl.session_cache.value


There are a few things to note here:

Line 2. A total of ``494`` from ``494`` records were converted.
Line 4. A total of ``8`` records were renamed.

Example 2
---------

Converting a file with no output. If any, errors are displayed.

.. code-block:: bash

   $ convert2yaml.py -f records.config -o records.yaml -m

.. note::

   Use -m, --mute to mute the output.

Non core records
================

With non core records the problem arises when a plugin register a new field which
is not registered in the core, the legacy implementation would read the record type
from the config file and then once the record is registered by a plugin it will be
properly registered inside ATS( * mismatched type gets the current value reset to
the default one*)

As said before in ATS when using YAML without explicit saying the type we have no
way to know the type till the registration happens(after reading the config).
We need to overcome this and to me there seems to be two options here:

Expect non core records to set the type (!!int, !!float, etc).

.. code-block:: bash

   ts:
      plugin_x:
         my_field_1: !!int '1'
         my_field_2: !!float '1.2'
         my_field_3: 'my string'


The convert2yaml tool can deal with this:

Example 1
---------

Use a type representation (``-t, --typerepr``) when converting a file.

Non core records expect the type to be specified as the YAML library used Internally
cannot assume the type. So in for cases like this you should set the type on the
non core records, for instance:

.. code-block:: bash

   CONFIG proxy.config.http.my_own_record_1 FLOAT 1.0
   CONFIG proxy.config.http.some_core_record INT 1


.. code-block:: bash

   $ convert2yaml.py -f records.config -o records.yaml -t float,int

   $ cat records.yaml
   ts:
      http:
         my_own_record_1: !!float '1.0'
         my_own_record_2: !!int '1'

Now the records parser knows the type of the non core records.

.. important::

   The issue with using type representer when converting a file is that this applies
   to all the records and not only the non core one. This will change in a future
   script update.



Final Notes
===========

Internally ATS still uses variables with record style names, querying any record can still
be done as usual. Either by using the ATS API or by using :program:`traffic_ctl`.


ATS only accepts a :file:`records.yaml` config file, if a legacy file ``records.config`` is found
then ATS will fail to start.

.. code-block:: bash

   traffic_server WARNING: **** Found a legacy config file (/your/ats/records.config). Please remove it and migrate to the new YAML format before continuing. ****