Advanced liminal.yml

In this section you will learn about advanced features of liminal.yml

Variables

Much like in programming languages, you can define variables for re-use across your liminal.yml file.

variables:
  myvar1: myvalue1
  myvar2: myvalue2

In the variables section of your liminal.yml you can define your variables as key value pairs

Placeholders

You can use placeholders of format {{myplaceholder}} in most any string value in your liminal.yml Make sure that any string that includes placeholders is surrounded by single or double quotes.

For example:

variables:
  image_name: myorg/myrepo:myapp
images:
  - image: '{{image_name}}'
    type: python
    source: .
pipelines:
  - pipeline: my_pipeline
    tasks:
      - task: my_python_task
        type: python
        image: "{{image_name}}"
        cmd: python -u my_module.py

Placeholders rendering

The process of rendering placeholders checks for values in several places, by order.

Pipeline placeholder rendering

When running pipelines placeholders will be rendered by replacing values from sources in the following order:

  1. Current DAG run conf (Airflow).
  2. liminal.yml variables section.
  3. Airflow variables (Airflow).
  4. Airflow macros (Airflow) . For example: "{{yesterday_ds}}". For more information see:

Airflow Jinja Templating (Airflow).

Build placeholder rendering

When using liminal build placeholders will be rendered by replacing values from sources in the following order:

  1. Environment variables.
  2. liminal.yml variables section.

Task variables

In addition to the variables section you can also set specific variables for specific tasks using the variables attribute of that task. For example:

  - task: my_task
    type: python
    cmd: python -u my_module.py
    variables:
      myvar1: myvalue1
      myvar2: myvalue2

You can also pass a reference to a variable map (dictionary) set in your variables section:

variables:
  my_variable_map:
    myvar1: myvalue1
    myvar2: myvalue2
  my_variable_map2:
    myvar1: myvalue_a
    myvar2: myvalue_x
pipelines:
  - pipeline: my_pipeline
    tasks:
      - task: my_task1
        type: python
        cmd: python -u my_module.py
        variables: my_variable_map1
      - task: my_task2
        type: python
        cmd: python -u my_module.py
        variables: my_variable_map2

Executors

For fully detailed information on executors see: executors.

Each task in your pipelines has a reference to an executor from the executors section of your liminal.yml file. If not specified, the appropriate default executor for that task type is used.

executors:
  - executor: my_kubernetes_executor
    type: kubernetes
    resources:
      request_memory: 128Mi
pipelines:
  pipeline: my_pipeline
  tasks:
    - task: my_python_task
      type: python
      image: myorg/myrepo:mypythonapp
      executor: my_kubernetes_executor
      cmd: python -u my_module.py

In the example above we define an executor of type kubernetes with custom resources configuration.

executors is a section in the root of your liminal.yml file and is a list of executors defined by the following attributes:

executor attributes

executor: name of your executor.

type: type of the executor. The current available image types are: kubernetes and emr.

Different executor types support their own additional configuration.

Task Defaults

task_defaults is a section in the root of your liminal.yml file in which default attributes can be set for each task type.

task_defaults:
  python:
    executor: my_kubernetes_executor
    env_vars:
      env: {{env}}
      foo: bar

In the example above we set default attributes for any python task in any pipeline in the liminal system defined by our liminal.yml file. Each python task that does not set these attributes will default to the setting in task_defaults.

If the same attribute is defined in both task_defaults and in the task, definitions from the task take precedence. If a map (dictionary) of values (for example env_vars) is defined in both task_defaults the two maps will be merged, with definitions in the task taking precedence in case key is defined in both maps.

Pipeline Defaults

pipeline_defaults is a section in the root of your liminal.yml file in which default attributes can be set for each pipeline in the liminal system defined in your liminal.yml file.

pipeline_defaults:
  start_date: 2021-01-01
  timeout_minutes: 30

In the example above we set default attributes for any pipeline defined in our liminal.yml file. Each pipeline that does not set these attributes will default to the setting in pipeline_defaults.

If the same attribute is defined in both pipeline_defaults and in the pipeline, definitions from the pipeline take precedence.

If tasks section is defined in pipeline_defaults each pipeline defined in our liminal.yml file will have the tasks defined in pipeline_defaults.

A special task type pipeline may be used in pipeline_defaults tasks section. This task type is interpreted as " tasks defined in pipeline go here". This allows flexibility of defining common tasks to be run before and after the tasks of each pipeline defined in our liminal.yml file. For example:

pipeline_defaults:
  before_tasks:
    - task: my_common_setup_task
      type: python
      image: myorg/myrepo:mypythonapp
      cmd: python -u my_setup_module.py
  after_tasks:
    - task: my_common_teardown_task1
      type: python
      image: myorg/myrepo:mypythonapp
      cmd: python -u my_teardown_module1.py
    - task: my_common_teardown_task2
      type: python
      image: myorg/myrepo:mypythonapp
      cmd: python -u my_teardown_module2.py

In the example above we set a list of tasks in pipeline_defaults which leads to each pipeline defined in our liminal.yml file will have my_common_setup_task run before its tasks and my_common_teardown_task1 and my_common_teardown_task2 after its tasks.

Inheritence

Each liminal.yml defines a subliminal layer of a liminal system. A subliminal layer can inherit attributes of a superliminal layer by specifying super: name_of_super in the root of the yml.

name: my_system
super: my_super

A super is found by this name if a liminal.yml file in your environment exists with that name. A superliminal layer liminal.yml file needs to define its type as super:

name: my_super
type: super

If not specified, the type of a liminal.yml file defaults to sub (subliminal).

A superliminal can also inherit from another superliminal by defining its own super attribute.

name: my_super
type: super
super: my_other_super

A liminal system can have 1 subliminal layer but many superliminal layers using inheritence. This allows us to chain common behaviors of our systems into several superliminal layers.

If no super is defined for a liminal.yml file then it defaults to having the base be its super.

superliminal attribtues

A superliminal layer can define the following attributes:

variables
executors
monitoring
task_defaults
pipeline_defaults

If the same attribute section is defined in both a superliminal and a lower layer of the system they will be merged, with key collisions favoring the lower level layer.