In this section you will learn about advanced features of liminal.yml
Much like in programming languages, you can define variables for re-use across your liminal.yml file.
variables: myvar1: myvalue1 myvar2: myvalue2
In the variables
section of your liminal.yml you can define your variables as key value pairs
You can use placeholders of format {{myplaceholder}}
in most any string value in your liminal.yml Make sure that any string that includes placeholders is surrounded by single or double quotes.
For example:
variables: image_name: myorg/myrepo:myapp images: - image: '{{image_name}}' type: python source: . pipelines: - pipeline: my_pipeline tasks: - task: my_python_task type: python image: "{{image_name}}" cmd: python -u my_module.py
The process of rendering placeholders checks for values in several places, by order.
When running pipelines placeholders will be rendered by replacing values from sources in the following order:
"{{yesterday_ds}}"
. For more information see:Airflow Jinja Templating (Airflow).
When using liminal build
placeholders will be rendered by replacing values from sources in the following order:
In addition to the variables section you can also set specific variables for specific task
s using the variables
attribute of that task. For example:
- task: my_task type: python cmd: python -u my_module.py variables: myvar1: myvalue1 myvar2: myvalue2
You can also pass a reference to a variable map (dictionary) set in your variables section:
variables: my_variable_map: myvar1: myvalue1 myvar2: myvalue2 my_variable_map2: myvar1: myvalue_a myvar2: myvalue_x pipelines: - pipeline: my_pipeline tasks: - task: my_task1 type: python cmd: python -u my_module.py variables: my_variable_map1 - task: my_task2 type: python cmd: python -u my_module.py variables: my_variable_map2
For fully detailed information on executors see: executors.
Each task
in your pipelines has a reference to an executor from the executors
section of your liminal.yml file. If not specified, the appropriate default executor for that task type is used.
executors: - executor: my_kubernetes_executor type: kubernetes resources: request_memory: 128Mi pipelines: pipeline: my_pipeline tasks: - task: my_python_task type: python image: myorg/myrepo:mypythonapp executor: my_kubernetes_executor cmd: python -u my_module.py
In the example above we define an executor
of type kubernetes
with custom resources configuration.
executors
is a section in the root of your liminal.yml file and is a list of executor
s defined by the following attributes:
executor
: name of your executor.
type
: type of the executor. The current available image types are: kubernetes
and emr
.
Different executor types support their own additional configuration.
task_defaults
is a section in the root of your liminal.yml file in which default attributes can be set for each task
type.
task_defaults: python: executor: my_kubernetes_executor env_vars: env: {{env}} foo: bar
In the example above we set default attributes for any python
task in any pipeline in the liminal system defined by our liminal.yml file. Each python
task that does not set these attributes will default to the setting in task_defaults
.
If the same attribute is defined in both task_defaults
and in the task
, definitions from the task
take precedence. If a map (dictionary) of values (for example env_vars
) is defined in both task_defaults
the two maps will be merged, with definitions in the task
taking precedence in case key is defined in both maps.
pipeline_defaults
is a section in the root of your liminal.yml file in which default attributes can be set for each pipeline
in the liminal system defined in your liminal.yml file.
pipeline_defaults: start_date: 2021-01-01 timeout_minutes: 30
In the example above we set default attributes for any pipeline
defined in our liminal.yml file. Each pipeline
that does not set these attributes will default to the setting in pipeline_defaults
.
If the same attribute is defined in both pipeline_defaults
and in the pipeline
, definitions from the pipeline
take precedence.
If tasks
section is defined in pipeline_defaults
each pipeline defined in our liminal.yml file will have the tasks defined in pipeline_defaults
.
A special task
type pipeline
may be used in pipeline_defaults
tasks
section. This task type is interpreted as " tasks defined in pipeline go here". This allows flexibility of defining common tasks to be run before and after the tasks of each pipeline defined in our liminal.yml file. For example:
pipeline_defaults: before_tasks: - task: my_common_setup_task type: python image: myorg/myrepo:mypythonapp cmd: python -u my_setup_module.py after_tasks: - task: my_common_teardown_task1 type: python image: myorg/myrepo:mypythonapp cmd: python -u my_teardown_module1.py - task: my_common_teardown_task2 type: python image: myorg/myrepo:mypythonapp cmd: python -u my_teardown_module2.py
In the example above we set a list of tasks
in pipeline_defaults
which leads to each pipeline defined in our liminal.yml file will have my_common_setup_task
run before its tasks and my_common_teardown_task1
and my_common_teardown_task2
after its tasks.
Each liminal.yml defines a subliminal layer of a liminal system. A subliminal layer can inherit attributes of a superliminal layer by specifying super: name_of_super
in the root of the yml.
name: my_system super: my_super
A super is found by this name if a liminal.yml file in your environment exists with that name. A superliminal layer liminal.yml file needs to define its type
as super
:
name: my_super type: super
If not specified, the type
of a liminal.yml file defaults to sub
(subliminal).
A superliminal can also inherit from another superliminal by defining its own super
attribute.
name: my_super type: super super: my_other_super
A liminal system can have 1 subliminal layer but many superliminal layers using inheritence. This allows us to chain common behaviors of our systems into several superliminal layers.
If no super
is defined for a liminal.yml file then it defaults to having the base be its super.
A superliminal layer can define the following attributes:
variables executors monitoring task_defaults pipeline_defaults
If the same attribute section is defined in both a superliminal and a lower layer of the system they will be merged, with key collisions favoring the lower level layer.