In the pipelines
section you can configure scheduled/manually run pipelines of tasks to be executed sequentially:
pipelines: - pipeline: my_data_pipeline timeout_minutes: 30 start_date: 2020-03-01 schedule: 0 10 * * * tasks: - task: my_sql_task type: sql query: "SELECT * FROM {{my_database_name}}.{{my_table_name}} WHERE event_date_prt >= '{{yesterday_ds}}'" AND cms_platform = 'xsite' output_table: my_db.my_out_table output_path: s3://my_bky/{{env}}/mydir - task: my_python_task type: python image: myorg/myrepo:pythonapp cmd: python my_python_app.py env_vars: env: {{env}} fizz: buzz
pipelines
is a section in the root lof your liminal.yml file and is a list of pipeline
s defined by the following attributes:
pipeline
: name of your pipeline (must be unique per liminal server).
timeout_minutes
: maximum allowed pipeline run time in minutes, if run exceeds this time, pipeline and all running tasks will fail.
start_date
: start date for the pipeline.
schedule
: to be configured if the pipeline should run on a schedule. Format is cron expression.
tasks
: list of task
s, defined by the following attributes:
For fully detailed information on tasks see: tasks.
task
: name of your task (must be made of alphanumeric, dash and/or underscore characters only).
type
: type of the task. Examples of available task types are: python
. and more..
Different task types require their own additional configuration. For example, python
task requires image
to be configured.