Definition of your own liminal system is done via yml configuration file named liminal.yml in your project. This yml is the one place in which you define all characteristics and behavior of your liminal system. In this section we'll go over the anatomy of the liminal.yml file.
At the root level of your limimal.yml you can define the most basic root characteristics of your liminal system such as name and owner:
name: MyLiminalSystem owner: Bosco Albert Baracus
For fully detailed information on pipelines see: images.
In the images section you can configure how to pack your code into docker images. This can be achieved in liminal easily, without any Docker knowledge needed by setting just a few attributes:
images: - image: myorg/myrepo:mypythonapp type: python source: . - image: myorg/myrepo:myserver type: python_server source: path/to/my/server/code endpoints: - endpoint: /myendpoint module: my_module function: my_function
images is a section in the root lof your liminal.yml file and is a list of images, defined by the following attributes:
image: name of your image, usually will be in the form of user/repository:tag common in docker image repositories such as Docker Hub, but for local development this can be any string.
type: type of the image. Usually this will match your coding language, or your coding langauge followed by an _ and a specific style of application (such as a server). The current available image types are: python.
source: location of source files to include in the image, this can be a relative path within your project, or even . which means the entire project should be packaged in the image.
build_cmd: certain languages require to be built or compiled, here you can provide your build command to be executed in order to build your code.
Other image types might require additional configuration, for example, servers require endpoints to be configured.
For fully detailed information on pipelines see: pipelines.
In the pipelines section you can configure scheduled/manually run pipelines of tasks to be executed sequentially:
pipelines: - pipeline: my_data_pipeline timeout_minutes: 30 start_date: 2020-03-01 schedule: 0 10 * * * tasks: - task: my_sql_task type: sql query: "SELECT * FROM {{my_database_name}}.{{my_table_name}} WHERE event_date_prt >= '{{yesterday_ds}}'" AND cms_platform = 'xsite' output_table: my_db.my_out_table output_path: s3://my_bky/{{env}}/mydir - task: my_python_task type: python image: myorg/myrepo:pythonapp cmd: python my_python_app.py env_vars: env: {{env}} fizz: buzz - task: my_python_task type: python image: myorg/myrepo:mypythonapp cmd: python -u my_module.py env_vars: env: {{env}} fizz: buzz
pipelines is a section in the root lof your liminal.yml file and is a list of pipelines defined by the following attributes:
pipeline: name of your pipeline (must be unique per liminal server).
timeout_minutes: maximum allowed pipeline run time in minutes, if run exceeds this time, pipeline and all running tasks will fail.
start_date: start date for the pipeline.
schedule: to be configured if the pipeline should run on a schedule. Format is cron expression.
tasks: list of tasks, defined by the following attributes:
For fully detailed information on tasks see: tasks.
task: name of your task (must be made of alphanumeric, dash and/or underscore characters only).
type: type of the task. Examples of available task types are: python. and more..
Different task types require their own additional configuration. For example, python task requires image to be configured.
For fully detailed information on services see: services.
In the services section you can configure constantly running applications such as servers.
services: - service: my_server image: myorg/myrepo:myserver
services is a section in the root lof your liminal.yml file and is a list of services, defined by the following attributes:
service: name of your service.
image: the service's docker image.
For fully detailed information on monitoring see: monitoring.
In the monitoring section you can configure monitoring for your pipelines and services.
monitoring: metrics_backends: - metrics_backend: cloudwatch_metrics type: aws_cloudwatch namespace: DataPipeline AWS_REGION_NAME: us-east-1 alerts_backends: - alerts_backend: cloudwatch_alerts type: aws_cloudwatch metrics_backend: cloudwatch_metrics ok_actions: ['arn:aws:sns:...'] alarm_actions: ['arn:aws:sns:...']
monitoring is a section in the root lof your liminal.yml file and is a list of metrics_backends and a list of alerts_backends.
metrics_backends are where metrics for your pipelines are automatically sent.
alerts_backendss automatically register alerts based on metrics_backend they are paired with.
metrics_backend: name of your metrics backend
type: type of the metrics backend. The current available metrics backends are: aws_cloudwatch.
Different metrics backend types require their own additional configuration. For example, aws_cloudwatch metrics backend requires namespace to be configured.
alerts_backend: name of your alerts backend
type: type of the alerts backend. The current available alerts backends are: aws_cloudwatch.
metrics_backend: name of the metrics backends to register alerts for.
Different alerts backend types require their own additional configuration. For example, aws_cloudwatch alerts backend requires alarm_actions to be configured.
For documentation of more advanced features of liminal.yml see: advanced liminal.yml