To facilitate management, Apache Airflow supports a range of REST API endpoints across its objects. This section provides an overview of the API design, methods, and supported use cases.
Most of the endpoints accept JSON as input and return JSON responses. This means that you must usually add the following headers to your request:
Content-type: application/json Accept: application/json
The term resource refers to a single type of object in the Airflow metadata. An API is broken up by its endpoint's corresponding resource. The name of a resource is typically plural and expressed in camelCase. Example: dagRuns.
Resource names are used as part of endpoint URLs, as well as in API parameters and responses.
The platform supports Create, Read, Update, and Delete operations on most resources. You can review the standards for these operations and their standard parameters below.
Some endpoints have special behavior as exceptions.
To create a resource, you typically submit an HTTP POST request with the resource‘s required metadata in the request body. The response returns a 201 Created response code upon success with the resource’s metadata, including its internal id, in the response body.
The HTTP GET request can be used to read a resource or to list a number of resources.
A resource‘s id can be submitted in the request parameters to read a specific resource. The response usually returns a 200 OK response code upon success, with the resource’s metadata in the response body.
If a GET request does not include a specific resource id, it is treated as a list request. The response usually returns a 200 OK response code upon success, with an object containing a list of resources' metadata in the response body.
When reading resources, some common query parameters are usually available. e.g.:
v1/connections?limit=25&offset=25
| Query Parameter | Type | Description |
|---|---|---|
| limit | integer | Maximum number of objects to fetch. Usually 25 by default |
| offset | integer | Offset after which to start returning objects. For use with limit query parameter. |
Updating a resource requires the resource id, and is typically done using an HTTP PATCH request, with the fields to modify in the request body. The response usually returns a 200 OK response code upon success, with information about the modified resource in the response body.
Deleting a resource requires the resource id and is typically executing via an HTTP DELETE request. The response usually returns a 204 No Content response code upon success.
Resource names are plural and expressed in camelCase.
Names are consistent between URL parameter name and field name.
Field names are in snake_case.
{ \"name\": \"string\", \"slots\": 0, \"occupied_slots\": 0, \"used_slots\": 0, \"queued_slots\": 0, \"open_slots\": 0 }
Update mask is available as a query parameter in patch endpoints. It is used to notify the API which fields you want to update. Using update_mask makes it easier to update objects by helping the server know which fields to update in an object instead of updating all fields. The update request ignores any fields that aren't specified in the field mask, leaving them with their current values.
Example:
resource = request.get('/resource/my-id').json()
resource['my_field'] = 'new-value'
request.patch('/resource/my-id?update_mask=my_field', data=json.dumps(resource))
You can use a third party client, such as curl, HTTPie, Postman or the Insomnia rest client to test the Apache Airflow API.
Note that you will need to pass credentials data.
For e.g., here is how to pause a DAG with curl, when basic authorization is used:
curl -X PATCH 'https://example.com/api/v1/dags/{dag_id}?update_mask=is_paused' \\ -H 'Content-Type: application/json' \\ --user \"username:password\" \\ -d '{ \"is_paused\": true }'
Using a graphical tool such as Postman or Insomnia, it is possible to import the API specifications directly:
Note that with Postman, you can also generate code snippets by selecting a request and clicking on the Code button.
Cross-origin resource sharing (CORS) is a browser security feature that restricts HTTP requests that are initiated from scripts running in the browser.
For details on enabling/configuring CORS, see Enabling CORS.
To be able to meet the requirements of many organizations, Airflow supports many authentication methods, and it is even possible to add your own method.
If you want to check which auth backend is currently set, you can use airflow config get-value api auth_backends command as in the example below.
$ airflow config get-value api auth_backends airflow.api.auth.backend.basic_auth
The default is to deny all requests.
For details on configuring the authentication, see API Authorization.
We follow the error response format proposed in RFC 7807 also known as Problem Details for HTTP APIs. As with our normal API responses, your client must be prepared to gracefully handle additional members of the response.
This indicates that the request has not been applied because it lacks valid authentication credentials for the target resource. Please check that you have valid credentials.
This response means that the server understood the request but refuses to authorize it because it lacks sufficient rights to the resource. It happens when you do not have the necessary permission to execute the action you performed. You need to get the appropriate permissions in other to resolve this error.
This response means that the server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing). To resolve this, please ensure that your syntax is correct.
This client error response indicates that the server cannot find the requested resource.
Indicates that the request method is known by the server but is not supported by the target resource.
The target resource does not have a current representation that would be acceptable to the user agent, according to the proactive negotiation header fields received in the request, and the server is unwilling to supply a default representation.
The request could not be completed due to a conflict with the current state of the target resource, e.g. the resource it tries to create already exists.
This means that the server encountered an unexpected condition that prevented it from fulfilling the request.
This API client was generated by the OpenAPI Generator project. By using the OpenAPI-spec from a remote server, you can easily generate an API client.
Install the following dependencies:
go get github.com/stretchr/testify/assert go get golang.org/x/oauth2 go get golang.org/x/net/context
Put the package under your project folder and add the following in import:
import sw "./airflow"
To use a proxy, set the environment variable HTTP_PROXY:
os.Setenv("HTTP_PROXY", "http://proxy_name:proxy_port")
Default configuration comes with Servers field that contains server objects as defined in the OpenAPI specification.
For using other server than the one defined on index 0 set context value sw.ContextServerIndex of type int.
ctx := context.WithValue(context.Background(), sw.ContextServerIndex, 1)
Templated server URL is formatted using default variables from configuration or from context value sw.ContextServerVariables of type map[string]string.
ctx := context.WithValue(context.Background(), sw.ContextServerVariables, map[string]string{ "basePath": "v2", })
Note, enum values are always validated and all unused variables are silently ignored.
Each operation can use different server URL defined using OperationServers map in the Configuration. An operation is uniquely identified by "{classname}Service.{nickname}" string. Similar rules for overriding default operation server index and variables applies by using sw.ContextOperationServerIndices and sw.ContextOperationServerVariables context maps.
ctx := context.WithValue(context.Background(), sw.ContextOperationServerIndices, map[string]int{
"{classname}Service.{nickname}": 2,
})
ctx = context.WithValue(context.Background(), sw.ContextOperationServerVariables, map[string]map[string]string{
"{classname}Service.{nickname}": {
"port": "8443",
},
})
All URIs are relative to http://localhost/api/v1
| Class | Method | HTTP request | Description |
|---|---|---|---|
| ConfigApi | GetConfig | Get /config | Get current configuration |
| ConnectionApi | DeleteConnection | Delete /connections/{connection_id} | Delete a connection |
| ConnectionApi | GetConnection | Get /connections/{connection_id} | Get a connection |
| ConnectionApi | GetConnections | Get /connections | List connections |
| ConnectionApi | PatchConnection | Patch /connections/{connection_id} | Update a connection |
| ConnectionApi | PostConnection | Post /connections | Create a connection |
| ConnectionApi | TestConnection | Post /connections/test | Test a connection |
| DAGApi | DeleteDag | Delete /dags/{dag_id} | Delete a DAG |
| DAGApi | GetDag | Get /dags/{dag_id} | Get basic information about a DAG |
| DAGApi | GetDagDetails | Get /dags/{dag_id}/details | Get a simplified representation of DAG |
| DAGApi | GetDagSource | Get /dagSources/{file_token} | Get a source code |
| DAGApi | GetDags | Get /dags | List DAGs |
| DAGApi | GetTask | Get /dags/{dag_id}/tasks/{task_id} | Get simplified representation of a task |
| DAGApi | GetTasks | Get /dags/{dag_id}/tasks | Get tasks for DAG |
| DAGApi | PatchDag | Patch /dags/{dag_id} | Update a DAG |
| DAGApi | PatchDags | Patch /dags | Update DAGs |
| DAGApi | PostClearTaskInstances | Post /dags/{dag_id}/clearTaskInstances | Clear a set of task instances |
| DAGApi | PostSetTaskInstancesState | Post /dags/{dag_id}/updateTaskInstancesState | Set a state of task instances |
| DAGApi | SetMappedTaskInstanceNote | Patch /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/{map_index}/setNote | Update the TaskInstance note. |
| DAGApi | SetTaskInstanceNote | Patch /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/setNote | Update the TaskInstance note. |
| DAGRunApi | ClearDagRun | Post /dags/{dag_id}/dagRuns/{dag_run_id}/clear | Clear a DAG run |
| DAGRunApi | DeleteDagRun | Delete /dags/{dag_id}/dagRuns/{dag_run_id} | Delete a DAG run |
| DAGRunApi | GetDagRun | Get /dags/{dag_id}/dagRuns/{dag_run_id} | Get a DAG run |
| DAGRunApi | GetDagRuns | Get /dags/{dag_id}/dagRuns | List DAG runs |
| DAGRunApi | GetDagRunsBatch | Post /dags/~/dagRuns/list | List DAG runs (batch) |
| DAGRunApi | GetUpstreamDatasetEvents | Get /dags/{dag_id}/dagRuns/{dag_run_id}/upstreamDatasetEvents | Get dataset events for a DAG run |
| DAGRunApi | PostDagRun | Post /dags/{dag_id}/dagRuns | Trigger a new DAG run |
| DAGRunApi | SetDagRunNote | Patch /dags/{dag_id}/dagRuns/{dag_run_id}/setNote | Update the DagRun note. |
| DAGRunApi | UpdateDagRunState | Patch /dags/{dag_id}/dagRuns/{dag_run_id} | Modify a DAG run |
| DagWarningApi | GetDagWarnings | Get /dagWarnings | List dag warnings |
| DatasetApi | GetDataset | Get /datasets/{uri} | Get a dataset |
| DatasetApi | GetDatasetEvents | Get /datasets/events | Get dataset events |
| DatasetApi | GetDatasets | Get /datasets | List datasets |
| DatasetApi | GetUpstreamDatasetEvents | Get /dags/{dag_id}/dagRuns/{dag_run_id}/upstreamDatasetEvents | Get dataset events for a DAG run |
| EventLogApi | GetEventLog | Get /eventLogs/{event_log_id} | Get a log entry |
| EventLogApi | GetEventLogs | Get /eventLogs | List log entries |
| ImportErrorApi | GetImportError | Get /importErrors/{import_error_id} | Get an import error |
| ImportErrorApi | GetImportErrors | Get /importErrors | List import errors |
| MonitoringApi | GetHealth | Get /health | Get instance status |
| MonitoringApi | GetVersion | Get /version | Get version information |
| PermissionApi | GetPermissions | Get /permissions | List permissions |
| PluginApi | GetPlugins | Get /plugins | Get a list of loaded plugins |
| PoolApi | DeletePool | Delete /pools/{pool_name} | Delete a pool |
| PoolApi | GetPool | Get /pools/{pool_name} | Get a pool |
| PoolApi | GetPools | Get /pools | List pools |
| PoolApi | PatchPool | Patch /pools/{pool_name} | Update a pool |
| PoolApi | PostPool | Post /pools | Create a pool |
| ProviderApi | GetProviders | Get /providers | List providers |
| RoleApi | DeleteRole | Delete /roles/{role_name} | Delete a role |
| RoleApi | GetRole | Get /roles/{role_name} | Get a role |
| RoleApi | GetRoles | Get /roles | List roles |
| RoleApi | PatchRole | Patch /roles/{role_name} | Update a role |
| RoleApi | PostRole | Post /roles | Create a role |
| TaskInstanceApi | GetExtraLinks | Get /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/links | List extra links |
| TaskInstanceApi | GetLog | Get /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/logs/{task_try_number} | Get logs |
| TaskInstanceApi | GetMappedTaskInstance | Get /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/{map_index} | Get a mapped task instance |
| TaskInstanceApi | GetMappedTaskInstances | Get /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/listMapped | List mapped task instances |
| TaskInstanceApi | GetTaskInstance | Get /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id} | Get a task instance |
| TaskInstanceApi | GetTaskInstances | Get /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances | List task instances |
| TaskInstanceApi | GetTaskInstancesBatch | Post /dags/ | List task instances (batch) |
| TaskInstanceApi | PatchMappedTaskInstance | Patch /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/{map_index} | Updates the state of a mapped task instance |
| TaskInstanceApi | PatchTaskInstance | Patch /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id} | Updates the state of a task instance |
| UserApi | DeleteUser | Delete /users/{username} | Delete a user |
| UserApi | GetUser | Get /users/{username} | Get a user |
| UserApi | GetUsers | Get /users | List users |
| UserApi | PatchUser | Patch /users/{username} | Update a user |
| UserApi | PostUser | Post /users | Create a user |
| VariableApi | DeleteVariable | Delete /variables/{variable_key} | Delete a variable |
| VariableApi | GetVariable | Get /variables/{variable_key} | Get a variable |
| VariableApi | GetVariables | Get /variables | List variables |
| VariableApi | PatchVariable | Patch /variables/{variable_key} | Update a variable |
| VariableApi | PostVariables | Post /variables | Create a variable |
| XComApi | GetXcomEntries | Get /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/xcomEntries | List XCom entries |
| XComApi | GetXcomEntry | Get /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/xcomEntries/{xcom_key} | Get an XCom entry |
Example
auth := context.WithValue(context.Background(), sw.ContextBasicAuth, sw.BasicAuth{ UserName: "username", Password: "password", }) r, err := client.Service.Operation(auth, args)
Due to the fact that model structure members are all pointers, this package contains a number of utility functions to easily obtain pointers to values of basic types. Each of these functions takes a value of the given basic type and returns a pointer to it:
PtrBoolPtrIntPtrInt32PtrInt64PtrFloatPtrFloat32PtrFloat64PtrStringPtrTime