Google Git
Sign in
apache / airflow-site / f81e48aadee354653679693fa54b04ac42d24bfd / . / docs-archive / 1.10.3 / _sources / _api / airflow / models / index.rst.txt
blob: 85aca2242ff4132d53f53bce7de02437f259d7be [file] [log] [blame]
:mod:`airflow.models`
=====================
.. py:module:: airflow.models
Submodules
----------
.. toctree::
:titlesonly:
:maxdepth: 1
base/index.rst
connection/index.rst
dagpickle/index.rst
errors/index.rst
kubernetes/index.rst
log/index.rst
skipmixin/index.rst
slamiss/index.rst
taskfail/index.rst
taskreschedule/index.rst
Package Contents
----------------
.. data:: Base
.. data:: ID_LEN
:annotation: = 250
.. function:: GetDefaultExecutor()
Creates a new instance of the configured executor if none exists and returns it
.. py:class:: LocalExecutor
Bases::class:`airflow.executors.base_executor.BaseExecutor`
LocalExecutor executes tasks locally in parallel. It uses the
multiprocessing Python library and queues to parallelize the execution
of tasks.
.. py:class:: _UnlimitedParallelism(executor)
Bases::class:`object`
Implements LocalExecutor with unlimited parallelism, starting one process
per each command to execute.
.. method:: start(self)
.. method:: execute_async(self, key, command)
:param key: the key to identify the TI
:type key: tuple(dag_id, task_id, execution_date)
:param command: the command to execute
:type command: str
.. method:: sync(self)
.. method:: end(self)
.. py:class:: _LimitedParallelism(executor)
Bases::class:`object`
Implements LocalExecutor with limited parallelism using a task queue to
coordinate work distribution.
.. method:: start(self)
.. method:: execute_async(self, key, command)
:param key: the key to identify the TI
:type key: tuple(dag_id, task_id, execution_date)
:param command: the command to execute
:type command: str
.. method:: sync(self)
.. method:: end(self)
.. method:: start(self)
.. method:: execute_async(self, key, command, queue=None, executor_config=None)
.. method:: sync(self)
.. method:: end(self)
.. py:exception:: AirflowDagCycleException
Bases::class:`airflow.exceptions.AirflowException`
.. py:exception:: AirflowException
Bases::class:`Exception`
Base class for all Airflow's errors.
Each custom exception should be derived from this class
.. attribute:: status_code
:annotation: = 500
.. py:exception:: AirflowSkipException
Bases::class:`airflow.exceptions.AirflowException`
.. py:exception:: AirflowTaskTimeout
Bases::class:`airflow.exceptions.AirflowException`
.. py:exception:: AirflowRescheduleException(reschedule_date)
Bases::class:`airflow.exceptions.AirflowException`
Raise when the task should be re-scheduled at a later time.
:param reschedule_date: The date when the task should be rescheduled
:type reschedule: datetime.datetime
.. py:class:: BaseDag
Bases::class:`object`
Base DAG object that both the SimpleDag and DAG inherit.
.. attribute:: __metaclass__
.. attribute:: dag_id
:return: the DAG ID
:rtype: unicode
.. attribute:: task_ids
:return: A list of task IDs that are in this DAG
:rtype: List[unicode]
.. attribute:: full_filepath
:return: The absolute path to the file that contains this DAG's definition
:rtype: unicode
.. method:: concurrency(self)
:return: maximum number of tasks that can run simultaneously from this DAG
:rtype: int
.. method:: is_paused(self)
:return: whether this DAG is paused or not
:rtype: bool
.. method:: pickle_id(self)
:return: The pickle ID for this DAG, if it has one. Otherwise None.
:rtype: unicode
.. py:class:: BaseDagBag
Bases::class:`object`
Base object that both the SimpleDagBag and DagBag inherit.
.. attribute:: dag_ids
:return: a list of DAG IDs in this bag
:rtype: List[unicode]
.. method:: get_dag(self, dag_id)
:return: whether the task exists in this bag
:rtype: airflow.dag.base_dag.BaseDag
.. function:: apply_lineage(func)
Saves the lineage to XCom and if configured to do so sends it
to the backend.
.. function:: prepare_lineage(func)
Prepares the lineage inlets and outlets. Inlets can be:
* "auto" -> picks up any outlets from direct upstream tasks that have outlets defined, as such that
if A -> B -> C and B does not have outlets but A does, these are provided as inlets.
* "list of task_ids" -> picks up outlets from the upstream task_ids
* "list of datasets" -> manually defined list of DataSet
.. py:class:: DagPickle(dag)
Bases::class:`airflow.models.base.Base`
Dags can originate from different places (user repos, master repo, ...)
and also get executed in different places (different executors). This
object represents a version of a DAG and becomes a source of truth for
a BackfillJob execution. A pickle is a native python serialized object,
and in this case gets stored in the database for the duration of the job.
The executors pick up the DagPickle id and read the dag definition from
the database.
.. attribute:: id
.. attribute:: pickle
.. attribute:: created_dttm
.. attribute:: pickle_hash
.. attribute:: __tablename__
:annotation: = dag_pickle
.. py:class:: ImportError
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = import_error
.. attribute:: id
.. attribute:: timestamp
.. attribute:: filename
.. attribute:: stacktrace
.. py:class:: SlaMiss
Bases::class:`airflow.models.base.Base`
Model that stores a history of the SLA that have been missed.
It is used to keep track of SLA failures over time and to avoid double
triggering alert emails.
.. attribute:: __tablename__
:annotation: = sla_miss
.. attribute:: task_id
.. attribute:: dag_id
.. attribute:: execution_date
.. attribute:: email_sent
.. attribute:: timestamp
.. attribute:: description
.. attribute:: notification_sent
.. attribute:: __table_args__
.. method:: __repr__(self)
.. py:class:: KubeWorkerIdentifier
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = kube_worker_uuid
.. attribute:: one_row_id
.. attribute:: worker_uuid
.. staticmethod:: get_or_create_current_kube_worker_uuid(session=None)
.. staticmethod:: checkpoint_kube_worker_uuid(worker_uuid, session=None)
.. py:class:: KubeResourceVersion
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = kube_resource_version
.. attribute:: one_row_id
.. attribute:: resource_version
.. staticmethod:: get_current_resource_version(session=None)
.. staticmethod:: checkpoint_resource_version(resource_version, session=None)
.. staticmethod:: reset_resource_version(session=None)
.. py:class:: Log(event, task_instance, owner=None, extra=None, **kwargs)
Bases::class:`airflow.models.base.Base`
Used to actively log events to the database
.. attribute:: __tablename__
:annotation: = log
.. attribute:: id
.. attribute:: dttm
.. attribute:: dag_id
.. attribute:: task_id
.. attribute:: event
.. attribute:: execution_date
.. attribute:: owner
.. attribute:: extra
.. attribute:: __table_args__
.. py:class:: TaskFail(task, execution_date, start_date, end_date)
Bases::class:`airflow.models.base.Base`
TaskFail tracks the failed run durations of each task instance.
.. attribute:: __tablename__
:annotation: = task_fail
.. attribute:: id
.. attribute:: task_id
.. attribute:: dag_id
.. attribute:: execution_date
.. attribute:: start_date
.. attribute:: end_date
.. attribute:: duration
.. attribute:: __table_args__
.. py:class:: TaskReschedule(task, execution_date, try_number, start_date, end_date, reschedule_date)
Bases::class:`airflow.models.base.Base`
TaskReschedule tracks rescheduled task instances.
.. attribute:: __tablename__
:annotation: = task_reschedule
.. attribute:: id
.. attribute:: task_id
.. attribute:: dag_id
.. attribute:: execution_date
.. attribute:: try_number
.. attribute:: start_date
.. attribute:: end_date
.. attribute:: duration
.. attribute:: reschedule_date
.. attribute:: __table_args__
.. staticmethod:: find_for_task_instance(task_instance, session)
Returns all task reschedules for the task instance and try number,
in ascending order.
:param task_instance: the task instance to find task reschedules for
:type task_instance: airflow.models.TaskInstance
.. py:class:: NotInRetryPeriodDep
Bases::class:`airflow.ti_deps.deps.base_ti_dep.BaseTIDep`
.. attribute:: NAME
:annotation: = Not In Retry Period
.. attribute:: IGNOREABLE
:annotation: = True
.. attribute:: IS_TASK_DEP
:annotation: = True
.. method:: _get_dep_statuses(self, ti, session, dep_context)
.. py:class:: PrevDagrunDep
Bases::class:`airflow.ti_deps.deps.base_ti_dep.BaseTIDep`
Is the past dagrun in a state that allows this task instance to run, e.g. did this
task instance's task in the previous dagrun complete if we are depending on past.
.. attribute:: NAME
:annotation: = Previous Dagrun State
.. attribute:: IGNOREABLE
:annotation: = True
.. attribute:: IS_TASK_DEP
:annotation: = True
.. method:: _get_dep_statuses(self, ti, session, dep_context)
.. py:class:: TriggerRuleDep
Bases::class:`airflow.ti_deps.deps.base_ti_dep.BaseTIDep`
Determines if a task's upstream tasks are in a state that allows a given task instance
to run.
.. attribute:: NAME
:annotation: = Trigger Rule
.. attribute:: IGNOREABLE
:annotation: = True
.. attribute:: IS_TASK_DEP
:annotation: = True
.. method:: _get_dep_statuses(self, ti, session, dep_context)
.. method:: _evaluate_trigger_rule(self, ti, successes, skipped, failed, upstream_failed, done, flag_upstream_failed, session)
Yields a dependency status that indicate whether the given task instance's trigger
rule was met.
:param ti: the task instance to evaluate the trigger rule of
:type ti: airflow.models.TaskInstance
:param successes: Number of successful upstream tasks
:type successes: bool
:param skipped: Number of skipped upstream tasks
:type skipped: bool
:param failed: Number of failed upstream tasks
:type failed: bool
:param upstream_failed: Number of upstream_failed upstream tasks
:type upstream_failed: bool
:param done: Number of completed upstream tasks
:type done: bool
:param flag_upstream_failed: This is a hack to generate
the upstream_failed state creation while checking to see
whether the task instance is runnable. It was the shortest
path to add the feature
:type flag_upstream_failed: bool
:param session: database session
:type session: sqlalchemy.orm.session.Session
.. py:class:: DepContext(deps=None, flag_upstream_failed=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_in_retry_period=False, ignore_in_reschedule_period=False, ignore_task_deps=False, ignore_ti_state=False)
Bases::class:`object`
A base class for contexts that specifies which dependencies should be evaluated in
the context for a task instance to satisfy the requirements of the context. Also
stores state related to the context that can be used by dependency classes.
For example there could be a SomeRunContext that subclasses this class which has
dependencies for:
- Making sure there are slots available on the infrastructure to run the task instance
- A task-instance's task-specific dependencies are met (e.g. the previous task
instance completed successfully)
- ...
:param deps: The context-specific dependencies that need to be evaluated for a
task instance to run in this execution context.
:type deps: set(airflow.ti_deps.deps.base_ti_dep.BaseTIDep)
:param flag_upstream_failed: This is a hack to generate the upstream_failed state
creation while checking to see whether the task instance is runnable. It was the
shortest path to add the feature. This is bad since this class should be pure (no
side effects).
:type flag_upstream_failed: bool
:param ignore_all_deps: Whether or not the context should ignore all ignoreable
dependencies. Overrides the other ignore_* parameters
:type ignore_all_deps: bool
:param ignore_depends_on_past: Ignore depends_on_past parameter of DAGs (e.g. for
Backfills)
:type ignore_depends_on_past: bool
:param ignore_in_retry_period: Ignore the retry period for task instances
:type ignore_in_retry_period: bool
:param ignore_in_reschedule_period: Ignore the reschedule period for task instances
:type ignore_in_reschedule_period: bool
:param ignore_task_deps: Ignore task-specific dependencies such as depends_on_past and
trigger rule
:type ignore_task_deps: bool
:param ignore_ti_state: Ignore the task instance's previous failure/success
:type ignore_ti_state: bool
.. data:: QUEUE_DEPS
.. data:: RUN_DEPS
.. function:: list_py_file_paths(directory, safe_mode=True, include_examples=None)
Traverse a directory and look for Python files.
:param directory: the directory to traverse
:type directory: unicode
:param safe_mode: whether to use a heuristic to determine whether a file
contains Airflow DAG definitions
:return: a list of paths to Python files in the specified directory
:rtype: list[unicode]
.. data:: cron_presets
.. function:: utils_date_range(start_date, end_date=None, num=None, delta=None)
Get a set of dates as a list based on a start, end and delta, delta
can be something that can be added to `datetime.datetime`
or a cron expression as a `str`
:Example::
date_range(datetime(2016, 1, 1), datetime(2016, 1, 3), delta=timedelta(1))
[datetime.datetime(2016, 1, 1, 0, 0), datetime.datetime(2016, 1, 2, 0, 0),
datetime.datetime(2016, 1, 3, 0, 0)]
date_range(datetime(2016, 1, 1), datetime(2016, 1, 3), delta='0 0 * * *')
[datetime.datetime(2016, 1, 1, 0, 0), datetime.datetime(2016, 1, 2, 0, 0),
datetime.datetime(2016, 1, 3, 0, 0)]
date_range(datetime(2016, 1, 1), datetime(2016, 3, 3), delta="0 0 0 * *")
[datetime.datetime(2016, 1, 1, 0, 0), datetime.datetime(2016, 2, 1, 0, 0),
datetime.datetime(2016, 3, 1, 0, 0)]
:param start_date: anchor date to start the series from
:type start_date: datetime.datetime
:param end_date: right boundary for the date range
:type end_date: datetime.datetime
:param num: alternatively to end_date, you can specify the number of
number of entries you want in the range. This number can be negative,
output will always be sorted regardless
:type num: int
.. function:: provide_session(func)
Function decorator that provides a session if it isn't provided.
If you want to reuse a session or run the function as part of a
database transaction, you pass it to the function, if not this wrapper
will create one and close it for you.
.. data:: apply_defaults
.. function:: send_email(to, subject, html_content, files=None, dryrun=False, cc=None, bcc=None, mime_subtype='mixed', mime_charset='us-ascii', **kwargs)
Send email using backend specified in EMAIL_BACKEND.
.. function:: as_tuple(obj)
If obj is a container, returns obj as a tuple.
Otherwise, returns a tuple containing obj.
.. function:: is_container(obj)
Test if an object is a container (iterable) but not a string
.. function:: validate_key(k, max_length=250)
.. function:: pprinttable(rows)
Returns a pretty ascii table from tuples
If namedtuple are used, the table will have headers
.. py:class:: Resources(cpus=configuration.conf.getint('operators', 'default_cpus'), ram=configuration.conf.getint('operators', 'default_ram'), disk=configuration.conf.getint('operators', 'default_disk'), gpus=configuration.conf.getint('operators', 'default_gpus'))
Bases::class:`object`
The resources required by an operator. Resources that are not specified will use the
default values from the airflow config.
:param cpus: The number of cpu cores that are required
:type cpus: long
:param ram: The amount of RAM required
:type ram: long
:param disk: The amount of disk space required
:type disk: long
:param gpus: The number of gpu units that are required
:type gpus: long
.. method:: __eq__(self, other)
.. method:: __repr__(self)
.. py:class:: State
Bases::class:`object`
Static class with task instance states constants and color method to
avoid hardcoding.
.. attribute:: NONE
.. attribute:: REMOVED
:annotation: = removed
.. attribute:: SCHEDULED
:annotation: = scheduled
.. attribute:: QUEUED
:annotation: = queued
.. attribute:: RUNNING
:annotation: = running
.. attribute:: SUCCESS
:annotation: = success
.. attribute:: SHUTDOWN
:annotation: = shutdown
.. attribute:: FAILED
:annotation: = failed
.. attribute:: UP_FOR_RETRY
:annotation: = up_for_retry
.. attribute:: UP_FOR_RESCHEDULE
:annotation: = up_for_reschedule
.. attribute:: UPSTREAM_FAILED
:annotation: = upstream_failed
.. attribute:: SKIPPED
:annotation: = skipped
.. attribute:: task_states
.. attribute:: dag_states
.. attribute:: state_color
.. classmethod:: color(cls, state)
.. classmethod:: color_fg(cls, state)
.. classmethod:: finished(cls)
A list of states indicating that a task started and completed a
run attempt. Note that the attempt could have resulted in failure or
have been interrupted; in any case, it is no longer running.
.. classmethod:: unfinished(cls)
A list of states indicating that a task either has not completed
a run or has not even started.
.. py:class:: UtcDateTime
Bases::class:`sqlalchemy.types.TypeDecorator`
Almost equivalent to :class:`~sqlalchemy.types.DateTime` with
``timezone=True`` option, but it differs from that by:
- Never silently take naive :class:`~datetime.datetime`, instead it
always raise :exc:`ValueError` unless time zone aware value.
- :class:`~datetime.datetime` value's :attr:`~datetime.datetime.tzinfo`
is always converted to UTC.
- Unlike SQLAlchemy's built-in :class:`~sqlalchemy.types.DateTime`,
it never return naive :class:`~datetime.datetime`, but time zone
aware value, even with SQLite or MySQL.
- Always returns DateTime in UTC
.. attribute:: impl
.. method:: process_bind_param(self, value, dialect)
.. method:: process_result_value(self, value, dialect)
Processes DateTimes from the DB making sure it is always
returning UTC. Not using timezone.convert_to_utc as that
converts to configured TIMEZONE while the DB might be
running with some other setting. We assume UTC datetimes
in the database.
.. py:class:: Interval
Bases::class:`sqlalchemy.types.TypeDecorator`
.. attribute:: impl
.. attribute:: attr_keys
.. method:: process_bind_param(self, value, dialect)
.. method:: process_result_value(self, value, dialect)
.. py:class:: timeout(seconds=1, error_message='Timeout')
Bases::class:`airflow.utils.log.logging_mixin.LoggingMixin`
To be used in a ``with`` block and timeout its content.
.. method:: handle_timeout(self, signum, frame)
.. method:: __enter__(self)
.. method:: __exit__(self, type, value, traceback)
.. py:class:: TriggerRule
Bases::class:`object`
.. attribute:: ALL_SUCCESS
:annotation: = all_success
.. attribute:: ALL_FAILED
:annotation: = all_failed
.. attribute:: ALL_DONE
:annotation: = all_done
.. attribute:: ONE_SUCCESS
:annotation: = one_success
.. attribute:: ONE_FAILED
:annotation: = one_failed
.. attribute:: NONE_FAILED
:annotation: = none_failed
.. attribute:: NONE_SKIPPED
:annotation: = none_skipped
.. attribute:: DUMMY
:annotation: = dummy
.. attribute:: _ALL_TRIGGER_RULES
.. classmethod:: is_valid(cls, trigger_rule)
.. classmethod:: all_triggers(cls)
.. py:class:: WeightRule
Bases::class:`object`
.. attribute:: DOWNSTREAM
:annotation: = downstream
.. attribute:: UPSTREAM
:annotation: = upstream
.. attribute:: ABSOLUTE
:annotation: = absolute
.. attribute:: _ALL_WEIGHT_RULES
.. classmethod:: is_valid(cls, weight_rule)
.. classmethod:: all_weight_rules(cls)
.. function:: get_hostname()
Fetch the hostname using the callable from the config or using
`socket.getfqdn` as a fallback.
.. py:class:: LoggingMixin(context=None)
Bases::class:`object`
Convenience super-class to have a logger configured with the class name
.. attribute:: logger
.. attribute:: log
.. method:: _set_context(self, context)
.. data:: XCOM_RETURN_KEY
:annotation: = return_value
.. data:: Stats
.. py:exception:: InvalidFernetToken
Bases::class:`Exception`
.. py:class:: NullFernet
Bases::class:`object`
A "Null" encryptor class that doesn't encrypt or decrypt but that presents
a similar interface to Fernet.
The purpose of this is to make the rest of the code not have to know the
difference, and to only display the message once, not 20 times when
`airflow initdb` is ran.
.. attribute:: is_encrypted
:annotation: = False
.. method:: decrpyt(self, b)
.. method:: encrypt(self, b)
.. data:: _fernet
.. function:: get_fernet()
Deferred load of Fernet key.
This function could fail either because Cryptography is not installed
or because the Fernet key is invalid.
:return: Fernet object
:raises: airflow.exceptions.AirflowException if there's a problem trying to load Fernet
.. data:: _CONTEXT_MANAGER_DAG
.. function:: clear_task_instances(tis, session, activate_dag_runs=True, dag=None)
Clears a set of task instances, but makes sure the running ones
get killed.
:param tis: a list of task instances
:param session: current session
:param activate_dag_runs: flag to check for active dag run
:param dag: DAG object
.. function:: get_last_dagrun(dag_id, session, include_externally_triggered=False)
Returns the last dag run for a dag, None if there was none.
Last dag run can be any type of run eg. scheduled or backfilled.
Overridden DagRuns are ignored.
.. py:class:: DagBag(dag_folder=None, executor=None, include_examples=configuration.conf.getboolean('core', 'LOAD_EXAMPLES'), safe_mode=configuration.conf.getboolean('core', 'DAG_DISCOVERY_SAFE_MODE'))
Bases::class:`airflow.dag.base_dag.BaseDagBag`, :class:`airflow.utils.log.logging_mixin.LoggingMixin`
A dagbag is a collection of dags, parsed out of a folder tree and has high
level configuration settings, like what database to use as a backend and
what executor to use to fire off tasks. This makes it easier to run
distinct environments for say production and development, tests, or for
different teams or security profiles. What would have been system level
settings are now dagbag level so that one system can run multiple,
independent settings sets.
:param dag_folder: the folder to scan to find DAGs
:type dag_folder: unicode
:param executor: the executor to use when executing task instances
in this DagBag
:param include_examples: whether to include the examples that ship
with airflow or not
:type include_examples: bool
:param has_logged: an instance boolean that gets flipped from False to True after a
file has been skipped. This is to prevent overloading the user with logging
messages about skipped files. Therefore only once per DagBag is a file logged
being skipped.
.. attribute:: CYCLE_NEW
:annotation: = 0
.. attribute:: CYCLE_IN_PROGRESS
:annotation: = 1
.. attribute:: CYCLE_DONE
:annotation: = 2
.. attribute:: dag_ids
.. method:: size(self)
:return: the amount of dags contained in this dagbag
.. method:: get_dag(self, dag_id)
Gets the DAG out of the dictionary, and refreshes it if expired
.. method:: process_file(self, filepath, only_if_updated=True, safe_mode=True)
Given a path to a python module or zip file, this method imports
the module and look for dag objects within it.
.. method:: kill_zombies(self, zombies, session=None)
Fail given zombie tasks, which are tasks that haven't
had a heartbeat for too long, in the current DagBag.
:param zombies: zombie task instances to kill.
:type zombies: airflow.utils.dag_processing.SimpleTaskInstance
:param session: DB session.
:type session: sqlalchemy.orm.session.Session
.. method:: bag_dag(self, dag, parent_dag, root_dag)
Adds the DAG into the bag, recurses into sub dags.
Throws AirflowDagCycleException if a cycle is detected in this dag or its subdags
.. method:: collect_dags(self, dag_folder=None, only_if_updated=True, include_examples=configuration.conf.getboolean('core', 'LOAD_EXAMPLES'), safe_mode=configuration.conf.getboolean('core', 'DAG_DISCOVERY_SAFE_MODE'))
Given a file path or a folder, this method looks for python modules,
imports them and adds them to the dagbag collection.
Note that if a ``.airflowignore`` file is found while processing
the directory, it will behave much like a ``.gitignore``,
ignoring files that match any of the regex patterns specified
in the file.
**Note**: The patterns in .airflowignore are treated as
un-anchored regexes, not shell-like glob patterns.
.. method:: dagbag_report(self)
Prints a report around DagBag loading stats
.. py:class:: User
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = users
.. attribute:: id
.. attribute:: username
.. attribute:: email
.. attribute:: superuser
.. method:: __repr__(self)
.. method:: get_id(self)
.. method:: is_superuser(self)
.. py:class:: TaskInstance(task, execution_date, state=None)
Bases::class:`airflow.models.base.Base`, :class:`airflow.utils.log.logging_mixin.LoggingMixin`
Task instances store the state of a task instance. This table is the
authority and single source of truth around what tasks have run and the
state they are in.
The SqlAlchemy model doesn't have a SqlAlchemy foreign key to the task or
dag model deliberately to have more control over transactions.
Database transactions on this table should insure double triggers and
any confusion around what task instances are or aren't ready to run
even while multiple schedulers may be firing task instances.
.. attribute:: __tablename__
:annotation: = task_instance
.. attribute:: task_id
.. attribute:: dag_id
.. attribute:: execution_date
.. attribute:: start_date
.. attribute:: end_date
.. attribute:: duration
.. attribute:: state
.. attribute:: _try_number
.. attribute:: max_tries
.. attribute:: hostname
.. attribute:: unixname
.. attribute:: job_id
.. attribute:: pool
.. attribute:: queue
.. attribute:: priority_weight
.. attribute:: operator
.. attribute:: queued_dttm
.. attribute:: pid
.. attribute:: executor_config
.. attribute:: __table_args__
.. attribute:: try_number
Return the try number that this task number will be when it is actually
run.
If the TI is currently running, this will match the column in the
databse, in all othercases this will be incremenetd
.. attribute:: next_try_number
.. attribute:: log_filepath
.. attribute:: log_url
.. attribute:: mark_success_url
.. attribute:: key
Returns a tuple that identifies the task instance uniquely
.. attribute:: is_premature
Returns whether a task is in UP_FOR_RETRY state and its retry interval
has elapsed.
.. attribute:: previous_ti
The task instance for the task that ran before this task instance.
.. method:: init_on_load(self)
Initialize the attributes that aren't stored in the DB.
.. method:: command(self, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, local=False, pickle_id=None, raw=False, job_id=None, pool=None, cfg_path=None)
Returns a command that can be executed anywhere where airflow is
installed. This command is part of the message sent to executors by
the orchestrator.
.. method:: command_as_list(self, mark_success=False, ignore_all_deps=False, ignore_task_deps=False, ignore_depends_on_past=False, ignore_ti_state=False, local=False, pickle_id=None, raw=False, job_id=None, pool=None, cfg_path=None)
Returns a command that can be executed anywhere where airflow is
installed. This command is part of the message sent to executors by
the orchestrator.
.. staticmethod:: generate_command(dag_id, task_id, execution_date, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, local=False, pickle_id=None, file_path=None, raw=False, job_id=None, pool=None, cfg_path=None)
Generates the shell command required to execute this task instance.
:param dag_id: DAG ID
:type dag_id: unicode
:param task_id: Task ID
:type task_id: unicode
:param execution_date: Execution date for the task
:type execution_date: datetime
:param mark_success: Whether to mark the task as successful
:type mark_success: bool
:param ignore_all_deps: Ignore all ignorable dependencies.
Overrides the other ignore_* parameters.
:type ignore_all_deps: bool
:param ignore_depends_on_past: Ignore depends_on_past parameter of DAGs
(e.g. for Backfills)
:type ignore_depends_on_past: bool
:param ignore_task_deps: Ignore task-specific dependencies such as depends_on_past
and trigger rule
:type ignore_task_deps: bool
:param ignore_ti_state: Ignore the task instance's previous failure/success
:type ignore_ti_state: bool
:param local: Whether to run the task locally
:type local: bool
:param pickle_id: If the DAG was serialized to the DB, the ID
associated with the pickled DAG
:type pickle_id: unicode
:param file_path: path to the file containing the DAG definition
:param raw: raw mode (needs more details)
:param job_id: job ID (needs more details)
:param pool: the Airflow pool that the task should run in
:type pool: unicode
:param cfg_path: the Path to the configuration file
:type cfg_path: basestring
:return: shell command that can be used to run the task instance
.. method:: current_state(self, session=None)
Get the very latest state from the database, if a session is passed,
we use and looking up the state becomes part of the session, otherwise
a new session is used.
.. method:: error(self, session=None)
Forces the task instance's state to FAILED in the database.
.. method:: refresh_from_db(self, session=None, lock_for_update=False)
Refreshes the task instance from the database based on the primary key
:param lock_for_update: if True, indicates that the database should
lock the TaskInstance (issuing a FOR UPDATE clause) until the
session is committed.
.. method:: clear_xcom_data(self, session=None)
Clears all XCom data from the database for the task instance
.. method:: set_state(self, state, session=None)
.. method:: are_dependents_done(self, session=None)
Checks whether the dependents of this task instance have all succeeded.
This is meant to be used by wait_for_downstream.
This is useful when you do not want to start processing the next
schedule of a task until the dependents are done. For instance,
if the task DROPs and recreates a table.
.. method:: _get_previous_ti(self, session=None)
.. method:: are_dependencies_met(self, dep_context=None, session=None, verbose=False)
Returns whether or not all the conditions are met for this task instance to be run
given the context for the dependencies (e.g. a task instance being force run from
the UI will ignore some dependencies).
:param dep_context: The execution context that determines the dependencies that
should be evaluated.
:type dep_context: DepContext
:param session: database session
:type session: sqlalchemy.orm.session.Session
:param verbose: whether log details on failed dependencies on
info or debug log level
:type verbose: bool
.. method:: get_failed_dep_statuses(self, dep_context=None, session=None)
.. method:: __repr__(self)
.. method:: next_retry_datetime(self)
Get datetime of the next retry if the task instance fails. For exponential
backoff, retry_delay is used as base and will be converted to seconds.
.. method:: ready_for_retry(self)
Checks on whether the task instance is in the right state and timeframe
to be retried.
.. method:: pool_full(self, session)
Returns a boolean as to whether the slot pool has room for this
task to run
.. method:: get_dagrun(self, session)
Returns the DagRun for this TaskInstance
:param session:
:return: DagRun
.. method:: _check_and_change_state_before_execution(self, verbose=True, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, mark_success=False, test_mode=False, job_id=None, pool=None, session=None)
Checks dependencies and then sets state to RUNNING if they are met. Returns
True if and only if state is set to RUNNING, which implies that task should be
executed, in preparation for _run_raw_task
:param verbose: whether to turn on more verbose logging
:type verbose: bool
:param ignore_all_deps: Ignore all of the non-critical dependencies, just runs
:type ignore_all_deps: bool
:param ignore_depends_on_past: Ignore depends_on_past DAG attribute
:type ignore_depends_on_past: bool
:param ignore_task_deps: Don't check the dependencies of this TI's task
:type ignore_task_deps: bool
:param ignore_ti_state: Disregards previous task instance state
:type ignore_ti_state: bool
:param mark_success: Don't run the task, mark its state as success
:type mark_success: bool
:param test_mode: Doesn't record success or failure in the DB
:type test_mode: bool
:param pool: specifies the pool to use to run the task instance
:type pool: str
:return: whether the state was changed to running or not
:rtype: bool
.. method:: _run_raw_task(self, mark_success=False, test_mode=False, job_id=None, pool=None, session=None)
Immediately runs the task (without checking or changing db state
before execution) and then sets the appropriate final state after
completion and runs any post-execute callbacks. Meant to be called
only after another function changes the state to running.
:param mark_success: Don't run the task, mark its state as success
:type mark_success: bool
:param test_mode: Doesn't record success or failure in the DB
:type test_mode: bool
:param pool: specifies the pool to use to run the task instance
:type pool: str
.. method:: run(self, verbose=True, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, mark_success=False, test_mode=False, job_id=None, pool=None, session=None)
.. method:: dry_run(self)
.. method:: _handle_reschedule(self, actual_start_date, reschedule_exception, test_mode=False, context=None, session=None)
.. method:: handle_failure(self, error, test_mode=False, context=None, session=None)
.. method:: is_eligible_to_retry(self)
Is task instance is eligible for retry
.. method:: get_template_context(self, session=None)
.. method:: overwrite_params_with_dag_run_conf(self, params, dag_run)
.. method:: render_templates(self)
.. method:: email_alert(self, exception)
.. method:: set_duration(self)
.. method:: xcom_push(self, key, value, execution_date=None)
Make an XCom available for tasks to pull.
:param key: A key for the XCom
:type key: str
:param value: A value for the XCom. The value is pickled and stored
in the database.
:type value: any pickleable object
:param execution_date: if provided, the XCom will not be visible until
this date. This can be used, for example, to send a message to a
task on a future date without it being immediately visible.
:type execution_date: datetime
.. method:: xcom_pull(self, task_ids=None, dag_id=None, key=XCOM_RETURN_KEY, include_prior_dates=False)
Pull XComs that optionally meet certain criteria.
The default value for `key` limits the search to XComs
that were returned by other tasks (as opposed to those that were pushed
manually). To remove this filter, pass key=None (or any desired value).
If a single task_id string is provided, the result is the value of the
most recent matching XCom from that task_id. If multiple task_ids are
provided, a tuple of matching values is returned. None is returned
whenever no matches are found.
:param key: A key for the XCom. If provided, only XComs with matching
keys will be returned. The default key is 'return_value', also
available as a constant XCOM_RETURN_KEY. This key is automatically
given to XComs returned by tasks (as opposed to being pushed
manually). To remove the filter, pass key=None.
:type key: str
:param task_ids: Only XComs from tasks with matching ids will be
pulled. Can pass None to remove the filter.
:type task_ids: str or iterable of strings (representing task_ids)
:param dag_id: If provided, only pulls XComs from this DAG.
If None (default), the DAG of the calling task is used.
:type dag_id: str
:param include_prior_dates: If False, only XComs from the current
execution_date are returned. If True, XComs from previous dates
are returned as well.
:type include_prior_dates: bool
.. method:: get_num_running_task_instances(self, session)
.. method:: init_run_context(self, raw=False)
Sets the log context.
.. py:class:: BaseOperator(task_id, owner=configuration.conf.get('operators', 'DEFAULT_OWNER'), email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=timedelta(seconds=300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, priority_weight=1, weight_rule=WeightRule.DOWNSTREAM, queue=configuration.conf.get('celery', 'default_queue'), pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=TriggerRule.ALL_SUCCESS, resources=None, run_as_user=None, task_concurrency=None, executor_config=None, inlets=None, outlets=None, *args, **kwargs)
Bases::class:`airflow.utils.log.logging_mixin.LoggingMixin`
Abstract base class for all operators. Since operators create objects that
become nodes in the dag, BaseOperator contains many recursive methods for
dag crawling behavior. To derive this class, you are expected to override
the constructor as well as the 'execute' method.
Operators derived from this class should perform or trigger certain tasks
synchronously (wait for completion). Example of operators could be an
operator that runs a Pig job (PigOperator), a sensor operator that
waits for a partition to land in Hive (HiveSensorOperator), or one that
moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these
operators (tasks) target specific operations, running specific scripts,
functions or data transfers.
This class is abstract and shouldn't be instantiated. Instantiating a
class derived from this one results in the creation of a task object,
which ultimately becomes a node in DAG objects. Task dependencies should
be set by using the set_upstream and/or set_downstream methods.
:param task_id: a unique, meaningful id for the task
:type task_id: str
:param owner: the owner of the task, using the unix username is recommended
:type owner: str
:param retries: the number of retries that should be performed before
failing the task
:type retries: int
:param retry_delay: delay between retries
:type retry_delay: datetime.timedelta
:param retry_exponential_backoff: allow progressive longer waits between
retries by using exponential backoff algorithm on retry delay (delay
will be converted into seconds)
:type retry_exponential_backoff: bool
:param max_retry_delay: maximum delay interval between retries
:type max_retry_delay: datetime.timedelta
:param start_date: The ``start_date`` for the task, determines
the ``execution_date`` for the first task instance. The best practice
is to have the start_date rounded
to your DAG's ``schedule_interval``. Daily jobs have their start_date
some day at 00:00:00, hourly jobs have their start_date at 00:00
of a specific hour. Note that Airflow simply looks at the latest
``execution_date`` and adds the ``schedule_interval`` to determine
the next ``execution_date``. It is also very important
to note that different tasks' dependencies
need to line up in time. If task A depends on task B and their
start_date are offset in a way that their execution_date don't line
up, A's dependencies will never be met. If you are looking to delay
a task, for example running a daily task at 2AM, look into the
``TimeSensor`` and ``TimeDeltaSensor``. We advise against using
dynamic ``start_date`` and recommend using fixed ones. Read the
FAQ entry about start_date for more information.
:type start_date: datetime.datetime
:param end_date: if specified, the scheduler won't go beyond this date
:type end_date: datetime.datetime
:param depends_on_past: when set to true, task instances will run
sequentially while relying on the previous task's schedule to
succeed. The task instance for the start_date is allowed to run.
:type depends_on_past: bool
:param wait_for_downstream: when set to true, an instance of task
X will wait for tasks immediately downstream of the previous instance
of task X to finish successfully before it runs. This is useful if the
different instances of a task X alter the same asset, and this asset
is used by tasks downstream of task X. Note that depends_on_past
is forced to True wherever wait_for_downstream is used.
:type wait_for_downstream: bool
:param queue: which queue to target when running this job. Not
all executors implement queue management, the CeleryExecutor
does support targeting specific queues.
:type queue: str
:param dag: a reference to the dag the task is attached to (if any)
:type dag: airflow.models.DAG
:param priority_weight: priority weight of this task against other task.
This allows the executor to trigger higher priority tasks before
others when things get backed up.
:type priority_weight: int
:param weight_rule: weighting method used for the effective total
priority weight of the task. Options are:
``{ downstream | upstream | absolute }`` default is ``downstream``
When set to ``downstream`` the effective weight of the task is the
aggregate sum of all downstream descendants. As a result, upstream
tasks will have higher weight and will be scheduled more aggressively
when using positive weight values. This is useful when you have
multiple dag run instances and desire to have all upstream tasks to
complete for all runs before each dag can continue processing
downstream tasks. When set to ``upstream`` the effective weight is the
aggregate sum of all upstream ancestors. This is the opposite where
downtream tasks have higher weight and will be scheduled more
aggressively when using positive weight values. This is useful when you
have multiple dag run instances and prefer to have each dag complete
before starting upstream tasks of other dags. When set to
``absolute``, the effective weight is the exact ``priority_weight``
specified without additional weighting. You may want to do this when
you know exactly what priority weight each task should have.
Additionally, when set to ``absolute``, there is bonus effect of
significantly speeding up the task creation process as for very large
DAGS. Options can be set as string or using the constants defined in
the static class ``airflow.utils.WeightRule``
:type weight_rule: str
:param pool: the slot pool this task should run in, slot pools are a
way to limit concurrency for certain tasks
:type pool: str
:param sla: time by which the job is expected to succeed. Note that
this represents the ``timedelta`` after the period is closed. For
example if you set an SLA of 1 hour, the scheduler would send an email
soon after 1:00AM on the ``2016-01-02`` if the ``2016-01-01`` instance
has not succeeded yet.
The scheduler pays special attention for jobs with an SLA and
sends alert
emails for sla misses. SLA misses are also recorded in the database
for future reference. All tasks that share the same SLA time
get bundled in a single email, sent soon after that time. SLA
notification are sent once and only once for each task instance.
:type sla: datetime.timedelta
:param execution_timeout: max time allowed for the execution of
this task instance, if it goes beyond it will raise and fail.
:type execution_timeout: datetime.timedelta
:param on_failure_callback: a function to be called when a task instance
of this task fails. a context dictionary is passed as a single
parameter to this function. Context contains references to related
objects to the task instance and is documented under the macros
section of the API.
:type on_failure_callback: callable
:param on_retry_callback: much like the ``on_failure_callback`` except
that it is executed when retries occur.
:type on_retry_callback: callable
:param on_success_callback: much like the ``on_failure_callback`` except
that it is executed when the task succeeds.
:type on_success_callback: callable
:param trigger_rule: defines the rule by which dependencies are applied
for the task to get triggered. Options are:
``{ all_success | all_failed | all_done | one_success |
one_failed | none_failed | none_skipped | dummy}``
default is ``all_success``. Options can be set as string or
using the constants defined in the static class
``airflow.utils.TriggerRule``
:type trigger_rule: str
:param resources: A map of resource parameter names (the argument names of the
Resources constructor) to their values.
:type resources: dict
:param run_as_user: unix username to impersonate while running the task
:type run_as_user: str
:param task_concurrency: When set, a task will be able to limit the concurrent
runs across execution_dates
:type task_concurrency: int
:param executor_config: Additional task-level configuration parameters that are
interpreted by a specific executor. Parameters are namespaced by the name of
executor.
**Example**: to run this task in a specific docker container through
the KubernetesExecutor ::
MyOperator(...,
executor_config={
"KubernetesExecutor":
{"image": "myCustomDockerImage"}
}
)
:type executor_config: dict
.. attribute:: template_fields
:annotation: = []
.. attribute:: template_ext
:annotation: = []
.. attribute:: ui_color
:annotation: = #fff
.. attribute:: ui_fgcolor
:annotation: = #000
.. attribute:: _base_operator_shallow_copy_attrs
:annotation: = ['user_defined_macros', 'user_defined_filters', 'params', '_log']
.. attribute:: shallow_copy_attrs
:annotation: = []
.. attribute:: dag
Returns the Operator's DAG if set, otherwise raises an error
.. attribute:: dag_id
.. attribute:: deps
Returns the list of dependencies for the operator. These differ from execution
context dependencies in that they are specific to tasks and can be
extended/overridden by subclasses.
.. attribute:: schedule_interval
The schedule interval of the DAG always wins over individual tasks so
that tasks within a DAG always line up. The task still needs a
schedule_interval as it may not be attached to a DAG.
.. attribute:: priority_weight_total
.. attribute:: upstream_list
@property: list of tasks directly upstream
.. attribute:: upstream_task_ids
.. attribute:: downstream_list
@property: list of tasks directly downstream
.. attribute:: downstream_task_ids
.. attribute:: task_type
.. method:: __eq__(self, other)
.. method:: __ne__(self, other)
.. method:: __lt__(self, other)
.. method:: __hash__(self)
.. method:: __rshift__(self, other)
Implements Self >> Other == self.set_downstream(other)
If "Other" is a DAG, the DAG is assigned to the Operator.
.. method:: __lshift__(self, other)
Implements Self << Other == self.set_upstream(other)
If "Other" is a DAG, the DAG is assigned to the Operator.
.. method:: __rrshift__(self, other)
Called for [DAG] >> [Operator] because DAGs don't have
__rshift__ operators.
.. method:: __rlshift__(self, other)
Called for [DAG] << [Operator] because DAGs don't have
__lshift__ operators.
.. method:: has_dag(self)
Returns True if the Operator has been assigned to a DAG.
.. method:: pre_execute(self, context)
This hook is triggered right before self.execute() is called.
.. method:: execute(self, context)
This is the main method to derive when creating an operator.
Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
.. method:: post_execute(self, context, result=None)
This hook is triggered right after self.execute() is called.
It is passed the execution context and any results returned by the
operator.
.. method:: on_kill(self)
Override this method to cleanup subprocesses when a task instance
gets killed. Any use of the threading, subprocess or multiprocessing
module within an operator needs to be cleaned up or it will leave
ghost processes behind.
.. method:: __deepcopy__(self, memo)
Hack sorting double chained task lists by task_id to avoid hitting
max_depth on deepcopy operations.
.. method:: __getstate__(self)
.. method:: __setstate__(self, state)
.. method:: render_template_from_field(self, attr, content, context, jinja_env)
Renders a template from a field. If the field is a string, it will
simply render the string and return the result. If it is a collection or
nested set of collections, it will traverse the structure and render
all elements in it. If the field has another type, it will return it as it is.
.. method:: render_template(self, attr, content, context)
Renders a template either from a file or directly in a field, and returns
the rendered result.
.. method:: get_template_env(self)
.. method:: prepare_template(self)
Hook that is triggered after the templated fields get replaced
by their content. If you need your operator to alter the
content of the file before the template is rendered,
it should override this method to do so.
.. method:: resolve_template_files(self)
.. method:: clear(self, start_date=None, end_date=None, upstream=False, downstream=False, session=None)
Clears the state of task instances associated with the task, following
the parameters specified.
.. method:: get_task_instances(self, session, start_date=None, end_date=None)
Get a set of task instance related to this task for a specific date
range.
.. method:: get_flat_relative_ids(self, upstream=False, found_descendants=None)
Get a flat list of relatives' ids, either upstream or downstream.
.. method:: get_flat_relatives(self, upstream=False)
Get a flat list of relatives, either upstream or downstream.
.. method:: run(self, start_date=None, end_date=None, ignore_first_depends_on_past=False, ignore_ti_state=False, mark_success=False)
Run a set of task instances for a date range.
.. method:: dry_run(self)
.. method:: get_direct_relative_ids(self, upstream=False)
Get the direct relative ids to the current task, upstream or
downstream.
.. method:: get_direct_relatives(self, upstream=False)
Get the direct relatives to the current task, upstream or
downstream.
.. method:: __repr__(self)
.. method:: add_only_new(self, item_set, item)
.. method:: _set_relatives(self, task_or_task_list, upstream=False)
.. method:: set_downstream(self, task_or_task_list)
Set a task or a task list to be directly downstream from the current
task.
.. method:: set_upstream(self, task_or_task_list)
Set a task or a task list to be directly upstream from the current
task.
.. method:: xcom_push(self, context, key, value, execution_date=None)
See TaskInstance.xcom_push()
.. method:: xcom_pull(self, context, task_ids=None, dag_id=None, key=XCOM_RETURN_KEY, include_prior_dates=None)
See TaskInstance.xcom_pull()
.. py:class:: DagModel
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = dag
These items are stored in the database for state related information
.. attribute:: dag_id
.. attribute:: is_paused_at_creation
.. attribute:: is_paused
.. attribute:: is_subdag
.. attribute:: is_active
.. attribute:: last_scheduler_run
.. attribute:: last_pickled
.. attribute:: last_expired
.. attribute:: scheduler_lock
.. attribute:: pickle_id
.. attribute:: fileloc
.. attribute:: owners
.. attribute:: description
.. attribute:: default_view
.. attribute:: schedule_interval
.. attribute:: timezone
.. attribute:: safe_dag_id
.. method:: __repr__(self)
.. staticmethod:: get_dagmodel(dag_id, session=None)
.. classmethod:: get_current(cls, dag_id, session=None)
.. method:: get_default_view(self)
.. method:: get_last_dagrun(self, session=None, include_externally_triggered=False)
.. method:: get_dag(self)
.. method:: create_dagrun(self, run_id, state, execution_date, start_date=None, external_trigger=False, conf=None, session=None)
Creates a dag run from this dag including the tasks associated with this dag.
Returns the dag run.
:param run_id: defines the the run id for this dag run
:type run_id: str
:param execution_date: the execution date of this dag run
:type execution_date: datetime.datetime
:param state: the state of the dag run
:type state: airflow.utils.state.State
:param start_date: the date this dag run should be evaluated
:type start_date: datetime.datetime
:param external_trigger: whether this dag run is externally triggered
:type external_trigger: bool
:param session: database session
:type session: sqlalchemy.orm.session.Session
.. py:class:: DAG(dag_id, description='', schedule_interval=timedelta(days=1), start_date=None, end_date=None, full_filepath=None, template_searchpath=None, user_defined_macros=None, user_defined_filters=None, default_args=None, concurrency=configuration.conf.getint('core', 'dag_concurrency'), max_active_runs=configuration.conf.getint('core', 'max_active_runs_per_dag'), dagrun_timeout=None, sla_miss_callback=None, default_view=None, orientation=configuration.conf.get('webserver', 'dag_orientation'), catchup=configuration.conf.getboolean('scheduler', 'catchup_by_default'), on_success_callback=None, on_failure_callback=None, doc_md=None, params=None)
Bases::class:`airflow.dag.base_dag.BaseDag`, :class:`airflow.utils.log.logging_mixin.LoggingMixin`
A dag (directed acyclic graph) is a collection of tasks with directional
dependencies. A dag also has a schedule, a start end an end date
(optional). For each schedule, (say daily or hourly), the DAG needs to run
each individual tasks as their dependencies are met. Certain tasks have
the property of depending on their own past, meaning that they can't run
until their previous schedule (and upstream tasks) are completed.
DAGs essentially act as namespaces for tasks. A task_id can only be
added once to a DAG.
:param dag_id: The id of the DAG
:type dag_id: str
:param description: The description for the DAG to e.g. be shown on the webserver
:type description: str
:param schedule_interval: Defines how often that DAG runs, this
timedelta object gets added to your latest task instance's
execution_date to figure out the next schedule
:type schedule_interval: datetime.timedelta or
dateutil.relativedelta.relativedelta or str that acts as a cron
expression
:param start_date: The timestamp from which the scheduler will
attempt to backfill
:type start_date: datetime.datetime
:param end_date: A date beyond which your DAG won't run, leave to None
for open ended scheduling
:type end_date: datetime.datetime
:param template_searchpath: This list of folders (non relative)
defines where jinja will look for your templates. Order matters.
Note that jinja/airflow includes the path of your DAG file by
default
:type template_searchpath: str or list[str]
:param user_defined_macros: a dictionary of macros that will be exposed
in your jinja templates. For example, passing ``dict(foo='bar')``
to this argument allows you to ``{{ foo }}`` in all jinja
templates related to this DAG. Note that you can pass any
type of object here.
:type user_defined_macros: dict
:param user_defined_filters: a dictionary of filters that will be exposed
in your jinja templates. For example, passing
``dict(hello=lambda name: 'Hello %s' % name)`` to this argument allows
you to ``{{ 'world' | hello }}`` in all jinja templates related to
this DAG.
:type user_defined_filters: dict
:param default_args: A dictionary of default parameters to be used
as constructor keyword parameters when initialising operators.
Note that operators have the same hook, and precede those defined
here, meaning that if your dict contains `'depends_on_past': True`
here and `'depends_on_past': False` in the operator's call
`default_args`, the actual value will be `False`.
:type default_args: dict
:param params: a dictionary of DAG level parameters that are made
accessible in templates, namespaced under `params`. These
params can be overridden at the task level.
:type params: dict
:param concurrency: the number of task instances allowed to run
concurrently
:type concurrency: int
:param max_active_runs: maximum number of active DAG runs, beyond this
number of DAG runs in a running state, the scheduler won't create
new active DAG runs
:type max_active_runs: int
:param dagrun_timeout: specify how long a DagRun should be up before
timing out / failing, so that new DagRuns can be created
:type dagrun_timeout: datetime.timedelta
:param sla_miss_callback: specify a function to call when reporting SLA
timeouts.
:type sla_miss_callback: types.FunctionType
:param default_view: Specify DAG default view (tree, graph, duration,
gantt, landing_times)
:type default_view: str
:param orientation: Specify DAG orientation in graph view (LR, TB, RL, BT)
:type orientation: str
:param catchup: Perform scheduler catchup (or only run latest)? Defaults to True
:type catchup: bool
:param on_failure_callback: A function to be called when a DagRun of this dag fails.
A context dictionary is passed as a single parameter to this function.
:type on_failure_callback: callable
:param on_success_callback: Much like the ``on_failure_callback`` except
that it is executed when the dag succeeds.
:type on_success_callback: callable
.. attribute:: dag_id
.. attribute:: full_filepath
.. attribute:: concurrency
.. attribute:: description
.. attribute:: pickle_id
.. attribute:: tasks
.. attribute:: task_ids
.. attribute:: filepath
File location of where the dag object is instantiated
.. attribute:: folder
Folder location of where the dag object is instantiated
.. attribute:: owner
.. attribute:: concurrency_reached
Returns a boolean indicating whether the concurrency limit for this DAG
has been reached
.. attribute:: is_paused
Returns a boolean indicating whether this DAG is paused
.. attribute:: latest_execution_date
Returns the latest date for which at least one dag run exists
.. attribute:: subdags
Returns a list of the subdag objects associated to this DAG
.. attribute:: roots
.. method:: __repr__(self)
.. method:: __eq__(self, other)
.. method:: __ne__(self, other)
.. method:: __lt__(self, other)
.. method:: __hash__(self)
.. method:: __enter__(self)
.. method:: __exit__(self, _type, _value, _tb)
.. method:: get_default_view(self)
This is only there for backward compatible jinja2 templates
.. method:: date_range(self, start_date, num=None, end_date=timezone.utcnow())
.. method:: is_fixed_time_schedule(self)
Figures out if the DAG schedule has a fixed time (e.g. 3 AM).
:return: True if the schedule has a fixed time, False if not.
.. method:: following_schedule(self, dttm)
Calculates the following schedule for this dag in UTC.
:param dttm: utc datetime
:return: utc datetime
.. method:: previous_schedule(self, dttm)
Calculates the previous schedule for this dag in UTC
:param dttm: utc datetime
:return: utc datetime
.. method:: get_run_dates(self, start_date, end_date=None)
Returns a list of dates between the interval received as parameter using this
dag's schedule interval. Returned dates can be used for execution dates.
:param start_date: the start date of the interval
:type start_date: datetime
:param end_date: the end date of the interval, defaults to timezone.utcnow()
:type end_date: datetime
:return: a list of dates within the interval following the dag's schedule
:rtype: list
.. method:: normalize_schedule(self, dttm)
Returns dttm + interval unless dttm is first interval then it returns dttm
.. method:: get_last_dagrun(self, session=None, include_externally_triggered=False)
.. method:: _get_concurrency_reached(self, session=None)
.. method:: _get_is_paused(self, session=None)
.. method:: handle_callback(self, dagrun, success=True, reason=None, session=None)
Triggers the appropriate callback depending on the value of success, namely the
on_failure_callback or on_success_callback. This method gets the context of a
single TaskInstance part of this DagRun and passes that to the callable along
with a 'reason', primarily to differentiate DagRun failures.
.. note: The logs end up in
``$AIRFLOW_HOME/logs/scheduler/latest/PROJECT/DAG_FILE.py.log``
:param dagrun: DagRun object
:param success: Flag to specify if failure or success callback should be called
:param reason: Completion reason
:param session: Database session
.. method:: get_active_runs(self, session=None)
Returns a list of dag run execution dates currently running
:param session:
:return: List of execution dates
.. method:: get_num_active_runs(self, external_trigger=None, session=None)
Returns the number of active "running" dag runs
:param external_trigger: True for externally triggered active dag runs
:type external_trigger: bool
:param session:
:return: number greater than 0 for active dag runs
.. method:: get_dagrun(self, execution_date, session=None)
Returns the dag run for a given execution date if it exists, otherwise
none.
:param execution_date: The execution date of the DagRun to find.
:param session:
:return: The DagRun if found, otherwise None.
.. method:: _get_latest_execution_date(self, session=None)
.. method:: resolve_template_files(self)
.. method:: get_template_env(self)
Returns a jinja2 Environment while taking into account the DAGs
template_searchpath, user_defined_macros and user_defined_filters
.. method:: set_dependency(self, upstream_task_id, downstream_task_id)
Simple utility method to set dependency between two tasks that
already have been added to the DAG using add_task()
.. method:: get_task_instances(self, session, start_date=None, end_date=None, state=None)
.. method:: topological_sort(self)
Sorts tasks in topographical order, such that a task comes after any of its
upstream dependencies.
Heavily inspired by:
http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/
:return: list of tasks in topological order
.. method:: set_dag_runs_state(self, state=State.RUNNING, session=None, start_date=None, end_date=None)
.. method:: clear(self, start_date=None, end_date=None, only_failed=False, only_running=False, confirm_prompt=False, include_subdags=True, include_parentdag=True, reset_dag_runs=True, dry_run=False, session=None, get_tis=False)
Clears a set of task instances associated with the current dag for
a specified date range.
.. classmethod:: clear_dags(cls, dags, start_date=None, end_date=None, only_failed=False, only_running=False, confirm_prompt=False, include_subdags=True, include_parentdag=False, reset_dag_runs=True, dry_run=False)
.. method:: __deepcopy__(self, memo)
.. method:: sub_dag(self, task_regex, include_downstream=False, include_upstream=True)
Returns a subset of the current dag as a deep copy of the current dag
based on a regex that should match one or many tasks, and includes
upstream and downstream neighbours based on the flag passed.
.. method:: has_task(self, task_id)
.. method:: get_task(self, task_id)
.. method:: pickle_info(self, session=None)
.. method:: pickle(self, session=None)
.. method:: tree_view(self)
Shows an ascii tree representation of the DAG
.. method:: add_task(self, task)
Add a task to the DAG
:param task: the task you want to add
:type task: task
.. method:: add_tasks(self, tasks)
Add a list of tasks to the DAG
:param tasks: a lit of tasks you want to add
:type tasks: list of tasks
.. method:: run(self, start_date=None, end_date=None, mark_success=False, local=False, executor=None, donot_pickle=configuration.conf.getboolean('core', 'donot_pickle'), ignore_task_deps=False, ignore_first_depends_on_past=False, pool=None, delay_on_limit_secs=1.0, verbose=False, conf=None, rerun_failed_tasks=False, run_backwards=False)
Runs the DAG.
:param start_date: the start date of the range to run
:type start_date: datetime.datetime
:param end_date: the end date of the range to run
:type end_date: datetime.datetime
:param mark_success: True to mark jobs as succeeded without running them
:type mark_success: bool
:param local: True to run the tasks using the LocalExecutor
:type local: bool
:param executor: The executor instance to run the tasks
:type executor: airflow.executor.BaseExecutor
:param donot_pickle: True to avoid pickling DAG object and send to workers
:type donot_pickle: bool
:param ignore_task_deps: True to skip upstream tasks
:type ignore_task_deps: bool
:param ignore_first_depends_on_past: True to ignore depends_on_past
dependencies for the first set of tasks only
:type ignore_first_depends_on_past: bool
:param pool: Resource pool to use
:type pool: str
:param delay_on_limit_secs: Time in seconds to wait before next attempt to run
dag run when max_active_runs limit has been reached
:type delay_on_limit_secs: float
:param verbose: Make logging output more verbose
:type verbose: bool
:param conf: user defined dictionary passed from CLI
:type conf: dict
:param rerun_failed_tasks:
:type: bool
:param run_backwards:
:type: bool
.. method:: cli(self)
Exposes a CLI specific to this DAG
.. method:: create_dagrun(self, run_id, state, execution_date=None, start_date=None, external_trigger=False, conf=None, session=None)
Creates a dag run from this dag including the tasks associated with this dag.
Returns the dag run.
:param run_id: defines the the run id for this dag run
:type run_id: str
:param execution_date: the execution date of this dag run
:type execution_date: datetime.datetime
:param state: the state of the dag run
:type state: airflow.utils.state.State
:param start_date: the date this dag run should be evaluated
:type start_date: datetime
:param external_trigger: whether this dag run is externally triggered
:type external_trigger: bool
:param session: database session
:type session: sqlalchemy.orm.session.Session
.. method:: sync_to_db(self, owner=None, sync_time=None, session=None)
Save attributes about this DAG to the DB. Note that this method
can be called for both DAGs and SubDAGs. A SubDag is actually a
SubDagOperator.
:param dag: the DAG object to save to the DB
:type dag: airflow.models.DAG
:param sync_time: The time that the DAG should be marked as sync'ed
:type sync_time: datetime
:return: None
.. staticmethod:: deactivate_unknown_dags(active_dag_ids, session=None)
Given a list of known DAGs, deactivate any other DAGs that are
marked as active in the ORM
:param active_dag_ids: list of DAG IDs that are active
:type active_dag_ids: list[unicode]
:return: None
.. staticmethod:: deactivate_stale_dags(expiration_date, session=None)
Deactivate any DAGs that were last touched by the scheduler before
the expiration date. These DAGs were likely deleted.
:param expiration_date: set inactive DAGs that were touched before this
time
:type expiration_date: datetime
:return: None
.. staticmethod:: get_num_task_instances(dag_id, task_ids, states=None, session=None)
Returns the number of task instances in the given DAG.
:param session: ORM session
:param dag_id: ID of the DAG to get the task concurrency of
:type dag_id: unicode
:param task_ids: A list of valid task IDs for the given DAG
:type task_ids: list[unicode]
:param states: A list of states to filter by if supplied
:type states: list[state]
:return: The number of running tasks
:rtype: int
.. method:: test_cycle(self)
Check to see if there are any cycles in the DAG. Returns False if no cycle found,
otherwise raises exception.
.. method:: _test_cycle_helper(self, visit_map, task_id)
Checks if a cycle exists from the input task using DFS traversal
.. py:class:: Chart
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = chart
.. attribute:: id
.. attribute:: label
.. attribute:: conn_id
.. attribute:: user_id
.. attribute:: chart_type
.. attribute:: sql_layout
.. attribute:: sql
.. attribute:: y_log_scale
.. attribute:: show_datatable
.. attribute:: show_sql
.. attribute:: height
.. attribute:: default_params
.. attribute:: owner
.. attribute:: x_is_date
.. attribute:: iteration_no
.. attribute:: last_modified
.. method:: __repr__(self)
.. py:class:: KnownEventType
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = known_event_type
.. attribute:: id
.. attribute:: know_event_type
.. method:: __repr__(self)
.. py:class:: KnownEvent
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = known_event
.. attribute:: id
.. attribute:: label
.. attribute:: start_date
.. attribute:: end_date
.. attribute:: user_id
.. attribute:: known_event_type_id
.. attribute:: reported_by
.. attribute:: event_type
.. attribute:: description
.. method:: __repr__(self)
.. py:class:: Variable
Bases::class:`airflow.models.base.Base`, :class:`airflow.utils.log.logging_mixin.LoggingMixin`
.. attribute:: __tablename__
:annotation: = variable
.. attribute:: __NO_DEFAULT_SENTINEL
.. attribute:: id
.. attribute:: key
.. attribute:: _val
.. attribute:: is_encrypted
.. attribute:: val
.. method:: __repr__(self)
.. method:: get_val(self)
.. method:: set_val(self, value)
.. classmethod:: setdefault(cls, key, default, deserialize_json=False)
Like a Python builtin dict object, setdefault returns the current value
for a key, and if it isn't there, stores the default value and returns it.
:param key: Dict key for this Variable
:type key: str
:param default: Default value to set and return if the variable
isn't already in the DB
:type default: Mixed
:param deserialize_json: Store this as a JSON encoded value in the DB
and un-encode it when retrieving a value
:return: Mixed
.. classmethod:: get(cls, key, default_var=__NO_DEFAULT_SENTINEL, deserialize_json=False, session=None)
.. classmethod:: set(cls, key, value, serialize_json=False, session=None)
.. method:: rotate_fernet_key(self)
.. py:class:: XCom
Bases::class:`airflow.models.base.Base`, :class:`airflow.utils.log.logging_mixin.LoggingMixin`
Base class for XCom objects.
.. attribute:: __tablename__
:annotation: = xcom
.. attribute:: id
.. attribute:: key
.. attribute:: value
.. attribute:: timestamp
.. attribute:: execution_date
.. attribute:: task_id
.. attribute:: dag_id
.. attribute:: __table_args__
TODO: "pickling" has been deprecated and JSON is preferred.
"pickling" will be removed in Airflow 2.0.
.. method:: init_on_load(self)
.. method:: __repr__(self)
.. classmethod:: set(cls, key, value, execution_date, task_id, dag_id, session=None)
Store an XCom value.
TODO: "pickling" has been deprecated and JSON is preferred.
"pickling" will be removed in Airflow 2.0.
:return: None
.. classmethod:: get_one(cls, execution_date, key=None, task_id=None, dag_id=None, include_prior_dates=False, session=None)
Retrieve an XCom value, optionally meeting certain criteria.
TODO: "pickling" has been deprecated and JSON is preferred.
"pickling" will be removed in Airflow 2.0.
:return: XCom value
.. classmethod:: get_many(cls, execution_date, key=None, task_ids=None, dag_ids=None, include_prior_dates=False, limit=100, session=None)
Retrieve an XCom value, optionally meeting certain criteria
TODO: "pickling" has been deprecated and JSON is preferred.
"pickling" will be removed in Airflow 2.0.
.. classmethod:: delete(cls, xcoms, session=None)
.. py:class:: DagRun
Bases::class:`airflow.models.base.Base`, :class:`airflow.utils.log.logging_mixin.LoggingMixin`
DagRun describes an instance of a Dag. It can be created
by the scheduler (for regular runs) or by an external trigger
.. attribute:: __tablename__
:annotation: = dag_run
.. attribute:: ID_PREFIX
:annotation: = scheduled__
.. attribute:: ID_FORMAT_PREFIX
.. attribute:: id
.. attribute:: dag_id
.. attribute:: execution_date
.. attribute:: start_date
.. attribute:: end_date
.. attribute:: _state
.. attribute:: run_id
.. attribute:: external_trigger
.. attribute:: conf
.. attribute:: dag
.. attribute:: __table_args__
.. attribute:: state
.. attribute:: is_backfill
.. method:: __repr__(self)
.. method:: get_state(self)
.. method:: set_state(self, state)
.. classmethod:: id_for_date(cls, date, prefix=ID_FORMAT_PREFIX)
.. method:: refresh_from_db(self, session=None)
Reloads the current dagrun from the database
:param session: database session
.. staticmethod:: find(dag_id=None, run_id=None, execution_date=None, state=None, external_trigger=None, no_backfills=False, session=None)
Returns a set of dag runs for the given search criteria.
:param dag_id: the dag_id to find dag runs for
:type dag_id: int, list
:param run_id: defines the the run id for this dag run
:type run_id: str
:param execution_date: the execution date
:type execution_date: datetime.datetime
:param state: the state of the dag run
:type state: airflow.utils.state.State
:param external_trigger: whether this dag run is externally triggered
:type external_trigger: bool
:param no_backfills: return no backfills (True), return all (False).
Defaults to False
:type no_backfills: bool
:param session: database session
:type session: sqlalchemy.orm.session.Session
.. method:: get_task_instances(self, state=None, session=None)
Returns the task instances for this dag run
.. method:: get_task_instance(self, task_id, session=None)
Returns the task instance specified by task_id for this dag run
:param task_id: the task id
.. method:: get_dag(self)
Returns the Dag associated with this DagRun.
:return: DAG
.. method:: get_previous_dagrun(self, session=None)
The previous DagRun, if there is one
.. method:: get_previous_scheduled_dagrun(self, session=None)
The previous, SCHEDULED DagRun, if there is one
.. method:: update_state(self, session=None)
Determines the overall state of the DagRun based on the state
of its TaskInstances.
:return: State
.. method:: verify_integrity(self, session=None)
Verifies the DagRun by checking for removed tasks or tasks that are not in the
database yet. It will set state to removed or add the task if required.
.. staticmethod:: get_run(session, dag_id, execution_date)
:param dag_id: DAG ID
:type dag_id: unicode
:param execution_date: execution date
:type execution_date: datetime
:return: DagRun corresponding to the given dag_id and execution date
if one exists. None otherwise.
:rtype: DagRun
.. classmethod:: get_latest_runs(cls, session)
Returns the latest DagRun for each DAG.
.. py:class:: Pool
Bases::class:`airflow.models.base.Base`
.. attribute:: __tablename__
:annotation: = slot_pool
.. attribute:: id
.. attribute:: pool
.. attribute:: slots
.. attribute:: description
.. method:: __repr__(self)
.. method:: to_json(self)
.. method:: used_slots(self, session)
Returns the number of slots used at the moment
.. method:: queued_slots(self, session)
Returns the number of slots used at the moment
.. method:: open_slots(self, session)
Returns the number of slots open at the moment
.. py:class:: Connection(conn_id=None, conn_type=None, host=None, login=None, password=None, schema=None, port=None, extra=None, uri=None)
Bases::class:`airflow.models.base.Base`, :class:`airflow.LoggingMixin`
Placeholder to store information about different database instances
connection information. The idea here is that scripts use references to
database instances (conn_id) instead of hard coding hostname, logins and
passwords when using operators or hooks.
.. attribute:: __tablename__
:annotation: = connection
.. attribute:: id
.. attribute:: conn_id
.. attribute:: conn_type
.. attribute:: host
.. attribute:: schema
.. attribute:: login
.. attribute:: _password
.. attribute:: port
.. attribute:: is_encrypted
.. attribute:: is_extra_encrypted
.. attribute:: _extra
.. attribute:: _types
:annotation: = [['docker', 'Docker Registry'], ['fs', 'File (path)'], ['ftp', 'FTP'], ['google_cloud_platform', 'Google Cloud Platform'], ['hdfs', 'HDFS'], ['http', 'HTTP'], ['hive_cli', 'Hive Client Wrapper'], ['hive_metastore', 'Hive Metastore Thrift'], ['hiveserver2', 'Hive Server 2 Thrift'], ['jdbc', 'Jdbc Connection'], ['jenkins', 'Jenkins'], ['mysql', 'MySQL'], ['postgres', 'Postgres'], ['oracle', 'Oracle'], ['vertica', 'Vertica'], ['presto', 'Presto'], ['s3', 'S3'], ['samba', 'Samba'], ['sqlite', 'Sqlite'], ['ssh', 'SSH'], ['cloudant', 'IBM Cloudant'], ['mssql', 'Microsoft SQL Server'], ['mesos_framework-id', 'Mesos Framework ID'], ['jira', 'JIRA'], ['redis', 'Redis'], ['wasb', 'Azure Blob Storage'], ['databricks', 'Databricks'], ['aws', 'Amazon Web Services'], ['emr', 'Elastic MapReduce'], ['snowflake', 'Snowflake'], ['segment', 'Segment'], ['azure_data_lake', 'Azure Data Lake'], ['azure_container_instances', 'Azure Container Instances'], ['azure_cosmos', 'Azure CosmosDB'], ['cassandra', 'Cassandra'], ['qubole', 'Qubole'], ['mongo', 'MongoDB'], ['gcpcloudsql', 'Google Cloud SQL']]
.. attribute:: password
.. attribute:: extra
.. attribute:: extra_dejson
Returns the extra property by deserializing json.
.. method:: parse_from_uri(self, uri)
.. method:: get_password(self)
.. method:: set_password(self, value)
.. method:: get_extra(self)
.. method:: set_extra(self, value)
.. method:: rotate_fernet_key(self)
.. method:: get_hook(self)
.. method:: __repr__(self)
.. method:: debug_info(self)
.. py:class:: SkipMixin
Bases::class:`airflow.utils.log.logging_mixin.LoggingMixin`
.. method:: skip(self, dag_run, execution_date, tasks, session=None)
Sets tasks instances to skipped from the same dag run.
:param dag_run: the DagRun for which to set the tasks to skipped
:param execution_date: execution_date
:param tasks: tasks to skip (not task_ids)
:param session: db session to use