| :py:mod:`airflow.models.serialized_dag` |
| ======================================= |
| |
| .. py:module:: airflow.models.serialized_dag |
| |
| .. autoapi-nested-parse:: |
| |
| Serialized DAG table in database. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.models.serialized_dag.SerializedDagModel |
| |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.models.serialized_dag.log |
| |
| |
| .. py:data:: log |
| |
| |
| |
| |
| .. py:class:: SerializedDagModel(dag, processor_subdir = None) |
| |
| Bases: :py:obj:`airflow.models.base.Base` |
| |
| A table for serialized DAGs. |
| |
| serialized_dag table is a snapshot of DAG files synchronized by scheduler. |
| This feature is controlled by: |
| |
| * ``[core] min_serialized_dag_update_interval = 30`` (s): |
| serialized DAGs are updated in DB when a file gets processed by scheduler, |
| to reduce DB write rate, there is a minimal interval of updating serialized DAGs. |
| * ``[scheduler] dag_dir_list_interval = 300`` (s): |
| interval of deleting serialized DAGs in DB when the files are deleted, suggest |
| to use a smaller interval such as 60 |
| * ``[core] compress_serialized_dags``: |
| whether compressing the dag data to the Database. |
| |
| It is used by webserver to load dags |
| because reading from database is lightweight compared to importing from files, |
| it solves the webserver scalability issue. |
| |
| .. py:attribute:: __tablename__ |
| :annotation: = serialized_dag |
| |
| |
| |
| .. py:attribute:: dag_id |
| |
| |
| |
| |
| .. py:attribute:: fileloc |
| |
| |
| |
| |
| .. py:attribute:: fileloc_hash |
| |
| |
| |
| |
| .. py:attribute:: last_updated |
| |
| |
| |
| |
| .. py:attribute:: dag_hash |
| |
| |
| |
| |
| .. py:attribute:: processor_subdir |
| |
| |
| |
| |
| .. py:attribute:: __table_args__ |
| |
| |
| |
| |
| .. py:attribute:: dag_runs |
| |
| |
| |
| |
| .. py:attribute:: dag_model |
| |
| |
| |
| |
| .. py:attribute:: load_op_links |
| :annotation: = True |
| |
| |
| |
| .. py:method:: __repr__() |
| |
| |
| .. py:method:: write_dag(dag, min_update_interval = None, processor_subdir = None, session = None) |
| :classmethod: |
| |
| Serializes a DAG and writes it into database. |
| If the record already exists, it checks if the Serialized DAG changed or not. If it is |
| changed, it updates the record, ignores otherwise. |
| |
| :param dag: a DAG to be written into database |
| :param min_update_interval: minimal interval in seconds to update serialized DAG |
| :param session: ORM Session |
| |
| :returns: Boolean indicating if the DAG was written to the DB |
| |
| |
| .. py:method:: read_all_dags(session = None) |
| :classmethod: |
| |
| Reads all DAGs in serialized_dag table. |
| |
| :param session: ORM Session |
| :returns: a dict of DAGs read from database |
| |
| |
| .. py:method:: data() |
| :property: |
| |
| |
| .. py:method:: dag() |
| :property: |
| |
| The DAG deserialized from the ``data`` column |
| |
| |
| .. py:method:: remove_dag(dag_id, session = None) |
| :classmethod: |
| |
| Deletes a DAG with given dag_id. |
| :param dag_id: dag_id to be deleted |
| :param session: ORM Session |
| |
| |
| .. py:method:: remove_deleted_dags(alive_dag_filelocs, processor_subdir = None, session=None) |
| :classmethod: |
| |
| Deletes DAGs not included in alive_dag_filelocs. |
| |
| :param alive_dag_filelocs: file paths of alive DAGs |
| :param session: ORM Session |
| |
| |
| .. py:method:: has_dag(dag_id, session = None) |
| :classmethod: |
| |
| Checks a DAG exist in serialized_dag table. |
| |
| :param dag_id: the DAG to check |
| :param session: ORM Session |
| |
| |
| .. py:method:: get_dag(dag_id, session = None) |
| :classmethod: |
| |
| |
| .. py:method:: get(dag_id, session = None) |
| :classmethod: |
| |
| Get the SerializedDAG for the given dag ID. |
| It will cope with being passed the ID of a subdag by looking up the |
| root dag_id from the DAG table. |
| |
| :param dag_id: the DAG to fetch |
| :param session: ORM Session |
| |
| |
| .. py:method:: bulk_sync_to_db(dags, processor_subdir = None, session = None) |
| :staticmethod: |
| |
| Saves DAGs as Serialized DAG objects in the database. Each |
| DAG is saved in a separate database query. |
| |
| :param dags: the DAG objects to save to the DB |
| :param session: ORM Session |
| :return: None |
| |
| |
| .. py:method:: get_last_updated_datetime(dag_id, session = None) |
| :classmethod: |
| |
| Get the date when the Serialized DAG associated to DAG was last updated |
| in serialized_dag table |
| |
| :param dag_id: DAG ID |
| :param session: ORM Session |
| |
| |
| .. py:method:: get_max_last_updated_datetime(session = None) |
| :classmethod: |
| |
| Get the maximum date when any DAG was last updated in serialized_dag table |
| |
| :param session: ORM Session |
| |
| |
| .. py:method:: get_latest_version_hash(dag_id, session = None) |
| :classmethod: |
| |
| Get the latest DAG version for a given DAG ID. |
| |
| :param dag_id: DAG ID |
| :param session: ORM Session |
| :return: DAG Hash, or None if the DAG is not found |
| :rtype: str | None |
| |
| |
| .. py:method:: get_dag_dependencies(session = None) |
| :classmethod: |
| |
| Get the dependencies between DAGs |
| |
| :param session: ORM Session |
| |
| |
| |