| :mod:`airflow.models.serialized_dag` |
| ==================================== |
| |
| .. py:module:: airflow.models.serialized_dag |
| |
| .. autoapi-nested-parse:: |
| |
| Serialzed DAG table in database. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| .. data:: log |
| |
| |
| |
| |
| .. py:class:: SerializedDagModel(dag) |
| |
| Bases: :class:`airflow.models.base.Base` |
| |
| A table for serialized DAGs. |
| |
| serialized_dag table is a snapshot of DAG files synchronized by scheduler. |
| This feature is controlled by: |
| |
| * ``[core] store_serialized_dags = True``: enable this feature |
| * ``[core] min_serialized_dag_update_interval = 30`` (s): |
| serialized DAGs are updated in DB when a file gets processed by scheduler, |
| to reduce DB write rate, there is a minimal interval of updating serialized DAGs. |
| * ``[scheduler] dag_dir_list_interval = 300`` (s): |
| interval of deleting serialized DAGs in DB when the files are deleted, suggest |
| to use a smaller interval such as 60 |
| |
| It is used by webserver to load dagbags when ``store_serialized_dags=True``. |
| Because reading from database is lightweight compared to importing from files, |
| it solves the webserver scalability issue. |
| |
| .. attribute:: __tablename__ |
| :annotation: = serialized_dag |
| |
| |
| |
| .. attribute:: dag_id |
| |
| |
| |
| |
| .. attribute:: fileloc |
| |
| |
| |
| |
| .. attribute:: fileloc_hash |
| |
| |
| |
| |
| .. attribute:: data |
| |
| |
| |
| |
| .. attribute:: last_updated |
| |
| |
| |
| |
| .. attribute:: __table_args__ |
| |
| |
| |
| |
| .. attribute:: dag |
| |
| |
| The DAG deserialized from the ``data`` column |
| |
| |
| |
| .. staticmethod:: dag_fileloc_hash(full_filepath) |
| |
| "Hashing file location for indexing. |
| |
| :param full_filepath: full filepath of DAG file |
| :return: hashed full_filepath |
| |
| |
| |
| |
| .. classmethod:: write_dag(cls, dag, min_update_interval=None, session=None) |
| |
| Serializes a DAG and writes it into database. |
| |
| :param dag: a DAG to be written into database |
| :param min_update_interval: minimal interval in seconds to update serialized DAG |
| :param session: ORM Session |
| |
| |
| |
| |
| .. classmethod:: read_all_dags(cls, session=None) |
| |
| Reads all DAGs in serialized_dag table. |
| |
| :param session: ORM Session |
| :returns: a dict of DAGs read from database |
| |
| |
| |
| |
| .. classmethod:: remove_dag(cls, dag_id, session=None) |
| |
| Deletes a DAG with given dag_id. |
| |
| :param dag_id: dag_id to be deleted |
| :type dag_id: str |
| :param session: ORM Session |
| |
| |
| |
| |
| .. classmethod:: remove_deleted_dags(cls, alive_dag_filelocs, session=None) |
| |
| Deletes DAGs not included in alive_dag_filelocs. |
| |
| :param alive_dag_filelocs: file paths of alive DAGs |
| :type alive_dag_filelocs: list |
| :param session: ORM Session |
| |
| |
| |
| |
| .. classmethod:: has_dag(cls, dag_id, session=None) |
| |
| Checks a DAG exist in serialized_dag table. |
| |
| :param dag_id: the DAG to check |
| :type dag_id: str |
| :param session: ORM Session |
| :rtype: bool |
| |
| |
| |
| |
| .. classmethod:: get(cls, dag_id, session=None) |
| |
| Get the SerializedDAG for the given dag ID. |
| It will cope with being passed the ID of a subdag by looking up the |
| root dag_id from the DAG table. |
| |
| :param dag_id: the DAG to fetch |
| :param session: ORM Session |
| |
| |
| |
| |