| :py:mod:`airflow.models.dagbag` |
| =============================== |
| |
| .. py:module:: airflow.models.dagbag |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.models.dagbag.FileLoadStat |
| airflow.models.dagbag.DagBag |
| |
| |
| |
| |
| .. py:class:: FileLoadStat |
| |
| Bases: :py:obj:`NamedTuple` |
| |
| Information about single file |
| |
| .. py:attribute:: file |
| :annotation: :str |
| |
| |
| |
| .. py:attribute:: duration |
| :annotation: :datetime.timedelta |
| |
| |
| |
| .. py:attribute:: dag_num |
| :annotation: :int |
| |
| |
| |
| .. py:attribute:: task_num |
| :annotation: :int |
| |
| |
| |
| .. py:attribute:: dags |
| :annotation: :str |
| |
| |
| |
| |
| .. py:class:: DagBag(dag_folder: Union[str, pathlib.Path, None] = None, include_examples: bool = conf.getboolean('core', 'LOAD_EXAMPLES'), include_smart_sensor: bool = conf.getboolean('smart_sensor', 'USE_SMART_SENSOR'), safe_mode: bool = conf.getboolean('core', 'DAG_DISCOVERY_SAFE_MODE'), read_dags_from_db: bool = False, store_serialized_dags: Optional[bool] = None, load_op_links: bool = True) |
| |
| Bases: :py:obj:`airflow.utils.log.logging_mixin.LoggingMixin` |
| |
| A dagbag is a collection of dags, parsed out of a folder tree and has high |
| level configuration settings, like what database to use as a backend and |
| what executor to use to fire off tasks. This makes it easier to run |
| distinct environments for say production and development, tests, or for |
| different teams or security profiles. What would have been system level |
| settings are now dagbag level so that one system can run multiple, |
| independent settings sets. |
| |
| :param dag_folder: the folder to scan to find DAGs |
| :type dag_folder: unicode |
| :param include_examples: whether to include the examples that ship |
| with airflow or not |
| :type include_examples: bool |
| :param include_smart_sensor: whether to include the smart sensor native |
| DAGs that create the smart sensor operators for whole cluster |
| :type include_smart_sensor: bool |
| :param read_dags_from_db: Read DAGs from DB if ``True`` is passed. |
| If ``False`` DAGs are read from python files. |
| :type read_dags_from_db: bool |
| :param load_op_links: Should the extra operator link be loaded via plugins when |
| de-serializing the DAG? This flag is set to False in Scheduler so that Extra Operator links |
| are not loaded to not run User code in Scheduler. |
| :type load_op_links: bool |
| |
| .. py:attribute:: DAGBAG_IMPORT_TIMEOUT |
| |
| |
| |
| |
| .. py:attribute:: SCHEDULER_ZOMBIE_TASK_THRESHOLD |
| |
| |
| |
| |
| .. py:method:: size(self) -> int |
| |
| :return: the amount of dags contained in this dagbag |
| |
| |
| .. py:method:: store_serialized_dags(self) -> bool |
| :property: |
| |
| Whether or not to read dags from DB |
| |
| |
| .. py:method:: dag_ids(self) -> List[str] |
| :property: |
| |
| :return: a list of DAG IDs in this bag |
| :rtype: List[unicode] |
| |
| |
| .. py:method:: get_dag(self, dag_id, session: sqlalchemy.orm.Session = None) |
| |
| Gets the DAG out of the dictionary, and refreshes it if expired |
| |
| :param dag_id: DAG Id |
| :type dag_id: str |
| |
| |
| .. py:method:: process_file(self, filepath, only_if_updated=True, safe_mode=True) |
| |
| Given a path to a python module or zip file, this method imports |
| the module and look for dag objects within it. |
| |
| |
| .. py:method:: bag_dag(self, dag, root_dag) |
| |
| Adds the DAG into the bag, recurses into sub dags. |
| |
| :raises: AirflowDagCycleException if a cycle is detected in this dag or its subdags. |
| :raises: AirflowDagDuplicatedIdException if this dag or its subdags already exists in the bag. |
| |
| |
| .. py:method:: collect_dags(self, dag_folder: Union[str, pathlib.Path, None] = None, only_if_updated: bool = True, include_examples: bool = conf.getboolean('core', 'LOAD_EXAMPLES'), include_smart_sensor: bool = conf.getboolean('smart_sensor', 'USE_SMART_SENSOR'), safe_mode: bool = conf.getboolean('core', 'DAG_DISCOVERY_SAFE_MODE')) |
| |
| Given a file path or a folder, this method looks for python modules, |
| imports them and adds them to the dagbag collection. |
| |
| Note that if a ``.airflowignore`` file is found while processing |
| the directory, it will behave much like a ``.gitignore``, |
| ignoring files that match any of the regex patterns specified |
| in the file. |
| |
| **Note**: The patterns in .airflowignore are treated as |
| un-anchored regexes, not shell-like glob patterns. |
| |
| |
| .. py:method:: collect_dags_from_db(self) |
| |
| Collects DAGs from database. |
| |
| |
| .. py:method:: dagbag_report(self) |
| |
| Prints a report around DagBag loading stats |
| |
| |
| .. py:method:: sync_to_db(self, session: Optional[sqlalchemy.orm.Session] = None) |
| |
| Save attributes about list of DAG to the DB. |
| |
| |
| |