| :py:mod:`airflow.providers.snowflake.operators.snowflake` |
| ========================================================= |
| |
| .. py:module:: airflow.providers.snowflake.operators.snowflake |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.snowflake.operators.snowflake.SnowflakeOperator |
| airflow.providers.snowflake.operators.snowflake.SnowflakeCheckOperator |
| airflow.providers.snowflake.operators.snowflake.SnowflakeValueCheckOperator |
| airflow.providers.snowflake.operators.snowflake.SnowflakeIntervalCheckOperator |
| |
| |
| |
| Functions |
| ~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.snowflake.operators.snowflake.get_db_hook |
| |
| |
| |
| .. py:function:: get_db_hook(self) |
| |
| Create and return SnowflakeHook. |
| |
| :return: a SnowflakeHook instance. |
| :rtype: SnowflakeHook |
| |
| |
| .. py:class:: SnowflakeOperator(*, sql, snowflake_conn_id = 'snowflake_default', parameters = None, autocommit = True, do_xcom_push = True, warehouse = None, database = None, role = None, schema = None, authenticator = None, session_parameters = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Executes SQL code in a Snowflake database |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:SnowflakeOperator` |
| |
| :param snowflake_conn_id: Reference to |
| :ref:`Snowflake connection id<howto/connection:snowflake>` |
| :param sql: the SQL code to be executed as a single string, or |
| a list of str (sql statements), or a reference to a template file. |
| Template references are recognized by str ending in '.sql' |
| :param autocommit: if True, each command is automatically committed. |
| (default value: True) |
| :param parameters: (optional) the parameters to render the SQL query with. |
| :param warehouse: name of warehouse (will overwrite any warehouse |
| defined in the connection's extra JSON) |
| :param database: name of database (will overwrite database defined |
| in connection) |
| :param schema: name of schema (will overwrite schema defined in |
| connection) |
| :param role: name of role (will overwrite any role defined in |
| connection's extra JSON) |
| :param authenticator: authenticator for Snowflake. |
| 'snowflake' (default) to use the internal Snowflake authenticator |
| 'externalbrowser' to authenticate using your web browser and |
| Okta, ADFS or any other SAML 2.0-compliant identify provider |
| (IdP) that has been defined for your account |
| 'https://<your_okta_account_name>.okta.com' to authenticate |
| through native Okta. |
| :param session_parameters: You can set session-level parameters at |
| the time you connect to Snowflake |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['sql'] |
| |
| |
| |
| .. py:attribute:: template_ext |
| :annotation: :Sequence[str] = ['.sql'] |
| |
| |
| |
| .. py:attribute:: template_fields_renderers |
| |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #ededed |
| |
| |
| |
| .. py:method:: get_db_hook(self) |
| |
| |
| .. py:method:: execute(self, context) |
| |
| Run query on snowflake |
| |
| |
| |
| .. py:class:: SnowflakeCheckOperator(*, sql, snowflake_conn_id = 'snowflake_default', parameters = None, autocommit = True, do_xcom_push = True, warehouse = None, database = None, role = None, schema = None, authenticator = None, session_parameters = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.operators.sql.SQLCheckOperator` |
| |
| Performs a check against Snowflake. The ``SnowflakeCheckOperator`` expects |
| a sql query that will return a single row. Each value on that |
| first row is evaluated using python ``bool`` casting. If any of the |
| values return ``False`` the check is failed and errors out. |
| |
| Note that Python bool casting evals the following as ``False``: |
| |
| * ``False`` |
| * ``0`` |
| * Empty string (``""``) |
| * Empty list (``[]``) |
| * Empty dictionary or set (``{}``) |
| |
| Given a query like ``SELECT COUNT(*) FROM foo``, it will fail only if |
| the count ``== 0``. You can craft much more complex query that could, |
| for instance, check that the table has the same number of rows as |
| the source table upstream, or that the count of today's partition is |
| greater than yesterday's partition, or that a set of metrics are less |
| than 3 standard deviation for the 7 day average. |
| |
| This operator can be used as a data quality check in your pipeline, and |
| depending on where you put it in your DAG, you have the choice to |
| stop the critical path, preventing from |
| publishing dubious data, or on the side and receive email alerts |
| without stopping the progress of the DAG. |
| |
| :param sql: the SQL code to be executed as a single string, or |
| a list of str (sql statements), or a reference to a template file. |
| Template references are recognized by str ending in '.sql' |
| :param snowflake_conn_id: Reference to |
| :ref:`Snowflake connection id<howto/connection:snowflake>` |
| :param autocommit: if True, each command is automatically committed. |
| (default value: True) |
| :param parameters: (optional) the parameters to render the SQL query with. |
| :param warehouse: name of warehouse (will overwrite any warehouse |
| defined in the connection's extra JSON) |
| :param database: name of database (will overwrite database defined |
| in connection) |
| :param schema: name of schema (will overwrite schema defined in |
| connection) |
| :param role: name of role (will overwrite any role defined in |
| connection's extra JSON) |
| :param authenticator: authenticator for Snowflake. |
| 'snowflake' (default) to use the internal Snowflake authenticator |
| 'externalbrowser' to authenticate using your web browser and |
| Okta, ADFS or any other SAML 2.0-compliant identify provider |
| (IdP) that has been defined for your account |
| 'https://<your_okta_account_name>.okta.com' to authenticate |
| through native Okta. |
| :param session_parameters: You can set session-level parameters at |
| the time you connect to Snowflake |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['sql'] |
| |
| |
| |
| .. py:attribute:: template_ext |
| :annotation: :Sequence[str] = ['.sql'] |
| |
| |
| |
| .. py:attribute:: ui_color |
| :annotation: = #ededed |
| |
| |
| |
| .. py:method:: get_db_hook(self) |
| |
| Get the database hook for the connection. |
| |
| :return: the database hook object. |
| :rtype: DbApiHook |
| |
| |
| |
| .. py:class:: SnowflakeValueCheckOperator(*, sql, pass_value, tolerance = None, snowflake_conn_id = 'snowflake_default', parameters = None, autocommit = True, do_xcom_push = True, warehouse = None, database = None, role = None, schema = None, authenticator = None, session_parameters = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.operators.sql.SQLValueCheckOperator` |
| |
| Performs a simple check using sql code against a specified value, within a |
| certain level of tolerance. |
| |
| :param sql: the sql to be executed |
| :param pass_value: the value to check against |
| :param tolerance: (optional) the tolerance allowed to accept the query as |
| passing |
| :param snowflake_conn_id: Reference to |
| :ref:`Snowflake connection id<howto/connection:snowflake>` |
| :param autocommit: if True, each command is automatically committed. |
| (default value: True) |
| :param parameters: (optional) the parameters to render the SQL query with. |
| :param warehouse: name of warehouse (will overwrite any warehouse |
| defined in the connection's extra JSON) |
| :param database: name of database (will overwrite database defined |
| in connection) |
| :param schema: name of schema (will overwrite schema defined in |
| connection) |
| :param role: name of role (will overwrite any role defined in |
| connection's extra JSON) |
| :param authenticator: authenticator for Snowflake. |
| 'snowflake' (default) to use the internal Snowflake authenticator |
| 'externalbrowser' to authenticate using your web browser and |
| Okta, ADFS or any other SAML 2.0-compliant identify provider |
| (IdP) that has been defined for your account |
| 'https://<your_okta_account_name>.okta.com' to authenticate |
| through native Okta. |
| :param session_parameters: You can set session-level parameters at |
| the time you connect to Snowflake |
| |
| .. py:method:: get_db_hook(self) |
| |
| Get the database hook for the connection. |
| |
| :return: the database hook object. |
| :rtype: DbApiHook |
| |
| |
| |
| .. py:class:: SnowflakeIntervalCheckOperator(*, table, metrics_thresholds, date_filter_column = 'ds', days_back = -7, snowflake_conn_id = 'snowflake_default', parameters = None, autocommit = True, do_xcom_push = True, warehouse = None, database = None, role = None, schema = None, authenticator = None, session_parameters = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.operators.sql.SQLIntervalCheckOperator` |
| |
| Checks that the values of metrics given as SQL expressions are within |
| a certain tolerance of the ones from days_back before. |
| |
| This method constructs a query like so :: |
| |
| SELECT {metrics_threshold_dict_key} FROM {table} |
| WHERE {date_filter_column}=<date> |
| |
| :param table: the table name |
| :param days_back: number of days between ds and the ds we want to check |
| against. Defaults to 7 days |
| :param metrics_thresholds: a dictionary of ratios indexed by metrics, for |
| example 'COUNT(*)': 1.5 would require a 50 percent or less difference |
| between the current day, and the prior days_back. |
| :param snowflake_conn_id: Reference to |
| :ref:`Snowflake connection id<howto/connection:snowflake>` |
| :param autocommit: if True, each command is automatically committed. |
| (default value: True) |
| :param parameters: (optional) the parameters to render the SQL query with. |
| :param warehouse: name of warehouse (will overwrite any warehouse |
| defined in the connection's extra JSON) |
| :param database: name of database (will overwrite database defined |
| in connection) |
| :param schema: name of schema (will overwrite schema defined in |
| connection) |
| :param role: name of role (will overwrite any role defined in |
| connection's extra JSON) |
| :param authenticator: authenticator for Snowflake. |
| 'snowflake' (default) to use the internal Snowflake authenticator |
| 'externalbrowser' to authenticate using your web browser and |
| Okta, ADFS or any other SAML 2.0-compliant identify provider |
| (IdP) that has been defined for your account |
| 'https://<your_okta_account_name>.okta.com' to authenticate |
| through native Okta. |
| :param session_parameters: You can set session-level parameters at |
| the time you connect to Snowflake |
| |
| .. py:method:: get_db_hook(self) |
| |
| Get the database hook for the connection. |
| |
| :return: the database hook object. |
| :rtype: DbApiHook |
| |
| |
| |
