docs-archive/apache-airflow/2.0.0/apache-airflow/stable/_sources/upgrade-check.rst.txt - airflow-site - Git at Google

  .. Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at

  ..   http://www.apache.org/licenses/LICENSE-2.0

  .. Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.


 Upgrade Check Script
 --------------------

 .. contents:: :local:

 .. _upgrade-check:

 Getting the Airflow Upgrade Check Package
 '''''''''''''''''''''''''''''''''''''''''

 Apache Airflow is published as ``apache-airflow`` package in PyPI. The Upgrade Check Script is part of a
 separate Python Package, since it is separate from the core Apache Airflow package and is only needed for
 a period of time and specifically only for upgrading from Airflow 1.10 releases to Airflow 2.0.

 While there has been a lot of work put into making this upgrade as easy as possible, there are some
 changes which are compatible between Airflow 1.10 and Airflow 2.0. In order to make this as simple to
 navigate, we recommend that people first upgrade to the latest release in the 1.10 series (at the
 time of writing: 1.10.14) and then to download this package and run the script as detailed below.


 .. note::

    On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
    does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
    of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
    ``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
    ``--use-deprecated legacy-resolver`` to your pip install command.

 .. code-block:: bash

     pip install apache-airflow-upgrade-check

 This will install the latest version of the Airflow Upgrade check package.


 Running the Airflow Upgrade Check Package
 '''''''''''''''''''''''''''''''''''''''''

 .. code-block:: bash

     airflow upgrade_check

 This will print out to the screen a number of actions that should be taken before upgrading the Airflow
 release to 2.0.0 or higher.

 The exit code of the command will be 0 (success) if no problems are found in running the command, or 1 if
 problems were encountered in running the check.

 A sample output as a result of a successful run of the upgrade check is shown below.

 .. code-block:: bash

     ============================================== STATUS ============================================

     Check for latest versions of apache-airflow and checker..................................SUCCESS
     Remove airflow.AirflowMacroPlugin class..................................................SUCCESS
     Chain between DAG and operator not allowed...............................................SUCCESS
     Connection.conn_id is not unique.........................................................SUCCESS
     Connection.conn_type is not nullable.....................................................SUCCESS
     Fernet is enabled by default.............................................................FAIL
     GCP service account key deprecation......................................................SUCCESS
     Changes in import paths of hooks, operators, sensors and others..........................FAIL
     Users must delete deprecated configs for KubernetesExecutor..............................FAIL
     Legacy UI is deprecated by default.......................................................SUCCESS
     Logging configuration has been moved to new section......................................FAIL
     Removal of Mesos Executor................................................................SUCCESS
     Users must set a kubernetes.pod_template_file value......................................FAIL
     SendGrid email uses old airflow.contrib module...........................................SUCCESS
     Changes in import path of remote task handlers...........................................SUCCESS
     Jinja Template Variables cannot be undefined.............................................FAIL
     Found 7 problems.

     ========================================== RECOMMENDATIONS ========================================

     Fernet is enabled by default
     ----------------------------
     The fernet mechanism is enabled by default to increase the security of the default installation.

     Problems:

     1.  fernet_key in airflow.cfg must be explicitly set empty as fernet mechanism is enabledby default. This means that the apache-airflow[crypto] extra-packages are always installed.However, this requires that your operating system has libffi-dev installed.

     Changes in import paths of hooks, operators, sensors and others
     ---------------------------------------------------------------
     Many hooks, operators and other classes has been renamed and moved. Those changes were part of unifying names and imports paths as described in AIP-21.
     The contrib folder has been replaced by providers directory and packages:
     https://github.com/apache/airflow#backport-packages

     Problems:

     1.  Using ``airflow.operators.python_operator.PythonOperator`` will be replaced by ``airflow.operators.python.PythonOperator``. Affected file:


 The following sections describe what is being done and how to apply the recommendations shown above.
 Please note that the above results shown are only a partial set, where only the first
 two of the seven problems identified are shown in the section above. In reality,
 all the problems are shown on the screen.


 Understanding what is being checked
 '''''''''''''''''''''''''''''''''''

 The Upgrade Check checks the configuration data from airflow.cfg, the meta data from the Airflow
 database, as well as the DAGs which have been set up in the current Airflow environment.

 Using the above results as an example, there are two specific problems which have
 been identified.

 The first problem is identified in the configuration file airflow.cfg where the current configuration
 option for the fernet_key is no longer acceptable and needs to be changed. This is because as of
 Airflow 2.0, the fernet_key cannot be left empty, but needs to have a defined value. Examining the
 problematic airflow.cfg and searching for the fernet_key entries would show the following:

 .. code-block:: bash

     fernet_key =

 The second problem was identified in one of the DAGs. In this case, this import
 statement for the ``PythonOperator`` needs to be changed, since the location is different
 in Airflow 2.0. Examining the DAG file would probably show the following:

 .. code-block:: python

     from airflow.operators.python_operator import PythonOperator

 We will discuss how to fix these and make them compatible with Airflow 2.0 in the next
 section.


 Applying the Recommendations
 ''''''''''''''''''''''''''''

 In most cases, the Recommendations result section of the Upgrade check contains
 enough information to make the change.

 For the first problem identified above with respect to the fernet_key, the solution is
 to enter a valid value in the Airflow Configuration file airflow.cfg for the fernet_key.

 For the second problem, looking at the source of the DAG file and changing the import
 statement for the Python Operator to be as follows will make this DAG work in Airflow 2.0.

 .. code-block:: python

     from airflow.operators.python import PythonOperator

 However, at the time of writing, this is incompatible in Airflow 1.10.14. So, this change
 can only be made while moving to Airflow 2.0.


 Turning off checks
 ''''''''''''''''''

 Advanced Airflow users or those with multiple Airflow deployments may want to
 customize the Upgrade Checks to their environment by turning off certain
 checks which are not applicable to them. And example of this is users with
 Airflow deployments not using the ``KubernetesPodOperator`` may want to turn off upgrade
 checks related to the ``KubernetesPodOperator``.

 This can be done by creating an "upgrade config file" in YAML as shown below:

 .. code-block::

     ignored_rules:
         - PodTemplateFileRule

 To use this configuration file (named ```upgrade-configuration.yaml`` for  this
 example) while running the upgrade check script, use the following command syntax:

 .. code-block:: bash

     airflow upgrade_check --config=./upgrade-configuration.yaml


 Adding custom checks
 ''''''''''''''''''''

 Advanced Airflow users or those with multiple Airflow deployments may also
 want to add additional upgrade checks for specific elements in their
 environment whether it is DAGs or configuration related.

 These additional checks should be defined in a Python class and added as
 ``custom_rules`` in the "upgrade config file" as shown below:

 .. code-block::

     custom_rules:
         - path.CustomCheckClass1
         - path.CustomCheckClass2

 Now, invoke the upgrade check script using this configuration file as shown
 below (the config file is named ``upgrade-configuration.yaml`` in
 this example):

 .. code-block:: bash

     airflow upgrade_check --config=./upgrade-configuration.yaml
	.. Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	.. http://www.apache.org/licenses/LICENSE-2.0

	.. Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.


	Upgrade Check Script
	--------------------

	.. contents:: :local:

	.. _upgrade-check:

	Getting the Airflow Upgrade Check Package
	'''''''''''''''''''''''''''''''''''''''''

	Apache Airflow is published as ``apache-airflow`` package in PyPI. The Upgrade Check Script is part of a
	separate Python Package, since it is separate from the core Apache Airflow package and is only needed for
	a period of time and specifically only for upgrading from Airflow 1.10 releases to Airflow 2.0.

	While there has been a lot of work put into making this upgrade as easy as possible, there are some
	changes which are compatible between Airflow 1.10 and Airflow 2.0. In order to make this as simple to
	navigate, we recommend that people first upgrade to the latest release in the 1.10 series (at the
	time of writing: 1.10.14) and then to download this package and run the script as detailed below.


	.. note::

	On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
	does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
	of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
	``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
	``--use-deprecated legacy-resolver`` to your pip install command.

	.. code-block:: bash

	pip install apache-airflow-upgrade-check

	This will install the latest version of the Airflow Upgrade check package.


	Running the Airflow Upgrade Check Package
	'''''''''''''''''''''''''''''''''''''''''

	.. code-block:: bash

	airflow upgrade_check

	This will print out to the screen a number of actions that should be taken before upgrading the Airflow
	release to 2.0.0 or higher.

	The exit code of the command will be 0 (success) if no problems are found in running the command, or 1 if
	problems were encountered in running the check.

	A sample output as a result of a successful run of the upgrade check is shown below.

	.. code-block:: bash

	============================================== STATUS ============================================

	Check for latest versions of apache-airflow and checker..................................SUCCESS
	Remove airflow.AirflowMacroPlugin class..................................................SUCCESS
	Chain between DAG and operator not allowed...............................................SUCCESS
	Connection.conn_id is not unique.........................................................SUCCESS
	Connection.conn_type is not nullable.....................................................SUCCESS
	Fernet is enabled by default.............................................................FAIL
	GCP service account key deprecation......................................................SUCCESS
	Changes in import paths of hooks, operators, sensors and others..........................FAIL
	Users must delete deprecated configs for KubernetesExecutor..............................FAIL
	Legacy UI is deprecated by default.......................................................SUCCESS
	Logging configuration has been moved to new section......................................FAIL
	Removal of Mesos Executor................................................................SUCCESS
	Users must set a kubernetes.pod_template_file value......................................FAIL
	SendGrid email uses old airflow.contrib module...........................................SUCCESS
	Changes in import path of remote task handlers...........................................SUCCESS
	Jinja Template Variables cannot be undefined.............................................FAIL
	Found 7 problems.

	========================================== RECOMMENDATIONS ========================================

	Fernet is enabled by default
	----------------------------
	The fernet mechanism is enabled by default to increase the security of the default installation.

	Problems:

	1. fernet_key in airflow.cfg must be explicitly set empty as fernet mechanism is enabledby default. This means that the apache-airflow[crypto] extra-packages are always installed.However, this requires that your operating system has libffi-dev installed.

	Changes in import paths of hooks, operators, sensors and others
	---------------------------------------------------------------
	Many hooks, operators and other classes has been renamed and moved. Those changes were part of unifying names and imports paths as described in AIP-21.
	The contrib folder has been replaced by providers directory and packages:
	https://github.com/apache/airflow#backport-packages

	Problems:

	1. Using ``airflow.operators.python_operator.PythonOperator`` will be replaced by ``airflow.operators.python.PythonOperator``. Affected file:


	The following sections describe what is being done and how to apply the recommendations shown above.
	Please note that the above results shown are only a partial set, where only the first
	two of the seven problems identified are shown in the section above. In reality,
	all the problems are shown on the screen.


	Understanding what is being checked
	'''''''''''''''''''''''''''''''''''

	The Upgrade Check checks the configuration data from airflow.cfg, the meta data from the Airflow
	database, as well as the DAGs which have been set up in the current Airflow environment.

	Using the above results as an example, there are two specific problems which have
	been identified.

	The first problem is identified in the configuration file airflow.cfg where the current configuration
	option for the fernet_key is no longer acceptable and needs to be changed. This is because as of
	Airflow 2.0, the fernet_key cannot be left empty, but needs to have a defined value. Examining the
	problematic airflow.cfg and searching for the fernet_key entries would show the following:

	.. code-block:: bash

	fernet_key =

	The second problem was identified in one of the DAGs. In this case, this import
	statement for the ``PythonOperator`` needs to be changed, since the location is different
	in Airflow 2.0. Examining the DAG file would probably show the following:

	.. code-block:: python

	from airflow.operators.python_operator import PythonOperator

	We will discuss how to fix these and make them compatible with Airflow 2.0 in the next
	section.


	Applying the Recommendations
	''''''''''''''''''''''''''''

	In most cases, the Recommendations result section of the Upgrade check contains
	enough information to make the change.

	For the first problem identified above with respect to the fernet_key, the solution is
	to enter a valid value in the Airflow Configuration file airflow.cfg for the fernet_key.

	For the second problem, looking at the source of the DAG file and changing the import
	statement for the Python Operator to be as follows will make this DAG work in Airflow 2.0.

	.. code-block:: python

	from airflow.operators.python import PythonOperator

	However, at the time of writing, this is incompatible in Airflow 1.10.14. So, this change
	can only be made while moving to Airflow 2.0.


	Turning off checks
	''''''''''''''''''

	Advanced Airflow users or those with multiple Airflow deployments may want to
	customize the Upgrade Checks to their environment by turning off certain
	checks which are not applicable to them. And example of this is users with
	Airflow deployments not using the ``KubernetesPodOperator`` may want to turn off upgrade
	checks related to the ``KubernetesPodOperator``.

	This can be done by creating an "upgrade config file" in YAML as shown below:

	.. code-block::

	ignored_rules:
	- PodTemplateFileRule

	To use this configuration file (named ```upgrade-configuration.yaml`` for this
	example) while running the upgrade check script, use the following command syntax:

	.. code-block:: bash

	airflow upgrade_check --config=./upgrade-configuration.yaml


	Adding custom checks
	''''''''''''''''''''

	Advanced Airflow users or those with multiple Airflow deployments may also
	want to add additional upgrade checks for specific elements in their
	environment whether it is DAGs or configuration related.

	These additional checks should be defined in a Python class and added as
	``custom_rules`` in the "upgrade config file" as shown below:

	.. code-block::

	custom_rules:
	- path.CustomCheckClass1
	- path.CustomCheckClass2

	Now, invoke the upgrade check script using this configuration file as shown
	below (the config file is named ``upgrade-configuration.yaml`` in
	this example):

	.. code-block:: bash

	airflow upgrade_check --config=./upgrade-configuration.yaml