| :py:mod:`tests.system.providers.cncf.kubernetes.example_spark_kubernetes` |
| ========================================================================= |
| |
| .. py:module:: tests.system.providers.cncf.kubernetes.example_spark_kubernetes |
| |
| .. autoapi-nested-parse:: |
| |
| This is an example DAG which uses SparkKubernetesOperator and SparkKubernetesSensor. |
| In this example, we create two tasks which execute sequentially. |
| The first task is to submit sparkApplication on Kubernetes cluster(the example uses spark-pi application). |
| and the second task is to check the final state of the sparkApplication that submitted in the first state. |
| |
| Spark-on-k8s operator is required to be already installed on Kubernetes |
| https://github.com/GoogleCloudPlatform/spark-on-k8s-operator |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| .. py:data:: ENV_ID |
| |
| |
| |
| |
| .. py:data:: DAG_ID |
| :annotation: = spark_pi |
| |
| |
| |
| .. py:data:: t1 |
| |
| |
| |
| |
| .. py:data:: test_run |
| |
| |
| |
| |
