| :py:mod:`airflow.providers.google.cloud.operators.mlengine` |
| =========================================================== |
| |
| .. py:module:: airflow.providers.google.cloud.operators.mlengine |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains Google Cloud MLEngine operators. |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.operators.mlengine.MLEngineStartBatchPredictionJobOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineCreateModelOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineGetModelOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineDeleteModelOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineManageVersionOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineCreateVersionOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineSetDefaultVersionOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineListVersionsOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineDeleteVersionOperator |
| airflow.providers.google.cloud.operators.mlengine.AIPlatformConsoleLink |
| airflow.providers.google.cloud.operators.mlengine.MLEngineStartTrainingJobOperator |
| airflow.providers.google.cloud.operators.mlengine.MLEngineTrainingCancelJobOperator |
| |
| |
| |
| |
| Attributes |
| ~~~~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.operators.mlengine.log |
| |
| |
| .. py:data:: log |
| |
| |
| |
| |
| .. py:class:: MLEngineStartBatchPredictionJobOperator(*, job_id, region, data_format, input_paths, output_path, model_name = None, version_name = None, uri = None, max_worker_count = None, runtime_version = None, signature_name = None, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, labels = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Start a Google Cloud ML Engine prediction job. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineStartBatchPredictionJobOperator` |
| |
| NOTE: For model origin, users should consider exactly one from the |
| three options below: |
| |
| 1. Populate ``uri`` field only, which should be a GCS location that |
| points to a tensorflow savedModel directory. |
| 2. Populate ``model_name`` field only, which refers to an existing |
| model, and the default version of the model will be used. |
| 3. Populate both ``model_name`` and ``version_name`` fields, which |
| refers to a specific version of a specific model. |
| |
| In options 2 and 3, both model and version name should contain the |
| minimal identifier. For instance, call:: |
| |
| MLEngineBatchPredictionOperator( |
| ..., |
| model_name='my_model', |
| version_name='my_version', |
| ...) |
| |
| if the desired model version is |
| ``projects/my_project/models/my_model/versions/my_version``. |
| |
| See https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs |
| for further documentation on the parameters. |
| |
| :param job_id: A unique id for the prediction job on Google Cloud |
| ML Engine. (templated) |
| :param data_format: The format of the input data. |
| It will default to 'DATA_FORMAT_UNSPECIFIED' if is not provided |
| or is not one of ["TEXT", "TF_RECORD", "TF_RECORD_GZIP"]. |
| :param input_paths: A list of GCS paths of input data for batch |
| prediction. Accepting wildcard operator ``*``, but only at the end. (templated) |
| :param output_path: The GCS path where the prediction results are |
| written to. (templated) |
| :param region: The Google Compute Engine region to run the |
| prediction job in. (templated) |
| :param model_name: The Google Cloud ML Engine model to use for prediction. |
| If version_name is not provided, the default version of this |
| model will be used. |
| Should not be None if version_name is provided. |
| Should be None if uri is provided. (templated) |
| :param version_name: The Google Cloud ML Engine model version to use for |
| prediction. |
| Should be None if uri is provided. (templated) |
| :param uri: The GCS path of the saved model to use for prediction. |
| Should be None if model_name is provided. |
| It should be a GCS path pointing to a tensorflow SavedModel. (templated) |
| :param max_worker_count: The maximum number of workers to be used |
| for parallel processing. Defaults to 10 if not specified. Should be a |
| string representing the worker count ("10" instead of 10, "50" instead |
| of 50, etc.) |
| :param runtime_version: The Google Cloud ML Engine runtime version to use |
| for batch prediction. |
| :param signature_name: The name of the signature defined in the SavedModel |
| to use for this job. |
| :param project_id: The Google Cloud project name where the prediction job is submitted. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID used for connection to Google |
| Cloud Platform. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param labels: a dictionary containing labels for the job; passed to BigQuery |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| :raises: ``ValueError``: if a unique model/version origin cannot be |
| determined. |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_job_id', '_region', '_input_paths', '_output_path', '_model_name',... |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineManageModelOperator(*, model, operation = 'create', project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Operator for managing a Google Cloud ML Engine model. |
| |
| .. warning:: |
| This operator is deprecated. Consider using operators for specific operations: |
| MLEngineCreateModelOperator, MLEngineGetModelOperator. |
| |
| :param model: A dictionary containing the information about the model. |
| If the `operation` is `create`, then the `model` parameter should |
| contain all the information about this model such as `name`. |
| |
| If the `operation` is `get`, the `model` parameter |
| should contain the `name` of the model. |
| :param operation: The operation to perform. Available operations are: |
| |
| * ``create``: Creates a new model as provided by the `model` parameter. |
| * ``get``: Gets a particular model where the name is specified in `model`. |
| :param project_id: The Google Cloud project name to which MLEngine model belongs. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineCreateModelOperator(*, model, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Creates a new model. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineCreateModelOperator` |
| |
| The model should be provided by the `model` parameter. |
| |
| :param model: A dictionary containing the information about the model. |
| :param project_id: The Google Cloud project name to which MLEngine model belongs. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineGetModelOperator(*, model_name, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Gets a particular model |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineGetModelOperator` |
| |
| The name of model should be specified in `model_name`. |
| |
| :param model_name: The name of the model. |
| :param project_id: The Google Cloud project name to which MLEngine model belongs. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model_name', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineDeleteModelOperator(*, model_name, delete_contents = False, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Deletes a model. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineDeleteModelOperator` |
| |
| The model should be provided by the `model_name` parameter. |
| |
| :param model_name: The name of the model. |
| :param delete_contents: (Optional) Whether to force the deletion even if the models is not empty. |
| Will delete all version (if any) in the dataset if set to True. |
| The default value is False. |
| :param project_id: The Google Cloud project name to which MLEngine model belongs. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model_name', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineManageVersionOperator(*, model_name, version_name = None, version = None, operation = 'create', project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Operator for managing a Google Cloud ML Engine version. |
| |
| .. warning:: |
| This operator is deprecated. Consider using operators for specific operations: |
| MLEngineCreateVersionOperator, MLEngineSetDefaultVersionOperator, |
| MLEngineListVersionsOperator, MLEngineDeleteVersionOperator. |
| |
| :param model_name: The name of the Google Cloud ML Engine model that the version |
| belongs to. (templated) |
| :param version_name: A name to use for the version being operated upon. |
| If not None and the `version` argument is None or does not have a value for |
| the `name` key, then this will be populated in the payload for the |
| `name` key. (templated) |
| :param version: A dictionary containing the information about the version. |
| If the `operation` is `create`, `version` should contain all the |
| information about this version such as name, and deploymentUrl. |
| If the `operation` is `get` or `delete`, the `version` parameter |
| should contain the `name` of the version. |
| If it is None, the only `operation` possible would be `list`. (templated) |
| :param operation: The operation to perform. Available operations are: |
| |
| * ``create``: Creates a new version in the model specified by `model_name`, |
| in which case the `version` parameter should contain all the |
| information to create that version |
| (e.g. `name`, `deploymentUrl`). |
| |
| * ``set_defaults``: Sets a version in the model specified by `model_name` to be the default. |
| The name of the version should be specified in the `version` |
| parameter. |
| |
| * ``list``: Lists all available versions of the model specified |
| by `model_name`. |
| |
| * ``delete``: Deletes the version specified in `version` parameter from the |
| model specified by `model_name`). |
| The name of the version should be specified in the `version` |
| parameter. |
| :param project_id: The Google Cloud project name to which MLEngine model belongs. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model_name', '_version_name', '_version', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineCreateVersionOperator(*, model_name, version, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Creates a new version in the model |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineCreateVersionOperator` |
| |
| Model should be specified by `model_name`, in which case the `version` parameter should contain all the |
| information to create that version |
| |
| :param model_name: The name of the Google Cloud ML Engine model that the version belongs to. (templated) |
| :param version: A dictionary containing the information about the version. (templated) |
| :param project_id: The Google Cloud project name to which MLEngine model belongs. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model_name', '_version', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineSetDefaultVersionOperator(*, model_name, version_name, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Sets a version in the model. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineSetDefaultVersionOperator` |
| |
| The model should be specified by `model_name` to be the default. The name of the version should be |
| specified in the `version_name` parameter. |
| |
| :param model_name: The name of the Google Cloud ML Engine model that the version belongs to. (templated) |
| :param version_name: A name to use for the version being operated upon. (templated) |
| :param project_id: The Google Cloud project name to which MLEngine model belongs. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model_name', '_version_name', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineListVersionsOperator(*, model_name, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Lists all available versions of the model |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineListVersionsOperator` |
| |
| The model should be specified by `model_name`. |
| |
| :param model_name: The name of the Google Cloud ML Engine model that the version |
| belongs to. (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param project_id: The Google Cloud project name to which MLEngine model belongs. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model_name', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineDeleteVersionOperator(*, model_name, version_name, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Deletes the version from the model. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineDeleteVersionOperator` |
| |
| The name of the version should be specified in `version_name` parameter from the model specified |
| by `model_name`. |
| |
| :param model_name: The name of the Google Cloud ML Engine model that the version |
| belongs to. (templated) |
| :param version_name: A name to use for the version being operated upon. (templated) |
| :param project_id: The Google Cloud project name to which MLEngine |
| model belongs. |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_model_name', '_version_name', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: AIPlatformConsoleLink |
| |
| Bases: :py:obj:`airflow.models.BaseOperatorLink` |
| |
| Helper class for constructing AI Platform Console link. |
| |
| .. py:attribute:: name |
| :annotation: = AI Platform Console |
| |
| |
| |
| .. py:method:: get_link(self, operator, dttm = None, ti_key = None) |
| |
| Link to external system. |
| |
| Note: The old signature of this function was ``(self, operator, dttm: datetime)``. That is still |
| supported at runtime but is deprecated. |
| |
| :param operator: airflow operator |
| :param ti_key: TaskInstance ID to return link for |
| :return: link to external system |
| |
| |
| |
| .. py:class:: MLEngineStartTrainingJobOperator(*, job_id, region, package_uris = None, training_python_module = None, training_args = None, scale_tier = None, master_type = None, master_config = None, runtime_version = None, python_version = None, job_dir = None, service_account = None, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, mode = 'PRODUCTION', labels = None, impersonation_chain = None, hyperparameters = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Operator for launching a MLEngine training job. |
| |
| .. seealso:: |
| For more information on how to use this operator, take a look at the guide: |
| :ref:`howto/operator:MLEngineStartTrainingJobOperator` |
| |
| :param job_id: A unique templated id for the submitted Google MLEngine |
| training job. (templated) |
| :param region: The Google Compute Engine region to run the MLEngine training |
| job in (templated). |
| :param package_uris: A list of Python package locations for the training |
| job, which should include the main training program and any additional |
| dependencies. This is mutually exclusive with a custom image specified |
| via master_config. (templated) |
| :param training_python_module: The name of the Python module to run within |
| the training job after installing the packages. This is mutually |
| exclusive with a custom image specified via master_config. (templated) |
| :param training_args: A list of command-line arguments to pass to the |
| training program. (templated) |
| :param scale_tier: Resource tier for MLEngine training job. (templated) |
| :param master_type: The type of virtual machine to use for the master |
| worker. It must be set whenever scale_tier is CUSTOM. (templated) |
| :param master_config: The configuration for the master worker. If this is |
| provided, master_type must be set as well. If a custom image is |
| specified, this is mutually exclusive with package_uris and |
| training_python_module. (templated) |
| :param runtime_version: The Google Cloud ML runtime version to use for |
| training. (templated) |
| :param python_version: The version of Python used in training. (templated) |
| :param job_dir: A Google Cloud Storage path in which to store training |
| outputs and other data needed for training. (templated) |
| :param service_account: Optional service account to use when running the training application. |
| (templated) |
| The specified service account must have the `iam.serviceAccounts.actAs` role. The |
| Google-managed Cloud ML Engine service account must have the `iam.serviceAccountAdmin` role |
| for the specified service account. |
| If set to None or missing, the Google-managed Cloud ML Engine service account will be used. |
| :param project_id: The Google Cloud project name within which MLEngine training job should run. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param mode: Can be one of 'DRY_RUN'/'CLOUD'. In 'DRY_RUN' mode, no real |
| training job will be launched, but the MLEngine training job request |
| will be printed out. In 'CLOUD' mode, a real MLEngine training job |
| creation request will be issued. |
| :param labels: a dictionary containing labels for the job; passed to BigQuery |
| :param hyperparameters: Optional HyperparameterSpec dictionary for hyperparameter tuning. |
| For further reference, check: |
| https://cloud.google.com/ai-platform/training/docs/reference/rest/v1/projects.jobs#HyperparameterSpec |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_job_id', '_region', '_package_uris', '_training_python_module',... |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: MLEngineTrainingCancelJobOperator(*, job_id, project_id = None, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Operator for cleaning up failed MLEngine training job. |
| |
| :param job_id: A unique templated id for the submitted Google MLEngine |
| training job. (templated) |
| :param project_id: The Google Cloud project name within which MLEngine training job should run. |
| If set to None or missing, the default project_id from the Google Cloud connection is used. |
| (templated) |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: :Sequence[str] = ['_project_id', '_job_id', '_impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
