| :mod:`airflow.contrib.operators.mlengine_operator` |
| ================================================== |
| |
| .. py:module:: airflow.contrib.operators.mlengine_operator |
| |
| |
| Module Contents |
| --------------- |
| |
| .. data:: log |
| |
| |
| |
| |
| .. function:: _normalize_mlengine_job_id(job_id) |
| Replaces invalid MLEngine job_id characters with '_'. |
| |
| This also adds a leading 'z' in case job_id starts with an invalid |
| character. |
| |
| Args: |
| job_id: A job_id str that may have invalid characters. |
| |
| Returns: |
| A valid job_id representation. |
| |
| |
| .. py:class:: MLEngineBatchPredictionOperator(project_id, job_id, region, data_format, input_paths, output_path, model_name=None, version_name=None, uri=None, max_worker_count=None, runtime_version=None, signature_name=None, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs) |
| |
| Bases: :class:`airflow.operators.BaseOperator` |
| |
| Start a Google Cloud ML Engine prediction job. |
| |
| NOTE: For model origin, users should consider exactly one from the |
| three options below: |
| |
| 1. Populate ``uri`` field only, which should be a GCS location that |
| points to a tensorflow savedModel directory. |
| 2. Populate ``model_name`` field only, which refers to an existing |
| model, and the default version of the model will be used. |
| 3. Populate both ``model_name`` and ``version_name`` fields, which |
| refers to a specific version of a specific model. |
| |
| In options 2 and 3, both model and version name should contain the |
| minimal identifier. For instance, call:: |
| |
| MLEngineBatchPredictionOperator( |
| ..., |
| model_name='my_model', |
| version_name='my_version', |
| ...) |
| |
| if the desired model version is |
| ``projects/my_project/models/my_model/versions/my_version``. |
| |
| See https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs |
| for further documentation on the parameters. |
| |
| :param project_id: The Google Cloud project name where the |
| prediction job is submitted. (templated) |
| :type project_id: str |
| |
| :param job_id: A unique id for the prediction job on Google Cloud |
| ML Engine. (templated) |
| :type job_id: str |
| |
| :param data_format: The format of the input data. |
| It will default to 'DATA_FORMAT_UNSPECIFIED' if is not provided |
| or is not one of ["TEXT", "TF_RECORD", "TF_RECORD_GZIP"]. |
| :type data_format: str |
| |
| :param input_paths: A list of GCS paths of input data for batch |
| prediction. Accepting wildcard operator ``*``, but only at the end. (templated) |
| :type input_paths: list[str] |
| |
| :param output_path: The GCS path where the prediction results are |
| written to. (templated) |
| :type output_path: str |
| |
| :param region: The Google Compute Engine region to run the |
| prediction job in. (templated) |
| :type region: str |
| |
| :param model_name: The Google Cloud ML Engine model to use for prediction. |
| If version_name is not provided, the default version of this |
| model will be used. |
| Should not be None if version_name is provided. |
| Should be None if uri is provided. (templated) |
| :type model_name: str |
| |
| :param version_name: The Google Cloud ML Engine model version to use for |
| prediction. |
| Should be None if uri is provided. (templated) |
| :type version_name: str |
| |
| :param uri: The GCS path of the saved model to use for prediction. |
| Should be None if model_name is provided. |
| It should be a GCS path pointing to a tensorflow SavedModel. (templated) |
| :type uri: str |
| |
| :param max_worker_count: The maximum number of workers to be used |
| for parallel processing. Defaults to 10 if not specified. |
| :type max_worker_count: int |
| |
| :param runtime_version: The Google Cloud ML Engine runtime version to use |
| for batch prediction. |
| :type runtime_version: str |
| |
| :param signature_name: The name of the signature defined in the SavedModel |
| to use for this job. |
| :type signature_name: str |
| |
| :param gcp_conn_id: The connection ID used for connection to Google |
| Cloud Platform. |
| :type gcp_conn_id: str |
| |
| :param delegate_to: The account to impersonate, if any. |
| For this to work, the service account making the request must |
| have domain-wide delegation enabled. |
| :type delegate_to: str |
| |
| :raises: ``ValueError``: if a unique model/version origin cannot be |
| determined. |
| |
| .. attribute:: template_fields |
| :annotation: = ['_project_id', '_job_id', '_region', '_input_paths', '_output_path', '_model_name', '_version_name', '_uri'] |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: MLEngineModelOperator(project_id, model, operation='create', gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs) |
| |
| Bases: :class:`airflow.operators.BaseOperator` |
| |
| Operator for managing a Google Cloud ML Engine model. |
| |
| :param project_id: The Google Cloud project name to which MLEngine |
| model belongs. (templated) |
| :type project_id: str |
| :param model: A dictionary containing the information about the model. |
| If the `operation` is `create`, then the `model` parameter should |
| contain all the information about this model such as `name`. |
| |
| If the `operation` is `get`, the `model` parameter |
| should contain the `name` of the model. |
| :type model: dict |
| :param operation: The operation to perform. Available operations are: |
| |
| * ``create``: Creates a new model as provided by the `model` parameter. |
| * ``get``: Gets a particular model where the name is specified in `model`. |
| :type operation: str |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :type gcp_conn_id: str |
| :param delegate_to: The account to impersonate, if any. |
| For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :type delegate_to: str |
| |
| .. attribute:: template_fields |
| :annotation: = ['_model'] |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: MLEngineVersionOperator(project_id, model_name, version_name=None, version=None, operation='create', gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs) |
| |
| Bases: :class:`airflow.operators.BaseOperator` |
| |
| Operator for managing a Google Cloud ML Engine version. |
| |
| :param project_id: The Google Cloud project name to which MLEngine |
| model belongs. |
| :type project_id: str |
| |
| :param model_name: The name of the Google Cloud ML Engine model that the version |
| belongs to. (templated) |
| :type model_name: str |
| |
| :param version_name: A name to use for the version being operated upon. |
| If not None and the `version` argument is None or does not have a value for |
| the `name` key, then this will be populated in the payload for the |
| `name` key. (templated) |
| :type version_name: str |
| |
| :param version: A dictionary containing the information about the version. |
| If the `operation` is `create`, `version` should contain all the |
| information about this version such as name, and deploymentUrl. |
| If the `operation` is `get` or `delete`, the `version` parameter |
| should contain the `name` of the version. |
| If it is None, the only `operation` possible would be `list`. (templated) |
| :type version: dict |
| |
| :param operation: The operation to perform. Available operations are: |
| |
| * ``create``: Creates a new version in the model specified by `model_name`, |
| in which case the `version` parameter should contain all the |
| information to create that version |
| (e.g. `name`, `deploymentUrl`). |
| |
| * ``get``: Gets full information of a particular version in the model |
| specified by `model_name`. |
| The name of the version should be specified in the `version` |
| parameter. |
| |
| * ``list``: Lists all available versions of the model specified |
| by `model_name`. |
| |
| * ``delete``: Deletes the version specified in `version` parameter from the |
| model specified by `model_name`). |
| The name of the version should be specified in the `version` |
| parameter. |
| :type operation: str |
| |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :type gcp_conn_id: str |
| |
| :param delegate_to: The account to impersonate, if any. |
| For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :type delegate_to: str |
| |
| .. attribute:: template_fields |
| :annotation: = ['_model_name', '_version_name', '_version'] |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |
| .. py:class:: MLEngineTrainingOperator(project_id, job_id, package_uris, training_python_module, training_args, region, scale_tier=None, master_type=None, runtime_version=None, python_version=None, job_dir=None, gcp_conn_id='google_cloud_default', delegate_to=None, mode='PRODUCTION', *args, **kwargs) |
| |
| Bases: :class:`airflow.operators.BaseOperator` |
| |
| Operator for launching a MLEngine training job. |
| |
| :param project_id: The Google Cloud project name within which MLEngine |
| training job should run (templated). |
| :type project_id: str |
| |
| :param job_id: A unique templated id for the submitted Google MLEngine |
| training job. (templated) |
| :type job_id: str |
| |
| :param package_uris: A list of package locations for MLEngine training job, |
| which should include the main training program + any additional |
| dependencies. (templated) |
| :type package_uris: str |
| |
| :param training_python_module: The Python module name to run within MLEngine |
| training job after installing 'package_uris' packages. (templated) |
| :type training_python_module: str |
| |
| :param training_args: A list of templated command line arguments to pass to |
| the MLEngine training program. (templated) |
| :type training_args: str |
| |
| :param region: The Google Compute Engine region to run the MLEngine training |
| job in (templated). |
| :type region: str |
| |
| :param scale_tier: Resource tier for MLEngine training job. (templated) |
| :type scale_tier: str |
| |
| :param master_type: Cloud ML Engine machine name. |
| Must be set when scale_tier is CUSTOM. (templated) |
| :type master_type: str |
| |
| :param runtime_version: The Google Cloud ML runtime version to use for |
| training. (templated) |
| :type runtime_version: str |
| |
| :param python_version: The version of Python used in training. (templated) |
| :type python_version: str |
| |
| :param job_dir: A Google Cloud Storage path in which to store training |
| outputs and other data needed for training. (templated) |
| :type job_dir: str |
| |
| :param gcp_conn_id: The connection ID to use when fetching connection info. |
| :type gcp_conn_id: str |
| |
| :param delegate_to: The account to impersonate, if any. |
| For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :type delegate_to: str |
| |
| :param mode: Can be one of 'DRY_RUN'/'CLOUD'. In 'DRY_RUN' mode, no real |
| training job will be launched, but the MLEngine training job request |
| will be printed out. In 'CLOUD' mode, a real MLEngine training job |
| creation request will be issued. |
| :type mode: str |
| |
| .. attribute:: template_fields |
| :annotation: = ['_project_id', '_job_id', '_package_uris', '_training_python_module', '_training_args', '_region', '_scale_tier', '_master_type', '_runtime_version', '_python_version', '_job_dir'] |
| |
| |
| |
| |
| .. method:: execute(self, context) |
| |
| |
| |
| |