| :py:mod:`airflow.providers.google.cloud.operators.vertex_ai.hyperparameter_tuning_job` |
| ====================================================================================== |
| |
| .. py:module:: airflow.providers.google.cloud.operators.vertex_ai.hyperparameter_tuning_job |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains Google Vertex AI operators. |
| |
| .. spelling:: |
| |
| irreproducible |
| codepoints |
| Tensorboard |
| aiplatform |
| myVPC |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.operators.vertex_ai.hyperparameter_tuning_job.CreateHyperparameterTuningJobOperator |
| airflow.providers.google.cloud.operators.vertex_ai.hyperparameter_tuning_job.GetHyperparameterTuningJobOperator |
| airflow.providers.google.cloud.operators.vertex_ai.hyperparameter_tuning_job.DeleteHyperparameterTuningJobOperator |
| airflow.providers.google.cloud.operators.vertex_ai.hyperparameter_tuning_job.ListHyperparameterTuningJobOperator |
| |
| |
| |
| |
| .. py:class:: CreateHyperparameterTuningJobOperator(*, project_id, region, display_name, metric_spec, parameter_spec, max_trial_count, parallel_trial_count, worker_pool_specs, base_output_dir = None, custom_job_labels = None, custom_job_encryption_spec_key_name = None, staging_bucket = None, max_failed_trial_count = 0, search_algorithm = None, measurement_selection = 'best', hyperparameter_tuning_job_labels = None, hyperparameter_tuning_job_encryption_spec_key_name = None, service_account = None, network = None, timeout = None, restart_job_on_worker_restart = False, enable_web_access = False, tensorboard = None, sync = True, gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Create Hyperparameter Tuning job |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param display_name: Required. The user-defined name of the HyperparameterTuningJob. The name can be |
| up to 128 characters long and can be consist of any UTF-8 characters. |
| :param metric_spec: Required. Dictionary representing metrics to optimize. The dictionary key is the |
| metric_id, which is reported by your training job, and the dictionary value is the optimization |
| goal of the metric('minimize' or 'maximize'). |
| example: metric_spec = {'loss': 'minimize', 'accuracy': 'maximize'} |
| :param parameter_spec: Required. Dictionary representing parameters to optimize. The dictionary key |
| is the metric_id, which is passed into your training job as a command line key word argument, and |
| the dictionary value is the parameter specification of the metric. |
| :param max_trial_count: Required. The desired total number of Trials. |
| :param parallel_trial_count: Required. The desired number of Trials to run in parallel. |
| :param worker_pool_specs: Required. The spec of the worker pools including machine type and Docker |
| image. Can provided as a list of dictionaries or list of WorkerPoolSpec proto messages. |
| :param base_output_dir: Optional. GCS output directory of job. If not provided a timestamped |
| directory in the staging directory will be used. |
| :param custom_job_labels: Optional. The labels with user-defined metadata to organize CustomJobs. |
| Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain |
| lowercase letters, numeric characters, underscores and dashes. International characters are |
| allowed. See https://goo.gl/xmQnxf for more information and examples of labels. |
| :param custom_job_encryption_spec_key_name: Optional.Customer-managed encryption key name for a |
| CustomJob. If this is set, then all resources created by the CustomJob will be encrypted with the |
| provided encryption key. |
| :param staging_bucket: Optional. Bucket for produced custom job artifacts. Overrides staging_bucket |
| set in aiplatform.init. |
| :param max_failed_trial_count: Optional. The number of failed Trials that need to be seen before |
| failing the HyperparameterTuningJob. If set to 0, Vertex AI decides how many Trials must fail |
| before the whole job fails. |
| :param search_algorithm: The search algorithm specified for the Study. Accepts one of the following: |
| `None` - If you do not specify an algorithm, your job uses the default Vertex AI algorithm. The |
| default algorithm applies Bayesian optimization to arrive at the optimal solution with a more |
| effective search over the parameter space. |
| |
| 'grid' - A simple grid search within the feasible space. This option is particularly useful if |
| you want to specify a quantity of trials that is greater than the number of points in the |
| feasible space. In such cases, if you do not specify a grid search, the Vertex AI default |
| algorithm may generate duplicate suggestions. To use grid search, all parameter specs must be of |
| type `IntegerParameterSpec`, `CategoricalParameterSpace`, or `DiscreteParameterSpec`. |
| |
| 'random' - A simple random search within the feasible space. |
| :param measurement_selection: This indicates which measurement to use if/when the service |
| automatically selects the final measurement from previously reported intermediate measurements. |
| Accepts: 'best', 'last' |
| Choose this based on two considerations: |
| A) Do you expect your measurements to monotonically improve? If so, choose 'last'. On the other |
| hand, if you're in a situation where your system can "over-train" and you expect the performance |
| to get better for a while but then start declining, choose 'best'. |
| B) Are your measurements significantly noisy and/or irreproducible? If so, 'best' will tend to be |
| over-optimistic, and it may be better to choose 'last'. |
| If both or neither of (A) and (B) apply, it doesn't matter which selection type is chosen. |
| :param hyperparameter_tuning_job_labels: Optional. The labels with user-defined metadata to organize |
| HyperparameterTuningJobs. Label keys and values can be no longer than 64 characters (Unicode |
| codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. |
| International characters are allowed. See https://goo.gl/xmQnxf for more information and examples |
| of labels. |
| :param hyperparameter_tuning_job_encryption_spec_key_name: Optional. Customer-managed encryption key |
| options for a HyperparameterTuningJob. If this is set, then all resources created by the |
| HyperparameterTuningJob will be encrypted with the provided encryption key. |
| :param service_account: Optional. Specifies the service account for workload run-as account. Users |
| submitting jobs must have act-as permission on this run-as account. |
| :param network: Optional. The full name of the Compute Engine network to which the job should be |
| peered. For example, projects/12345/global/networks/myVPC. Private services access must already |
| be configured for the network. If left unspecified, the job is not peered with any network. |
| :param timeout: The maximum job running time in seconds. The default is 7 days. |
| :param restart_job_on_worker_restart: Restarts the entire CustomJob if a worker gets restarted. This |
| feature can be used by distributed training jobs that are not resilient to workers leaving and |
| joining a job. |
| :param enable_web_access: Whether you want Vertex AI to enable interactive shell access to training |
| containers. https://cloud.google.com/vertex-ai/docs/training/monitor-debug-interactive-shell |
| :param tensorboard: Optional. The name of a Vertex AI |
| [Tensorboard][google.cloud.aiplatform.v1beta1.Tensorboard] resource to which this CustomJob will |
| upload Tensorboard logs. Format: |
| ``projects/{project}/locations/{location}/tensorboards/{tensorboard}`` The training script should |
| write Tensorboard to following Vertex AI environment variable: AIP_TENSORBOARD_LOG_DIR |
| `service_account` is required with provided `tensorboard`. For more information on configuring |
| your service account please visit: |
| https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training |
| :param sync: Whether to execute this method synchronously. If False, this method will unblock and it |
| will be executed in a concurrent Future. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| .. py:method:: on_kill(self) |
| |
| Callback called when the operator is killed. |
| Cancel any running job. |
| |
| |
| |
| .. py:class:: GetHyperparameterTuningJobOperator(*, region, project_id, hyperparameter_tuning_job_id, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Gets a HyperparameterTuningJob |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param hyperparameter_tuning_job_id: Required. The name of the HyperparameterTuningJob resource. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| :param gcp_conn_id: The connection ID to use connecting to Google Cloud. |
| :param delegate_to: The account to impersonate using domain-wide delegation of authority, |
| if any. For this to work, the service account making the request must have |
| domain-wide delegation enabled. |
| :param impersonation_chain: Optional service account to impersonate using short-term |
| credentials, or chained list of accounts required to get the access_token |
| of the last account in the list, which will be impersonated in the request. |
| If set as a string, the account must grant the originating account |
| the Service Account Token Creator IAM role. |
| If set as a sequence, the identities from the list must grant |
| Service Account Token Creator IAM role to the directly preceding identity, with first |
| account from the list granting this role to the originating account (templated). |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'hyperparameter_tuning_job_id', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: DeleteHyperparameterTuningJobOperator(*, hyperparameter_tuning_job_id, region, project_id, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Deletes a HyperparameterTuningJob. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param hyperparameter_tuning_job_id: Required. The name of the HyperparameterTuningJob resource to be |
| deleted. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'project_id', 'hyperparameter_tuning_job_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |
| .. py:class:: ListHyperparameterTuningJobOperator(*, region, project_id, page_size = None, page_token = None, filter = None, read_mask = None, retry = DEFAULT, timeout = None, metadata = (), gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None, **kwargs) |
| |
| Bases: :py:obj:`airflow.models.BaseOperator` |
| |
| Lists HyperparameterTuningJobs in a Location. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param filter: The standard list filter. |
| Supported fields: |
| - ``display_name`` supports = and !=. |
| - ``state`` supports = and !=. |
| - ``model_display_name`` supports = and != |
| Some examples of using the filter are: |
| - ``state="JOB_STATE_SUCCEEDED" AND display_name="my_job"`` |
| - ``state="JOB_STATE_RUNNING" OR display_name="my_job"`` |
| - ``NOT display_name="my_job"`` |
| - ``state="JOB_STATE_FAILED"`` |
| :param page_size: The standard list page size. |
| :param page_token: The standard list page token. |
| :param read_mask: Mask specifying which fields to read. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| .. py:attribute:: template_fields |
| :annotation: = ['region', 'project_id', 'impersonation_chain'] |
| |
| |
| |
| .. py:attribute:: operator_extra_links |
| |
| |
| |
| |
| .. py:method:: execute(self, context) |
| |
| This is the main method to derive when creating an operator. |
| Context is the same dictionary used as when rendering jinja templates. |
| |
| Refer to get_template_context for more context. |
| |
| |
| |