| :py:mod:`airflow.providers.google.cloud.hooks.vertex_ai.hyperparameter_tuning_job` |
| ================================================================================== |
| |
| .. py:module:: airflow.providers.google.cloud.hooks.vertex_ai.hyperparameter_tuning_job |
| |
| .. autoapi-nested-parse:: |
| |
| This module contains a Google Cloud Vertex AI hook. |
| |
| .. spelling:: |
| |
| irreproducible |
| codepoints |
| Tensorboard |
| aiplatform |
| myVPC |
| |
| |
| |
| Module Contents |
| --------------- |
| |
| Classes |
| ~~~~~~~ |
| |
| .. autoapisummary:: |
| |
| airflow.providers.google.cloud.hooks.vertex_ai.hyperparameter_tuning_job.HyperparameterTuningJobHook |
| |
| |
| |
| |
| .. py:class:: HyperparameterTuningJobHook(gcp_conn_id = 'google_cloud_default', delegate_to = None, impersonation_chain = None) |
| |
| Bases: :py:obj:`airflow.providers.google.common.hooks.base_google.GoogleBaseHook` |
| |
| Hook for Google Cloud Vertex AI Hyperparameter Tuning Job APIs. |
| |
| .. py:method:: get_job_service_client(region = None) |
| |
| Returns JobServiceClient. |
| |
| |
| .. py:method:: get_hyperparameter_tuning_job_object(display_name, custom_job, metric_spec, parameter_spec, max_trial_count, parallel_trial_count, max_failed_trial_count = 0, search_algorithm = None, measurement_selection = 'best', project = None, location = None, labels = None, encryption_spec_key_name = None) |
| |
| Returns HyperparameterTuningJob object |
| |
| |
| .. py:method:: get_custom_job_object(display_name, worker_pool_specs, base_output_dir = None, project = None, location = None, labels = None, encryption_spec_key_name = None, staging_bucket = None) |
| |
| Returns CustomJob object |
| |
| |
| .. py:method:: extract_hyperparameter_tuning_job_id(obj) |
| :staticmethod: |
| |
| Returns unique id of the hyperparameter_tuning_job. |
| |
| |
| .. py:method:: wait_for_operation(operation, timeout = None) |
| |
| Waits for long-lasting operation to complete. |
| |
| |
| .. py:method:: cancel_hyperparameter_tuning_job() |
| |
| Cancel HyperparameterTuningJob |
| |
| |
| .. py:method:: create_hyperparameter_tuning_job(project_id, region, display_name, metric_spec, parameter_spec, max_trial_count, parallel_trial_count, worker_pool_specs, base_output_dir = None, custom_job_labels = None, custom_job_encryption_spec_key_name = None, staging_bucket = None, max_failed_trial_count = 0, search_algorithm = None, measurement_selection = 'best', hyperparameter_tuning_job_labels = None, hyperparameter_tuning_job_encryption_spec_key_name = None, service_account = None, network = None, timeout = None, restart_job_on_worker_restart = False, enable_web_access = False, tensorboard = None, sync = True) |
| |
| Create a HyperparameterTuningJob. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param display_name: Required. The user-defined name of the HyperparameterTuningJob. The name can be |
| up to 128 characters long and can be consist of any UTF-8 characters. |
| :param metric_spec: Required. Dictionary representing metrics to optimize. The dictionary key is the |
| metric_id, which is reported by your training job, and the dictionary value is the optimization |
| goal of the metric('minimize' or 'maximize'). |
| example: metric_spec = {'loss': 'minimize', 'accuracy': 'maximize'} |
| :param parameter_spec: Required. Dictionary representing parameters to optimize. The dictionary key |
| is the metric_id, which is passed into your training job as a command line key word argument, and |
| the dictionary value is the parameter specification of the metric. |
| :param max_trial_count: Required. The desired total number of Trials. |
| :param parallel_trial_count: Required. The desired number of Trials to run in parallel. |
| :param worker_pool_specs: Required. The spec of the worker pools including machine type and Docker |
| image. Can provided as a list of dictionaries or list of WorkerPoolSpec proto messages. |
| :param base_output_dir: Optional. GCS output directory of job. If not provided a timestamped |
| directory in the staging directory will be used. |
| :param custom_job_labels: Optional. The labels with user-defined metadata to organize CustomJobs. |
| Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain |
| lowercase letters, numeric characters, underscores and dashes. International characters are |
| allowed. See https://goo.gl/xmQnxf for more information and examples of labels. |
| :param custom_job_encryption_spec_key_name: Optional.Customer-managed encryption key name for a |
| CustomJob. If this is set, then all resources created by the CustomJob will be encrypted with the |
| provided encryption key. |
| :param staging_bucket: Optional. Bucket for produced custom job artifacts. Overrides staging_bucket |
| set in aiplatform.init. |
| :param max_failed_trial_count: Optional. The number of failed Trials that need to be seen before |
| failing the HyperparameterTuningJob. If set to 0, Vertex AI decides how many Trials must fail |
| before the whole job fails. |
| :param search_algorithm: The search algorithm specified for the Study. Accepts one of the following: |
| `None` - If you do not specify an algorithm, your job uses the default Vertex AI algorithm. The |
| default algorithm applies Bayesian optimization to arrive at the optimal solution with a more |
| effective search over the parameter space. |
| |
| 'grid' - A simple grid search within the feasible space. This option is particularly useful if |
| you want to specify a quantity of trials that is greater than the number of points in the |
| feasible space. In such cases, if you do not specify a grid search, the Vertex AI default |
| algorithm may generate duplicate suggestions. To use grid search, all parameter specs must be of |
| type `IntegerParameterSpec`, `CategoricalParameterSpace`, or `DiscreteParameterSpec`. |
| |
| 'random' - A simple random search within the feasible space. |
| :param measurement_selection: This indicates which measurement to use if/when the service |
| automatically selects the final measurement from previously reported intermediate measurements. |
| Accepts: 'best', 'last' |
| Choose this based on two considerations: |
| A) Do you expect your measurements to monotonically improve? If so, choose 'last'. On the other |
| hand, if you're in a situation where your system can "over-train" and you expect the performance |
| to get better for a while but then start declining, choose 'best'. |
| B) Are your measurements significantly noisy and/or irreproducible? If so, 'best' will tend to be |
| over-optimistic, and it may be better to choose 'last'. |
| If both or neither of (A) and (B) apply, it doesn't matter which selection type is chosen. |
| :param hyperparameter_tuning_job_labels: Optional. The labels with user-defined metadata to organize |
| HyperparameterTuningJobs. Label keys and values can be no longer than 64 characters (Unicode |
| codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. |
| International characters are allowed. See https://goo.gl/xmQnxf for more information and examples |
| of labels. |
| :param hyperparameter_tuning_job_encryption_spec_key_name: Optional. Customer-managed encryption key |
| options for a HyperparameterTuningJob. If this is set, then all resources created by the |
| HyperparameterTuningJob will be encrypted with the provided encryption key. |
| :param service_account: Optional. Specifies the service account for workload run-as account. Users |
| submitting jobs must have act-as permission on this run-as account. |
| :param network: Optional. The full name of the Compute Engine network to which the job should be |
| peered. For example, projects/12345/global/networks/myVPC. Private services access must already |
| be configured for the network. If left unspecified, the job is not peered with any network. |
| :param timeout: The maximum job running time in seconds. The default is 7 days. |
| :param restart_job_on_worker_restart: Restarts the entire CustomJob if a worker gets restarted. This |
| feature can be used by distributed training jobs that are not resilient to workers leaving and |
| joining a job. |
| :param enable_web_access: Whether you want Vertex AI to enable interactive shell access to training |
| containers. https://cloud.google.com/vertex-ai/docs/training/monitor-debug-interactive-shell |
| :param tensorboard: Optional. The name of a Vertex AI |
| [Tensorboard][google.cloud.aiplatform.v1beta1.Tensorboard] resource to which this CustomJob will |
| upload Tensorboard logs. Format: |
| ``projects/{project}/locations/{location}/tensorboards/{tensorboard}`` The training script should |
| write Tensorboard to following Vertex AI environment variable: AIP_TENSORBOARD_LOG_DIR |
| `service_account` is required with provided `tensorboard`. For more information on configuring |
| your service account please visit: |
| https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training |
| :param sync: Whether to execute this method synchronously. If False, this method will unblock and it |
| will be executed in a concurrent Future. |
| |
| |
| .. py:method:: get_hyperparameter_tuning_job(project_id, region, hyperparameter_tuning_job, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Gets a HyperparameterTuningJob |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param hyperparameter_tuning_job: Required. The name of the HyperparameterTuningJob resource. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: list_hyperparameter_tuning_jobs(project_id, region, filter = None, page_size = None, page_token = None, read_mask = None, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Lists HyperparameterTuningJobs in a Location. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param filter: The standard list filter. |
| Supported fields: |
| - ``display_name`` supports = and !=. |
| - ``state`` supports = and !=. |
| - ``model_display_name`` supports = and != |
| Some examples of using the filter are: |
| - ``state="JOB_STATE_SUCCEEDED" AND display_name="my_job"`` |
| - ``state="JOB_STATE_RUNNING" OR display_name="my_job"`` |
| - ``NOT display_name="my_job"`` |
| - ``state="JOB_STATE_FAILED"`` |
| :param page_size: The standard list page size. |
| :param page_token: The standard list page token. |
| :param read_mask: Mask specifying which fields to read. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| .. py:method:: delete_hyperparameter_tuning_job(project_id, region, hyperparameter_tuning_job, retry = DEFAULT, timeout = None, metadata = ()) |
| |
| Deletes a HyperparameterTuningJob. |
| |
| :param project_id: Required. The ID of the Google Cloud project that the service belongs to. |
| :param region: Required. The ID of the Google Cloud region that the service belongs to. |
| :param hyperparameter_tuning_job: Required. The name of the HyperparameterTuningJob resource to be |
| deleted. |
| :param retry: Designation of what errors, if any, should be retried. |
| :param timeout: The timeout for this request. |
| :param metadata: Strings which should be sent along with the request as metadata. |
| |
| |
| |