Cluster Cluster Config Args
Constructors
Properties
The autoscaling policy config associated with the cluster. Note that once set, if autoscaling_config
is the only field set in cluster_config
, it can only be removed by setting policy_uri = ""
, rather than removing the whole block. Structure defined below.
The Compute Engine accelerator (GPU) configuration for these instances. Can be specified multiple times. Structure defined below.
The Customer managed encryption keys settings for the cluster. Structure defined below.
The config settings for port access on the cluster. Structure defined below.
Common config settings for resources of Google Compute Engine cluster instances, applicable to all instances in the cluster. Structure defined below.
Commands to execute on each node after config is completed. You can specify multiple versions of these. Structure defined below.
The settings for auto deletion cluster schedule. Structure defined below.
The Google Compute Engine config settings for the master instances in a cluster. Structure defined below.
The config setting for metastore service with the cluster. Structure defined below.
The Google Compute Engine config settings for the additional instances in a cluster. Structure defined below.
Security related configuration. Structure defined below.
The config settings for software inside the cluster. Structure defined below.
The Cloud Storage staging bucket used to stage files, such as Hadoop jars, between client machines and the cluster. Note: If you don't explicitly specify a staging_bucket
then GCP will auto create / assign one for you. However, you are not guaranteed an auto generated bucket which is solely dedicated to your cluster; it may be shared with other clusters in the same region/zone also choosing to use the auto generation option.
The Cloud Storage temp bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. Note: If you don't explicitly specify a temp_bucket
then GCP will auto create / assign one for you.
The Google Compute Engine config settings for the worker instances in a cluster. Structure defined below.