pulumi-google-native-kotlin/com.pulumi.googlenative.dataproc.v1.kotlin.outputs

Package-level declarations

Types

data class AcceleratorConfigResponse(val acceleratorCount: Int, val acceleratorTypeUri: String)

Specifies the type and number of accelerator cards attached to the instances of an instance. See GPUs on Compute Engine (https://cloud.google.com/compute/docs/gpus/).

AutoscalingConfigResponse

data class AutoscalingConfigResponse(val policyUri: String)

Autoscaling Policy config associated with the cluster.

AuxiliaryNodeGroupResponse

data class AuxiliaryNodeGroupResponse(val nodeGroup: NodeGroupResponse, val nodeGroupId: String)

Node group identification and configuration information.

AuxiliaryServicesConfigResponse

data class AuxiliaryServicesConfigResponse(val metastoreConfig: MetastoreConfigResponse, val sparkHistoryServerConfig: SparkHistoryServerConfigResponse)

Auxiliary services configuration for a Cluster.

BasicAutoscalingAlgorithmResponse

data class BasicAutoscalingAlgorithmResponse(val cooldownPeriod: String, val sparkStandaloneConfig: SparkStandaloneAutoscalingConfigResponse, val yarnConfig: BasicYarnAutoscalingConfigResponse)

Basic algorithm for autoscaling.

BasicYarnAutoscalingConfigResponse

data class BasicYarnAutoscalingConfigResponse(val gracefulDecommissionTimeout: String, val scaleDownFactor: Double, val scaleDownMinWorkerFraction: Double, val scaleUpFactor: Double, val scaleUpMinWorkerFraction: Double)

Basic autoscaling configurations for YARN.

BindingResponse

data class BindingResponse(val condition: ExprResponse, val members: List<String>, val role: String)

Associates members, or principals, with a role.

ClusterConfigResponse

data class ClusterConfigResponse(val autoscalingConfig: AutoscalingConfigResponse, val auxiliaryNodeGroups: List<AuxiliaryNodeGroupResponse>, val configBucket: String, val dataprocMetricConfig: DataprocMetricConfigResponse, val encryptionConfig: EncryptionConfigResponse, val endpointConfig: EndpointConfigResponse, val gceClusterConfig: GceClusterConfigResponse, val gkeClusterConfig: GkeClusterConfigResponse, val initializationActions: List<NodeInitializationActionResponse>, val lifecycleConfig: LifecycleConfigResponse, val masterConfig: InstanceGroupConfigResponse, val metastoreConfig: MetastoreConfigResponse, val secondaryWorkerConfig: InstanceGroupConfigResponse, val securityConfig: SecurityConfigResponse, val softwareConfig: SoftwareConfigResponse, val tempBucket: String, val workerConfig: InstanceGroupConfigResponse)

The cluster config.

ClusterMetricsResponse

data class ClusterMetricsResponse(val hdfsMetrics: Map<String, String>, val yarnMetrics: Map<String, String>)

Contains cluster daemon metrics, such as HDFS and YARN stats.Beta Feature: This report is available for testing purposes only. It may be changed before final release.

ClusterSelectorResponse

data class ClusterSelectorResponse(val clusterLabels: Map<String, String>, val zone: String)

A selector that chooses target cluster for jobs based on metadata.

ClusterStatusResponse

data class ClusterStatusResponse(val detail: String, val state: String, val stateStartTime: String, val substate: String)

The status of a cluster and its instances.

ConfidentialInstanceConfigResponse

data class ConfidentialInstanceConfigResponse(val enableConfidentialCompute: Boolean)

Confidential Instance Config for clusters using Confidential VMs (https://cloud.google.com/compute/confidential-vm/docs)

DataprocMetricConfigResponse

data class DataprocMetricConfigResponse(val metrics: List<MetricResponse>)

Dataproc metric config.

DiskConfigResponse

data class DiskConfigResponse(val bootDiskSizeGb: Int, val bootDiskType: String, val localSsdInterface: String, val numLocalSsds: Int)

Specifies the config of disk options for a group of VM instances.

DriverSchedulingConfigResponse

data class DriverSchedulingConfigResponse(val memoryMb: Int, val vcores: Int)

Driver scheduling configuration.

EncryptionConfigResponse

data class EncryptionConfigResponse(val gcePdKmsKeyName: String, val kmsKey: String)

Encryption settings for the cluster.

EndpointConfigResponse

data class EndpointConfigResponse(val enableHttpPortAccess: Boolean, val httpPorts: Map<String, String>)

Endpoint config for this cluster

EnvironmentConfigResponse

data class EnvironmentConfigResponse(val executionConfig: ExecutionConfigResponse, val peripheralsConfig: PeripheralsConfigResponse)

Environment configuration for a workload.

ExecutionConfigResponse

data class ExecutionConfigResponse(val idleTtl: String, val kmsKey: String, val networkTags: List<String>, val networkUri: String, val serviceAccount: String, val stagingBucket: String, val subnetworkUri: String, val ttl: String)

Execution configuration for a workload.

ExprResponse

data class ExprResponse(val description: String, val expression: String, val location: String, val title: String)

Represents a textual expression in the Common Expression Language (CEL) syntax. CEL is a C-like expression language. The syntax and semantics of CEL are documented at https://github.com/google/cel-spec.Example (Comparison): title: "Summary size limit" description: "Determines if a summary is less than 100 chars" expression: "document.summary.size() < 100" Example (Equality): title: "Requestor is owner" description: "Determines if requestor is the document owner" expression: "document.owner == request.auth.claims.email" Example (Logic): title: "Public documents" description: "Determine whether the document should be publicly visible" expression: "document.type != 'private' && document.type != 'internal'" Example (Data Manipulation): title: "Notification string" description: "Create a notification string with a timestamp." expression: "'New message received at ' + string(document.create_time)" The exact variables and functions that may be referenced within an expression are determined by the service that evaluates it. See the service documentation for additional information.

GceClusterConfigResponse

data class GceClusterConfigResponse(val confidentialInstanceConfig: ConfidentialInstanceConfigResponse, val internalIpOnly: Boolean, val metadata: Map<String, String>, val networkUri: String, val nodeGroupAffinity: NodeGroupAffinityResponse, val privateIpv6GoogleAccess: String, val reservationAffinity: ReservationAffinityResponse, val serviceAccount: String, val serviceAccountScopes: List<String>, val shieldedInstanceConfig: ShieldedInstanceConfigResponse, val subnetworkUri: String, val tags: List<String>, val zoneUri: String)

Common config settings for resources of Compute Engine cluster instances, applicable to all instances in the cluster.

GetAutoscalingPolicyIamPolicyResult

data class GetAutoscalingPolicyIamPolicyResult(val bindings: List<BindingResponse>, val etag: String, val version: Int)

GetAutoscalingPolicyResult

data class GetAutoscalingPolicyResult(val basicAlgorithm: BasicAutoscalingAlgorithmResponse, val labels: Map<String, String>, val name: String, val secondaryWorkerConfig: InstanceGroupAutoscalingPolicyConfigResponse, val workerConfig: InstanceGroupAutoscalingPolicyConfigResponse)

GetBatchResult

data class GetBatchResult(val createTime: String, val creator: String, val environmentConfig: EnvironmentConfigResponse, val labels: Map<String, String>, val name: String, val operation: String, val pysparkBatch: PySparkBatchResponse, val runtimeConfig: RuntimeConfigResponse, val runtimeInfo: RuntimeInfoResponse, val sparkBatch: SparkBatchResponse, val sparkRBatch: SparkRBatchResponse, val sparkSqlBatch: SparkSqlBatchResponse, val state: String, val stateHistory: List<StateHistoryResponse>, val stateMessage: String, val stateTime: String, val uuid: String)

GetClusterResult

data class GetClusterResult(val clusterName: String, val clusterUuid: String, val config: ClusterConfigResponse, val labels: Map<String, String>, val metrics: ClusterMetricsResponse, val project: String, val status: ClusterStatusResponse, val statusHistory: List<ClusterStatusResponse>, val virtualClusterConfig: VirtualClusterConfigResponse)

GetJobResult

data class GetJobResult(val done: Boolean, val driverControlFilesUri: String, val driverOutputResourceUri: String, val driverSchedulingConfig: DriverSchedulingConfigResponse, val hadoopJob: HadoopJobResponse, val hiveJob: HiveJobResponse, val jobUuid: String, val labels: Map<String, String>, val pigJob: PigJobResponse, val placement: JobPlacementResponse, val prestoJob: PrestoJobResponse, val pysparkJob: PySparkJobResponse, val reference: JobReferenceResponse, val scheduling: JobSchedulingResponse, val sparkJob: SparkJobResponse, val sparkRJob: SparkRJobResponse, val sparkSqlJob: SparkSqlJobResponse, val status: JobStatusResponse, val statusHistory: List<JobStatusResponse>, val trinoJob: TrinoJobResponse, val yarnApplications: List<YarnApplicationResponse>)

GetNodeGroupResult

data class GetNodeGroupResult(val labels: Map<String, String>, val name: String, val nodeGroupConfig: InstanceGroupConfigResponse, val roles: List<String>)

GetRegionAutoscalingPolicyIamPolicyResult

data class GetRegionAutoscalingPolicyIamPolicyResult(val bindings: List<BindingResponse>, val etag: String, val version: Int)

GetRegionClusterIamPolicyResult

data class GetRegionClusterIamPolicyResult(val bindings: List<BindingResponse>, val etag: String, val version: Int)

GetRegionJobIamPolicyResult

data class GetRegionJobIamPolicyResult(val bindings: List<BindingResponse>, val etag: String, val version: Int)

GetRegionOperationIamPolicyResult

data class GetRegionOperationIamPolicyResult(val bindings: List<BindingResponse>, val etag: String, val version: Int)

GetRegionWorkflowTemplateIamPolicyResult

data class GetRegionWorkflowTemplateIamPolicyResult(val bindings: List<BindingResponse>, val etag: String, val version: Int)

GetWorkflowTemplateIamPolicyResult

data class GetWorkflowTemplateIamPolicyResult(val bindings: List<BindingResponse>, val etag: String, val version: Int)

GetWorkflowTemplateResult

data class GetWorkflowTemplateResult(val createTime: String, val dagTimeout: String, val jobs: List<OrderedJobResponse>, val labels: Map<String, String>, val name: String, val parameters: List<TemplateParameterResponse>, val placement: WorkflowTemplatePlacementResponse, val updateTime: String, val version: Int)

GkeClusterConfigResponse

data class GkeClusterConfigResponse(val gkeClusterTarget: String, val namespacedGkeDeploymentTarget: NamespacedGkeDeploymentTargetResponse, val nodePoolTarget: List<GkeNodePoolTargetResponse>)

The cluster's GKE config.

GkeNodeConfigResponse

data class GkeNodeConfigResponse(val accelerators: List<GkeNodePoolAcceleratorConfigResponse>, val bootDiskKmsKey: String, val localSsdCount: Int, val machineType: String, val minCpuPlatform: String, val preemptible: Boolean, val spot: Boolean)

Parameters that describe cluster nodes.

GkeNodePoolAcceleratorConfigResponse

data class GkeNodePoolAcceleratorConfigResponse(val acceleratorCount: String, val acceleratorType: String, val gpuPartitionSize: String)

A GkeNodeConfigAcceleratorConfig represents a Hardware Accelerator request for a node pool.

GkeNodePoolAutoscalingConfigResponse

data class GkeNodePoolAutoscalingConfigResponse(val maxNodeCount: Int, val minNodeCount: Int)

GkeNodePoolAutoscaling contains information the cluster autoscaler needs to adjust the size of the node pool to the current cluster usage.

GkeNodePoolConfigResponse

data class GkeNodePoolConfigResponse(val autoscaling: GkeNodePoolAutoscalingConfigResponse, val config: GkeNodeConfigResponse, val locations: List<String>)

The configuration of a GKE node pool used by a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/concepts/jobs/dataproc-gke#create-a-dataproc-on-gke-cluster).

GkeNodePoolTargetResponse

data class GkeNodePoolTargetResponse(val nodePool: String, val nodePoolConfig: GkeNodePoolConfigResponse, val roles: List<String>)

GKE node pools that Dataproc workloads run on.

HadoopJobResponse

data class HadoopJobResponse(val archiveUris: List<String>, val args: List<String>, val fileUris: List<String>, val jarFileUris: List<String>, val loggingConfig: LoggingConfigResponse, val mainClass: String, val mainJarFileUri: String, val properties: Map<String, String>)

A Dataproc job for running Apache Hadoop MapReduce (https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html) jobs on Apache Hadoop YARN (https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html).

HiveJobResponse

data class HiveJobResponse(val continueOnFailure: Boolean, val jarFileUris: List<String>, val properties: Map<String, String>, val queryFileUri: String, val queryList: QueryListResponse, val scriptVariables: Map<String, String>)

A Dataproc job for running Apache Hive (https://hive.apache.org/) queries on YARN.

IdentityConfigResponse

data class IdentityConfigResponse(val userServiceAccountMapping: Map<String, String>)

Identity related configuration, including service account based secure multi-tenancy user mappings.

InstanceGroupAutoscalingPolicyConfigResponse

data class InstanceGroupAutoscalingPolicyConfigResponse(val maxInstances: Int, val minInstances: Int, val weight: Int)

Configuration for the size bounds of an instance group, including its proportional size to other groups.

InstanceGroupConfigResponse

data class InstanceGroupConfigResponse(val accelerators: List<AcceleratorConfigResponse>, val diskConfig: DiskConfigResponse, val imageUri: String, val instanceNames: List<String>, val instanceReferences: List<InstanceReferenceResponse>, val isPreemptible: Boolean, val machineTypeUri: String, val managedGroupConfig: ManagedGroupConfigResponse, val minCpuPlatform: String, val numInstances: Int, val preemptibility: String)

The config settings for Compute Engine resources in an instance group, such as a master or worker group.

InstanceReferenceResponse

data class InstanceReferenceResponse(val instanceId: String, val instanceName: String, val publicEciesKey: String, val publicKey: String)

A reference to a Compute Engine instance.

JobPlacementResponse

data class JobPlacementResponse(val clusterLabels: Map<String, String>, val clusterName: String, val clusterUuid: String)

Dataproc job config.

JobReferenceResponse

data class JobReferenceResponse(val jobId: String, val project: String)

Encapsulates the full scoping used to reference a job.

JobSchedulingResponse

data class JobSchedulingResponse(val maxFailuresPerHour: Int, val maxFailuresTotal: Int)

Job scheduling options.

JobStatusResponse

data class JobStatusResponse(val details: String, val state: String, val stateStartTime: String, val substate: String)

Dataproc job status.

KerberosConfigResponse

data class KerberosConfigResponse(val crossRealmTrustAdminServer: String, val crossRealmTrustKdc: String, val crossRealmTrustRealm: String, val crossRealmTrustSharedPasswordUri: String, val enableKerberos: Boolean, val kdcDbKeyUri: String, val keyPasswordUri: String, val keystorePasswordUri: String, val keystoreUri: String, val kmsKeyUri: String, val realm: String, val rootPrincipalPasswordUri: String, val tgtLifetimeHours: Int, val truststorePasswordUri: String, val truststoreUri: String)

Specifies Kerberos related configuration.

KubernetesClusterConfigResponse

data class KubernetesClusterConfigResponse(val gkeClusterConfig: GkeClusterConfigResponse, val kubernetesNamespace: String, val kubernetesSoftwareConfig: KubernetesSoftwareConfigResponse)

The configuration for running the Dataproc cluster on Kubernetes.

KubernetesSoftwareConfigResponse

data class KubernetesSoftwareConfigResponse(val componentVersion: Map<String, String>, val properties: Map<String, String>)

The software configuration for this Dataproc cluster running on Kubernetes.

LifecycleConfigResponse

data class LifecycleConfigResponse(val autoDeleteTime: String, val autoDeleteTtl: String, val idleDeleteTtl: String, val idleStartTime: String)

Specifies the cluster auto-delete schedule configuration.

LoggingConfigResponse

data class LoggingConfigResponse(val driverLogLevels: Map<String, String>)

The runtime logging config of the job.

ManagedClusterResponse

data class ManagedClusterResponse(val clusterName: String, val config: ClusterConfigResponse, val labels: Map<String, String>)

Cluster that is managed by the workflow.

ManagedGroupConfigResponse

data class ManagedGroupConfigResponse(val instanceGroupManagerName: String, val instanceTemplateName: String)

Specifies the resources used to actively manage an instance group.

MetastoreConfigResponse

data class MetastoreConfigResponse(val dataprocMetastoreService: String)

Specifies a Metastore configuration.

MetricResponse

data class MetricResponse(val metricOverrides: List<String>, val metricSource: String)

A Dataproc custom metric.

NamespacedGkeDeploymentTargetResponse

data class NamespacedGkeDeploymentTargetResponse(val clusterNamespace: String, val targetGkeCluster: String)

Deprecated. Used only for the deprecated beta. A full, namespace-isolated deployment target for an existing GKE cluster.

NodeGroupAffinityResponse

data class NodeGroupAffinityResponse(val nodeGroupUri: String)

Node Group Affinity for clusters using sole-tenant node groups. The Dataproc NodeGroupAffinity resource is not related to the Dataproc NodeGroup resource.

NodeGroupResponse

data class NodeGroupResponse(val labels: Map<String, String>, val name: String, val nodeGroupConfig: InstanceGroupConfigResponse, val roles: List<String>)

Dataproc Node Group. The Dataproc NodeGroup resource is not related to the Dataproc NodeGroupAffinity resource.

NodeInitializationActionResponse

data class NodeInitializationActionResponse(val executableFile: String, val executionTimeout: String)

Specifies an executable to run on a fully configured node and a timeout period for executable completion.

OrderedJobResponse

data class OrderedJobResponse(val hadoopJob: HadoopJobResponse, val hiveJob: HiveJobResponse, val labels: Map<String, String>, val pigJob: PigJobResponse, val prerequisiteStepIds: List<String>, val prestoJob: PrestoJobResponse, val pysparkJob: PySparkJobResponse, val scheduling: JobSchedulingResponse, val sparkJob: SparkJobResponse, val sparkRJob: SparkRJobResponse, val sparkSqlJob: SparkSqlJobResponse, val stepId: String, val trinoJob: TrinoJobResponse)

A job executed by the workflow.

ParameterValidationResponse

data class ParameterValidationResponse(val regex: RegexValidationResponse, val values: ValueValidationResponse)

Configuration for parameter validation.

PeripheralsConfigResponse

data class PeripheralsConfigResponse(val metastoreService: String, val sparkHistoryServerConfig: SparkHistoryServerConfigResponse)

Auxiliary services configuration for a workload.

PigJobResponse

data class PigJobResponse(val continueOnFailure: Boolean, val jarFileUris: List<String>, val loggingConfig: LoggingConfigResponse, val properties: Map<String, String>, val queryFileUri: String, val queryList: QueryListResponse, val scriptVariables: Map<String, String>)

A Dataproc job for running Apache Pig (https://pig.apache.org/) queries on YARN.

PrestoJobResponse

data class PrestoJobResponse(val clientTags: List<String>, val continueOnFailure: Boolean, val loggingConfig: LoggingConfigResponse, val outputFormat: String, val properties: Map<String, String>, val queryFileUri: String, val queryList: QueryListResponse)

A Dataproc job for running Presto (https://prestosql.io/) queries. IMPORTANT: The Dataproc Presto Optional Component (https://cloud.google.com/dataproc/docs/concepts/components/presto) must be enabled when the cluster is created to submit a Presto job to the cluster.

PySparkBatchResponse

data class PySparkBatchResponse(val archiveUris: List<String>, val args: List<String>, val fileUris: List<String>, val jarFileUris: List<String>, val mainPythonFileUri: String, val pythonFileUris: List<String>)

A configuration for running an Apache PySpark (https://spark.apache.org/docs/latest/api/python/getting_started/quickstart.html) batch workload.

PySparkJobResponse

data class PySparkJobResponse(val archiveUris: List<String>, val args: List<String>, val fileUris: List<String>, val jarFileUris: List<String>, val loggingConfig: LoggingConfigResponse, val mainPythonFileUri: String, val properties: Map<String, String>, val pythonFileUris: List<String>)

A Dataproc job for running Apache PySpark (https://spark.apache.org/docs/0.9.0/python-programming-guide.html) applications on YARN.

QueryListResponse

data class QueryListResponse(val queries: List<String>)

A list of queries to run on a cluster.

RegexValidationResponse

data class RegexValidationResponse(val regexes: List<String>)

Validation based on regular expressions.

ReservationAffinityResponse

data class ReservationAffinityResponse(val consumeReservationType: String, val key: String, val values: List<String>)

Reservation Affinity for consuming Zonal reservation.

RuntimeConfigResponse

data class RuntimeConfigResponse(val containerImage: String, val properties: Map<String, String>, val version: String)

Runtime configuration for a workload.

RuntimeInfoResponse

data class RuntimeInfoResponse(val approximateUsage: UsageMetricsResponse, val currentUsage: UsageSnapshotResponse, val diagnosticOutputUri: String, val endpoints: Map<String, String>, val outputUri: String)

Runtime information about workload execution.

SecurityConfigResponse

data class SecurityConfigResponse(val identityConfig: IdentityConfigResponse, val kerberosConfig: KerberosConfigResponse)

Security related configuration, including encryption, Kerberos, etc.

ShieldedInstanceConfigResponse

data class ShieldedInstanceConfigResponse(val enableIntegrityMonitoring: Boolean, val enableSecureBoot: Boolean, val enableVtpm: Boolean)

Shielded Instance Config for clusters using Compute Engine Shielded VMs (https://cloud.google.com/security/shielded-cloud/shielded-vm).

SoftwareConfigResponse

data class SoftwareConfigResponse(val imageVersion: String, val optionalComponents: List<String>, val properties: Map<String, String>)

Specifies the selection and config of software inside the cluster.

SparkBatchResponse

data class SparkBatchResponse(val archiveUris: List<String>, val args: List<String>, val fileUris: List<String>, val jarFileUris: List<String>, val mainClass: String, val mainJarFileUri: String)

A configuration for running an Apache Spark (https://spark.apache.org/) batch workload.

SparkHistoryServerConfigResponse

data class SparkHistoryServerConfigResponse(val dataprocCluster: String)

Spark History Server configuration for the workload.

SparkJobResponse

data class SparkJobResponse(val archiveUris: List<String>, val args: List<String>, val fileUris: List<String>, val jarFileUris: List<String>, val loggingConfig: LoggingConfigResponse, val mainClass: String, val mainJarFileUri: String, val properties: Map<String, String>)

A Dataproc job for running Apache Spark (https://spark.apache.org/) applications on YARN.

SparkRBatchResponse

data class SparkRBatchResponse(val archiveUris: List<String>, val args: List<String>, val fileUris: List<String>, val mainRFileUri: String)

A configuration for running an Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) batch workload.

SparkRJobResponse

data class SparkRJobResponse(val archiveUris: List<String>, val args: List<String>, val fileUris: List<String>, val loggingConfig: LoggingConfigResponse, val mainRFileUri: String, val properties: Map<String, String>)

A Dataproc job for running Apache SparkR (https://spark.apache.org/docs/latest/sparkr.html) applications on YARN.

SparkSqlBatchResponse

data class SparkSqlBatchResponse(val jarFileUris: List<String>, val queryFileUri: String, val queryVariables: Map<String, String>)

A configuration for running Apache Spark SQL (https://spark.apache.org/sql/) queries as a batch workload.

SparkSqlJobResponse

data class SparkSqlJobResponse(val jarFileUris: List<String>, val loggingConfig: LoggingConfigResponse, val properties: Map<String, String>, val queryFileUri: String, val queryList: QueryListResponse, val scriptVariables: Map<String, String>)

A Dataproc job for running Apache Spark SQL (https://spark.apache.org/sql/) queries.

SparkStandaloneAutoscalingConfigResponse

data class SparkStandaloneAutoscalingConfigResponse(val gracefulDecommissionTimeout: String, val scaleDownFactor: Double, val scaleDownMinWorkerFraction: Double, val scaleUpFactor: Double, val scaleUpMinWorkerFraction: Double)

Basic autoscaling configurations for Spark Standalone.

StateHistoryResponse

data class StateHistoryResponse(val state: String, val stateMessage: String, val stateStartTime: String)

Historical state information.

TemplateParameterResponse

data class TemplateParameterResponse(val description: String, val fields: List<String>, val name: String, val validation: ParameterValidationResponse)

A configurable parameter that replaces one or more fields in the template. Parameterizable fields: - Labels - File uris - Job properties - Job arguments - Script variables - Main class (in HadoopJob and SparkJob) - Zone (in ClusterSelector)

TrinoJobResponse

data class TrinoJobResponse(val clientTags: List<String>, val continueOnFailure: Boolean, val loggingConfig: LoggingConfigResponse, val outputFormat: String, val properties: Map<String, String>, val queryFileUri: String, val queryList: QueryListResponse)

A Dataproc job for running Trino (https://trino.io/) queries. IMPORTANT: The Dataproc Trino Optional Component (https://cloud.google.com/dataproc/docs/concepts/components/trino) must be enabled when the cluster is created to submit a Trino job to the cluster.

UsageMetricsResponse

data class UsageMetricsResponse(val milliDcuSeconds: String, val shuffleStorageGbSeconds: String)

Usage metrics represent approximate total resources consumed by a workload.

UsageSnapshotResponse

data class UsageSnapshotResponse(val milliDcu: String, val shuffleStorageGb: String, val snapshotTime: String)

The usage snaphot represents the resources consumed by a workload at a specified time.

ValueValidationResponse

data class ValueValidationResponse(val values: List<String>)

Validation based on a list of allowed values.

VirtualClusterConfigResponse

data class VirtualClusterConfigResponse(val auxiliaryServicesConfig: AuxiliaryServicesConfigResponse, val kubernetesClusterConfig: KubernetesClusterConfigResponse, val stagingBucket: String)

The Dataproc cluster config for a cluster that does not directly control the underlying compute resources, such as a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/guides/dpgke/dataproc-gke-overview).

WorkflowTemplatePlacementResponse

data class WorkflowTemplatePlacementResponse(val clusterSelector: ClusterSelectorResponse, val managedCluster: ManagedClusterResponse)

Specifies workflow execution target.Either managed_cluster or cluster_selector is required.

YarnApplicationResponse

data class YarnApplicationResponse(val name: String, val progress: Double, val state: String, val trackingUrl: String)

A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.Beta Feature: This report is available for testing purposes only. It may be changed before final release.