pulumi-google-native-kotlin/com.pulumi.googlenative.dataflow.v1b3.kotlin.outputs

Package-level declarations

Types

AutoscalingSettingsResponse

data class AutoscalingSettingsResponse(val algorithm: String, val maxNumWorkers: Int)

Settings for WorkerPool autoscaling.

BigQueryIODetailsResponse

data class BigQueryIODetailsResponse(val dataset: String, val project: String, val query: String, val table: String)

Metadata for a BigQuery connector used by the job.

BigTableIODetailsResponse

data class BigTableIODetailsResponse(val instanceId: String, val project: String, val tableId: String)

Metadata for a Cloud Bigtable connector used by the job.

ComponentSourceResponse

data class ComponentSourceResponse(val name: String, val originalTransformOrCollection: String, val userName: String)

Description of an interstitial value between transforms in an execution stage.

ComponentTransformResponse

data class ComponentTransformResponse(val name: String, val originalTransform: String, val userName: String)

Description of a transform executed as part of an execution stage.

DatastoreIODetailsResponse

data class DatastoreIODetailsResponse(val namespace: String, val project: String)

Metadata for a Datastore connector used by the job.

DebugOptionsResponse

data class DebugOptionsResponse(val enableHotKeyLogging: Boolean)

Describes any options that have an effect on the debugging of pipelines.

DiskResponse

data class DiskResponse(val diskType: String, val mountPoint: String, val sizeGb: Int)

Describes the data disk used by a workflow job.

DisplayDataResponse

data class DisplayDataResponse(val boolValue: Boolean, val durationValue: String, val floatValue: Double, val int64Value: String, val javaClassValue: String, val key: String, val label: String, val namespace: String, val shortStrValue: String, val strValue: String, val timestampValue: String, val url: String)

Data provided with a pipeline or transform to provide descriptive info.

EnvironmentResponse

data class EnvironmentResponse(val clusterManagerApiService: String, val dataset: String, val debugOptions: DebugOptionsResponse, val experiments: List<String>, val flexResourceSchedulingGoal: String, val internalExperiments: Map<String, String>, val sdkPipelineOptions: Map<String, String>, val serviceAccountEmail: String, val serviceKmsKeyName: String, val serviceOptions: List<String>, val shuffleMode: String, val tempStoragePrefix: String, val userAgent: Map<String, String>, val version: Map<String, String>, val workerPools: List<WorkerPoolResponse>, val workerRegion: String, val workerZone: String)

Describes the environment in which a Dataflow Job runs.

ExecutionStageStateResponse

data class ExecutionStageStateResponse(val currentStateTime: String, val executionStageName: String, val executionStageState: String)

A message describing the state of a particular execution stage.

ExecutionStageSummaryResponse

data class ExecutionStageSummaryResponse(val componentSource: List<ComponentSourceResponse>, val componentTransform: List<ComponentTransformResponse>, val inputSource: List<StageSourceResponse>, val kind: String, val name: String, val outputSource: List<StageSourceResponse>, val prerequisiteStage: List<String>)

Description of the composing transforms, names/ids, and input/outputs of a stage of execution. Some composing transforms and sources may have been generated by the Dataflow service during execution planning.

FileIODetailsResponse

data class FileIODetailsResponse(val filePattern: String)

Metadata for a File connector used by the job.

GetJobResult

data class GetJobResult(val clientRequestId: String, val createTime: String, val createdFromSnapshotId: String, val currentState: String, val currentStateTime: String, val environment: EnvironmentResponse, val executionInfo: JobExecutionInfoResponse, val jobMetadata: JobMetadataResponse, val labels: Map<String, String>, val location: String, val name: String, val pipelineDescription: PipelineDescriptionResponse, val project: String, val replaceJobId: String, val replacedByJobId: String, val requestedState: String, val runtimeUpdatableParams: RuntimeUpdatableParamsResponse, val satisfiesPzs: Boolean, val stageStates: List<ExecutionStageStateResponse>, val startTime: String, val steps: List<StepResponse>, val stepsLocation: String, val tempFiles: List<String>, val transformNameMapping: Map<String, String>, val type: String)

GetTemplateResult

data class GetTemplateResult(val metadata: TemplateMetadataResponse, val runtimeMetadata: RuntimeMetadataResponse, val status: StatusResponse, val templateType: String)

JobExecutionInfoResponse

data class JobExecutionInfoResponse(val stages: Map<String, String>)

Additional information about how a Cloud Dataflow job will be executed that isn't contained in the submitted job.

JobMetadataResponse

data class JobMetadataResponse(val bigTableDetails: List<BigTableIODetailsResponse>, val bigqueryDetails: List<BigQueryIODetailsResponse>, val datastoreDetails: List<DatastoreIODetailsResponse>, val fileDetails: List<FileIODetailsResponse>, val pubsubDetails: List<PubSubIODetailsResponse>, val sdkVersion: SdkVersionResponse, val spannerDetails: List<SpannerIODetailsResponse>, val userDisplayProperties: Map<String, String>)

Metadata available primarily for filtering jobs. Will be included in the ListJob response and Job SUMMARY view.

PackageResponse

data class PackageResponse(val location: String, val name: String)

The packages that must be installed in order for a worker to run the steps of the Cloud Dataflow job that will be assigned to its worker pool. This is the mechanism by which the Cloud Dataflow SDK causes code to be loaded onto the workers. For example, the Cloud Dataflow Java SDK might use this to install jars containing the user's code and all of the various dependencies (libraries, data files, etc.) required in order for that code to run.

ParameterMetadataResponse

data class ParameterMetadataResponse(val customMetadata: Map<String, String>, val groupName: String, val helpText: String, val isOptional: Boolean, val label: String, val name: String, val paramType: String, val parentName: String, val parentTriggerValues: List<String>, val regexes: List<String>)

Metadata for a specific parameter.

PipelineDescriptionResponse

data class PipelineDescriptionResponse(val displayData: List<DisplayDataResponse>, val executionPipelineStage: List<ExecutionStageSummaryResponse>, val originalPipelineTransform: List<TransformSummaryResponse>, val stepNamesHash: String)

A descriptive representation of submitted pipeline as well as the executed form. This data is provided by the Dataflow service for ease of visualizing the pipeline and interpreting Dataflow provided metrics.

PubSubIODetailsResponse

data class PubSubIODetailsResponse(val subscription: String, val topic: String)

Metadata for a Pub/Sub connector used by the job.

RuntimeMetadataResponse

data class RuntimeMetadataResponse(val parameters: List<ParameterMetadataResponse>, val sdkInfo: SDKInfoResponse)

RuntimeMetadata describing a runtime environment.

RuntimeUpdatableParamsResponse

data class RuntimeUpdatableParamsResponse(val maxNumWorkers: Int, val minNumWorkers: Int)

Additional job parameters that can only be updated during runtime using the projects.jobs.update method. These fields have no effect when specified during job creation.

SdkHarnessContainerImageResponse

data class SdkHarnessContainerImageResponse(val capabilities: List<String>, val containerImage: String, val environmentId: String, val useSingleCorePerContainer: Boolean)

Defines an SDK harness container for executing Dataflow pipelines.

SDKInfoResponse

data class SDKInfoResponse(val language: String, val version: String)

SDK Information.

SdkVersionResponse

data class SdkVersionResponse(val sdkSupportStatus: String, val version: String, val versionDisplayName: String)

The version of the SDK used to run the job.

SpannerIODetailsResponse

data class SpannerIODetailsResponse(val databaseId: String, val instanceId: String, val project: String)

Metadata for a Spanner connector used by the job.

StageSourceResponse

data class StageSourceResponse(val name: String, val originalTransformOrCollection: String, val sizeBytes: String, val userName: String)

Description of an input or output of an execution stage.

StatusResponse

data class StatusResponse(val code: Int, val details: List<Map<String, String>>, val message: String)

The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC. Each Status message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the API Design Guide.

StepResponse

data class StepResponse(val kind: String, val name: String, val properties: Map<String, String>)

Defines a particular step within a Cloud Dataflow job. A job consists of multiple steps, each of which performs some specific operation as part of the overall job. Data is typically passed from one step to another as part of the job. Note: The properties of this object are not stable and might change. Here's an example of a sequence of steps which together implement a Map-Reduce job: * Read a collection of data from some source, parsing the collection's elements. * Validate the elements. * Apply a user-defined function to map each element to some value and extract an element-specific key value. * Group elements with the same key into a single element with that key, transforming a multiply-keyed collection into a uniquely-keyed collection. * Write the elements out to some data sink. Note that the Cloud Dataflow service may be used to run many different types of jobs, not just Map-Reduce.

TaskRunnerSettingsResponse

data class TaskRunnerSettingsResponse(val alsologtostderr: Boolean, val baseTaskDir: String, val baseUrl: String, val commandlinesFileName: String, val continueOnException: Boolean, val dataflowApiVersion: String, val harnessCommand: String, val languageHint: String, val logDir: String, val logToSerialconsole: Boolean, val logUploadLocation: String, val oauthScopes: List<String>, val parallelWorkerSettings: WorkerSettingsResponse, val streamingWorkerMainClass: String, val taskGroup: String, val taskUser: String, val tempStoragePrefix: String, val vmId: String, val workflowFileName: String)

Taskrunner configuration settings.

TemplateMetadataResponse

data class TemplateMetadataResponse(val description: String, val name: String, val parameters: List<ParameterMetadataResponse>)

Metadata describing a template.

TransformSummaryResponse

data class TransformSummaryResponse(val displayData: List<DisplayDataResponse>, val inputCollectionName: List<String>, val kind: String, val name: String, val outputCollectionName: List<String>)

Description of the type, names/ids, and input/outputs for a transform.

WorkerPoolResponse

data class WorkerPoolResponse(val autoscalingSettings: AutoscalingSettingsResponse, val dataDisks: List<DiskResponse>, val defaultPackageSet: String, val diskSizeGb: Int, val diskSourceImage: String, val diskType: String, val ipConfiguration: String, val kind: String, val machineType: String, val metadata: Map<String, String>, val network: String, val numThreadsPerWorker: Int, val numWorkers: Int, val onHostMaintenance: String, val packages: List<PackageResponse>, val poolArgs: Map<String, String>, val sdkHarnessContainerImages: List<SdkHarnessContainerImageResponse>, val subnetwork: String, val taskrunnerSettings: TaskRunnerSettingsResponse, val teardownPolicy: String, val workerHarnessContainerImage: String, val zone: String)

Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.

WorkerSettingsResponse

data class WorkerSettingsResponse(val baseUrl: String, val reportingEnabled: Boolean, val servicePath: String, val shuffleServicePath: String, val tempStoragePrefix: String, val workerId: String)

Provides data to pass through to the worker harness.