Flex
Example Usage
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
const bigDataJob = new gcp.dataflow.FlexTemplateJob("big_data_job", {
name: "dataflow-flextemplates-job",
containerSpecGcsPath: "gs://my-bucket/templates/template.json",
parameters: {
inputSubscription: "messages",
},
});import pulumi
import pulumi_gcp as gcp
big_data_job = gcp.dataflow.FlexTemplateJob("big_data_job",
name="dataflow-flextemplates-job",
container_spec_gcs_path="gs://my-bucket/templates/template.json",
parameters={
"inputSubscription": "messages",
})using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
return await Deployment.RunAsync(() =>
{
var bigDataJob = new Gcp.Dataflow.FlexTemplateJob("big_data_job", new()
{
Name = "dataflow-flextemplates-job",
ContainerSpecGcsPath = "gs://my-bucket/templates/template.json",
Parameters =
{
{ "inputSubscription", "messages" },
},
});
});package main
import (
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/dataflow"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
_, err := dataflow.NewFlexTemplateJob(ctx, "big_data_job", &dataflow.FlexTemplateJobArgs{
Name: pulumi.String("dataflow-flextemplates-job"),
ContainerSpecGcsPath: pulumi.String("gs://my-bucket/templates/template.json"),
Parameters: pulumi.StringMap{
"inputSubscription": pulumi.String("messages"),
},
})
if err != nil {
return err
}
return nil
})
}package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.dataflow.FlexTemplateJob;
import com.pulumi.gcp.dataflow.FlexTemplateJobArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var bigDataJob = new FlexTemplateJob("bigDataJob", FlexTemplateJobArgs.builder()
.name("dataflow-flextemplates-job")
.containerSpecGcsPath("gs://my-bucket/templates/template.json")
.parameters(Map.of("inputSubscription", "messages"))
.build());
}
}resources:
bigDataJob:
type: gcp:dataflow:FlexTemplateJob
name: big_data_job
properties:
name: dataflow-flextemplates-job
containerSpecGcsPath: gs://my-bucket/templates/template.json
parameters:
inputSubscription: messagesNote on "destroy" / "apply"
There are many types of Dataflow jobs. Some Dataflow jobs run constantly, getting new data from (e.g.) a GCS bucket, and outputting data continuously. Some jobs process a set amount of data then terminate. All jobs can fail while running due to programming errors or other issues. In this way, Dataflow jobs are different from most other provider / Google resources. The Dataflow resource is considered 'existing' while it is in a nonterminal state. If it reaches a terminal state (e.g. 'FAILED', 'COMPLETE', 'CANCELLED'), it will be recreated on the next 'apply'. This is as expected for jobs which run continuously, but may surprise users who use this resource for other kinds of Dataflow jobs. A Dataflow job which is 'destroyed' may be "cancelled" or "drained". If "cancelled", the job terminates - any data written remains where it is, but no new data will be processed. If "drained", no new data will enter the pipeline, but any data currently in the pipeline will finish being processed. The default is "cancelled", but if a user sets on_delete to "drain" in the configuration, you may experience a long wait for your pulumi destroy to complete. You can potentially short-circuit the wait by setting skip_wait_on_job_termination to true, but beware that unless you take active steps to ensure that the job name parameter changes between instances, the name will conflict and the launch of the new job will fail. One way to do this is with a random_id resource, for example:
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as random from "@pulumi/random";
const config = new pulumi.Config();
const bigDataJobSubscriptionId = config.get("bigDataJobSubscriptionId") || "projects/myproject/subscriptions/messages";
const bigDataJobNameSuffix = new random.RandomId("big_data_job_name_suffix", {
byteLength: 4,
keepers: {
region: region,
subscription_id: bigDataJobSubscriptionId,
},
});
const bigDataJob = new gcp.dataflow.FlexTemplateJob("big_data_job", {
name: pulumi.interpolate`dataflow-flextemplates-job-${bigDataJobNameSuffix.dec}`,
region: region,
containerSpecGcsPath: "gs://my-bucket/templates/template.json",
skipWaitOnJobTermination: true,
parameters: {
inputSubscription: bigDataJobSubscriptionId,
},
});import pulumi
import pulumi_gcp as gcp
import pulumi_random as random
config = pulumi.Config()
big_data_job_subscription_id = config.get("bigDataJobSubscriptionId")
if big_data_job_subscription_id is None:
big_data_job_subscription_id = "projects/myproject/subscriptions/messages"
big_data_job_name_suffix = random.RandomId("big_data_job_name_suffix",
byte_length=4,
keepers={
"region": region,
"subscription_id": big_data_job_subscription_id,
})
big_data_job = gcp.dataflow.FlexTemplateJob("big_data_job",
name=big_data_job_name_suffix.dec.apply(lambda dec: f"dataflow-flextemplates-job-{dec}"),
region=region,
container_spec_gcs_path="gs://my-bucket/templates/template.json",
skip_wait_on_job_termination=True,
parameters={
"inputSubscription": big_data_job_subscription_id,
})using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
using Random = Pulumi.Random;
return await Deployment.RunAsync(() =>
{
var config = new Config();
var bigDataJobSubscriptionId = config.Get("bigDataJobSubscriptionId") ?? "projects/myproject/subscriptions/messages";
var bigDataJobNameSuffix = new Random.RandomId("big_data_job_name_suffix", new()
{
ByteLength = 4,
Keepers =
{
{ "region", region },
{ "subscription_id", bigDataJobSubscriptionId },
},
});
var bigDataJob = new Gcp.Dataflow.FlexTemplateJob("big_data_job", new()
{
Name = bigDataJobNameSuffix.Dec.Apply(dec => $"dataflow-flextemplates-job-{dec}"),
Region = region,
ContainerSpecGcsPath = "gs://my-bucket/templates/template.json",
SkipWaitOnJobTermination = true,
Parameters =
{
{ "inputSubscription", bigDataJobSubscriptionId },
},
});
});package main
import (
"fmt"
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/dataflow"
"github.com/pulumi/pulumi-random/sdk/v4/go/random"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
cfg := config.New(ctx, "")
bigDataJobSubscriptionId := "projects/myproject/subscriptions/messages"
if param := cfg.Get("bigDataJobSubscriptionId"); param != "" {
bigDataJobSubscriptionId = param
}
bigDataJobNameSuffix, err := random.NewRandomId(ctx, "big_data_job_name_suffix", &random.RandomIdArgs{
ByteLength: pulumi.Int(4),
Keepers: pulumi.StringMap{
"region": pulumi.Any(region),
"subscription_id": pulumi.String(bigDataJobSubscriptionId),
},
})
if err != nil {
return err
}
_, err = dataflow.NewFlexTemplateJob(ctx, "big_data_job", &dataflow.FlexTemplateJobArgs{
Name: bigDataJobNameSuffix.Dec.ApplyT(func(dec string) (string, error) {
return fmt.Sprintf("dataflow-flextemplates-job-%v", dec), nil
}).(pulumi.StringOutput),
Region: pulumi.Any(region),
ContainerSpecGcsPath: pulumi.String("gs://my-bucket/templates/template.json"),
SkipWaitOnJobTermination: pulumi.Bool(true),
Parameters: pulumi.StringMap{
"inputSubscription": pulumi.String(bigDataJobSubscriptionId),
},
})
if err != nil {
return err
}
return nil
})
}package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.random.RandomId;
import com.pulumi.random.RandomIdArgs;
import com.pulumi.gcp.dataflow.FlexTemplateJob;
import com.pulumi.gcp.dataflow.FlexTemplateJobArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
final var config = ctx.config();
final var bigDataJobSubscriptionId = config.get("bigDataJobSubscriptionId").orElse("projects/myproject/subscriptions/messages");
var bigDataJobNameSuffix = new RandomId("bigDataJobNameSuffix", RandomIdArgs.builder()
.byteLength(4)
.keepers(Map.ofEntries(
Map.entry("region", region),
Map.entry("subscription_id", bigDataJobSubscriptionId)
))
.build());
var bigDataJob = new FlexTemplateJob("bigDataJob", FlexTemplateJobArgs.builder()
.name(bigDataJobNameSuffix.dec().applyValue(dec -> String.format("dataflow-flextemplates-job-%s", dec)))
.region(region)
.containerSpecGcsPath("gs://my-bucket/templates/template.json")
.skipWaitOnJobTermination(true)
.parameters(Map.of("inputSubscription", bigDataJobSubscriptionId))
.build());
}
}configuration:
bigDataJobSubscriptionId:
type: string
default: projects/myproject/subscriptions/messages
resources:
bigDataJobNameSuffix:
type: random:RandomId
name: big_data_job_name_suffix
properties:
byteLength: 4
keepers:
region: ${region}
subscription_id: ${bigDataJobSubscriptionId}
bigDataJob:
type: gcp:dataflow:FlexTemplateJob
name: big_data_job
properties:
name: dataflow-flextemplates-job-${bigDataJobNameSuffix.dec}
region: ${region}
containerSpecGcsPath: gs://my-bucket/templates/template.json
skipWaitOnJobTermination: true
parameters:
inputSubscription: ${bigDataJobSubscriptionId}Import
This resource does not support import.
Constructors
Properties
List of experiments that should be used by the job. An example value is ["enable_stackdriver_agent_metrics"].
The algorithm to use for autoscaling.
The GCS path to the Dataflow job Flex Template.
Immutable. Indicates if the job should use the streaming engine feature.
The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PRIVATE".
The name for the Cloud KMS key for the job. Key format is: projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY
User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. Note: This field is marked as deprecated as the API does not currently support adding labels. NOTE: Google-provided Dataflow templates often provide default labels that begin with goog-dataflow-provided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.
The machine type to use for launching the job. The default is n1-standard-1.
The machine type to use for the job.
Immutable. The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
Immutable. The initial number of Google Compute Engine instances for the job.
Template specific Key/Value pairs to be forwarded to the pipeline's options; keys are case-sensitive based on the language on which the pipeline is coded, mostly Java. Note: do not configure Dataflow options here in parameters.
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
Service account email to run the workers as. This should be just an email e.g. myserviceaccount@myproject.iam.gserviceaccount.com. Do not include any serviceAccount: or other prefix.
The Cloud Storage path to use for staging files. Must be a valid Cloud Storage URL, beginning with gs://.
The subnetwork to which VMs will be assigned. Should be of the form "regions/REGION/subnetworks/SUBNETWORK".
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.Only applicable when updating a pipeline. Map of transform name prefixes of the job to be replaced with the corresponding name prefixes of the new job.