pulumi-aws-kotlin/com.pulumi.aws.glue.kotlin/Job

Job

class Job : KotlinCustomResource

Provides a Glue Job resource.

Glue functionality, such as monitoring and logging of jobs, is typically managed with the default_arguments argument. See the Special Parameters Used by AWS Glue topic in the Glue developer guide for additional information.

Example Usage

Python Job

package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.glue.Job;
import com.pulumi.aws.glue.JobArgs;
import com.pulumi.aws.glue.inputs.JobCommandArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }
    public static void stack(Context ctx) {
        var example = new Job("example", JobArgs.builder()
            .roleArn(aws_iam_role.example().arn())
            .command(JobCommandArgs.builder()
                .scriptLocation(String.format("s3://%s/example.py", aws_s3_bucket.example().bucket()))
                .build())
            .build());
    }
}

Scala Job

package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.glue.Job;
import com.pulumi.aws.glue.JobArgs;
import com.pulumi.aws.glue.inputs.JobCommandArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }
    public static void stack(Context ctx) {
        var example = new Job("example", JobArgs.builder()
            .roleArn(aws_iam_role.example().arn())
            .command(JobCommandArgs.builder()
                .scriptLocation(String.format("s3://%s/example.scala", aws_s3_bucket.example().bucket()))
                .build())
            .defaultArguments(Map.of("--job-language", "scala"))
            .build());
    }
}

Streaming Job

package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.glue.Job;
import com.pulumi.aws.glue.JobArgs;
import com.pulumi.aws.glue.inputs.JobCommandArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }
    public static void stack(Context ctx) {
        var example = new Job("example", JobArgs.builder()
            .roleArn(aws_iam_role.example().arn())
            .command(JobCommandArgs.builder()
                .name("gluestreaming")
                .scriptLocation(String.format("s3://%s/example.script", aws_s3_bucket.example().bucket()))
                .build())
            .build());
    }
}

Enabling CloudWatch Logs and Metrics

package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.cloudwatch.LogGroup;
import com.pulumi.aws.cloudwatch.LogGroupArgs;
import com.pulumi.aws.glue.Job;
import com.pulumi.aws.glue.JobArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }
    public static void stack(Context ctx) {
        var exampleLogGroup = new LogGroup("exampleLogGroup", LogGroupArgs.builder()
            .retentionInDays(14)
            .build());
        var exampleJob = new Job("exampleJob", JobArgs.builder()
            .defaultArguments(Map.ofEntries(
                Map.entry("--continuous-log-logGroup", exampleLogGroup.name()),
                Map.entry("--enable-continuous-cloudwatch-log", "true"),
                Map.entry("--enable-continuous-log-filter", "true"),
                Map.entry("--enable-metrics", "")
            ))
            .build());
    }
}

Import

Glue Jobs can be imported using name, e.g.,

$ pulumi import aws:glue/job:Job MyJob MyJob

Properties

arn

val arn: Output<String>

Amazon Resource Name (ARN) of Glue Job

command

val command: Output<JobCommand>

The command of the job. Defined below.

connections

val connections: Output<List<String>>?

The list of connections used for this job.

defaultArguments

val defaultArguments: Output<Map<String, String>>?

The map of default arguments for this job. You can specify arguments here that your own job-execution script consumes, as well as arguments that AWS Glue itself consumes. For information about how to specify and consume your own Job arguments, see the Calling AWS Glue APIs in Python topic in the developer guide. For information about the key-value pairs that AWS Glue consumes to set up your job, see the Special Parameters Used by AWS Glue topic in the developer guide.

description

val description: Output<String>?

Description of the job.

executionClass

val executionClass: Output<String>?

Indicates whether the job is run with a standard or flexible execution class. The standard execution class is ideal for time-sensitive workloads that require fast job startup and dedicated resources. Valid value: FLEX, STANDARD.

executionProperty

val executionProperty: Output<JobExecutionProperty>

Execution property of the job. Defined below.

glueVersion

val glueVersion: Output<String>

The version of glue to use, for example "1.0". For information about available versions, see the AWS Glue Release Notes.

val id: Output<String>

maxCapacity

val maxCapacity: Output<Double>

The maximum number of AWS Glue data processing units (DPUs) that can be allocated when this job runs. Required when pythonshell is set, accept either 0.0625 or 1.0. Use number_of_workers and worker_type arguments instead with glue_version 2.0 and above.