S3Endpoint
Provides a DMS (Data Migration Service) S3 endpoint resource. DMS S3 endpoints can be created, updated, deleted, and imported.
Note: AWS is deprecating
extra_connection_attributes, such as used withaws.dms.Endpoint. This resource is an alternative toaws.dms.Endpointand does not useextra_connection_attributes. (AWS currently includesextra_connection_attributesin the raw responses to the AWS Provider requests and so they may be visible in the logs.) Note: Some of this resource's arguments have default values that come from the AWS Provider. Other default values are provided by AWS and subject to change without notice. When relying on AWS defaults, the provider state will often have a zero value. For example, the AWS Provider does not provide a default forcdc_max_batch_intervalbut the AWS default is60(seconds). However, the provider state will show0since this is the value return by AWS when no value is present. Below, we aim to flag the defaults that come from AWS (e.g., "AWS default...").
Example Usage
Minimal Configuration
This is the minimal configuration for an aws.dms.S3Endpoint. This endpoint will rely on the AWS Provider and AWS defaults.
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.dms.S3Endpoint;
import com.pulumi.aws.dms.S3EndpointArgs;
import com.pulumi.resources.CustomResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var example = new S3Endpoint("example", S3EndpointArgs.builder()
.endpointId("donnedtipi")
.endpointType("target")
.bucketName("beckut_name")
.serviceAccessRoleArn(aws_iam_role.example().arn())
.build(), CustomResourceOptions.builder()
.dependsOn(aws_iam_role_policy.example())
.build());
}
}Complete Configuration
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.dms.S3Endpoint;
import com.pulumi.aws.dms.S3EndpointArgs;
import com.pulumi.resources.CustomResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var example = new S3Endpoint("example", S3EndpointArgs.builder()
.endpointId("donnedtipi")
.endpointType("target")
.sslMode("none")
.tags(Map.ofEntries(
Map.entry("Name", "donnedtipi"),
Map.entry("Update", "to-update"),
Map.entry("Remove", "to-remove")
))
.addColumnName(true)
.addTrailingPaddingCharacter(false)
.bucketFolder("folder")
.bucketName("bucket_name")
.cannedAclForObjects("private")
.cdcInsertsAndUpdates(true)
.cdcInsertsOnly(false)
.cdcMaxBatchInterval(100)
.cdcMinFileSize(16)
.cdcPath("cdc/path")
.compressionType("GZIP")
.csvDelimiter(";")
.csvNoSupValue("x")
.csvNullValue("?")
.csvRowDelimiter("\\r\\n")
.dataFormat("parquet")
.dataPageSize(1100000)
.datePartitionDelimiter("UNDERSCORE")
.datePartitionEnabled(true)
.datePartitionSequence("yyyymmddhh")
.datePartitionTimezone("Asia/Seoul")
.dictPageSizeLimit(1000000)
.enableStatistics(false)
.encodingType("plain")
.encryptionMode("SSE_S3")
.expectedBucketOwner(data.aws_caller_identity().current().account_id())
.externalTableDefinition("etd")
.ignoreHeaderRows(1)
.includeOpForFullLoad(true)
.maxFileSize(1000000)
.parquetTimestampInMillisecond(true)
.parquetVersion("parquet-2-0")
.preserveTransactions(false)
.rfc4180(false)
.rowGroupLength(11000)
.serverSideEncryptionKmsKeyId(aws_kms_key.example().arn())
.serviceAccessRoleArn(aws_iam_role.example().arn())
.timestampColumnName("tx_commit_time")
.useCsvNoSupValue(false)
.useTaskStartTimeForFullLoadTimestamp(true)
.build(), CustomResourceOptions.builder()
.dependsOn(aws_iam_role_policy.example())
.build());
}
}Import
Endpoints can be imported using the endpoint_id, e.g.,
$ pulumi import aws:dms/s3Endpoint:S3Endpoint example example-dms-endpoint-tfConstructors
Functions
Properties
Only applies if output files for a CDC load are written in .csv format. If use_csv_no_sup_value is set to true, string to use for all columns not included in the supplemental log. If you do not specify a string value, DMS uses the null value for these columns regardless of use_csv_no_sup_value. (Ignored for source endpoints.)
ARN for the KMS key that will be used to encrypt the connection parameters. If you do not specify a value for kms_key_arn, then AWS DMS will use your default encryption key. AWS KMS creates the default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS region.
When set to true, uses the task start time as the timestamp column value instead of the time data is written to target. For full load, when set to true, each row of the timestamp column contains the task start time. For CDC loads, each row of the timestamp column contains the transaction commit time.When set to false, the full load timestamp in the timestamp column increments with the time data arrives at the target. Default is false.