pulumi-aws-native-kotlin/com.pulumi.awsnative.bedrock.kotlin.inputs/DataSourceWebCrawlerConfigurationArgs

DataSourceWebCrawlerConfigurationArgs

data class DataSourceWebCrawlerConfigurationArgs(val crawlerLimits: Output<DataSourceWebCrawlerLimitsArgs>? = null, val exclusionFilters: Output<List<String>>? = null, val inclusionFilters: Output<List<String>>? = null, val scope: Output<DataSourceWebScopeType>? = null, val userAgent: Output<String>? = null, val userAgentHeader: Output<String>? = null) : ConvertibleToJava<DataSourceWebCrawlerConfigurationArgs>

Configuration for the web crawler.

Constructors

constructor(crawlerLimits: Output<DataSourceWebCrawlerLimitsArgs>? = null, exclusionFilters: Output<List<String>>? = null, inclusionFilters: Output<List<String>>? = null, scope: Output<DataSourceWebScopeType>? = null, userAgent: Output<String>? = null, userAgentHeader: Output<String>? = null)

Properties

crawlerLimits

val crawlerLimits: Output<DataSourceWebCrawlerLimitsArgs>? = null

The configuration of crawl limits for the web URLs.

exclusionFilters

val exclusionFilters: Output<List<String>>? = null

A list of one or more exclusion regular expression patterns to exclude certain URLs. If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.

inclusionFilters

val inclusionFilters: Output<List<String>>? = null

A list of one or more inclusion regular expression patterns to include certain URLs. If you specify an inclusion and exclusion filter/pattern and both match a URL, the exclusion filter takes precedence and the web content of the URL isn’t crawled.

scope

val scope: Output<DataSourceWebScopeType>? = null

The scope of what is crawled for your URLs. You can choose to crawl only web pages that belong to the same host or primary domain. For example, only web pages that contain the seed URL "https://docs.aws.amazon.com/bedrock/latest/userguide/" and no other domains. You can choose to include sub domains in addition to the host or primary domain. For example, web pages that contain "aws.amazon.com" can also include sub domain "docs.aws.amazon.com".

userAgent

val userAgent: Output<String>? = null

The suffix that will be included in the user agent header.

userAgentHeader

val userAgentHeader: Output<String>? = null

The full user agent header, including UUID and suffix.

Functions

toJava

open override fun toJava(): DataSourceWebCrawlerConfigurationArgs