web Crawler Mode
Parameters
value
The default mode is set to HOST_ONLY
. You can choose one of the following modes:
HOST_ONLY
– crawl only the website host names. For example, if the seed URL is"abc.example.com"
, then only URLs with host name"abc.example.com"
are crawled.SUBDOMAINS
– crawl the website host names with subdomains. For example, if the seed URL is"abc.example.com"
, then"a.abc.example.com"
and"b.abc.example.com"
are also crawled.EVERYTHING
– crawl the website host names with subdomains and other domains that the webpages link to.