Open topic with navigation

# FollowRobotProtocol

A Boolean value that specifies whether the connector respects instructions provided by Web sites to Web crawlers. By default, the connector respects the following instructions:

- The file `robots.txt`, if present in the root directory of a Web site.
- NOINDEX and NOCRAWL instructions in the robots meta element of a Web page.

To ignore these instructions, set `FollowRobotProtocol=False`.

Type: | Boolean
Default: | True
Required: | No
Configuration Section: | TaskName or FetchTasks or Default
Example: | `FollowRobotProtocol=False`
See Also: |

© Copyright 2019 Micro Focus or one of its affiliates | Send documentation feedback Close Send documentation feedback To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to swpdl.idoldocsfeedback@microfocus.com. We appreciate your feedback! Product: Topic Title: Help Topic: Feedback:

_FT_HTML5_bannerTitle.htm