# IgnoreRobotProtocolErrors

A Boolean value that specifies whether to ignore robot protocol errors.

When you set FollowRobotProtocol to `TRUE`, Web Connector requests a file named `robots.txt` from the root directory of a web site. The web server should return a correctly formatted `robots.txt` or should respond with an HTTP 404 (not found) response code. If the server gives any other response, for example returning HTTP 200 (success) but providing a malformed file, Web Connector attempts to retrieve the `robots.txt` file again before each page request.

To ignore invalid robots protocol files and synchronize the web site as if no `robots.txt` had been provided, set this parameter to `TRUE`.

Type: | Boolean
Default: | False
Required: | No
Configuration Section: | TaskName or FetchTasks
Example: | `IgnoreRobotProtocolErrors=TRUE`
See Also: | FollowRobotProtocol

_FT_HTML5_bannerTitle.htm