IgnoreRobotProtocolErrors
A Boolean value that specifies whether to ignore robot protocol errors.
When you set FollowRobotProtocol to TRUE
, Web Connector requests a file named robots.txt
from the root directory of a web site. The web server should return a correctly formatted robots.txt
or should respond with an HTTP 404 (not found) response code. If the server gives any other response, for example returning HTTP 200 (success) but providing a malformed file, Web Connector attempts to retrieve the robots.txt
file again before each page request.
To ignore invalid robots protocol files and synchronize the web site as if no robots.txt
had been provided, set this parameter to TRUE
.
Type: | Boolean |
Default: | False |
Required: | No |
Configuration Section: | TaskName or FetchTasks |
Example: | IgnoreRobotProtocolErrors=TRUE
|
See Also: | FollowRobotProtocol |