If enabled, the connector follows the Robot META tags. If FollowRobotProtocol
is enabled, use FollowRobotMeta
to determine whether the connector follows the Robot META tags.
The Robot META tag, placed in the HTML <HEAD>
section of a page, can specify whether a page should be indexed or links on the page should be followed.
Type: | Boolean |
Default: | False |
Required: | No |
Configuration Section: | TaskName or Default |
Example: | FollowRobotMeta=True
|
See Also: |
|