FollowRobotProtocol
If enabled, the connector follows Robot protocol. Use the parameter FollowRobotMeta
to determine whether the connector follows the Robot META tags.
If the robot’s protocol disallows your spider access to the Web site or to certain areas of the Web Site, disable this parameter so your spider ignores the robot’s protocol.
Type: | Boolean |
Default: | True |
Required: | No |
Configuration Section: | TaskName or Default |
Example: | FollowRobotProtocol=False
|
See Also: |