Open topic with navigation

# Limit the Download Rate

To reduce the load on the Web site you are indexing, you can limit the maximum rate at which the connector downloads content, using the configuration parameter `MaxKBytesPerSec`. This parameter does not limit the download rate for any individual page, it limits the maximum download rate by pausing between pages.

A Web server might be configured to deny requests that arrive from the same host with little or no time between them. The connector requests information from a Web site much faster than a human visitor, so you might find that the connector's requests are denied. To prevent a server from denying requests from the connector, you can specify a time interval that a synchronize thread must wait between requests. To do this, set the configuration parameter `PageDelay`. If you set a delay of one second, each synchronize thread must wait at least one second between requests. By default, the connector uses five synchronize threads so might request up to five pages per second.

You can also limit the maximum number of pages that are ingested in a single synchronize cycle, by setting the configuration parameter `MaxPages`. If the limit is reached, the remaining pages are ingested in future synchronize cycles.

TIP:

For more information about these configuration parameters, refer to the Web Connector Reference.

© 2017 Hewlett Packard Enterprise Development LP | Send documentation feedback to HPE Close Send documentation feedback To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to AutonomyTPFeedback@hpe.com. We appreciate your feedback! Product: Topic Title: Help Topic: Feedback:

_HP_HTML5_bannerTitle.htm