Web Connector 26.3
Web Connector Help

# SpiderUrlCantHaveRegex

A Perl-compatible regular expression to restrict the pages that are crawled by the connector. If the full URL of a page matches the regular expression, the page is not crawled and is not ingested.

Type: | String
Default: |
Required: | No
Configuration Section: | TaskName or FetchTasks or Default
Example: | `SpiderUrlCantHaveRegex=.*private\.mywebsite\.com.*|.*internal\.mywebsite\.com.*`
See Also: | SpiderUrlMustHaveRegex

Copyright 2026 Open Text
Version 26.3 | Last updated July 2026

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery Web Connector 26.3

Topic Title: SpiderUrlCantHaveRegex