Web Connector 25.4
Web Connector Help

# SpiderUrlCantHaveRegex

A Perl-compatible regular expression to restrict the pages that are crawled by the connector. If the full URL of a page matches the regular expression, the page is not crawled and is not ingested.

Type: | String
Default: |
Required: | No
Configuration Section: | TaskName or FetchTasks or Default
Example: | `SpiderUrlCantHaveRegex=.*private\.mywebsite\.com.*|.*internal\.mywebsite\.com.*`
See Also: | SpiderUrlMustHaveRegex

Copyright 2025 Open Text
Version 25.4 | Last updated October 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery Web Connector 25.4

Topic Title: SpiderUrlCantHaveRegex