Data Discovery & Risk Insights 26.2.0 Help Center

# Manage datasets

In OpenText Core Data Discovery & Risk Insights, a dataset defines the location of data on a source, the rules and schedule for processing, and the grammar rules to identify within the data during processing.

Once you have created at least one source, you can create as many datasets as necessary for that source. For example, you want to process data from only specific directories on a given file system. You would create a source for the file system and then create a dataset for each of the directories on that file system that contain data you want to process. This lets you focus the processing on only the desired data, omitting known irrelevant data.

On the Manage Datasets page, you can filter the list of datasets by dataset TYPE, choose whether to VIEW the list sorted by sources, and show datasets for one or more specific SOURCES. The analysis information specifies the processing conducted for the dataset. The data count represents the number of emails or documents for unstructured data sources or the number of tables for structured data sources. When the count represents documents, the document count reflects the number of parent documents processed (extracted attachments are not included in the count). The size of each dataset represents the size on disk.

TIP: For unstructured sources, the size displayed shows for either "Root Documents" or "Root + Child Documents". To change the view, click "Root Documents" or "Root + Child Documents" above the data cards and select the other view. Root documents are top-level documents. Child documents are documents within a container, such as a ZIP file.

From the Manage Datasets page, you can view additional information about each dataset.

- View the processing options (analysis) initially selected for the dataset.

  Initial analysis options

  Analysis | Description
  --- | ---
  Metadata Scan | Indicates that only the document metadata has been analyzed and indexed for all documents in this dataset.
  n% SmartScan | Indicates that document metadata has been indexed and document metadata and body content has been analyzed to identify tag values for the defined percentage of documents in this dataset.
  Full Scan | Indicates that document metadata has been indexed and document metadata and body content has been analyzed to identify tag values for all documents in this dataset.
  C | Indicates that content has been stored for documents in this dataset. The original document itself was not originally stored.
  E | Indicates that grammar values have been extracted for documents in this dataset. Because the content has not been stored, you can see that grammar values exist, but you cannot see the document content in Analyze or Manage.
  CE | Indicates that the content has been stored and grammar values have been extracted for documents in this dataset.

- Click anywhere in the row for the desired dataset and then click the open detail pane icon () to display dataset details. The detail pane includes the options defined for the dataset as well as key information about the documents in the dataset.

  From the detail pane, you can edit, scan, activate/deactivate, or delete the dataset. For datasets with data, you can view the data items in Analyze or go to the data volume chart () focused on the selected dataset. For file system and SharePoint datasets containing data, you can go to the sensitive data heat map () in Analyze, focused on the dataset.

  - Click the Change link next to the schedule information to open the Edit dataset dialog to the Schedule information.

  - If the grammar sets have been updated since the dataset was last scanned, a re-analyze icon () and a Re-analyze link display next to the Grammar Sets information. The grammar sets applied to the dataset that have been updated (and therefore "out of sync") display in bold text.

    IMPORTANT: Re-analyzing the dataset to update the grammar sets is optional. Changes to the grammar sets for this dataset are not automatically updated for the dataset. Re-analyzing a dataset for which the content has not been stored requires reprocessing the full dataset, may take additional time, and impact server load.

  - On the METRICS tab, view metrics related to the selected dataset.

  - On the GRAMMARS tab, view the grammar types and grammar rules defined for the dataset.

  - On the ACTIVITY tab, view the details of the last 10 activities performed. If more than 10 activities have been performed, click the MORE link to see the full list for the dataset on the Agent Activity page.

    If errors have occurred for an activity, click the error link view the details of any errors. For more information about viewing agent activity errors, see Activity error details.

  - On the SECURITY tab, view the security policy and associated users and groups that have been given specific access to the dataset.

## Create a dataset

You can create datasets as needed for existing sources. The available options vary based on the dataset type. Follow the instructions for the desired dataset type.

To create a file system dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Complete the Primary Capture Rules options to capture data.

   Option | Description
   --- | ---
   - - Sub-directory | Type the subpath to the directory you want to process data from. Type only the sub-directory path beneath the defined source. May be left blank to process items at the defined source level. If supported by the agent cluster assigned to the source, click to add additional sub-directories to be processed. Limits: The sub-directory path cannot contain any of the following special characters. `<` (less than) `>` (greater than) `:` (colon) `"` (double quote) `|` (vertical bar or pipe) `?` (question mark) `*` (asterisk) / (slash) The path cannot contain `.` or `..` before, after, or in between slashes (`\`) with no other characters and cannot end with `.` . Not valid:
       `..\` | `.\abc` | `abc\..` | `abc\.\def`
     - Valid:
       `ab..c` | `.abc` | `abc\d.e.f` | `abc\.def\gh`
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions. One by one, type the desired extension without the leading dot (.) and then press Enter on your keyboard. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. Select Exclude extensions to exclude all files with the defined file extension. Type the desired extension without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. When disabled (deselected) all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. SmartScan (Analyzed). Specifies to index the metadata and to analyze the metadata and body content to identify tag values for a defined percentage of individual files. Sampling Rate. Define the percentage of individual files in the datast to process, including index of metadata and body content. Use the slider to set the percentage, or type the desired percentage Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to index the metadata and to analyze the metadata and body content to identify tag values. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. NOTE: In this release, password protected PST files cannot be expanded and the contents processed. If encountered, the PST container is processed.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   - External metadata capture | Specify whether to identify and store metadata above and beyond the metadata that is captured by default. The additional metadata is stored with the original document. Click the toggle to select () or deselect () the option. IMPORTANT: For requirements and additional information about external metadata file capture, review Process external metadata. When enabled (selected), additional metadata is identified and stored for individual files. Complete the following options.
     Option | Description
     --- | ---
     External metadata file processor | Type the name of the file (without the file extension) that will be used to identify the additional metadata and apply the information. This file must be a `.ps1` file and exist on the agent host machine at the location defined by the `ScriptBasePath` in the Agent Admin UI under Advanced Settings > Run Script.
     Dynamic | Select to scan each file and gather metadata according to the configuration defined by the external metadata file processor.
     From file | Select to identify metadata read from a metadata file. In External metadata file extensions, type the extensions of the files that contain the external metadata information. These files are not processed as part of the primary capture. Limits: do not include the leading dot (.). For example, type `idx` not `.idx`. For multiple file extensions, press Enter on your keyboard or type a comma (,) and a space after each entry. To remove an extension, click the associated remove icon ().
   When disabled, (deselected), only the original document metadata is processed.

   Click NEXT.

2. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. If one or more existing datasets within the selected source have defined advanced capture rules, you can create new advanced capture rules, proceed without advanced capture rules, or select advanced capture rules defined for existing datasets within the selected source.

      Option | Description
      --- | ---
      Create a unique set of rules or skip this step | Select to create new advanced capture rules or to proceed without advanced capture rules. To create new advanced capture rules, proceed to the next step to complete the advanced capture rules. To proceed without advanced capture rules, click NEXT and continue creating the dataset.
      View and select rules from other datasets | Select to view advanced capture rules defined for existing datasets within the selected source. To remove listed advanced capture rules so they do not apply to the new dataset, click the delete icon associated with the rule you want to remove. To add another rule, click the add rule icon and then proceed to the next step to complete the advanced capture rules.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

3. Complete the Credential options for accessing the new dataset.

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. NOTE: Options cannot be edited. Select Specify Credentials to define credentials other than that defined for the parent source.
   Username | Defines the username for the user that has access to the dataset. If different than the source, defaults to the username defined for the source but can be edited. Type the username of the user that has access to the defined dataset directory. Limits: use the format `domain\user` or `user@domain`
   Password | Defines the password for the user that has access to the dataset. If different than the source, type the password for user defined.

   Click NEXT.

4. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

5. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

6. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

7. Complete the Schedule options to define the schedule for processing this dataset.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

8. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

9. Click FINISH.

   The new dataset is created.

   NOTE: Initialization begins to identify potential duplicate items as a result of datasets with overlapping paths. Once fully initialized, the status is updated in the dataset detail pane and the selected default file action begins.

To create an Exchange or Exchange Online (O365) dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. If this dataset is associated with an Exchange source with the dataset scan option based on a list of mailbox addresses, define the list of Mailboxes to include in the dataset.

   NOTE: Keep the following in mind.

   - An individual mailbox cannot be associated with more than one dataset.

   - Once the dataset has been scanned, you can still add mailboxes. You can remove mailboxes only if there is no scanned data associated with them.

   1. Type, or copy and paste, the individual email addresses of each mailbox to be scanned.

      - Separate each email address with a comma.

      - Maximum 254 characters allowed for each email address.

      - Maximum 500 individual email addresses allowed.

   2. Click ADD.

      - Use the search box to search the currently displayed list of email addresses.

      - Click the delete icon associated with an email address to remove it from the List of mailbox addresses list.

      - Click REMOVE ALL to clear all email addresses in the List of mailbox addresses list.

   3. Click NEXT.

2. Complete the Primary Capture Rules options to capture data.

   Option | Description
   --- | ---
   Default action will be taken on mailboxes from the following group | Select the OpenText Core Data Discovery & Risk Insights group that will be acted upon based on the selection you make for Default action on mailboxes. This is the group created as part of the Exchange connection tasks. This option supports a single group. Limits: Displays only when the dataset scan option set at the source level is to scan Based on a pre-defined user group. IMPORTANT: Avoid creating datasets that define groups with users who span multiple groups.
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the date the last modifies. Click the toggle to select () or deselect () the option. When enabled (selected), items with a modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Default action on mailboxes | Select one of the following actions to take when processing documents from this dataset. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. The selected action is the default action on all mailboxes for the defined OpenText Core Data Discovery & Risk Insights group. NOTE: In this release, password protected PST files cannot be expanded and the contents processed. If encountered, the PST container is processed.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.

   Click NEXT.

3. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. If one or more existing datasets within the selected source have defined advanced capture rules, you can create new advanced capture rules, proceed without advanced capture rules, or select advanced capture rules defined for existing datasets within the selected source.

      Option | Description
      --- | ---
      Create a unique set of rules or skip this step | Select to create new advanced capture rules or to proceed without advanced capture rules. To create new advanced capture rules, proceed to the next step to complete the advanced capture rules. To proceed without advanced capture rules, click NEXT and continue creating the dataset.
      View and select rules from other datasets | Select to view advanced capture rules defined for existing datasets within the selected source. To remove listed advanced capture rules so they do not apply to the new dataset, click the delete icon associated with the rule you want to remove. To add another rule, click the add rule icon and then proceed to the next step to complete the advanced capture rules.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

4. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

5. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

6. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

7. Complete the Schedule options to define the schedule for processing this dataset.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

8. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

9. Click FINISH.

   The new dataset is created.

To create a SharePoint or SharePoint Online (O365) dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Complete the Primary Capture Rules options to capture data.

   For SharePoint datasets

   Option | Description
   --- | ---
   SharePoint URL type | Select whether this dataset is a Site collection, a Web application, or a Document library.
   - Site Sub-URL | Type the sub-URL relative to the base of the SharePoint site defined for the source. May be left blank to process items at the defined source level. Example sub-URLs: Site collection: If the source SharePoint URL is `https://company.sharepoint.com`, type `/sites/team01` to connect to the SharePoint site at `https://company.sharepoint.com/sites/team01`. IMPORTANT: SharePoint subsites of this site are not processed as part of this site. To process from SharePoint site and its subsite, create another dataset and define the full sub-URL for the subsite. For example, create another dataset and type `/sites/team01/projectA` as the sub-URL to process data from the subsite of the "team01" site. Document library: If the source SharePoint URL is `https://company.sharepoint.com`, type `/sites/team01/Shared%20Documents` to connect to the SharePoint site at `https://company.sharepoint.com/sites/team01/Shared%20Documents`. NOTE: Ensure the sub-URL must reflect the direct location of the site or document library and not a redirect URL. Limits: The path cannot contain `.` or `..` before, after, or in between slashes (`\` or `/`) with no other characters. Not valid:
     `..\` | `.\abc` | `abc\..` | `abc\.\def`
   - Valid:
     `abc..` | `.abc` | `abc\def.` | `abc\.def\gh`
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions. One by one, type the desired extension without the leading dot (.) and then press Enter on your keyboard. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. Select Exclude extensions to exclude all files with the defined file extension. Type the desired extension without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. When disabled (deselected) all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. SmartScan (Analyzed). Specifies to process and index the metadata and the body content for a defined percentage of individual files. Sampling Rate. Define the percentage of individual files in the datast to process, including index of metadata and body content. Use the slider to set the percentage, or type the desired percentage Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   - External metadata capture | Specify whether to identify and store metadata above and beyond the metadata that is captured by default. The additional metadata is stored with the original document. Click the toggle to select () or deselect () the option. IMPORTANT: For requirements and additional information about external metadata file capture, review Process external metadata. When enabled (selected), additional metadata is identified and stored for individual files. Complete the following options.
     Option | Description
     --- | ---
     External metadata file processor | Type the name of the file (without the file extension) that will be used to identify the additional metadata and apply the information. This file must be a `.ps1` file and exist on the agent host machine at the location defined by the `ScriptBasePath` in the Agent Admin UI under Advanced Settings > Run Script.
     Dynamic | Select to scan each file and gather metadata according to the configuration defined by the external metadata file processor.
     From file | Select to identify metadata read from a metadata file. In External metadata file extensions, type the extensions of the files that contain the external metadata information. These files are not processed as part of the primary capture. Limits: do not include the leading dot (.). For example, type `idx` not `.idx`. For multiple file extensions, press Enter on your keyboard or type a comma (,) and a space after each entry. To remove an extension, click the associated remove icon ().
   When disabled, (deselected), only the original document metadata is processed.

   For SharePoint Online (O365) datasets

   Option | Description
   --- | ---
   SharePoint URL type | Select whether this dataset is a Site collection or a Document library.
   Site Sub-URL | Type the sub-URL relative to the base of the SharePoint site defined for the source and start with a forward slash (/). May be left blank to process items at the defined source level. Example site sub-URL: If the source SharePoint URL is `https://company.sharepoint.com`, type `/sites/team01` to connect to the SharePoint site at `https://company.sharepoint.com/sites/team01`. IMPORTANT: SharePoint subsites of this site are not processed as part of this site. To process from SharePoint site and its subsite, create another dataset and define the full sub-URL for the subsite. For example, create another dataset type `/sites/team01/projectA` as the sub-URL to process data from the subsite of the "teams01" site.
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions. One by one, type the desired extension without the leading dot (.) and then press Enter on your keyboard. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. Select Exclude extensions to exclude all files with the defined file extension. Type the desired extension without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. When disabled (deselected) all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. SmartScan (Analyzed). Specifies to process and index the metadata and the body content for a defined percentage of individual files. Sampling Rate. Define the percentage of individual files in the datast to process, including index of metadata and body content. Use the slider to set the percentage, or type the desired percentage Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Sampling Rate | Define the percentage of individual files in the dataset to process, including index of metadata and body content. Use the slider to set the percentage, or type the desired percentage. Limits: Only available when SmartScan is selected as the default action on files.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   - External metadata capture | Specify whether to identify and store metadata above and beyond the metadata that is captured by default. The additional metadata is stored with the original document. Click the toggle to select () or deselect () the option. IMPORTANT: For requirements and additional information about external metadata file capture, review Process external metadata. When enabled (selected), additional metadata is identified and stored for individual files. Complete the following options.
     Option | Description
     --- | ---
     External metadata file processor | Type the name of the file (without the file extension) that will be used to identify the additional metadata and apply the information. This file must be a `.ps1` file and exist on the agent host machine at the location defined by the `ScriptBasePath` in the Agent Admin UI under Advanced Settings > Run Script.
     Dynamic | Select to scan each file and gather metadata according to the configuration defined by the external metadata file processor.
     From file | Select to identify metadata read from a metadata file. In External metadata file extensions, type the extensions of the files that contain the external metadata information. These files are not processed as part of the primary capture. Limits: do not include the leading dot (.). For example, type `idx` not `.idx`. For multiple file extensions, press Enter on your keyboard or type a comma (,) and a space after each entry. To remove an extension, click the associated remove icon ().
   When disabled, (deselected), only the original document metadata is processed.

   Click NEXT.

2. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

3. Complete the Credential options for accessing the new dataset.

   For SharePoint datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. NOTE: Options cannot be edited. Select Specify Credentials to define credentials other than that defined for the source.
   Username | Defines the username for the user that has access to the dataset. If different than the source, defaults to the username defined for the source but can be edited. Type the user information of the SharePoint service user that has access to the SharePoint directory you want to process data from. Limits: Use the format `domain\user` or `machine_name\user`. Maximum 63 characters for the `user`. Cannot contain `" / \ [ ] : ; | = , + * ? < >` within the `user`.
   Password | Defines the password for the user that has access to the dataset. If different than the source, type the password for user defined.

   For SharePoint Online (O365) datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. NOTE: Options cannot be edited. Select Specify Credentials to define credentials other than that defined for the source.
   Authentication Type | Select the authentication type to connect to SharePoint Online as appropriate for your environment and the connection tasks performed. SharePoint App Only Microsoft Entra ID
   Application ID | Defines the application ID for the dataset. If different than the source, defaults to the Application ID defined for the source but can be edited. Type the value for the appropriate Application (client) ID.
   Client Secret | Defines the client secret for the dataset. If different than the source, defaults to the Client Secret defined for the source and can be edited. Type the value for the Client Secret that corresponds to the defined Application (client) ID. Limits: Displays only when SharePoint App Only is selected as the authentication type.
   Directory ID | Defines the Directory ID. If different than the source, defaults to the Directory ID defined for the source and can be edited. Type the value for the Directory ID generated as part of the connection tasks. Limits: Displays only when Microsoft Entra ID is selected as the authentication type.
   PKCS12 keystore Upload | Click BROWSE and then navigate to and select the PKCS12 keystore generated as part of the connection tasks. Limits: Displays only when Microsoft Entra ID is selected as the authentication type.
   PKCS12 keystore Password | Defines the password for the selected PKCS keystore. If different than the source, defaults to the Certificate Password defined for the source but can be edited. Type the password that corresponds to the selected certificate. Limits: Displays only when Microsoft Entra ID is selected as the authentication type.

   Click NEXT.

4. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

5. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

6. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

7. Complete the Schedule options to define the schedule for processing this dataset.

   IMPORTANT: Ensure that the dataset is scanned more often than the maximum number of days SharePoint change logs are kept. The logs are necessary for tracking deleted items. For example, if your SharePoint change log retention is 60 days, your SharePoint datasets should be scanned at lease every 59 days. For more information, see "SharePoint deletion tracking" under SharePoint connection.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

8. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

9. Click FINISH.

   The new dataset is created.

   NOTE: Initialization begins to identify potential duplicate items as a result of datasets with overlapping paths. Once fully initialized, the status is updated in the dataset detail pane and the selected default file action begins.

To create a Content Manager dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Complete the Primary Capture Rules options to capture data.

   IMPORTANT: Avoid creating datasetswith the same record type as an existing dataset. Creating a dataset without defining record types results in ALL record types being processed, which in effect creates a duplicate selection if you create another dataset.

   Option | Description
   --- | ---
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   Capture additional metadata | Specify whether to include addition metadata fields when processing. Click the toggle to select () or deselect () the option. When enabled (selected), the Record Notes metadata field is also processed. CAUTION: Capturing this additional metadata will impact Content Manager scan performance. When not enabled (deselected), the Record Notes metadata field is not processed.
   Classifications | Type the name of the Content Manager classifications to be processed. Press Enter on your keyboard to commit the classification you entered. You can type the classification name exactly as it exists in Content Manager or specify a fully structured query using Content Manager's query syntax. Example using Content Manager query syntax: To process all records (documents) with the "Accommodation" classification or any of its sub classifications, define the classification field on the record and then the classification on the container (folder) that the record is in. Line breaks in the syntax are for display purposes only.
       classification:[title:"accommodation" or
       parent:[title:"accommodation"]+]
       or container:[classification:[title:"accommodation" or
       parent:[title:"accommodation"]+]]
   To remove a classification, click the associated remove icon ().
   Record types | Type the record types to be processed. To remove a record type, click the associated remove icon ().
   - External metadata capture | Specify whether to identify and store metadata above and beyond the metadata that is captured by default. The additional metadata is stored with the original document. Click the toggle to select () or deselect () the option. IMPORTANT: For requirements and additional information about external metadata file capture, review Process external metadata. When enabled (selected), additional metadata is identified and stored for individual files. Complete the following options.
     Option | Description
     --- | ---
     External metadata file processor | Type the name of the file (without the file extension) that will be used to identify the additional metadata and apply the information. This file must be a `.ps1` file and exist on the agent host machine at the location defined by the `ScriptBasePath` in the Agent Admin UI under Advanced Settings > Run Script.
     Dynamic | Select to scan each file and gather metadata according to the configuration defined by the external metadata file processor.
   When disabled, (deselected), only the original document metadata is processed.

   Click NEXT.

2. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

3. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

4. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

5. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

6. Complete the Schedule options to define the schedule for processing this dataset.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

7. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

8. Click FINISH.

   The new dataset is created.

To create a Google Drive dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Define the Drives to include in the dataset.

   NOTE: Keep the following in mind.

   - An individual user account cannot be associated with more than one dataset.

   - Once the dataset has been scanned, you can still add accounts. You can remove accounts only if there is no data associated with them.

   Select whether this dataset includes one or more User Drives or a Shared Drive.

   - If a User Drive,

     1. Type, or copy and paste, the individual access account addresses for the drives to be scanned.

        - Separate each account with a comma.

        - Maximum 254 characters allowed for each account.

        - Maximum 500 individual accounts allowed.

     2. Click ADD.

        - Use the search box to search the currently displayed list of accounts.

        - Click the delete icon associated with an email address to remove it from the List of access accounts list.

        - Click REMOVE ALL to clear all email addresses in the List of access accounts list.

     3. Click NEXT.

   - If a Shared Drive,

     1. Type the access account address for the shared drive to be scanned.

     2. Select whether to Scan all drives or to Scan a specific shared drive associated with the defined access account

     3. If you selected to Scan a specific shared drive, type the Shared Drive ID to be scanned.

     4. Click NEXT.

2. Complete the Primary Capture Rules options to capture data.

   Option | Description
   --- | ---
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions. One by one, type the desired extension without the leading dot (.) and then press Enter on your keyboard. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. Select Exclude extensions to exclude all files with the defined file extension. Type the desired extension without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. When disabled (deselected) all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   - External metadata capture | Specify whether to identify and store metadata above and beyond the metadata that is captured by default. The additional metadata is stored with the original document. Click the toggle to select () or deselect () the option. IMPORTANT: For requirements and additional information about external metadata file capture, review Process external metadata. When enabled (selected), additional metadata is identified and stored for individual files. Complete the following options.
     Option | Description
     --- | ---
     External metadata file processor | Type the name of the file (without the file extension) that will be used to identify the additional metadata and apply the information. This file must be a `.ps1` file and exist on the agent host machine at the location defined by the `ScriptBasePath` in the Agent Admin UI under Advanced Settings > Run Script.
     Dynamic | Select to scan each file and gather metadata according to the configuration defined by the external metadata file processor.
     From file | Select to identify metadata read from a metadata file. In External metadata file extensions, type the extensions of the files that contain the external metadata information. These files are not processed as part of the primary capture. Limits: do not include the leading dot (.). For example, type `idx` not `.idx`. For multiple file extensions, press Enter on your keyboard or type a comma (,) and a space after each entry. To remove an extension, click the associated remove icon ().
   When disabled, (deselected), only the original document metadata is processed.

   Click NEXT.

3. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

4. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

5. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

6. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

7. Complete the Schedule options to define the schedule for processing this dataset.

   IMPORTANT: Ensure that they are scanned more often than the maximum number of days Google Drive change logs are kept. The logs are necessary for tracking deleted items. For example, if your Google Drive change log retention is 60 days, your Google Drive datasets should be scanned at lease every 59 days. For more information, see "Google Drive deletion tracking" under Google Drive connection.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

8. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

9. Click FINISH.

   The new dataset is created.

To create a OneDrive dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Define the OneDrive Drives to include in the dataset.

   NOTE: Keep the following in mind.

   - An individual drive cannot be associated with more than one dataset.

   - Once the dataset has been scanned, you can still add sites/UPNs. You can remove sites/UPNs only if there is no data associated with them.

   For OneDrive datasets

   1. Type, or copy and paste, the individual site names for each drive to be scanned.

      - Use the appropriate format based on the user profile service application settings.

      - Separate each site name with a comma.

      - Maximum 254 characters allowed for each site name.

      - Maximum 500 individual site names allowed.

   2. Click ADD.

      - Use the search box to search the currently displayed list of site names.

      - Click the delete icon associated with a site name to remove it from the List of site names list.

      - Click REMOVE ALL to clear all site names in the List of site names list.

   3. Click NEXT.

   For OneDrive Online (O365) datasets

   1. Type, or copy and paste, the individual User Principal Name (UPN) for each drive to be scanned.

      - Separate each UPN with a comma.

      - Maximum 254 characters allowed for each UPN.

      - Maximum 500 individual UPNs allowed.

   2. Click ADD.

      - Use the search box to search the currently displayed list of account UPNs.

      - Click the delete icon associated with a UPN to remove it from the List of account UPNs list.

      - Click REMOVE ALL to clear all UPNs in the List of account UPNs list.

   3. Click NEXT.

2. Complete the Primary Capture Rules options to capture data.

   Option | Description
   --- | ---
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions. One by one, type the desired extension without the leading dot (.) and then press Enter on your keyboard. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. Select Exclude extensions to exclude all files with the defined file extension. Type the desired extension without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. When disabled (deselected) all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. SmartScan (Analyzed). Specifies to process and index the metadata and the body content for a defined percentage of individual files. Sampling Rate. Define the percentage of individual files in the datast to process, including index of metadata and body content. Use the slider to set the percentage, or type the desired percentage. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   - External metadata capture | Specify whether to identify and store metadata above and beyond the metadata that is captured by default. The additional metadata is stored with the original document. Click the toggle to select () or deselect () the option. IMPORTANT: For requirements and additional information about external metadata file capture, review Process external metadata. When enabled (selected), additional metadata is identified and stored for individual files. Complete the following options.
     Option | Description
     --- | ---
     External metadata file processor | Type the name of the file (without the file extension) that will be used to identify the additional metadata and apply the information. This file must be a `.ps1` file and exist on the agent host machine at the location defined by the `ScriptBasePath` in the Agent Admin UI under Advanced Settings > Run Script.
     Dynamic | Select to scan each file and gather metadata according to the configuration defined by the external metadata file processor.
     From file | Select to identify metadata read from a metadata file. In External metadata file extensions, type the extensions of the files that contain the external metadata information. These files are not processed as part of the primary capture. Limits: do not include the leading dot (.). For example, type `idx` not `.idx`. For multiple file extensions, press Enter on your keyboard or type a comma (,) and a space after each entry. To remove an extension, click the associated remove icon ().
   When disabled, (deselected), only the original document metadata is processed.

   Click NEXT.

3. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

4. Complete the Credential options for accessing the new dataset.

   For OneDrive datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. NOTE: Options cannot be edited. Select Specify Credentials to define credentials other than that defined for the source.
   Username | Defines the username for the user that has access to the dataset. If different than the source, defaults to the username defined for the source but can be edited. Type the user information of the OneDrive service user that has access to the OneDrive directory you want to process data from. Limits: Use the format `domain\user` or `machine_name\user`. Maximum 63 characters for the `user`. Cannot contain `" / \ [ ] : ; | = , + * ? < >` within the `user`.
   Password | Defines the password for the user that has access to the dataset. If different than the source, type the password for user defined.

   For OneDrive Online (O365) datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. NOTE: Options cannot be edited. Select Specify Credentials to define credentials other than that defined for the source.
   Authentication Type | Select Microsoft Entra ID as the authentication type to connect to OneDrive Online.
   Application ID | Defines the application ID for the dataset. If different than the source, defaults to the Application ID defined for the source but can be edited. Type the value for the appropriate Application (client) ID.
   Directory ID | Defines the Directory ID. If different than the source, defaults to the Directory ID defined for the source and can be edited. Type the value for the Directory ID generated as part of the connection tasks. Limits: Displays only when Microsoft Entra ID is selected as the authentication type.
   PKCS12 keystore Upload | Click BROWSE and then navigate to and select the PKCS12 keystore generated as part of the connection tasks. Limits: Displays only when Microsoft Entra ID is selected as the authentication type.
   PKCS12 keystore Password | Defines the password for the selected PKCS keystore. If different than the source, defaults to the Certificate Password defined for the source but can be edited. Type the password that corresponds to the selected certificate. Limits: Displays only when Microsoft Entra ID is selected as the authentication type.

   Click NEXT.

5. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

6. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

7. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

8. Complete the Schedule options to define the schedule for processing this dataset.

   IMPORTANT: Ensure that the dataset is scanned more often than the maximum number of days SharePoint change logs are kept. The logs are necessary for tracking deleted items. For example, if your SharePoint change log retention is 60 days, your OneDrive datasets should be scanned at lease every 59 days. For more information, see "Deletion tracking" under SharePoint connection.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

9. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

10. Click FINISH.

    The new dataset is created.

    NOTE: Initialization begins to identify potential duplicate items as a result of datasets with overlapping paths. Once fully initialized, the status is updated in the dataset detail pane and the selected default file action begins.

To create a Documentum dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Complete the Primary Capture Rules options to capture data.

   Option | Description
   --- | ---
   Repository | Type the name of the Documentum repository you want to process data from. NOTE: The Documentum repository must be unique across datasets within the source.
   DQL | Type the DQL query that defines the field and value to query against. Leave blank to process all data. Example: To process items that are one of three specific Documentum types, type `title='ITdoc' or title='HRdoc' or title='FinDoc'`.
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.

   Click NEXT.

2. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

3. Complete the Credential options for accessing the new dataset.

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. NOTE: Options cannot be edited. Select Specify Credentials to define credentials other than that defined for the source.
   Username | Defines the username for the user that has access to the dataset. If different than the source, defaults to the username defined for the source and can be edited. Type the username of the user that has access to the defined dataset directory.
   Password | Defines the password for the user that has access to the dataset. If different than the source, type the password for user defined.

   Click NEXT.

4. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

5. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

6. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

7. Complete the Schedule options to define the schedule for processing this dataset.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

8. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

9. Click FINISH.

   The new dataset is created.

To create an Extended ECM dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Complete the Primary Capture Rules options to capture data.

   NOTE: You can have only one Extended ECM dataset per source.

   Option | Description
   --- | ---
   Nodes | Type the numeric node ID for the desired Extended ECM node you want to process data from. The node ID is the value of the "Item ID" property for the object in Extended ECM. To add another node, click the add icon () and type the numeric node. Continue to add nodes as desired. To remove a node, click the associated remove icon ().
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions. One by one, type the desired extension without the leading dot (.) and then press Enter on your keyboard. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. Select Exclude extensions to exclude all files with the defined file extension. Type the desired extension without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). Limits: Only alphanumeric characters and _ - & # + = ^ $ ~ @ are allowed. When disabled (deselected) all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.

   Click NEXT.

2. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

3. Complete the Credential options for accessing the new dataset.

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. NOTE: Options cannot be edited. Select Specify Credentials to define credentials other than that defined for the source.
   Username | Defines the username for the user that has access to the dataset. If different than the source, defaults to the username defined for the source and can be edited. Type the username of the user that has access to the defined dataset directory.
   Password | Defines the password for the user that has access to the dataset. If different than the source, type the password for user defined.

   Click NEXT.

4. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

5. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

6. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

7. Complete the Schedule options to define the schedule for processing this dataset.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

8. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

9. Click FINISH.

   The new dataset is created.

To create a Confluence dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Define the Confluence spaces to include in the dataset.

   NOTE: An individual space key can be defined for only one dataset per Confluence source.

   1. Type, or copy and paste, the individual alphanumeric keys for each Confluence space to be scanned.

      - Separate each key with a comma or a new line.

      - Maximum 255 characters allowed for each key.

      - Maximum 500 unique, individual space keys allowed.

   2. Click ADD.

      - Use the search box to search the currently displayed list of space keys.

      - Click the delete icon associated with a key to remove it from the List of spaces list.

      - Click REMOVE ALL to clear all space keys in the List of spaces list.

   3. Click NEXT.

2. Complete the Primary Capture Rules options to capture data.

   Option | Description
   --- | ---
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.

   Click NEXT.

3. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

4. Complete the Credential options for accessing the new dataset.

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. NOTE: Options cannot be edited. Select Specify Credentials to define credentials other than that defined for the source.
   Authentication Type | Select Username and password or Personal access token as the authentication type to connect to Confluence. Limits: Editable only when Specify Credentials is selected as the credential input type.
   Username | Defines the username for the user that has access to the defined spaces in the dataset. If different than the source, defaults to the username defined for the source and can be edited. Type the username of the user that has access to the defined spaces in the dataset. This user was created as part of the Confluence connection tasks. Limits: Editable only when Specify Credentials is selected as the credential input type and Username and password is selected as the authentication type. Maximum 255 characters.
   Password | Defines the password for the user that has access to the dataset. If different than the source, type the password for user defined. Limits: Editable only when Specify Credentials is selected as the credential input type and Username and password is selected as the authentication type.
   Personal access token | Defines the personal access token generated as part of the Confluence connection tasks. If different than the source, type the personal access token. Limits: Editable only when Specify Credentials is selected as the credential input type and Personal access token is selected as the authentication type. Maximum 256 characters.

   Click NEXT.

5. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

6. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

7. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

8. Complete the Schedule options to define the schedule for processing this dataset.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

9. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

10. Click FINISH.

    The new dataset is created.

To create an unstructured data dataset based on a custom adapter

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Complete the Primary Capture Rules options to capture data.

   The Dataset Options defined when you created the custom adapter are included in the Primary Capture Rules. For more information about custom adapters, see Custom adapters.

   Option | Description
   --- | ---
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   [Dataset Options] | Type the desired value for the dataset options for the custom adapter. Hover over an information icon () to review the description for the associated option.

   Click NEXT.

2. (Optional) Complete the Advanced Capture Rules to refine the analysis options. You can refine document analysis based on create date, modify date, file extensions, and file size to perform analysis on specific documents in different ways. Once the primary capture rules are applied and the initial list of files is fetched from the source, the advance capture rules evaluate each document fetched, one document at a time. You will rank the advanced capture rules so the first rule that is matched is applied to a the document.

   1. Complete the advanced capture rule options.

      Option | Description
      --- | ---
      Select action | Select one of the following actions to take when processing documents matching the filter criteria. Analyze. Specifies to process and index the metadata and the body content of individual files that match the filter criteria. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Store content as text. Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing content increases the size of your index. Extract grammar rules. Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. TIP: You can extract grammar values later for files within a workbook. Metadata only. Specifies to process and index only the metadata from individual files that match the filter criteria. You cannot preview the document content in Analyze or Manage. Skip. Specifies to skip processing for individual files matching the filter criteria.
      Filter | Click Filter and define the desired filter criteria. To add criteria to the desired section, click the add icon and select the filter fields as desired. For fields added to the ANY section, each entry is combined with the OR operator. For fields added to the ALL section, each entry is combined with the AND operator. For extensions, each entry in the field is combined with the OR operator, then combined with any other fields with the AND operator. For fields added to the NONE section, each entry is combined with the OR operator.

   2. Set the priority order for the advanced capture rules. The first rule that is matched is applied to a the document..

      Click the drag icon () for the desired rule and then drag the rule up or down to the desired order.

   Click NEXT.

3. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. When selected, all additional options are unavailable. Click NEXT to continue. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

4. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

5. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

6. Complete the Schedule options to define the schedule for processing this dataset.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

7. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

8. Click FINISH.

   The new dataset is created.

To create a structured data (database) dataset

1. From the primary navigation pane, click Sources > Manage Datasets.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the dataset .

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

1. Select the tables to include in the dataset.

   1. In the list of available schemas, click the desired schema.

      The schema name displays in the SCHEMA row at the top. The list refreshes to show the available inventoried tables associated with the selected schema.

      NOTE: A table can be included in only one dataset. Tables that are included in other datasets do not display in the list of tables.

   2. In the list of available tables, click the checkboxes for the desired tables.

      The selected schema and table names display in the right pane.

   3. To select tables in another schema, click SCHEMA. The schemas display in the list. Click the desired schema and then click the checkboxes for the desired tables in that schema.

   4. To clear tables from the list of selected tables in the right pane, click CLEAR.

      - Click Clear all to clear the entire list of selected tables.

      - Click Clear for selected schema to clear all tables for the schema that currently displays in the SCHEMA row.

   5. Click NEXT.

2. Complete the Primary Capture Rules options to capture data.

   NOTE: The maximum number of rows that can be scanned is 999,999,999 (< 1 billion) rows per table.

   Option | Description
   --- | ---
   Default action | Select one of the following actions to take when processing table rows in this dataset. SmartScan (Analyzed). (Default) Specifies to process a defined percentage of rows in the selected tables. Percentage of rows to sample. Define the percentage of rows to process. Use the slider to set the percentage, or type the desired percentage. Number of rows to sample. Type the Minimum and Maximum number of rows to sample. NOTE: If the percentage of rows to sample results in fewer rows than defined by the minimum number of rows to sample, the defined minimum number of rows are scanned. Full Scan (Analyzed). Specifies to process all rows in the selected tables
   Enable randomized cell scanning | Specify whether to randomly scan cell values across rows and columns. Click the toggle to select () or deselect () the option. When enabled (selected), table cells are scanned randomly. This ensures a record is not captured and presented in its entirety. When not enabled (deselected), table rows are scanned across all columns and are captured as a complete record. NOTE: When SmartScan is selected, entire rows in the table are sampled randomly. Limits: Only available for datasets assigned to sources using the built-in structured data processor.
   Store content as text | Specify whether to store the contents of the tables as text. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the tables are stored in the index as text. When not enabled (deselected), tag values are identified but the table content cannot be viewed. Limits: Only available for datasets assigned to sources associated with an agent cluster that is not enabled for Structured Data Manager capabilities. NOTE: Changes to grammar, tag, and workspace keyword search criteria will not be automatically updated for tables that have already been analyzed. Storing content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identified by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for tables within a workbook.

   Click NEXT.

3. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

   Click NEXT.

4. (Optional) Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

5. Complete the Schedule options to define the schedule for processing this dataset.

   Option | Description
   --- | ---
   Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset. When not enabled, all other scheduling options are unavailable. Click NEXT to continue.
   Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
   Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
   Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

   Click NEXT.

6. (Optional) Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

   1. From the Select attribute list, do one of the following:

      - Select the desired attribute from the list.

      - To create a new attribute, click the New Attribute link in the list.

        In the New Attribute dialog, type a name for the new attribute and then click CREATE.

      The list of values for the selected attribute populate the value list.

      NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

   2. From the Select value list, do one of the following:

      - Select the desired value from the list.

      - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

        In the New Value dialog, type a name for the new value and then click CREATE.

      NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

   3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

      Repeat until all desired attribute values are selected.

      To remove a selected attribute and value pair, click the associated remove icon ().

7. Click FINISH.

   The new dataset is created

## Initiate Processing

Datasets are not automatically scanned when the dataset is created. If you defined a schedule, the dataset scan will run according to that schedule. If a schedule was not defined for the dataset, you must initiate a scan of the dataset to process the data. Wait for the dataset initialization to complete before initiating a scan of a newly created dataset.

If you need to process a dataset outside of the scheduled run time, you can manually start a scan of the dataset. If you request to scan a dataset and the dataset is currently processing, the scan request is not acted upon.

NOTE: Re-scanning a structured dataset may return a different set of items. Previous review changes, such as marking grammar values as false positive, may be invalidated.

To scan a dataset to initiate processing or to rescan off schedule

1. On the Manage Datasets page, click or hover over the row for the desired dataset.

   Additional icons display in the right column.

2. Click the scan icon ( ) associated with the desired dataset.

   TIP: You can also click in the row for the desired dataset, open the detail pane () and then click SCAN.

3. In the confirmation dialog, click YES to confirm the action.

   A message confirming the scan request. Click the Agent Activity link in the message to go to the Agent Activity page to review the progress.

   If processing is not currently running, the scan is started.

   NOTE: For file system and SharePoint dataset, initialization begins to identify potential duplicate items as a result of datasets with overlapping paths. Once fully initialized, the status is updated in the dataset detail pane and the scan action begins.

## Edit a dataset

Datasets can be edited as needed. When editing a dataset, keep the following in mind.

- If items exist in this dataset, some options may be dimmed and cannot be edited. Options may also be dimmed based on current and previous selections.

- If items exist in this dataset, you cannot change the agent cluster type from a cloud-based cluster to an on-premises cluster, or from an on-premises cluster to a cloud-based cluster.

- If items exist in this dataset and you add advanced capture rules, the rules will be only be applied to newly processed items.

- If you change any of the credentials, you are required to re-enter the password for the defined user.

- You can load the schedule, attribute, and grammar options from a template, even if the dataset is based on a different template. This action overrides any existing schedule, attribute, or grammar options and can be refined further as needed.

- Changes to existing grammar selections trigger a reprocessing of the dataset.

  If you have datasets that include grammars applied before the introduction of grammar sets, you will see the originally applied grammars in the list of selected grammars. If you select grammar sets, the originally applied grammars are overridden.

  If you remove the association of any grammar rules, the grammar values previously identified by the removed grammar rule will no longer be associated with items in the dataset. Tags based on removed grammar rules will no longer be assigned to items in the dataset.

CAUTION: Do not change the dataset location—File System directory, SharePoint site URL, or Content Manager dataset—unless the location has actually changed. The physical location (or Exchange group name) must have changed (or be changed) prior to updating the location in this dataset.

Do not change an Exchange group name unless the name of the group has changed. The group name in Exchange must be changed prior to updating the group name in this dataset. Changes to group names may affect tracking of delete activities.

If you want to create a dataset with the same definitions of the dataset you are editing, create a new dataset.

To edit a dataset

1. On the Manage Datasets page, click the name of the dataset you want to edit.

   TIP: You can also do one of the following:

   - Click or hover over the row for the desired dataset and then click the edit icon ().

   - Click the row for the desired dataset, click the open detail pane icon (), and then click EDIT.

   To edit the schedule for the dataset, open the detail pane and click the Change link next to the schedule information.

   The Edit Dataset dialog opens.

2. Make the necessary changes.

   - If you change the Security permissions, it may take up to five minutes to take affect. Alternately, the user can update their roles and permissions more quickly by logging out of OpenText Core Data Discovery & Risk Insights and then logging back in.

     NOTE: If an associated user is currently viewing a workspace affected by the change, the user must navigate away from and back to the workspace to view the change.

   - If you change a path for a dataset that uses specific credentials and does not inherit from the source, you are required to re-enter the password for the defined user.

   - For datasets for which you define mailboxes or drives, you can add mailboxes, access accounts, site names, or UPNs as necessary. For these same datasets, you cannot remove a mailbox, access account, site name, or UPN if the dataset has been scanned and data is associated with the mailbox, access account, site name, or UPN.

   - To import the advanced capture rules from a template click LOAD FROM TEMPLATE on the Advanced Capture Rules page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the advanced capture rule options from the selected template are loaded to the Advanced Capture Rules page and override any options previously selected. Make any additional changes as desired.

   - To import the security options from a template, click LOAD FROM TEMPLATE on the Security page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the security options from the selected template are loaded to the Security page and override any options previously selected. Make any additional changes as desired.

   - To import the grammar sets from a template, click LOAD FROM TEMPLATE on the Grammar Sets page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the grammar sets from the selected template are loaded to the Grammar Sets page and override any options previously selected. Make any additional changes as desired.

   - If you change any selections on the Grammar Sets page and then click PREVIOUS, NEXT, or FINISH, a message displays informing you that the dataset will be reprocessed and the number of documents affected.

     - Click OK to acknowledge that the dataset will be reprocessed and continue or finish your changes to the dataset.

     - Click CANCEL to not move forward with the grammar changes and return to the Grammar Sets page. Undo the changes you made to the grammar sets or make additional changes as desired. If you make additional changes, you will see the message again when you click PREVIOUS, NEXT or FINISH. You cannot move forward until you acknowledge that the dataset will be reprocessed due to grammar set changes.

   - To import the schedule from a template, click LOAD FROM TEMPLATE on the Schedule page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the schedule options from the selected template are loaded to the Schedule page and override any options previously selected. Make any additional changes as desired.

   - To import the attributes from a template, click LOAD FROM TEMPLATE on the Attributes page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the attributes from the selected template are loaded to the Attributes page and override any options previously selected. Make any additional changes as desired.

3. Click FINISH.

   The dataset information is edited.

## Activate or deactivate a dataset

You can deactivate and then activate datasets as needed. A deactivated dataset cannot be processed. If the dataset was already processing data, no additional data is processed once the dataset is deactivated. Deactivated datasets cannot be edited either. Deactivated datasets display a gray icon next to the dataset name.

To activate or deactivate a dataset

1. On the Manage Datasets page, click or hover over the row for the desired dataset.

   Additional icons display in the right column.

   TIP: You can also click in the row for the desired dataset, open the detail pane () and then click ACTIVATE or DEACTIVATE.

2. Click the activate () or deactivate icon () associated with the desired dataset.

3. In the confirmation dialog, click YES to confirm the action.

   The dataset is activated or deactivated as appropriate.

## Remove connection to a dataset

You can remove the connection to a dataset ("delete" the dataset) if there are no workspaces associated with the dataset and no documents associated with the dataset are on hold. If the dataset has associated documents, you can deactivate the dataset but you cannot delete it.

To delete the connection to a dataset

1. On the Manage Datasets page, click or hover over the row for the desired dataset. Action icons display in the right column.

   Click the delete icon ().

   TIP: You can also click in the row for the desired dataset, open the detail pane () and then click DELETE.

2. In the confirmation dialog, click YES to confirm the action.

   The connection to the dataset is deleted.

Copyright 2019-2026 Open Text
Version 26.2.0 | Last updated April 2026
Connect
Marketplace
Learn
Support portal
Documentation
Contact
Submit or view service request