Fusion 24.1.0 Connect Help Center

# Manage datasets

In Fusion, a dataset defines the location of data on a source, the rules and schedule for processing, and the grammar rules to identify within the data during processing.

Once you have created at least one source, you can create as many datasets as necessary for that source. For example, you want to process data from only specific directories on a given file system. You would create a source for the file system and then create a dataset for each of the directories on that file system that contain data you want to process. This lets you focus the processing on only the desired data, omitting known irrelevant data.

On the Manage Datasets page, you can filter the list of datasets by dataset TYPE and choose whether to VIEW the list sorted by sources. The analysis information specifies the processing conducted for the dataset. The data count represents the number of emails or documents for unstructured data sources or the number of tables for structured data sources. When the count represents documents, the document count reflects the number of parent documents processed (extracted attachments are not included in the count). The size of each dataset represents the size on disk.

From the Manage Datasets page, you can view additional information about each dataset.

- View the processing options (analysis) initially selected for the dataset.

  Initial analysis options

  Analysis | Description
  --- | ---
  Metadata Scan | Indicates that only the document metadata has been analyzed and indexed for all documents in this dataset.
  n% SmartScan | Indicates that document metadata has been indexed and document metadata and body content has been analyzed to identify tag values for the defined percentage of documents in this dataset.
  Full Scan | Indicates that document metadata has been indexed and document metadata and body content has been analyzed to identify tag values for all documents in this dataset.
  C | Indicates that content has been stored for documents in this dataset. The original document itself was not originally stored.
  E | Indicates that grammar values have been extracted for documents in this dataset. Because the content has not been stored, you can see that grammar values exist, but you cannot see the document content in Analyze or Manage.
  CE | Indicates that the content has been stored and grammar values have been extracted for documents in this dataset.

- Click anywhere in the row for the desired dataset and then click the open detail panel icon () to display dataset details. The detail panel includes the options defined for the dataset as well as key information about the documents in the dataset.

  From the detail panel, you can edit, scan, activate/deactivate, and delete the dataset. For datasets with data, you can go to the data volume chart (), focused on the selected dataset. For file system and SharePoint datasets containing data, you can go to the sensitive data heat map () in Analyze, focused on the dataset.

  - Click the Change link next to the schedule information to open the Edit dataset dialog to the Schedule information.

  - If the grammar sets have been updated since the dataset was created, a warning icon () and a Re-analyze link display next to the Grammar Sets information. The grammar set applied to the dataset that has been updated (is "out of sync") displays in red text.

    IMPORTANT: Re-analyzing the dataset to update the grammar sets is optional. Changes to the grammar sets for this dataset are not automatically updated for the dataset. Re-analyzing a dataset for which the content has not been stored requires reprocessing the full dataset and may take additional time and impact server load.

  - On the METRICS tab, view metrics related to the selected dataset.

  - On the GRAMMARS tab, view the grammar types and grammar rules defined for the dataset.

  - On the ACTIVITY tab, view the details of the last 10 activities performed. If more than 10 activities have been performed, click the MORE link to see the full list for the dataset on the Agent Activity page.

  - On the SECURITY tab, view the security policy and associated users and groups that have been given specific access to the dataset.

To create a dataset

1. From the primary navigation panel, click Sources > Manage Datasets.

   The Manage Datasets page opens.

2. Click NEW DATASET.

   TIP: You can also hover over the row for the source for which you want to add a dataset. Action icons display in the right column. Click the new dataset icon ().

   The New Dataset dialog opens to the General page.

3. Complete the General options for the new dataset and then follow the dialog prompts for the remaining options.

   Option | Description
   --- | ---
   Dataset name | Type a meaningful, unique name for the dataset. Limits: Maximum 50 characters.
   Source | Select the source on which the dataset exists. If you had an existing dataset row selected when you clicked NEW DATASET, the source of the selected dataset is pre-selected, but can be changed as desired. TIP: To filter the list of available sources, start typing in the field instead of scrolling through the list. As you type, the list is filtered based on your entry.
   Dataset template | Select a dataset template on which to base the new dataset or select None to create the dataset from scratch. NOTE: This option defaults to the selection you made the last time you created a dataset. Make sure the desired option is selected. For more information about dataset templates, see Dataset templates.
   Description | (Optional) Type a meaningful description of the dataset. Limits: Maximum 250 characters.

   Click NEXT.

4. (Structured data datasets only) Select the tables to include in the dataset.

   1. In the list of available schemas, click the desired schema.

      The schema name displays in the SCHEMA row at the top. The list refreshes to show the available inventoried tables associated with the selected schema.

      NOTE: A table can be included in only one dataset. Tables that are included in other datasets do not display in the list of tables.

   2. In the list of available tables, click the checkboxes for the desired tables.

      The selected schema and table names display in the right panel.

   3. To select tables in another schema, click SCHEMA. The schemas display in the list. Click the desired schema and then click the checkboxes for the desired tables in that schema.

   4. To clear tables from the list of selected tables in the right panel, click CLEAR.

      - Click Clear all to clear the entire list of selected tables.

      - Click Clear for selected schema to clear all tables for the schema that currently displays in the SCHEMA row.

   5. Click NEXT.

5. Complete the Primary Capture Rules options to capture data from the selected dataset type.

   For File System datasets

   Option | Description
   --- | ---
   - - Sub-directory | Type the subpath to the directory you want to process data from. Type only the sub-directory path beneath the defined source. May be left blank to process items at the defined source level. Limits: The sub-directory path cannot contain any of the following special characters. `<` (less than) `>` (greater than) `:` (colon) `"` (double quote) `|` (vertical bar or pipe) `?` (question mark) `*` (asterisk) / (slash) The path cannot contain `.` or `..` before, after, or in between slashes (`\`) with no other characters. Not valid:
       `..\` | `.\abc` | `abc\..` | `abc\.\def`
     - Valid:
       `abc..` | `.abc` | `abc\def.` | `abc\.def\gh`
   Default action on files | Select one of the following actions to take when processing documents from this dataset. SmartScan (Analyzed). Specifies to index the metadata and to analyze the metadata and body content to identify tag values for a defined percentage of individual files. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. Full Scan (Analyzed). (Default) Specifies to index the metadata and to analyze the metadata and body content to identify tag values. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. NOTE: In this release, password protected PST files cannot be expanded and the contents processed. If encountered, the PST container is processed.
   Sampling Rate | Define the percentage of individual files in the datast to process, including index of metadata and body content. Use the slider to set the percentage, or type the desired percentage. Limits: Only available when SmartScan (Analyzed) is selected as the default action on files.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions and then type the desired extensions. As you type, suggestions display. Click the desired extension to add it to the list of extensions. To remove an extension, click the associated remove icon (). Select Exclude extensions to exclude all files with the defined file extension and then type the desired extensions without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit yoru entry. To remove an extension, click the associated remove icon (). When disabled (deselected) all items matching the primary capture rules are processed.
   - External metadata capture | Specify whether to identify and store metadata above and beyond the metadata that Fusion captures by default. The additional metadata is stored with the original document. Click the toggle to select () or deselect () the option. IMPORTANT: For requirements and additional information about external metadata file capture, review Process external metadata. When enabled (selected), additional metadata is identified and stored for individual files. Complete the following options.
     Option | Description
     --- | ---
     External metadata file processor | Type the name of the file (without the file extension) that will be used to identify the additional metadata and apply the information. This file must be a `.ps1` file and exist on the agent host machine at the location defined by the `ScriptBasePath` in the Agent Admin UI under Advanced Settings > Run Script.
     Dynamic | Select to scan each file and gather metadata according to the configuration defined by the external metadata file processor.
     From file | Select to identify metadata read from a metadata file and then saved with the corresponding original item. In External metadata file extensions, type the extensions of the files that contain the external metadata information. These files are not processed as part of the primary capture. Limits: do not include the leading dot (.). For example, type `idx` not `.idx`. For multiple file extensions, press Enter on your keyboard or type a comma (,) and a space after each entry. To remove an extension, click the associated remove icon ().
   When disabled, (deselected), only the original document metadata is processed.

   For Exchange and Exchange Online (O365) datasets

   IMPORTANT: Avoid creating datasets that define groups with users who span multiple groups.

   The Fusion group created as part of the Exchange connection tasks is used in this step.

   Option | Description
   --- | ---
   Default action will be taken on mailboxes from the following group | Select the Fusion group that will be acted upon based on the selection you make for Default action on mailboxes. This is a Fusion group created as part of the Exchange connection tasks. This option supports a single group.
   Default action on mailboxes | Select one of the following actions to take when processing documents from this dataset. Full Scan (Analyzed). Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage. The selected action is the default action on all mailboxes for the defined Fusion group. NOTE: In this release, password protected PST files cannot be expanded and the contents processed. If encountered, the PST container is processed.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   Limit by sent date | Specify whether to set a date range for files to be processed. The dates represent the date the item was sent. Click the toggle to select () or deselect () the option. When enabled (selected), items with a sent date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.

   For SharePoint datasets

   Option | Description
   --- | ---
   SharePoint URL type | Select whether this dataset is a Site collection or a Web application.
   - Site Sub-URL | Type the sub-URL relative to the base of the SharePoint site defined for the source. May be left blank to process items at the defined source level. Example site sub-URL: If the source SharePoint URL is `https://company.sharepoint.com`, type `/sites/team01` to connect to the SharePoint site at `https://company.sharepoint.com/sites/team01`. IMPORTANT: SharePoint subsites of this site are not processed as part of this site. To process from SharePoint site and its subsite, create another dataset and define the full sub-URL for the subsite. For example, create another dataset and type `/sites/team01/projectA` as the sub-URL to process data from the subsite of the "team01" site. Limits: The path cannot contain `.` or `..` before, after, or in between slashes (`\` or `/`) with no other characters. Not valid:
     `..\` | `.\abc` | `abc\..` | `abc\.\def`
   - Valid:
     `abc..` | `.abc` | `abc\def.` | `abc\.def\gh`
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. SmartScan (Analyzed). Specifies to process and index the metadata and the body content for a defined percentage of individual files. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage.
   Sampling Rate | Define the percentage of individual files in the dataset to process, including index of metadata and body content. Use the slider to set the percentage, or type the desired percentage. Limits: Only available when SmartScan is selected as the default action on files.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions and then type the desired extensions. As you type, suggestions display. Click the desired extension to add it to the list of extensions. To remove an extension, click the associated remove icon (). Select Exclude extensions to exclude all files with the defined file extension and then type the desired extensions without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). When disabled (deselected) all items matching the primary capture rules are processed.

   For SharePoint Online (O365) datasets

   Option | Description
   --- | ---
   Site Sub-URL | Type the sub-URL relative to the base of the SharePoint site defined for the source and start with a forward slash (/). May be left blank to process items at the defined source level. Example site sub-URL: If the source SharePoint URL is `https://company.sharepoint.com`, type `/sites/team01` to connect to the SharePoint site at `https://company.sharepoint.com/sites/team01`. IMPORTANT: SharePoint subsites of this site are not processed as part of this site. To process from SharePoint site and its subsite, create another dataset and define the full sub-URL for the subsite. For example, create another dataset type `/sites/team01/projectA` as the sub-URL to process data from the subsite of the "teams01" site.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. SmartScan (Analyzed). Specifies to process and index the metadata and the body content for a defined percentage of individual files. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage.
   Sampling Rate | Define the percentage of individual files in the dataset to process, including index of metadata and body content. Use the slider to set the percentage, or type the desired percentage. Limits: Only available when SmartScan is selected as the default action on files.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit maximum file size | Specify whether to set a maximum limit on files to be processed. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. Items larger than the defined file size will not be processed. The maximum file size that can be defined is 10000 MB (10 GB). When not enabled (deselected), all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions and then type the desired extensions. As you type, suggestions display. Click the desired extension to add it to the list of extensions. To remove an extension, click the associated remove icon (). Select Exclude extensions to exclude all files with the defined file extension and then type the desired extensions without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). When disabled (deselected) all items matching the primary capture rules are processed.

   For Content Manager datasets

   IMPORTANT: Avoid creating datasetswith the same record type as an existing dataset. Creating a dataset without defining record types results in ALL record types being processed, which in effect creates a duplicate selection if you create another dataset.

   Option | Description
   --- | ---
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   Capture additional metadata | Specify whether to include addition metadata fields when processing. Click the toggle to select () or deselect () the option. When enabled (selected), the Record Notes metadata field is also processed. CAUTION: Capturing this additional metadata will impact Content Manager scan performance. When not enabled (deselected), the Record Notes metadata field is not processed.
   Classifications | Type the name of the Content Manager classifications to be processed. Press Enter on your keyboard to commit the classification you entered. You can type the classification name exactly as it exists in Content Manager or specify a fully structured query using Content Manager's query syntax. Example using Content Manager query syntax: To process all records (documents) with the "Accommodation" classification or any of its sub classifications, define the classification field on the record and then the classification on the container (folder) that the record is in. Line breaks in the syntax are for display purposes only.
       classification:[title:"accommodation" or
       parent:[title:"accommodation"]+]
       or container:[classification:[title:"accommodation" or
       parent:[title:"accommodation"]+]]
   To remove a classification, click the associated remove icon ().
   Record types | Type the record types to be processed. To remove a record type, click the associated remove icon ().
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.

   For Google Drive datasets

   Option | Description
   --- | ---
   [Type] | Specify whether this dataset connects to a User Drive or a Shared Drive in your organization. TIP: If connecting to a User Drive, the dataset is associated with a single user's Google Drive.
   Access Account | For User Drive connections, type the email address of the user whose Google Drive this dataset connects to. For Shared Drive connections, type the email address of an account (member) that has access to the shared drive you will define.
   Scan all drives this account is a member of | Specifies whether to process all Google Drives the defined user is a member of. Limits: Applies only when Shared Drive is selected.
   Scan a specific shared drive | Specifies to process only the defined Google Drive the defined user is a member of. Limits: Applies only when Shared Drive is selected.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions and then type the desired extensions. As you type, suggestions display. Click the desired extension to add it to the list of extensions. To remove an extension, click the associated remove icon (). Select Exclude extensions to exclude all files with the defined file extension and then type the desired extensions without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). When disabled (deselected) all items matching the primary capture rules are processed.

   For Documentum datasets

   Option | Description
   --- | ---
   Repository | Type the name of the Documentum repository you want to process data from. NOTE: The Documentum repository must be unique across datasets within the source.
   DQL | Type the DQL query that defines the field and value to query against. Leave blank to process all data. Example: To process items that are one of three specific Documentum types, type `title='ITdoc' or title='HRdoc' or title='FinDoc'`.
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   Limit by modified date | Specify whether to set a date range for files to be processed. The dates represent the last time the items were modified. Click the toggle to select () or deselect () the option. When enabled (selected), items with a last modified date within, and including, the defined dates are processed. When not enabled, (deselected) all items matching the primary capture rules are processed.
   Limit file extensions | Specify whether to process items based on file extension. Click the toggle to select () or deselect () the option. When enabled (selected), items with the defined file extensions are included or excluded from processing as specified. Select Include extensions to process only files with the defined file extensions and then type the desired extensions. As you type, suggestions display. Click the desired extension to add it to the list of extensions. To remove an extension, click the associated remove icon (). Select Exclude extensions to exclude all files with the defined file extension and then type the desired extensions without the leading dot (.) and then press Enter on your keyboard. Type additional extensions as needed, pressing Enter on your keyboard after each extension to commit your entry. To remove an extension, click the associated remove icon (). When disabled (deselected) all items matching the primary capture rules are processed.

   For Extended ECM datasets

   NOTE: You can have only one Extended ECM dataset per source.

   Option | Description
   --- | ---
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.

   For unstructured data datasets based on custom adapters

   The Dataset Options defined when you created the custom adapter are included in the Primary Capture Rules. For more information about custom adapters, see Custom adapters.

   Option | Description
   --- | ---
   Default action on files | Select one of the following actions to take when processing documents from this dataset. Full Scan (Analyzed). (Default) Specifies to process and index the metadata and the body content of individual files. You can preview the document content in a plain text view in Analyze and Manage, but you cannot preview any attachments or embedded images. Full Scan (Metadata only). Specifies to process and index only the metadata from individual files. You cannot preview the document content in Analyze or Manage.
   Store content as text | Specify whether to store the contents of the individual files as text; the file itself is not stored. Click the toggle to select () or deselect () the option. When enabled (selected), the contents of the files are stored in the index as text. When not enabled (deselected), tag values are identified but the file content cannot be viewed or re-analyzed. Limits: Only available when the selected default action includes "Analyzed". NOTE: Changes to grammar, tag and workspace keyword search criteria will not be automatically updated for files that have already been analyzed. Storing file content increases the size of your index.
   Extract grammar values | Specify whether to extract, index, and count the values identifies by the grammar rules. Click the toggle to select () or deselect () the option. When enabled (selected), grammar values are identified, extracted, and indexed. When not enabled (deselected), grammar values are not extracted and masked view of file content cannot be shown. Limits: Only available when the selected default action includes "Analyzed". TIP: You can extract grammar values later for files within a workbook.
   [Dataset Options] | Type the desired value for the dataset options for the custom adapter. Hover over an information icon () to review the description for the associated option.

   For structured data (database) datasets

   NOTE: The maximum number of rows that can be scanned is 999,999,999 (< 1 billion) rows per table.

   Option | Description
   --- | ---
   Default action | Select one of the following actions to take when processing table rows in this dataset. Full Scan (Analyzed). Specifies to process all rows in the selected tables SmartScan (Analyzed). (Default) Specifies to process a defined percentage of rows in the selected tables.
   Sampling Rate | Define the percentage of rows to process. Use the slider to set the percentage, or type the desired percentage. Limits: Only available when SmartScan is selected as the default action.
   Number of rows to sample | Type the Minimum and Maximum number of rows to sample. NOTE: If the percentage of rows to sample results in fewer rows than defined by the minimum number of rows to sample, the defined minimum number of rows are scanned. Limits: Only available when SmartScan is selected as the default action.

   Click NEXT.

6. Complete the Credential options for accessing the new dataset.

   NOTE: Exchange and Exchange datasets use the credentials from the parent source. Datasets for Content Manager, Google Drive, custom adapter unstructured sources, and structured data do not require credentials. This step is not included for datasets for these sources.

   For File System datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. Select Specify Credentials to define credentials other than that defined for the parent source.
   Username | Define the username for the user that has access to the dataset. If inherited from the source, defaults to the username defined for the source and cannot be edited. If different than the source, defaults to the username defined for the source but can be edited. Type the username of the user that has access to the defined dataset directory. Limits: use the format `domain\user` or `user@domain`
   Password | Define the password for the user that has access to the dataset. If inherited from the source, defaults to the password defined for the source and cannot be edited. If different than the source, type the password for user defined.

   For SharePoint datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. Select Specify Credentials to define credentials other than that defined for the source.
   Username | Define the username for the user that has access to the dataset. If inherited from the source, defaults to the username defined for the source and cannot be edited. If different than the source, defaults to the username defined for the source but can be edited. Type the user information of the SharePoint service user that has access to the SharePoint directory you want to process data from. Limits: use the format `domain\user` or `machine_name\user`
   Password | Define the password for the user that has access to the dataset. If inherited from the source, defaults to the password defined for the source and cannot be edited. If different than the source, type the password for user defined.

   For SharePoint Online (O365) datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. Select Specify Credentials to define credentials other than that defined for the source.
   Application ID | Define the application ID for the dataset. If inherited from the source, defaults to the Application ID defined for the source and cannot be edited. If different than the source, defaults to the Application ID defined for the source but can be edited. Type the value for the appropriate Application (client) ID.
   Client Secret | Define the client secret for the dataset. If inherited from the source, defaults to the Client Secret defined for the source but cannot be edited. If different than the source, defaults to the Client Secret defined for the source and can be edited. Type the value for the Client Secret that corresponds to the defined Application (client) ID.

   For Documentum datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. Select Specify Credentials to define credentials other than that defined for the source.
   Username | Define the username for the user that has access to the dataset. If inherited from the source, defaults to the username defined for the source but cannot be edited. If different than the source, defaults to the username defined for the source and can be edited. Type the username of the user that has access to the defined dataset directory.
   Password | Define the password for the user that has access to the dataset. If inherited from the source, defaults to the password defined for the source and cannot be edited. If different than the source, type the password for user defined.

   For Extended ECM datasets

   Option | Description
   --- | ---
   [credential input type] | Select how credentials are input. Select Inherit From Source to use the credentials defined for the source. Select Specify Credentials to define credentials other than that defined for the source.
   Username | Define the username for the user that has access to the dataset. If inherited from the source, defaults to the username defined for the source but cannot be edited. If different than the source, defaults to the username defined for the source and can be edited. Type the username of the user that has access to the defined dataset directory.
   Password | Define the password for the user that has access to the dataset. If inherited from the source, defaults to the password defined for the source and cannot be edited. If different than the source, type the password for user defined.

   Click NEXT.

7. Complete the OCR Settings options for documents originating from the new dataset.

   NOTE: OCR options are not available for structured datasets or unstructured datasets associated with agent clusters that are not enabled for OCR. If the agent cluster associated with this dataset does not have OCR enabled, this step is not included.

   Option | Description
   --- | ---
   [Processing type] | Select how OCR processing is conducted on documents originating from this dataset. Disabled. Specifies to not take any OCR action on documents originating from this dataset. If OCR is requested for a workbook, OCR will be skipped for documents originating from this dataset. On Demand. Specifies to allow OCR action on documents originating from this dataset when requested for a workbook within a workspace that allows OCR. On Ingest. Specifies to perform OCR on documents originating from this dataset upon initial processing as defined by the primary capture rules. Limits: On Ingest is not available for datasets for Exchange, Exchange Online, SharePoint, or SharePoint Online sources. NOTE: Once the dataset is created and documents processed, OCR options cannot be edited.
   Configuration | If not using the included configuration file, type the name of the Media Server configuration file to be used for OCR task analysis. If left blank, the default configuration file is used. Unless otherwise defined, the default configuration file is `OCR_Only.cfg`. This configuration performs basic OCR. If defining a custom configuration file, the custom configuration file must exist on the agent host machine in the defined location, typically under `\Config` at the root of the agent installation directory. If the configuration file is within a subfolder in the defined location, include the subfolder in your entry - the entry is relative to the defined location. Examples: If the custom configuration file is `myConfig.cfg` and is placed within `\Config`, type myConfig.cfg. If the custom configuration file is `myConfig.cfg` and is placed within `\Config\myFolder`, type myFolder\myConfig.cfg.
   Limit maximum file size | Specify whether to limit OCR processing based on a maximum file size. Click the toggle to select () or deselect () the option. When enabled (selected), enter a maximum file size, in MB, that will be processed. OCR will not be processed on documents larger than the defined file size. The maximum file size that can be defined is 100000 MB. When not enabled (deselected), OCR is performed on all documents matching the primary capture rules and the remaining OCR options.
   Limit file extensions | Specify whether to perform OCR on documents based on file extension. Click the toggle to select () or deselect () the option. When disabled (deselected), OCR is performed on items with file extensions of `BMP`, `JPEG`, `JPG`, `PNG`, `TIF`, and `TIFF`. When enabled (selected), items with the defined file extensions are included in OCR processing. In the file extension field, type the desired file extensions to process. OCR will only be performed on items with the defined file extensions . As you type, suggestions display for supported file types. Click the desired extension to add it to the list of extensions. The first 10 supported file types display in a scrollable list. Supported extensions include `BMP` (compressed `BMP` files are not supported), `DOC`, `DOCX`, `GIF` (only applied to the first frame of an animated `GIF`), `ICO`, `JPEG`, `JPG`, `ODP`, `ODS`, `ODT`, `PBM`, `PDF`, `PGM`, `PPM`, `PNG`, `PPT`, `PPTX`, `RTF`, `TIF`, `TIFF`, `WEBP`, `XLS`, and `XLSX`. To remove an extension, click the associated .

   Click NEXT.

8. Complete the Security options to define whether you want to limit access to the dataset to specific users and groups.

   CAUTION: If limiting access to a source and an underlying dataset, users without access will not be able to view workspaces with a data source that includes the dataset or view individual items that originated in the dataset.

   Option | Description
   --- | ---
   Inherit from Source | Select to inherit dataset access from the source. (default)
   Grant access to all users | Select to not limit access to this dataset.
   Specify the users and groups that will have access | Select to limit access to the dataset to only the defined users and groups.
   List of Users/Groups | Define the users and groups that will have access to items originating from this dataset. In the Enter name or email address box, begin typing a group name or a name or email address of a user. As you enter a string in the field, the interface displays names or email addresses matching the string. Click Add to add the selected user or group to the source access list. To remove a user or group from the source access, hover over the name in the User/Group column and then click the corresponding remove icon ().

9. Select the Grammar Sets to include in the analysis of documents in this dataset.

   1. In the Grammar Sets list, do one of the following:

      - Click the desired grammar set name.

      - Type the name of the desired grammar set. As you type, grammar set names that match your entry display. Click the desired grammar set name.

      The grammar set is added to the top of the field. To remove a grammar set, click the associated .

   2. Repeat as necessary to add more grammar sets.

   3. Click outside of the Grammar Sets field to view the grammar rules for the selected grammar sets.

   4. Click NEXT.

10. Complete the Schedule options to define the schedule for processing this dataset.

    NOTE: Exchange datasets do not include Schedule options.

    Option | Description
    --- | ---
    Enable scheduling | Specify whether to enable or disable scheduling. Click the toggle to select () or deselect () the option. When enabled (selected), processing of this dataset occurs as defined by the default schedule set by the associated agent cluster. When not enabled (deselected), processing does not run on a schedule. You must manually start processing for this dataset.
    Override default schedule | Specify whether to override the default schedule defined by the associated agent cluster. When enabled (selected), the selections you make in the schedule on this page override the default schedule defined by the associated agent cluster. When not enabled (deselected), the default schedule for the associated agent cluster takes precedence.
    Only run during this time range | Define the details for when the scan runs. Select the days of the week that the scan will run. Define the Daily Start Time (GMT) and Daily End Time (GMT) for the scan. The time defined is relative to server on which you will install the agent. At the End Time, any task currently running is stopped; items in mid-processing are allowed to complete. Specify to Run once every day or Run continuously with delay interval. If running continuously with a delay interval, the task runs repeatedly during the defined days and times and pauses between runs for the defined interval (runs every n minutes.) Limits: This option does not show if you selected "Cloud Cluster" as the agent cluster.
    Blackout period (never run during this time range) | (Optional) Define the details for when no new scans will run. Scans already in progress will continue until completed. Select the days of the week that the scan will not run. Define the Daily start time (GMT) and Daily end time (GMT) for the scan. The time defined is relative to the server on which you will install the agent.

    Click NEXT.

11. Complete the Attributes options to associate the desired attributes and values with this dataset. For more information about dataset attributes, see Dataset attributes.

    1. From the Select attribute list, do one of the following:

       - Select the desired attribute from the list.

       - To create a new attribute, click the New Attribute link in the list.

         In the New Attribute dialog, type a name for the new attribute and then click CREATE.

       The list of values for the selected attribute populate the value list.

       NOTE: Any attributes you create here are automatically saved even if you cancel out of creating the dataset.

    2. From the Select value list, do one of the following:

       - Select the desired value from the list.

       - If the selected attribute does not currently have any values or to create a new value, click the Add Value link in the list

         In the New Value dialog, type a name for the new value and then click CREATE.

       NOTE: Any values you create here are automatically saved even if you cancel out of creating the dataste.

    3. To add more attributes, click the add icon () and then select the desired attribute and corresponding value.

       Repeat until all desired attribute values are selected.

       To remove a selected attribute and value pair, click the associated remove icon ().

    1. Click NEXT.

12. Click FINISH.

    The new dataset is created.

    NOTE: For file system and SharePoint datasets, initialization begins to identify potential duplicate items as a result of datasets with overlapping paths. Once fully initialized, the status is updated in the dataset detail panel and the selected default file action begins.

Datasets are not automatically scanned when the dataset is created. If you defined a schedule, the dataset scan will run according to that schedule. If a schedule was not defined for the dataset, you must initiate a scan of the dataset to process the data. Wait for the dataset initialization to complete before initiating a scan of a newly created dataset.

If you need to process a dataset outside of the scheduled run time, you can manually start a scan of the dataset. If you request to scan a dataset and the dataset is currently processing, the scan request is not acted upon.

To scan a dataset to initiate processing

1. On the Manage Datasets page, click or hover over the row for the desired dataset.

   Additional icons display in the right column.

2. Click the scan icon ( ) associated with the desired dataset.

   TIP: You can also click in the row for the desired dataset, open the detail panel () and then click SCAN.

3. In the confirmation dialog, click YES to confirm the action.

   A message confirming the scan request. Click the Agent Activity link in the message to go to the Agent Activity page to review the progress.

   If processing is not currently running, the scan is started.

   NOTE: For file system and SharePoint dataset, initialization begins to identify potential duplicate items as a result of datasets with overlapping paths. Once fully initialized, the status is updated in the dataset detail panel and the scan action begins.

When editing a dataset, keep the following in mind.

- If items exist in this dataset, some options may be dimmed and cannot be edited.

- If items exist in this dataset, you cannot change the agent cluster type from a cloud-based cluster to an on-premises cluster, or from an on-premises cluster to a cloud-based cluster.

- If you change any of the credentials, you are required to re-enter the password for the defined user.

- You can load the schedule, attribute, and grammar options from a template, even if the dataset is based on a different template. This action overrides any existing schedule, attribute, or grammar options and can be refined further as needed.

- Changes to existing grammar selections trigger a reprocessing of the dataset.

  If you have datasets that include grammars applied before the introduction of grammar sets, you will see the originally applied grammars in the list of selected grammars. If you select grammar sets, the originally applied grammars are overridden.

CAUTION: Do not change the dataset location—File System directory, SharePoint site URL, or Content Manager dataset—unless the location has actually changed. The physical location (or Exchange group name) must have changed (or be changed) prior to updating the location in this dataset.

Do not change an Exchange group name unless the name of the group has changed. The group name in Exchange must be changed prior to updating the group name in this dataset. Changes to group names may affect tracking of delete activities.

If you want to create a dataset with the same definitions of the dataset you are editing, create a new dataset.

To edit a dataset

1. On the Manage Datasets page, click the name of the dataset you want to edit.

   TIP: You can also do one of the following:

   - Click or hover over the row for the desired dataset and then click the edit icon ().

   - Click the row for the desired dataset, click the open detail panel icon (), and then click EDIT.

   To edit the schedule for the dataset, open the detail panel and click the Change link next to the schedule information.

   The Edit Dataset dialog opens.

2. Make the necessary changes.

   - To import the security options from a template, click LOAD FROM TEMPLATE on the Security page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the security options from the selected template are loaded to the Security page and override any options previously selected. Make any additional changes as desired.

   - To import the grammar sets from a template, click LOAD FROM TEMPLATE on the Grammar Sets page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the grammar sets from the selected template are loaded to the Grammar Sets page and override any options previously selected. Make any additional changes as desired.

     If you change any selections on the Grammar Sets page and then click PREVIOUS, NEXT, or FINISH, a message displays informing you that the dataset will be reprocessed and the number of documents affected.

     - Click OK to acknowledge that the dataset will be reprocessed and continue or finish your changes to the dataset.

     - Click CANCEL to not move forward with the grammar changes and return to the Grammar Sets page. Undo the changes you made to the grammar sets or make additional changes as desired. If you make additional changes, you will see the message again when you click PREVIOUS, NEXT or FINISH. You cannot move forward until you acknowledge that the dataset will be reprocessed due to grammar set changes.

   - To import the schedule from a template, click LOAD FROM TEMPLATE on the Schedule page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the schedule options from the selected template are loaded to the Schedule page and override any options previously selected. Make any additional changes as desired.

   - To import the attributes from a template, click LOAD FROM TEMPLATE on the Attributes page.

     In the resulting dialog, select the desired dataset template and then click OK. Only the attributes from the selected template are loaded to the Attributes page and override any options previously selected. Make any additional changes as desired.

3. Click FINISH.

   The dataset information is edited.

You can deactivate and then activate datasets as needed. A deactivated dataset cannot be processed. If the dataset was already processing data, no additional data is processed once the dataset is deactivated. Deactivated datasets cannot be edited either. Deactivated datasets display a gray icon next to the dataset name.

To activate or deactivate a dataset

1. On the Manage Datasets page, click or hover over the row for the desired dataset.

   Additional icons display in the right column.

   TIP: You can also click in the row for the desired dataset, open the detail panel () and then click ACTIVATE or DEACTIVATE.

2. Click the activate () or deactivate icon () associated with the desired dataset.

3. In the confirmation dialog, click YES to confirm the action.

   The dataset is activated or deactivated as appropriate.

You can remove the connection to a dataset ("delete" the dataset) if there are no workspaces associated with the dataset and no documents associated with the dataset are on hold. If the dataset has associated documents, you can deactivate the dataset but you cannot delete it.

To delete the connection to a dataset

1. On the Manage Datasets page, click or hover over the row for the desired dataset. Action icons display in the right column.

   Click the delete icon ().

   TIP: You can also click in the row for the desired dataset, open the detail panel () and then click DELETE.

2. In the confirmation dialog, click YES to confirm the action.

   The connection to the dataset is deleted.

Copyright 2019-2024 Open Text
Version 24.1.0 | Last updated December 2023
Connect
Marketplace
Learn
Support portal
Documentation
Contact
Submit or view service request