Data Discovery & Risk Insights 25.2.0 Help Center

SharePoint connection

If you will be creating sources and datasets that process SharePoint or SharePoint Online (O365) data, you must complete additional tasks to enable processing by the processing agent.

Complete the tasks for SharePoint or SharePoint Online as appropriate for the data to be managed by OpenText Core Data Discovery & Risk Insights. If you will process data by both SharePoint and SharePoint Online, complete the tasks for both implementations.

For the SharePoint version supported, see the supported data resources under Agent installation and configuration.

SharePoint connection tasks

Complete the following to process data from SharePoint.

Create SharePoint service user
OpenText Core Data Discovery & Risk Insights performs processes in SharePoint through a SharePoint service user. The SharePoint service user is used by the processing agent to connect to and perform actions in your SharePoint environment related to processing data. A SharePoint service user is also necessary to send data managed by OpenText Core Data Discovery & Risk Insights to a defined location in SharePoint.

The exact level of permissions required depends on the actions you will be performing through OpenText Core Data Discovery & Risk Insights.
- To process and collect items from SharePoint source, the SharePoint service user must have a minimum permission of Reading.
- To delete items from a SharePoint source, the SharePoint service user must have a minimum permission of Collaboration.
- To process, collect, and delete items from OneDrive, the SharePoint service user must be added to the Site Collection Administrators for the OneDrive site to be processed.
- To send items to a SharePoint target, the SharePoint service user must have a minimum permission of Collaboration.
When creating a source, you will define the credentials for the SharePoint service user that will be used to process data (read, collect, delete). When creating a target, you will define the credentials for the SharePoint service user that will be used to send data to SharePoint. For auditing in SharePoint, you can choose to create two separate service users—one for processing and one for sending to targets.
Update the logon account for OpenText Core Data Discovery & Risk Insights services

SharePoint Online connection tasks

OpenText Core Data Discovery & Risk Insights requires non-user based access to SharePoint Online using either the Microsoft Entra ID access method (recommended) or the SharePoint app-only access method.

CAUTION: Microsoft is ending support for the SharePoint app-only authentication method. This authentication method will stop working for new SharePoint tenants as of November 1st, 2024; it will stop working for existing tenants and be fully retired as of April 2nd, 2026.

For more information, see Azure ACS retirement in Microsoft 365.

To connect to SharePoint Online (O365) using the Microsoft Entra ID method

Prepare credentials for certificate based authorization. The following must be available.
1. A pkcs12 keystore (.p12 or .pks) containing a single certificate and corresponding private key. The certificate must at least be issued with a maximum expiration of 2 years, and bear the Client Authentication Enhanced Key Usage and Application policy extensions.
2. The public certificate (.pem or .cer) matching the above keystore.
If you do not yet have files that meet the above criteria, create a PowerShell script with the following content. It may then be used to assist in generating self-signed files that are valid for this use.

NOTE: The PowerShell to create the pkcs12 keystore must be generated and run on a Windows Server 2019 or later.

CAUTION: When creating the script, the -TextExtension line must not contain line breaks. Any breaks in the -TextExtension line in the sample below are a result of line wrapping for viewing.

If you copy and paste the contents below into your file, you must manually remove line breaks in the -TextExtension line.
```
# Variables to be used

[string] $certName="<certificateName>"
```
```
#Example: cddri-spo-access-rw-20240530
```
```
[string] $requesterEmail="<requesterEmailAddress>"
```
```
#Example: johnDoe.admin@company.com
```
```
[string] $p12PW="<password>"
```
```
[string] $outputDir="<localPath>\"
```
```
# Example: c:\certificates
```
```
# Create local self-signed client cert, storage in current user's certs.
```
```
# -------------------------
```
```
$Certificate=New-SelfSignedCertificate -Type Custom `
```
```
-Subject "E=$requesterEmail,CN=$certName" `
```
```
-KeyUsage DigitalSignature `
```
```
-KeyAlgorithm RSA -KeyLength 2048 `
```
```
-KeyExportPolicy Exportable `
```
```
-NotAfter (Get-Date).AddYears(2) -HashAlgorithm sha256 `
```
```
-CertStoreLocation Cert:\CurrentUser\My `
```
```
-TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.2","1.3.6.1.4.1.311.21.10={text}OID=1.3.6.1.5.5.7.3.2")
```
```
 
```
```
# Export public side of above certificate
```
```
# -------------------------
```
```
Export-Certificate -Cert $Certificate -FilePath "$outputDir$certName.der"
```
```
certutil -encode "$outputDir$certName.der" "$outputDir$certName.pem.cer"
```
```
# Export Private side pkcs12 keystore of above certificate
```
```
# -------------------------
```
```
$mypwd = ConvertTo-SecureString -String $p12PW -Force -AsPlainText
```
```
Export-PfxCertificate -Cert $Certificate `
```
```
-FilePath "$outputDir$certName.p12" `
```
```
-Password $mypwd
```
where
- <certificateName> is the file name for the certificate file that will be generated. Do not include the file extension.
- <requesterEmailAddress> is the email address of the person requesting the certificate and keystore.
- <password> is the password required to access the certificate.
- <localPath> is the location on the local machine where the certificate and pkcs12 keystore are generated.
Open a browser and connect to the Microsoft Entra admin center for your organization as an account with appropriate privileges.

Create a new app registration.

In the left pane, expand Identity > Applications, and then click App registrations.

Click New registration and complete the options as follows.

Option	Description
Name	Type FAS-DD_SharePoint_Collection as the application name.
Support account types	Select Accounts in this organizational directory only.
Redirect URI	make no selections or entries

Click Register. The overview page for the new app registration displays.

Take note of the resulting Application (client) ID and Directory (tenant) ID values. These will be required when defining the source in the Connect UI.

Set permissions for the new app registration.
1. In the left pane of the new application registration page, expand Manage and then click API permissions.
  
  Click Add a permission.
  1. In the Request API permissions pane, click Microsoft Graph.
  2. For the permission type, click Delegated permissions.
  3. In the list of permissions, expand Users and select the check box for User.Read.
  4. Click Add Permissions.
2. On the API permissions page, click Add a permission.
  1. In the Request API permissions pane, click Microsoft Graph.
  2. For the permission type, click Application permissions.
  3. In the list of permissions, expand Sites and do one of the following.
    - To process and collect data items from SharePoint sources, select the check box for Sites.Read.All.
    - To process and collect data items from SharePoint sources and send data items to SharePoint targets, select the check box for Sites.Manage.All.
  4. Click Add permissions.
3. On the API permissions page, click Add a permission.
  1. In the Request API permissions pane, click SharePoint.
  2. For the permission type, click Application permissions.
  3. In the list of permissions, expand Sites and do one of the following.
    - To process and collect data items from SharePoint sources, select the check box for Sites.Read.All.
    - To process and collect data items from SharePoint sources and send data items to SharePoint targets, select the check box for Sites.Manage.All.
  4. Click Add permissions.
Define the access credentials.
1. In the left pane of the new application registration page, expand Manage and then click Certificates & secrets.
2. Click Upload certificate.
3. In the Upload certificate panel, click the folder selection button. Navigate to and select the certificate (public key, .cer) file from of step 1.

When you create SharePoint Online (O365) sources in OpenText Core Data Discovery & Risk Insights, you will need to provide the Client Id (listed in OpenText Core Data Discovery & Risk Insights as Application ID), the pkcs12 keystore file, and the password for the keystore.

To connect to SharePoint Online (O365) using the SharePoint app-only method

Create or identify an O365 user with, at a minimum, "SharePoint Administrator" permission at the O365 level. OpenText Core Data Discovery & Risk Insights requires this user in order to access your SharePoint Online environment.

You must perform the remainder of these tasks as this user.

Create a “SharePoint app-only” definition at either the SharePoint Online tenant level or at the SharePoint site level.

To create a “SharePoint app-only” definition at the SharePoint Online tenant level

Open https://<O365Tenant>.sharepoint.com/_layouts/15/appregnew.aspx, where <O365Tenant> is your O365 tenant name.

Complete the App Information options.

Option	Description
Client Id	Click the Generate button. IMPORTANT: Make note of the generated Client Id. This is needed later.
Client Secret	Click the Generate button. IMPORTANT: Make note of the generated Client Secret. This is needed later.
Title	Type a descriptive title for the App-Only principal.
App Domain	Type a placeholder domain URL, such as `www.localhost.com`.
App Redirect URL	Type a placeholder redirect URL, such as `https://www.localhost.com/`.

Click Create.

To create a “SharePoint app-only” definition at the SharePoint site level

The user performing this must be an admin of the specific SharePoint site.

Open https://<O365Tenant>.sharepoint.com/sites/<site>/_layouts/15/appregnew.aspx where,
- <O365Tenant> is your O365 tenant name.
- <site> is the SharePoint site name.

Complete the App Information options.

Option	Description
Client Id	Click the Generate button. IMPORTANT: Make note of the generated Client Id value. This is needed later.
Client Secret	Click the Generate button. IMPORTANT: Make note of the generated Client Secret value. This is needed later.
Title	Type a descriptive title for the App-Only principal.
App Domain	Type a placeholder domain URL, such as `www.localhost.com`.
App Redirect URL	Type a placeholder redirect URL, such as `https://www.localhost.com/`.

Click Create.

Grant permission to the created “SharePoint app-only” definition at either the SharePoint Online tenant level or at the SharePoint site level, as appropriate for the definition created.
- To grant permission to the definition at the SharePoint Online tenant level
  1. Open https://<O365Tenant>-admin.sharepoint.com/_layouts/15/appinv.aspx, where <O365Tenant> is your O365 tenant name.
  2. On the App Information page, type the generated Client Id value in the App Id field.
    
    The Title, App Domain, and Redirect URL fields populate based on the Client Id defined.
  3. Type a valid Permission Request XML.
    
    Syntax
    <AppPermissionRequests AllowAppOnlyPolicy="true">
    <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="<ReadOrManage>"/>
    </AppPermissionRequests>
    
    To allow OpenText Core Data Discovery & Risk Insights to collect from a SharePoint Online source, the Read right is required.
    
    To allow OpenText Core Data Discovery & Risk Insights to send to a SharePoint Online target, the Manage right is required.
    
    OpenText Core Data Discovery & Risk Insights creates a new SharePoint list for output. This requirement correlates with the regular SharePoint user permission “designer”.
    
    To allow OpenText Core Data Discovery & Risk Insights collect from SharePoint Online and send to a SharePoint target, the Manage right is required.
    
    Example: allow collect and send to target
    <AppPermissionRequests AllowAppOnlyPolicy="true">
    <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="Manage"/>
    </AppPermissionRequests>
  4. Click Create.
    
    In the consent confirmation dialog, click Trust It to grant permission.
- To grant permission to the definition at the SharePoint site level
  1. Open https://<O365Tenant>.sharepoint.com/sites/<site>/_layouts/15/appinv.aspx where,
    
    <O365Tenant> is your O365 tenant name.
    
    <site> is the SharePoint site name.
  2. On the App Information page, type the generated Client Id value in the App Id field.
    
    The Title, App Domain, and Redirect URL fields populate based on the Client Id defined.
  3. Type a valid Permission Request XML.
    
    Syntax
    <AppPermissionRequests AllowAppOnlyPolicy="true">
    <AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="<ReadOrManage>"/>
    </AppPermissionRequests>
    
    To allow OpenText Core Data Discovery & Risk Insights to collect from a SharePoint Online source, the Read right is required.
    
    To allow OpenText Core Data Discovery & Risk Insights to send to a SharePoint Online target, the Manage right is required.
    
    OpenText Core Data Discovery & Risk Insights creates a new SharePoint list for output. This requirement correlates with the regular SharePoint user permission “designer”.
    
    To allow OpenText Core Data Discovery & Risk Insights collect from SharePoint Online and send to a SharePoint target, the Manage right is required.
    
    Example: allow collect and send to target
    <AppPermissionRequests AllowAppOnlyPolicy="true">
    <AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Manage"/>
    </AppPermissionRequests>
  4. Click Create.
    
    In the consent confirmation dialog, click Trust It to grant permission.
  NOTE: If you want to process data from multiple sites, create a permission grant for each site.

Configure web proxy settings (optional)

The SharePoint processor service controlled by the processing agent requires connectivity to the OpenText Core Data Discovery & Risk Insights cloud components, often located away from the local network where the agent host servers are located. Although direct connectivity is ideal, use of a web proxy may be required in some environments for the agent systems to reach the OpenText Core Data Discovery & Risk Insights cloud.

NOTE: Authenticated proxies are supported for SharePoint Online (O365) only.

To configure the web proxy for the SharePoint processor

On the machine hosting the OpenText Core Data Discovery & Risk Insights processing agent, log on to the agent administration UI.

From the Start menu, click OpenText Core Data Discovery & Risk Insights Agent > Agent Admin.
In the navigation panel, click Advanced Settings.

The Advanced Settings page opens.
In the Category list, click SharePoint Processor.

Complete the following options.

Option	Description
Proxy address URL	Type the URL of the web proxy.
Proxy bypass list	Type a comma separated list of addresses that do not use the proxy server..
Proxy Bypass on Local	Specify whether to bypass the proxy for local addresses. Select True to bypass for local addresses. Select False to not bypass for local addresses.
Proxy Password	If the proxy requires authentication, type the password for the user account that will access the proxy server. IMPORTANT: Authenticated proxies are only supported for SharePoint Online (O365) resources. If not connecting to SharePoint Online (O365), leave this field blank.
Proxy Username	If the proxy requires authentication, type the username for the user account that will access the proxy server. IMPORTANT: Authenticated proxies are only supported for SharePoint Online (O365) resources. If not connecting to SharePoint Online (O365), leave this field blank.

Click Save. You can close the agent administration UI.

SharePoint processing

When processing Microsoft Office items from SharePoint, OpenText Core Data Discovery & Risk Insights uses the last modified date in the de-duplication calculation. Due to the way modified dates are handled in Office items and in SharePoint (including OneDrive), OpenText Core Data Discovery & Risk Insights will not identify documents with different dates as duplicates.

When an Office item is uploaded to a SharePoint or OneDrive site web interface, the item's modified date is changed.
When an Office item is added to a local system and is then synchronized to the SharePoint site, the item's modified date is not changed.

SharePoint Lists are comprised of form records, called items in SharePoint, that contain various text fields and can have attachments. When deleting SharePoint files, OpenText Core Data Discovery & Risk Insights does not delete attachments to items from SharePoint Lists.

SharePoint item counts

The document and item counts in OpenText Core Data Discovery & Risk Insights may differ from the "item" count as seen in the SharePoint site interface. This difference relates to the following.

In OpenText Core Data Discovery & Risk Insights, a document is an original file processed by OpenText Core Data Discovery & Risk Insights and an item is an attachment to an original file. In SharePoint, an item is a row in a table, or a record in a database and a document is a type of item.
In SharePoint, item counts are derived from the total number folders, documents, and items (each entry in a SharePoint Item List). In OpenText Core Data Discovery & Risk Insights, document counts are derived from the total number of documents from SharePoint Document Libraries and attachments in a SharePoint Item List. OpenText Core Data Discovery & Risk Insights does not process the field, or entry, in an Item List, only the attachments from the Item List.

For example, if SharePoint Item has zero attachments, SharePoint records this as one item. If a SharePoint Item has 10 attachments, SharePoint also records this as one item.
The item count listed on the SharePoint Site Contents page for libraries includes all items in the library, including folders. Folders in which files exist in SharePoint are not included in counts in OpenText Core Data Discovery & Risk Insights.
When processing SharePoint content, OpenText Core Data Discovery & Risk Insights does not process SharePoint library items that include the UIVersion field. These SharePoint items are SharePoint UI elements and are skipped. For example, the Form Templates and List Template Gallery library items are UI elements and therefore not processed by OpenText Core Data Discovery & Risk Insights. However, when viewing the SharePoint Site Contents page, these SharePoint items are included in the item count.

SharePoint deletion tracking

OpenText Core Data Discovery & Risk Insights tracks the deletion of managed SharePoint items made at the original source location using the SharePoint change logs. Each time processing is run on a dataset—on a schedule, or on demand—OpenText Core Data Discovery & Risk Insights checks the SharePoint change logs for deleted items. For each managed item that is deleted in SharePoint, that item is deleted from OpenText Core Data Discovery & Risk Insights. If an item within a container file (such as ZIP) is deleted in SharePoint, the item is removed from the application as part of updating the container file when the job run occurs.

To ensure accurate tracking of items deleted from SharePoint, ensure that the SharePoint datasets in OpenText Core Data Discovery & Risk Insights are updated more often than the maximum number of days SharePoint change logs are kept. OpenText Core Data Discovery & Risk Insights uses information from the SharePoint change logs to identify deleted SharePoint items to be removed from the application. Without this information, items deleted from your SharePoint environment cannot be removed from the application. SharePoint items that have been added or modified are appropriately updated in OpenText Core Data Discovery & Risk Insights.

For example, if your SharePoint change logs are configured to be stored for 60 days, verify that your SharePoint datasets are updated at least every 59 days.

CAUTION: Failure to rescan SharePoint datasets before SharePoint change logs are purged will result in items being tracked incorrectly in OpenText Core Data Discovery & Risk Insights. Using the same example, if your SharePoint change logs are configured to be stored for 60 days and your SharePoint datasets are updated every 90 days, you will lose 30 days of important information about deleted items—items deleted during this 30 day time frame will not be removed from the application.

The loss of information cannot be reconciled in OpenText Core Data Discovery & Risk Insights; you would have to create a new dataset and start over.