Structured data connection

If you will be creating datasets that process data from structured data sources (databases), you must complete additional tasks to enable this processing. OpenText Core Data Discovery & Risk Insights accesses structured data sources through a built-in structured data processor or through connection to Structured Data Manager. Review the requirements for each and complete the connection tasks as appropriate for your environment.

Built-in structured data processor connection tasks

NOTE: JDBC is not supported by the built-in structured data processor.

OpenText Core Data Discovery & Risk Insights includes a built-in adapter to access and process supported structured data types using both off-cloud agent clusters and cloud agent clusters.

  • Oracle

    • Oracle 19c, Oracle 21c

  • IBM Db2

    • Db2 11.5.x

  • SQL Server

    • SQL Server 2019, SQL Server 2022

  • PostgreSQL

    • PostgreSQL 12.x, PostgreSQL 13.x, PostgreSQL 14.x, PostgreSQL 15.x, PostgreSQL 16.x

  • MySQL

    • MySQL 8.x

  • Sybase

    • Sybase 16.0

Prior to creating agent clusters, sources, and datasets that use the built-in structured data processor, review the following.

  • Agent clusters assigned to process structured data must have the "Enable Structured Data Manager integration" option disabled. This ensures that the agents in the cluster are not expecting a connection to Structured Data Manager.

  • You will define a user to access the structured data sources. On the Connection page of the Source creation wizard, you must define a user that has the appropriate permissions to access and read from the selected data type and connection.

  • For IBM Db2 databases that include tables with one or more columns being a user defined type (UDT), the transform definition must exist and be applied to the database prior to being assigned to a OpenText Core Data Discovery & Risk Insights dataset.

Structured Data Manager connection tasks

OpenText Core Data Discovery & Risk Insights supports processing supported structured data types by connecting to Structured Data Manager and using off-cloud agent clusters.

NOTE: For supported versions, see the Structured Data Manager Certification Matrix.

  • Oracle

  • IBM Db2

  • SQL Server

  • PostgreSQL

  • MySQL

  • JDBC

  • Sybase

Requirements

Prior to beginning the Structured Data Manager connection tasks, you must have the following in place.

  • Structured Data Manager 23.2.0 fully installed, with Discovery enabled and the Discovery Service installed.

  • The OpenText Core Data Discovery & Risk Insights processing agents intended to process structured data must be installed within the same network as Structured Data Manager.

  • The OpenText Core Data Discovery & Risk Insights processing agents intended to process structured data must be organized into one or more agent clusters.

  • Agent clusters assigned to process data from Structured Data Manager must have the "Enable Structured Data Manager integration" option enabled. If this option is not enabled, OpenText Core Data Discovery & Risk Insights will use the built-in structured data processor instead and you will not be able to perform additional Structured Data Manager actions on data processed by the built-in processor.

Configure Structured Data Manager

Structured Data Manager must be configured for connection to OpenText Core Data Discovery & Risk Insights. This must be performed on every Structured Data Manager host server.

  1. On the Structured Data Manager host server, run the configuration script appropriate for your environment.

    • On Windows servers, run <SDM_HOME>\obt\bin\configure_fusion_integration.bat.

    • On Linux servers, run <SDM_HOME>\obt\bin\configure_fusion_integration.sh.

    where <SDM_HOME> is the installation path for Structured Data Manager.

  2. Follow the prompts as appropriate for your environment unless specified otherwise below.

    1. Make note of the Web Console URL.

    2. When prompted to enable certificate base secure communication with the OpenText Core Data Discovery & Risk Insights agent, press Enter on your keyboard to accept the default Yes response.

    3. When prompted for the details for the self-signed certificate, respond as follows.

      1. For the SDM Server Hostname/IP, type the host name, IP address, or fully qualified domain name of the Structured Data Manager server you are configuring.

      2. For the OpenText Core Data Discovery & Risk Insights Agent Hostname/IP, type the hostname, IP address, or fully qualified domain name of the machine hosting the OpenText Core Data Discovery & Risk Insights agent that will process structured data.

      3. For the Organization Unit, type the desired name of the organization unit. The default entry is SDMOrgUnit.

      4. For the Organization, type the desired name of the organization. The default entry is SDMorg.

      5. For the Certificate Password,type the desired password required to access the certificate.

      The custom root certificate authority, server certificate, OpenText Core Data Discovery & Risk Insights agent (client) certificate are generated.

  3. On the OpenText Core Data Discovery & Risk Insights agent host server you defined, create the C:\Windows\System32\config\systemprofile\.fusionsdmcerts directory.

  4. From the Structured Data Manager server, move the resulting certificates, client.pfx and ca.crt, from the location identified in the response to the \.fusionsdmcerts directory on the OpenText Core Data Discovery & Risk Insights agent host.

  5. To connect this Structured Data Manager server to another OpenText Core Data Discovery & Risk Insights agent host, run <SDM_HOME>\obt\bin\configure_fusion_integration.bat -c (or <SDM_HOME>\obt\bin\configure_fusion_integration.bat -c on Linux) and specify the hostname, IP address, or fully qualified domain name of the additional OpenText Core Data Discovery & Risk Insights agent host to generate another certificate.

    TIP: When running the script, the option -c and -genclientsearch provide the same end result.

  6. Move the <SDM_HOME>\obt\webconsole\apache-tomcat\fusionconf\rootCA_keystore.jks file to a secure location and record the password used to create the certificate.

  7. Restart the Structured Data Manager WebConsole server.

Configure OpenText Core Data Discovery & Risk Insights for Structured Data Manager connection

The OpenText Core Data Discovery & Risk Insights processing agent must be configured to connect to Structured Data Manager using the generated certificate files. This must be completed on each processing agent host that will be processing structured data.

Complete one of the following tasks to configure the connection based on whether you will define a certificate store (more secure) or the individual generated certificate files.

  1. On the agent host server, load the certificates to the certificate store.

    1. Open MMC (Microsoft Management Console) as the user the agent is running as.

      Unless you changed the run-as user for the agent, this would be the "Local System" user.

    2. If not already added, add the Certificates snap-in.

      Unless you changed the run-as user for the agent, add the snap-in for the Computer account.

    3. Import the generated certificate files, client.pfx and ca.cert, into the certificate store for the agent run-as user.

      Unless you changed the run-as user for the agent, this would be Certificates (Local Computer) > Personal.

    4. Edit the properties for the certificate files now in the certificate store to add a Friendly Name. Make note of the Friendly Names defined.

    5. (Optional) Delete the generated certificate files from the agent host. The certificate information is securely stored in the certificate store and the individual files are no longer needed.

  2. On the agent host server, open the Agent administration UI.

    From the Start menu, click OpenText Core Data Discovery & Risk Insights Agent > Agent Admin. The Advanced process settings page opens.

    1. In the category list, click Structured Data Manager Processor.

    2. Complete the following options to define the individual certificate files.

      Option Description
      Base Path

      Type the fully qualified path to Structured Data Manager's OpenText Core Data Discovery & Risk Insights Integration API.
      Client Certificate Friendly Name Type the friendly name of the generated client.pfx certificate file added to the certificate store on this agent host machine.
      Client Certificate Store Location

      Do one of the following:

      • Type LocalMachine if you did not change the run-as user for the agent.

      • Type CurrentUser if you changed the run-as user for the agent.
      Client Certificate Store Name

      Type My.

      NOTE: This is the default store name when the certificate is imported to the certificate store under Certificates (Local Computer) > Personal.
      Custom CA Certificate Friendly Name

      Type the friendly name of the generated ca.cert certificate file added to the certificate store on this agent host machine.
      Custom CA Certificate Store Location

      Do one of the following:

      • Type LocalMachine if you did not change the run-as user for the agent.

      • Type CurrentUser if you changed the run-as user for the agent.
      Custom CA Certificate Store Name

      Type My.

      NOTE: This is the default store name when the certificate is imported to the certificate store under Certificates (Local Computer) > Personal.

    3. Click Save.

  3. Restart the OpenText Core Data Discovery & Risk Insights Structured Data Processor service.

  4. Repeat on all remaining agent host machines that will process structured data.

  1. On the agent host server, open the Agent administration UI.

    From the Start menu, click OpenText Core Data Discovery & Risk Insights Agent > Agent Admin. The Advanced process settings page opens.

  2. In the category list, click Structured Data Manager Processor.

  3. Complete the following options to define the individual certificate files.

    Option Description
    Base Path

    Type the fully qualified path to Structured Data Manager's OpenText Core Data Discovery & Risk Insights Integration API.
    Client Certificate File Path Type the fully qualified path to the generated client.pfx certificate file on this agent host machine.
    Client Certificate Password Type the password defined for the client certificate file on this agent host machine.
    Custom CA Certificate File Path

    Type the fully qualified path to the generated ca.cert certificate file on this agent host machine.

  4. Click Save.

  5. Open and review the StructuredDataProcessor.log log file to confirm that the connection has been established. By default, this file is located in C:\Program Files\OpenText\Core Data Discovery and Risk Insights\Agent\Logs\.

    When the connection is successful, you will see `Starting the engine for adapter type SdmAdapter`. If you don’t see this line, something isn’t configured correctly and communication has not been established.

    CAUTION: Do not move forward until you have successfully established the connection between OpenText Core Data Discovery & Risk Insights and Structured Data Manager.

  6. Repeat on all remaining agent host machines that will process structured data.

Manage structured data in OpenText Core Data Discovery & Risk Insights

After configuring Structured Data Manager and OpenText Core Data Discovery & Risk Insights, complete the following steps to begin managing your structured data in OpenText Core Data Discovery & Risk Insights.

  1. Create agent clusters in Connect. Ensure the Requirements for this connection are met.

  2. Create structured data sources and datasets in Connect.

    The creation of a structured data source in OpenText Core Data Discovery & Risk Insights triggers an inventory and creates an environment in Structured Data Manager.

  3. If you created custom grammar rules in Connect, you must create identical grammar rules in Structured Data Managerand create the mapping.

    NOTE: If you did not create custom grammar rules in Connect, skip this step.

    1. For every custom grammar rule in Connect, make note of the grammar rule ID. The ID can be found on the GENERAL tab of the detail panel for the custom grammar rule in Connect.

    2. In Structured Data Manager Discovery, recreate an identical grammar rule.

      For more information, see Grammar Workbench > Grammar Types > Grammar Rules in the Structured Data Manager Discovery Guide.

    3. Log on to the Structured Data Manager host server.

    4. Add a <Grammar> entry to <OBT_HOME>\obt\bin\fusion_grammar_mapping.xml to define the grammar mapping. <OBT_HOME> is the directory where Structured Data Manager stores application data, such as configuration files and log files.

      Define values for each of the following elements in the <Grammar> entry.

      XML element Description
      SDMType Defines the Structured Data Manager grammar type that this custom rule is assigned to.
      SDMLocale Defines the locale of this rule. Type ww for world wide.
      SDMName Defines the display name of the Structured Data Manager grammar rule.
      FusionRule Defines the corresponding OpenText Core Data Discovery & Risk Insights grammar rule ID.
      FusionRuleName Defines the display name of the corresponding OpenText Core Data Discovery & Risk Insights grammar rule.
      Type

      Type RULE or TYPE, as appropriate.

      NOTE: Currently, only "RULE" and "TYPE" are supported.

      Example grammar mapping entry

      <Grammar>
        <SDMType>Street Address</SDMType>
        <SDMLocale>br</SDMLocale>
        <SDMName>Brazil Street</SDMName>
        <FusionRule>contact_data/address/br</FusionRule>
        <FusionRuleName>Address (Brazil)</FusionRuleName>
        <Type>RULE</Type>
      </Grammar>

    5. Run one of the following scripts as appropriate for your environment. This ingests the newly added mapping.

      • On Windows servers, run <OBT_HOME>\obt\bin\FusionGrammarMapping.bat.

      • On Linux servers, run <OBT_HOME>\obt\bin\FusionGrammarMapping.sh.

      where <OBT_HOME> is the directory where Structured Data Manager stores application data, such as configuration files and log files.

    6. Restart the WebConsole tomcat.

  4. Scan the datasets in Connect.

    The initial scan of a dataset within that source in OpenText Core Data Discovery & Risk Insights creates a project within that environment in Structured Data Manager Discovery.

  5. When the scan in OpenText Core Data Discovery & Risk Insights completes, perform a review of the scan results in the Structured Data Manager Web Console.

  6. Scan the datasets in Connect to synchronize the review details.

Re-scan structured data (optional)

You may want to re-scan your structured datasets if the data in the managed tables has significantly increased or changed, grammars assigned to the structured datasets have changed, if you added custom grammar rules to datasets that have already been scanned, or the scan settings in Structured Data Manager have changed.

  1. In Structured Data Manager, review and re-scan the OpenText Core Data Discovery & Risk Insights-managed tables.

    1. In the Web Console, go to Discovery > Projects. Open a project with the same name as a OpenText Core Data Discovery & Risk Insights dataset.

    2. Click the Tables tab to view the tables in the project.

    3. Do one of the following:

      • Click the reset all tables icon () to remove the previous scan results and return the tables to a state in which they can be rescanned.

      • Click the deep reset all icon () to remove all scan history.

  2. Scan the datasets in Connect to synchronize the review details.

  1. If you created custom grammar rules in Connect, you must create identical grammar rules in Structured Data Managerand create the mapping.

    NOTE: If you did not create custom grammar rules in Connect, skip this step.

    1. For every custom grammar rule in Connect, make note of the grammar rule ID. The ID can be found on the GENERAL tab of the detail panel for the custom grammar rule in Connect.

    2. In Structured Data Manager Discovery, recreate an identical grammar rule.

      For more information, see Grammar Workbench > Grammar Types > Grammar Rules in the Structured Data Manager Discovery Guide.

    3. Log on to the Structured Data Manager host server.

    4. Add a <Grammar> entry to <OBT_HOME>\obt\bin\fusion_grammar_mapping.xml to define the grammar mapping. <OBT_HOME> is the directory where Structured Data Manager stores application data, such as configuration files and log files.

      Define values for each of the following elements in the <Grammar> entry.

      XML element Description
      SDMType Defines the Structured Data Manager grammar type that this custom rule is assigned to.
      SDMLocale Defines the locale of this rule. Type ww for world wide.
      SDMName Defines the display name of the Structured Data Manager grammar rule.
      FusionRule Defines the corresponding OpenText Core Data Discovery & Risk Insights grammar rule ID.
      FusionRuleName Defines the display name of the corresponding OpenText Core Data Discovery & Risk Insights grammar rule.
      Type

      Type RULE or TYPE, as appropriate.

      NOTE: Currently, only "RULE" and "TYPE" are supported.

      Example grammar mapping entry

      <Grammar>
        <SDMType>Street Address</SDMType>
        <SDMLocale>br</SDMLocale>
        <SDMName>Brazil Street</SDMName>
        <FusionRule>contact_data/address/br</FusionRule>
        <FusionRuleName>Address (Brazil)</FusionRuleName>
        <Type>RULE</Type>
      </Grammar>

    5. Run one of the following scripts as appropriate for your environment. This ingests the newly added mapping.

      • On Windows servers, run <OBT_HOME>\obt\bin\FusionGrammarMapping.bat.

      • On Linux servers, run <OBT_HOME>\obt\bin\FusionGrammarMapping.sh.

      where <OBT_HOME> is the directory where Structured Data Manager stores application data, such as configuration files and log files.

    6. Restart the WebConsole tomcat.

  2. In Structured Data Manager, review and re-scan the OpenText Core Data Discovery & Risk Insights-managed tables.

    1. In the Web Console, go to Discovery > Projects. Open a project with the same name as a OpenText Core Data Discovery & Risk Insights dataset.

    2. Click the Tables tab to view the tables in the project.

    3. Do one of the following:

      • Click the reset all tables icon () to remove the previous scan results and return the tables to a state in which they can be rescanned.

      • Click the deep reset all icon () to remove all scan history.

  3. Scan the datasets in Connect to synchronize the review details.