Structured data connection

If you will be creating datasets that process data from structured data sources (databases), you must complete additional tasks to enable this processing. Fusion accesses structured data sources through connection to Structured Data Manager. Review the following requirements and complete the connection tasks.

The following structured data types are supported.

  • Db2

  • Oracle

  • PostgeSQL

  • SQL Server

  • Sybase

  • MySQL

  • JDBC

Structured data connection requirements

Prior to beginning the connection tasks, you must have the following in place.

  • Structured Data Manager 23.2.0 fully installed, with Discovery enabled and the Discovery Service installed.

  • The Fusion processing agents intended to process structured data must be installed within the same network as Structured Data Manager.

  • The Fusion processing agents intended to process structured data must be organized into one or more agent clusters.

  • Agent clusters assigned to process structured data must have the "Enable structured data capabilities" option enabled. When you create datasets for structured data, you will only be able to assign agent clusters that have this option enabled.

Configure Structured Data Manager

Structured Data Manager must be configured for connection to Fusion. This must be performed on every Structured Data Manager host server.

  1. On the Structured Data Manager host server, run the configuration script appropriate for your environment.

    • On Windows servers, run <SDM_HOME>\obt\bin\configure_fusion_integration.bat.

    • On Linux servers, run <SDM_HOME>\obt\bin\configure_fusion_integration.sh.

    where <SDM_HOME> is the installation path for Structured Data Manager.

  2. Follow the prompts as appropriate for your environment unless specified otherwise below.

    1. Make note of the Web Console URL.

    2. When prompted to enable certificate base secure communication with the Fusion agent, press Enter on your keyboard to accept the default Yes response.

    3. When prompted for the details for the self-signed certificate, respond as follows.

      1. For the SDM Server Hostname/IP, type the host name, IP address, or fully qualified domain name of the SDM server you are configuring.

      2. For the Fusion Agent Hostname/IP, type the hostname, IP address, or fully qualified domain name of the machine hosting the Fusion agent that will process structured data.

      3. For the Organization Unit, type the desired name of the organization unit. The default entry is SDMOrgUnit.

      4. For the Organization, type the desired name of the organization. The default entry is SDMorg.

      5. For the Certificate Password,type the desired password required to access the certificate.

      The custom root certificate authority, server certificate, Fusion agent (client) certificate are generated.

  3. On the Fusion agent host server you defined, create the C:\Windows\System32\config\systemprofile\.fusionsdmcerts directory.

  4. From the Structured Data Manager server, move the resulting certificates, client.pfx and ca.crt, from the location identified in the response to the \.fusionsdmcerts directory on the Fusion agent host.

  5. To connect this Structured Data Manager server to another Fusion agent host, run <SDM_HOME>\obt\bin\configure_fusion_integration.bat -c (or <SDM_HOME>\obt\bin\configure_fusion_integration.bat -c on Linux) and specify the hostname, IP address, or fully qualified domain name of the additional Fusion agent host to generate another certificate.

    TIP: When running the script, the option -c and -genclientsearch provide the same end result.

  6. Move the <SDM_HOME>\obt\webconsole\apache-tomcat\fusionconf\rootCA_keystore.jks file to a secure location and record the password used to create the certificate.

  7. Restart the Structured Data Manager WebConsole server.

Configure Fusion for structured data connection

The Fusion processing agent must be configured to connect to Structured Data Manager using the generated certificate files. This must be completed on each processing agent host that will be processing structured data.

Complete one of the following tasks to configure the connection based on whether you will define a certificate store (more secure) or the individual generated certificate files.

  1. On the agent host server, load the certificates to the certificate store.

    1. Open MMC (Microsoft Management Console) as the user the agent is running as.

      Unless you changed the run-as user for the agent, this would be the "Local System" user.

    2. If not already added, add the Certificates snap-in.

      Unless you changed the run-as user for the agent, add the snap-in for the Computer account.

    3. Import the generated certificate files, client.pfx and ca.cert, into the certificate store for the agent run-as user.

      Unless you changed the run-as user for the agent, this would be Certificates (Local Computer) > Personal.

    4. Edit the properties for the certificate files now in the certificate store to add a Friendly Name. Make note of the Friendly Names defined.

    5. (Optional) Delete the generated certificate files from the agent host. The certificate information is securely stored in the certificate store and the individual files are no longer needed.

  2. On the agent host server, open the Agent administration UI.

    From the Start menu, click Fusion Agent > Agent Admin. The Advanced process settings page opens.

    1. In the Category list, click Structured Data Processor.

    2. Complete the following options to define the individual certificate files.

      Option Description
      Base Path

      Type the fully qualified path to Structured Data Manager's Fusion Integration API.
      Client Certificate Friendly Name Type the friendly name of the generated client.pfx certificate file added to the certificate store on this agent host machine.
      Client Certificate Store Location

      Do one of the following:

      • Type LocalMachine if you did not change the run-as user for the agent.

      • Type CurrentUser if you changed the run-as user for the agent.
      Client Certificate Store Name

      Type My.

      NOTE: This is the default store name when the certificate is imported to the certificate store under Certificates (Local Computer) > Personal.
      Custom CA Certificate Friendly Name

      Type the friendly name of the generated ca.cert certificate file added to the certificate store on this agent host machine.
      Custom CA Certificate Store Location

      Do one of the following:

      • Type LocalMachine if you did not change the run-as user for the agent.

      • Type CurrentUser if you changed the run-as user for the agent.
      Custom CA Certificate Store Name

      Type My.

      NOTE: This is the default store name when the certificate is imported to the certificate store under Certificates (Local Computer) > Personal.

    3. Click Save.

  3. Restart the Fusion Structured Data Processor service.

  4. Repeat on all remaining agent host machines that will process structured data.

  1. On the agent host server, open the Agent administration UI.

    From the Start menu, click Fusion Agent > Agent Admin. The Advanced process settings page opens.

  2. In the Category list, click Structured Data Processor.

  3. Complete the following options to define the individual certificate files.

    Option Description
    Base Path

    Type the fully qualified path to Structured Data Manager's Fusion Integration API.
    Client Certificate File Path Type the fully qualified path to the generated client.pfx certificate file on this agent host machine.
    Client Certificate Password Type the password defined for the client certificate file on this agent host machine.
    Custom CA Certificate File Path

    Type the fully qualified path to the generated ca.cert certificate file on this agent host machine.

  4. Click Save.

  5. Open and review the StructuredDataProcessor.log log file to confirm that the connection has been established. By default, this file is located in C:\Program Files\Fusion\Agent\Logs\.

    When the connection is successful, you will see `Starting the engine for adapter type SdmAdapter`. If you don’t see this line, something isn’t configured correctly and communication has not been established.

    CAUTION: Do not move forward until you have successfully established the connection between Fusion and Structured Data Manager.

  6. Repeat on all remaining agent host machines that will process structured data.

Manage structured data in Fusion

After configuring Structured Data Manager and Fusion, complete the following steps to begin managing your structured data in Fusion.

  1. Create agent clusters in Connect. Ensure the Structured data connection requirements for this connection are met.

  2. Create structured data sources and datasets in Connect.

    The creation of a structured data source in Fusion triggers an inventory and creates an environment in Structured Data Manager.

  3. If you created custom grammar rules in Connect, you must create identical grammar rules in Structured Data Manager and create the mapping.

    NOTE: If you did not create custom grammar rules in Connect, skip this step.

    1. For every custom grammar rule in Connect, make note of the grammar rule ID. The ID can be found on the GENERAL tab of the detail panel for the custom grammar rule in Connect.

    2. In Structured Data Manager Discovery, recreate an identical grammar rule.

      For more information, see Grammar Workbench > Grammar Types > Grammar Rules in the Structured Data Manager Discovery Guide.

    3. Log on to the Structured Data Manager host server.

    4. Add a <Grammar> entry to <OBT_HOME>\obt\bin\fusion_grammar_mapping.xml to define the grammar mapping. <OBT_HOME> is the directory where Structured Data Manager stores application data, such as configuration files and log files.

      Define values for each of the following elements in the <Grammar> entry.

      XML element Description
      SDMType Defines the Structured Data Manager grammar type that this custom rule is assigned to.
      SDMCode Defines the Structured Data Manager grammar rule ID.
      SDMLocale Defines the locale of this rule. Type ww for world wide.
      SDMName Defines the display name of the Structured Data Manager grammar rule.
      FusionRule Defines the corresponding Fusion grammar rule ID.
      FusionRuleName Defines the display name of the corresponding Fusion grammar rule.
      Type

      Type RULE.

      NOTE: Currently, only "RULE" is supported.

      Example grammar mapping entry

      <Grammar>
        <SDMType>Street Address</SDMType>
        <SDMCode>0015-0004</SDMCode>
        <SDMLocale>br</SDMLocale>
        <SDMName>Brazil Street</SDMName>
        <FusionRule>contact_data/address/br</FusionRule>
        <FusionRuleName>Address (Brazil)</FusionRuleName>
        <Type>RULE</Type>
      </Grammar>

    5. Run one of the following scripts as appropriate for your environment. This ingests the newly added mapping.

      • On Windows servers, run <OBT_HOME>\obt\bin\FusionGrammarMapping.bat.

      • On Linux servers, run <OBT_HOME>\obt\bin\FusionGrammarMapping.sh.

      where <OBT_HOME> is the directory where Structured Data Manager stores application data, such as configuration files and log files.

  4. Scan the datasets in Connect.

    The initial scan of a dataset within that source in Fusion creates a project within that environment in Structured Data Manager Discovery.

  5. When the scan in Fusion completes, perform a review of the scan results in the Structured Data Manager Web Console.

  6. Scan the datasets in Connect to synchronize the review details.

Re-scan structured data (optional)

You may want to re-scan your structured datasets if the data in the managed tables has significantly increased or changed, grammars assigned to the structured datasets have changed, if you added custom grammar rules to datasets that have already been scanned, or the scan settings in Structured Data Manager have changed.

  1. In Structured Data Manager, review and re-scan the Fusion-managed tables.

    1. In the Web Console, go to Discovery > Projects. Open a project with the same name as a Fusion dataset.

    2. Click the Tables tab to view the tables in the project.

    3. Do one of the following:

      • Click the reset all tables icon () to remove the previous scan results and return the tables to a state in which they can be rescanned.

      • Click the deep reset all icon () to remove all scan history.

  2. Scan the datasets in Connect to synchronize the review details.

  1. If you created custom grammar rules in Connect, you must create identical grammar rules in Structured Data Manager and create the mapping.

    NOTE: If you did not create custom grammar rules in Connect, skip this step.

    1. For every custom grammar rule in Connect, make note of the grammar rule ID. The ID can be found on the GENERAL tab of the detail panel for the custom grammar rule in Connect.

    2. In Structured Data Manager Discovery, recreate an identical grammar rule.

      For more information, see Grammar Workbench > Grammar Types > Grammar Rules in the Structured Data Manager Discovery Guide.

    3. Log on to the Structured Data Manager host server.

    4. Add a <Grammar> entry to <OBT_HOME>\obt\bin\fusion_grammar_mapping.xml to define the grammar mapping. <OBT_HOME> is the directory where Structured Data Manager stores application data, such as configuration files and log files.

      Define values for each of the following elements in the <Grammar> entry.

      XML element Description
      SDMType Defines the Structured Data Manager grammar type that this custom rule is assigned to.
      SDMCode Defines the Structured Data Manager grammar rule ID.
      SDMLocale Defines the locale of this rule. Type ww for world wide.
      SDMName Defines the display name of the Structured Data Manager grammar rule.
      FusionRule Defines the corresponding Fusion grammar rule ID.
      FusionRuleName Defines the display name of the corresponding Fusion grammar rule.
      Type

      Type RULE.

      NOTE: Currently, only "RULE" is supported.

      Example grammar mapping entry

      <Grammar>
        <SDMType>Street Address</SDMType>
        <SDMCode>0015-0004</SDMCode>
        <SDMLocale>br</SDMLocale>
        <SDMName>Brazil Street</SDMName>
        <FusionRule>contact_data/address/br</FusionRule>
        <FusionRuleName>Address (Brazil)</FusionRuleName>
        <Type>RULE</Type>
      </Grammar>

    5. Run one of the following scripts as appropriate for your environment. This ingests the newly added mapping.

      • On Windows servers, run <OBT_HOME>\obt\bin\FusionGrammarMapping.bat.

      • On Linux servers, run <OBT_HOME>\obt\bin\FusionGrammarMapping.sh.

      where <OBT_HOME> is the directory where Structured Data Manager stores application data, such as configuration files and log files.

  2. In Structured Data Manager, review and re-scan the Fusion-managed tables.

    1. In the Web Console, go to Discovery > Projects. Open a project with the same name as a Fusion dataset.

    2. Click the Tables tab to view the tables in the project.

    3. Do one of the following:

      • Click the reset all tables icon () to remove the previous scan results and return the tables to a state in which they can be rescanned.

      • Click the deep reset all icon () to remove all scan history.

  3. Scan the datasets in Connect to synchronize the review details.