The material in this chapter supplements Chapter 2 of the Databridge Client Administrator's Guide. As stated previously, the Databridge Client for Kafka uses client control tables in the same way as relational database clients. These control tables are common to all clients and have the same layout. See Databridge Client Control Tables in the Databridge Client Administrator's Guide for a complete description of these tables.
All Databridge Client commands must be run with the current directory set to the working directory associated with the data source, as these commands rely on the files in the config sub-directory and write the output to the logs folder. To customize how DMSII data sets get mapped to their corresponding tables, you must write user scripts in SQL. For instructions, see Customizing with User Scripts in the Databridge Client Administrator's Guide.
Creating Client Control Tables
Before you clone a DMSII data source, you must create the Client control
tables using the Client
Creating Client Control Tables
Create the global working directory.
Create the data source working directory.
Set the current directory to be the data source working directory.
exportcommand to create a text configuration file in the
Edit the text configuration file to setup the Oracle connection parameters as well as the Kafka specific parameters. Refer to the Getting Started chapter of the Databridge Client Administrator's Guide.
importcommand to create a binary configuration file in the
configfolder which the Client will use by default.
configurecommand to create the Client control tables.
The material in the Creating Client Control Tables chapter of the Databridge Client Administrator's Guide applies to the Kafka Client.
Defining a Data Source
Refer to Defining a Data Source chapter of the Databridge Client Administrator's Guide. All information in this chapter applies to the Kafka Client except for the data table creation and index creation user scripts as these do not apply to the Kafka Client.
The Decoding DMSII Dates, Times, and Date/Times section of the
Databridge Client Administrator's Guide applies to the Kafka Client
with the exception that the dates are written out using the format
specified by the
span_date_format parameter in the
of the client configuration file.
The Creating Indexes for Tables section of the Databridge Client Administrator's Guide does not apply to the Kafka Client.
The Databridge Client Administrator's Guide section on Adding a Non DMSII Column applies to the Kafka Client; in fact it supports more types of external columns than the relational database clients do. The table below shows the values for all the external column types supported by the Kafka Client.
|User Column Name
|Database update type, as follows:
0 for extract
1 for create
2 for delete (bit 10 must also be enabled)
3 for modify
Note: You cannot use this value at the same time as bit 11.
|Time the update was written to the flat file.
|DMSII audit file timestamp. This column is set to NULL during the initial clone.
Note: You cannot use this bit at the same time as bit 13.
|Audit file number
|Audit block serial number (ABSN)
|Data source name
|Data source identifier as defined in the DATASOURCES Client control table
|Expanded database update type as follows:
0 for extract
1 for create
2 for delete
3 for modify
|Data source identifier as defined in the DATASOURCES Client control table (key item)
|Expanded audit file time. This column contains the DMSII audit file timestamp during updates and the starting time of the data extraction during extraction.
NOTE: You cannot use this bit at the same time as bit 4.
|Generic user column whose entry is left as NULL.
|Time when the record was created in the flat file (PC time).
|The one letter change code for an update A=Add, D=Delete, M=Modify
|The structure number for the data set.
|The data set name.
|The record type for variable format data sets.
|A flag that indicates if an update is part of a MODIFY BI/AI pair. A value of 1 for an ADD or a DELETE, indicates that the ADD was originally the AI or DELETE was originally the BI of a MODIFY BI/AI pair.
|The mode of the data set as stored in the ds_mode column of the DATASETS Client control table. 0=CLONE, 1=FIXUP, 2=UPDATE.
|The format level for the data set.
|The segment number of the audit location within the audit file (audit_seg column in DATASETS Client control table)
|The index of the audit location within the segment of the audit file (audit_inx column in DATASETS Client control table)
|The stack number of the corresponding task on the MCP.
Values for All the Externals Column Types Supported by the Kafka Client
The value for the Bit column in this table is equal to the value in the
dms_subtype column of the DATAITEMS Client control table. The exception
are bits 14, 18, 19, and 20 which result in a
dms_subtype of 0. Bits
are numbered from right to left; the right-most bit is 1.
The Generate Command
generate command creates a schema file for every table associated
with the data sets being replicated. The files, which are named
tablename_schema.json, reside in the "dbscripts" directory. This file describes the layout
of the corresponding table in JSON format. An example of such a file for
the table customer is below:
In addition to creating the schema files, the
generate command also
topic_config.ini file in the config folder that allows you
to associate a topic name with one or more tables. By default the client
uses the topic name of
source_tablename for the various tables. This
can be changed by editing the file and setting the topic name to use the
desired names. For an example of such a file for the data source TESTDB
refer to example 1-1 in chapter 1.
Under normal circumstance the above mentioned files are only updated
when the parent data set has bit 2 of the
status_bit column set.
Adding the -u option to the command causes all the schema files and the
topic_config.ini file to be updated. The client always reads the
topic_config.ini file before saving the topic names. If the file does
not exist it gets created with the default topic names.
The Process Command
process command is the main command of the Databridge Kafka
Client. This command creates output streams for all data set tables
where the active column is 1 in the corresponding entries of the
DATASETS client control table. Since the
define command initializes
the ds_mode column to 0 (indicating that the data set needs to be
cloned), all the selected data sets are cloned the first time you run a
If you do not select specific data sets in the data set global mapping script, the Databridge Client automatically clones all data sets except for remaps, the restart data set, and the global data set.
Once the data sets are cloned, subsequent runs of the
will store updates in the Kafka repository. For insert operations all
columns will be present, for delete operations, only columns that are
keys will be present in the update record. In the case of updates, the
keys and changed columns will be present in the record. All records
contain the update_type field, which is 0 for extract records, 1 for
insert operations after the data extraction completes, 2 for delete
operations, and 3 for update operations.
process command to update the data in the Kafka repository can be
scheduled. The schedule becomes effective after the
process command is
run for the first time. For more information, see the Scheduling
Updates section in the Databridge Client Administrator's Guide.
Before running the
process command, ensure that the current directory
is set to the working directory that was created for this data source.
For more information on the
process command refer to the Databridge
Client Administrator's Guide.
The Clone Command
Refer to The Clone Command section of the Databridge Client Administrator's Guide for documentation on this command.