Chapter 2: Engine
This chapter describes the Databridge Engine: what it is, how to configure it, and the commands you can use with it.
How DBEngine Works
The Databridge Engine (also referred to as DBEngine) is a host library program that uses the DMSII Support Library to retrieve data records from the DMSII database for cloning.
Sources of information for database records for DBEngine are as follows:
- Database data sets
- DMSII audit trail
When an Accessory requests a clone (initial extraction), DBEngine reads each record in the specified data set and sends it to the Accessory. This is called data set extraction.
DBEngine’s source of information about updates to database records is the DMSII audit trail, which contains before and after images for all changed records. DBEngine reads the audit trail and selects information for data sets the Accessory requested. It sends a copy of each updated record back to the Accessory. By default, DBEngine processes only closed audit files. If the Read Active Audit option is set to true, it will also access the current (in use) audit file.
DBEngine initializes when Databridge Server and other Accessories call it (a single copy of DBEngine is linked to each Accessory). DBEngine shares the same code stack as other copies of the Engine, but it does not share the same data stack.
DBEngine Visibility to Accessories and Usercodes
If you install Databridge Host and its Accessories under a privileged usercode as recommended, Accessories you use for replication can access DBEngine by title as needed. Accessories call entry points in DBEngine as specified in the Databridge API. The DBEngine component is shipped as a privileged program so that it can access guard files under all usercodes without any additional configuration.
However, if you need to run Databridge Host under multiple usercodes, or if the Databridge Accessories are installed under a different user code from DBEngine, you must do one of the following:
- Include a copy of DBEngine and its parameter file under each Accessory's usercode.
- Install Databridge Host without a usercode to a common disk family and use family substitution statements that reference that disk family. For information, see the Unisys documentation.
- Establish DBEngine as a system library. (See Establish DBEngine as a System Library (SL))
Establish DBEngine as a System Library (SL)
Use the following procedure to make DBEngine visible to Databridge Accessories and programs running under different user codes. For more information, see DBEngine Visibility to Accessories and Usercodes.
To establish DBEngine as an SL
Enter the following command from the Action line in MARC:
SL DBENGINE = (usercode)OBJECT/DATABRIDGE/ENGINE ON familyname
To make the DBEngine parameter file available to Accessories under a different usercode, copy the DBEngine parameter file to the usercode of the Accessories.
To remove the function name assignment
Use this procedure after previously establishing it as a system library in the following situations:
- To access DBEngine by title (as per our recommendations)
To install a new version or patch of DBEngine
From the Action line in MARC, enter the following command:
SL - DBENGINE
The host responds with the following:
FUNCTION "Databridge Engine" IS NO LONGER ESTABLISHED
If you're installing a new version or patch and want to re-establish DBEngine as a system library, from the Action line in MARC enter the following command:
SL DBENGINE = (usercode)OBJECT/DATABRIDGE/ENGINE ON familyname
DBEngine validates the following:
- The property level in the DESCRIPTION file. This prevents the Databridge software from running with incompatible levels of the DMSII software.
- The audit level of each audit file. This ensures that it is compatible with the level of DMSII software for which DBEngine was compiled.
- The audit update level when the requested audit file is opened for tracking. This ensures that Databridge does not return records that have a different format than what the Accessory is expecting. Databridge starts by opening the oldest requested audit file first, and then moving forward; this ensures that the Accessory receives the updates in the proper order.
That the update levels of the DMSUPPORT library and DESCRIPTION file match the audit file. When DBEngine must look for a DESCRIPTION file at a new update level, it checks for the file in the following sequence:
Likewise, when DBEngine is searching for DMSUPPORT, it will try:
To ensure that the DMSII DESCRIPTION file and the Support Library file include the correct update level for the database, run WFL/DATABRIDGE/BACKUPTAILORED after every DASDL update or reorganization. This WFL adds the update level to the last node of the DESCRIPTION and Support Library filenames. See “Chapter 10: Utilities” on page177.
Configuring DBEngine Parameters
There are three types of DBEngine parameter files that you can configure, depending on your needs:
|The common parameter file (DATA/ENGINE/CONTROL)||Use to specify availability, checkpoint frequency, etc., for all databases that do not set the parameter value in a database-specific parameter file or a logical database-specific parameter file.|
|A database-specific parameter file (DATA/ENGINE/databasename/CONTROL)||Use to specify availability, checkpoint frequency, etc., at the database level. Options in the databasespecific parameter file override the options in the common parameter file.|
|A logical–database-specific parameter file (DATA/ ENGINE/databasename/logicaldatabasename/CONTROL||Use this type of DBEngine parameter file to specify availability, checkpoint frequency, etc., at the logical database level. Options in the logical databasespecific parameter file override the options in the database-specific parameter file and the options in the common parameter file.|
DBEngine reads the database-specific parameter file and the logical–database-specific parameter file, if they exist, in addition to the common parameter file (DATA/ENGINE/CONTROL). The database- specific parameter file and the logical–database-specific parameter file can have the same options as the common parameter file. Options in the database-specific parameter file and the logical– database-specific parameter file override the options in the common parameter file. This allows you to configure different options for different databases (for example, to have different checkpoint frequencies).
Parameter files must be visible to DBEngine and in one of the following locations:
- Under the current usercode (that is, the Accessory’s usercode)
- Under the DBEngine codefile’s usercode and pack name
Parameter Change Limits
The Databridge Client and Enterprise Server can change the values of many of the parameters in this file. The parameter declaration can specify the allowable changes.
Boolean Parameter Change Limits
The syntax for Boolean parameters will have the following form.
parameter [ [ = ] default [ ONLY ] ]
where default is the default value before any changes by the Databridge Client or Enterprise Server. The default value can be any of the following:
If the parameter is specified without a default value it is equivalent to parameter = TRUE.
The ONLY keyword makes the parameter read-only.
Numeric Parameter Change Limits
The syntax for numeric parameters has the following form:
parameter [ = ] default [ limits ]
where default is the default value before any changes by the Databridge Client or Enterprise Server.
The optional limits clause can be any of the following. (The parentheses are required.)
(ALLOW min - max)
where min is the minimum value the parameter can be set to and max is the maximum. ALLOW ANY means there is no minimum or maximum restrictions. For the MAX max form there is no minimum value limitation.
If no limits clause is specified, there is no minimum value limitation but the maximum value is default.
Audit and Property Levels
Do not make changes to Audit level or Property level unless you are instructed to do so by Micro Focus. This information is for reference only.
Audit level = number
Property level = "numberstring"
| Where | Is | |:--------- |:----------- | | *number* | An integer that represents the audit level DBEngine can use even if it was compiled with a different audit level. | | *"numberstring"* | The value that represents the property level of the DESCRIPTION file that DBEngine can use even if it was compiled with a different property level. |
Audit levels and property levels can have multiple entries, as in this example:
Audit level = 7
Property level = "040230"
Property level = "040710"
The DBEngine options that follow are available for these circumstances:
- When an audit file has a different audit level than the one with which DBEngine was compiled
- When the DESCRIPTION file has a different property level than any of the values in the Databridge internal property level list
In either of these cases, DBEngine checks its parameter file for an entry that indicates the software level for which DBEngine is compatible. If an audit level or property level mismatch remains after DBEngine checks the parameter file, DBEngine displays a message about the mismatch and returns an error.
Available From... To....
AVAILABLE day FROM time TO time
|day||One of the following:
|time||One of the following:
The Available From...To...option limits when a Databridge Accessory can use DBEngine to access a database. If an Accessory attempts to access the database outside of the specified time interval, error 41, “Access to database denied” is returned.
The following example shows how you can specify multiple Available From...To... options:
Available weekdays from 9:20 PM to 11:50 PM Available Saturday from 4:00 AM to 10:50 PM`
If there are conflicting entries, DBEngine will use the least restrictive (most available) entry.
The Checkpoint Frequency option can be based on the following:
- Size of a transaction group
- Elapsed time
- Idle database
The Checkpoint frequency option applies to all Accessories. Databridge Clients can override the DBEngine Checkpoint values using parameters in the Client configuration files.
CHECKPOINT [ CLIENT ] [ EVERY ] n [ limits ] [ SECONDS | MINUTES ]
Checkpoint client every 0 seconds
This Checkpoint option specifies the elapsed time (in seconds or minutes) when Databridge is processing the audit trail, not the time difference when the updates actually occurred.
When this option is specified, DBEngine checks the current time of day at each QPT. If that amount of time has elapsed since the last commit, DBEngine causes another commit.
Elapsed-time checkpoint frequency is useful for making transactions available to the client sooner when the mainframe is heavily loaded and replication slows.
Checkpoint client every 5 minutes Checkpoint every 20 seconds
CHECKPOINT [ CLIENT ] [ DURING ] IDLE DATABASE [ [ = ] value [ ONLY ]
This option determines if DBEngine should do a commit if the audit trail ends with an End Transaction. If the database is idle (open for update but no programs have a transaction in progress) the last transaction group will not be committed because no Begin Transaction follows the last End Transaction. As soon as a program starts a transaction or DMSII does a syncpoint or controlpoint or the database is closed, that last transaction group will be committed.
If the site wants to ensure that the last transaction group before the database became idle is replicated, the recommended procedure is to run a program that forces a syncpoint. For databases that do not have the DASDL option REAPPLYCOMPLETED set, this will also ensure that the last transaction is actually written to the audit trail and the database.
Again, this procedure is necessary only if one or more programs have the database open update but none of them are actually doing any updates. If running a special program to force a syncpoint is not desirable in this situation, the site can set the following option in the DBEngine parameter file:
Checkpoint client during idle database = true
This will cause a commit if the last available audit record is an End Transaction and no other transactions are in progress.
Setting this option runs the risk that DMSII will not treat that position in the audit trail as a quiet point because it (incorrectly) writes the next Begin Transaction record with a transaction count other than 1. A halt/load recovery could then discard those committed transactions and the client database would not match the DMSII database.
Pseudo Quiet Points
CHECKPOINT [ CLIENT ] [ DURING ] LONG TRANSACTIONS [ = value [ ONLY ] ]
where value is
Checkpoint client during long transactions = false only
When this option is true, DBEngine treats a pseudo quiet point as a normal quiet point and a possible place to commit a transaction group.
A pseudo quiet point occurs when DMS forces all programs out of transaction state in order to perform a sync point when the SYNCWAIT time specified in the DASDL is exceeded. If a program terminates (or there is a halt/load) before finishing a long transaction, DMS rolls back the entire long transaction, but the client database still has the committed portions of the partial transaction. DBEngine eventually reverses these updates when the recovery region of the audit is processed by sending update reversals. In these cases, the history tables will contain both the original update and its reversal.
only option prevents Databridge Clients or Accessories from changing the LONG
Transaction Group Size
CHECKPOINT [ CLIENT ] [ EVERY ] n [ limits ] [ AUDIT ] BLOCKS CHECKPOINT [ CLIENT ] [ EVERY ] n [ limits ] TRANSACTIONS CHECKPOINT [ CLIENT ] [ EVERY ] n [ limits ] [ UPDATE ] RECORDS
Checkpoint options can be combined with "or." For example:
CHECKPOINT [ CLIENT ] [ EVERY ] n [ limits ] [ AUDIT ] BLOCKS OR [ EVERY ] n [ limits ] TRANSACTIONS OR [ EVERY ] n [ limits ] [ UPDATE ] RECORDS
|n||The default number of audit bocks or updates or transactions to be allowed in a transaction group. n can be any valid integer.|
||See Numeric Parameter Change Limits for an explanation of the limits option.|
||A required word|
||An optional word|
||An optional word|
||An optional word|
||This indicates that n refers to the number of audit blocks processed in a transaction group.|
||A required word if you are checking combinations of checkpoint options.|
||An optional word|
||This indicates that n refers to the number of record updates in a transaction group.|
||This indicates that n refers to the number of transactions in a transaction group.|
CHECKPOINT CLIENT EVERY 100 (ALLOW any) AUDIT BLOCKS CHECKPOINT CLIENT EVERY 1000 (ALLOW any) RECORDS CHECKPOINT CLIENT EVERY 0 TRANSACTIONS
DATA/ENGINE/SAMPLE/CONTROL may have different settings.
This Checkpoint option specifies the size of a transaction group. DBEngine ends a transaction group at the first quiet point (natural or super) after reaching any Checkpoint value. This ensures that transaction groups are roughly the same size and reduces the overhead associated with ending transaction groups due to frequent quiet points.
Updated records in a transaction group are not available to Accessories until the end of the transaction group is reached and the whole group is committed to the client database. Therefore, smaller transaction groups make the individual updates available sooner.
The disadvantage of smaller transaction groups is that DBEngine must send state information records at the end of each group to reflect the new location in the audit trail. The Accessory must update its control tables with this state information, which is additional overhead. The target database might have some constraints on how many records can be updated in one group. If so, the Checkpoint option would need to be low enough to stay within those constraints.
Syntax: CONVERT REVERSALS TO UPDATES = value [ ONLY ]
where value is
If this option is true, reversals are treated as normal updates such that the original update and its reversal will both be sent to the Databridge Client or the Accessory. This removes the need to abort and reposition the audit trail when an aborted transaction is encountered. The Client database will show the original update and its reversal. A history table will include both.
DMSII Program Titles
DMCONTROL [ = ] "filetitle" DMUTILITY [ = ] "filetitle" DMRECOVERY [ = ] "filetitle"
Titles as specified in WFL/DATABRIDGE/INCLUDE/SSRTITLES
The DMSII Program Titles option allows you to specify a different title for the DMCONTROL, DMUTILITY, and DMRECOVERY programs. This option allows you to use nonstandard names for either program, in addition to specifying usercodes and packnames.
We strongly suggest, however, that you use WFL/DATABRIDGE/INCLUDE/SSRTITLES to specify these titles instead, as this allows you to specify different titles for different MCP and DMSII releases.
DMUTILITY "(581)SYSTEM/DMUTILITY ON SYSPACK"
DYNAMIC NAMES [[ = ] value ]
where value is either
If false or no, names of the Extract Workers are EXTRACT/WORKER/n where n is a relative Worker number.
If true or yes, the names of the Extract Workers will include the database name and the current data
set name in this form:
EXTRACT/WORKERn/databasename/structurenumber. Note there is no slash between WORKER and n.
Syntax: ENTERPRISE WORKERS = number [ limits ]
where number is the default number of Databridge Enterprise extract threads. For an explanation of limits see Numeric Parameter Change Limits.
Enterprise Workers = 20 (allow any)
The Enterprise Workers option enables Databridge Enterprise to process multiple Worker threads to perform data set extracts during a clone. When more than one Worker thread is running, the Databridge Client receives records interspersed from multiple data sets.
The actual number of Databridge Enterprise threads will not exceed the number of data sets to extract.
LINKS [ = ] value [ ONLY ]
where value is either
The LINKS option enables DBEngine to process link data items for unsectioned data sets.
If you enable the
enable_dms_links parameter in the Databridge Client configuration file, you
must also enable the DBEngine LINKS option.
The Span and Snapshot Accessories do not support links and will discard any link values they receive.
MANUAL COMPILE [[ = ] value ]
where value is either
no, DBEngine will automatically recompile DBSupport and DBDMSIISupport when needed
for a particular database update level.
yes, DBEngine will return the error message “Compile of title failed”, and you will need to
manually initiate the compile of that program. This is useful when the DMALGOL compiler is not
resident on the system where DBEngine is running.
MIRRORED AUDIT [ AND DATABASE ] FOR sourcename AT hostname PORT "portnumber"
|sourcename||The name of a DBServer SOURCE on the primary system (where the live database is)|
|hostname||The primary system (where the live database is)|
|portnumber||The port number for communicating with DBServer on the primary system|
The Mirrored Audit option allows an Accessory to process transaction groups on a secondary system containing the mirrored audit files with minimal impact on the primary system. Special disk hardware or software duplicates the audit files on the secondary system. DBEngine will communicate with DBServer on the primary system to determine the current (logical) end of the audit trail.
Put the MIRRORED option in a database-specific DBEngine parameter file (DATA/ENGINE/database/ CONTROL). If you put it in DATA/ENGINE/CONTROL, then no matter what database the Accessory wants to use, it will always try to get the information from the specified remote source.
Mirrored Audit and Database for BANKDB at PROD port "3000" Mirrored Audit for CUSTOMERDB at GREENHOST port "4101"
Print Statistics Report.
[ PRINT ] STATISTICS [ = ] value [ ONLY ]
where value is either
yes, DBEngine will print a statistics report when it finishes. The optional word "only"
ensures that the Databridge Client or Accessory cannot change this option.
READAHEAD [ = ] value [ ONLY ]
where value is either
The ReadAhead option can be used to increase throughput when DBEngine accesses a remote audit file via DBServer, such as when running Databridge Span with the “SOURCE sourcename ...” option, or the “MIRRORED AUDIT ...” DBEngine option. If the ReadAhead option is enabled, DBEngine will request the next audit region before the Accessory actually needs it. This overlaps the network time and the processing time for greater throughput. The optional word "only" ensures that the Databridge Client or Accessory cannot change this option.
Read Active Audit
READ ACTIVE AUDIT [ = ] value [ ONLY ]
where value is either
Yes, Databridge will return updates during tracking from the active audit file. Databridge
reads the active audit file as needed during the extract and fixup phases. This parameter may only be
set to True on installations running SSR 50.1 or later.
The optional word "only" ensures that the Databridge Client or a Databridge Accessory cannot change this option.
This option must be set to true for Databridge Enterprise Server to read the active audit file during tracking.
WORKERS = number [ limits ]
where number is the default number of Workers. For an explanation of limits see Numeric Parameter Change Limits.
Workers = 10 (allow 1 - 4095)
The Workers option enables DBEngine to process one or more Extract Worker stacks to perform the data set extracts during a clone. On large systems with multiple processors and/or extra available resources, this might increase the speed of the cloning process. When there is more than one Worker running, the Accessory or Databridge Client receives records interspersed from multiple data sets.
AX WORKERS = number command can dynamically change the number of Extract Workers
allowed, but must not exceed limits. An Accessory or Databridge Client can also specify the number
of Workers by calling the DBParameters entry point, but it must not exceed the specified limits.
The actual number of Workers running will not exceed the number of data sets to extract.
Sample DBEngine Parameter File
%----------------------------------------------------------------------- % % (C) Copyright 2019 Micro Focus or one of its affiliates. % % Module: DATA/ENGINE/SAMPLE/CONTROL % % Project: Databridge % % Description: Databridge Engine Parameter File % % (C) Copyright 2019 Micro Focus or one of its affiliates. % %----------------------------------------------------------------------- % Software compatibilities Audit level = 10 Property level = "050710" % Frequency of commits Checkpoint client every 100 (allow any) audit blocks Checkpoint client every 1000 (allow 1 - 9999999) records % Checkpoint client every 200 (allow any) transactions % Checkpoint client every 10 (allow 1 - 36000) seconds Checkpoint client during long transactions = false only Checkpoint client during idle database = false % Number of concurrent Extract workers Workers = 4 (max 10) Dynamic names = false % Workers named .../database/dataset Enterprise Workers = 20 (max 100) % threads reading datasets % Restrict access to Databridge to certain times % Available daily from 11:30 PM to 7:00 AM % Monday from 12:00 PM to 1:30 PM % Available Saturday from 4:00 AM to 10:50 PM % Available weekdays from 9:20 PM to 3:00 AM % When retrieving audit regions from another host % across a network, ReadAhead will send a request for % the next region before the Accessory asks for it. ReadAhead = true % Print statistics report at EOJ Print Statistics = false % Include/exclude link data items Links = false % Access/don't access the active (current) audit file Read active audit = false % Treat reversals as normal updates Convert reversals to updates = false % Enable/disable automatic recompile of DBSupport and % DMSIISupport Manual compile = false % DMSII program titles % (We recommend using WFL/DATABRIDGE/INCLUDE/SSRTITLES instead.) %- DMCONTROL "*SYSTEM/DMCONTROL ON DISK" %- DMUTILITY "*SYSTEM/DMUTILITY ON DISK" %- DMRECOVERY "*SYSTEM/DMRECOVERY ON DISK" % Mirroring % WARNING: The MIRRORED option should only appear in a % database-specific DBEngine control file, i.e. % DATA/ENGINE/<database>/CONTROL. % MIRRORED AUDIT [ AND DATABASE ] FOR <sourcename> % AT "<hostname>" PORT "<portnumber>" % <hostname> is the primary system, i.e. where the live database is. % <sourcename> is name of a DBServer SOURCE on that primary system. % Mirrored Audit and Database for BANKDB at PROD port "3000" % % Mirrored Audit for CUSTOMERDB at GREENHOST port "4101"
DBEngine responds to the following AX commands. The mix number in the AX command must be
the job number of the DBEngine itself, not an Accessory calling DBEngine. You can use the ODT or
LIBS NAME =ENGINE= to find the DBEngine job number
After entering any AX command, enter
MSG from the ODT to see the response.
The DBEngine responses appear in the system messages.
||If DBEngine has the database open during tracking, this command will close it. This allows other programs exclusive use of the database.|
||Displays the same commit-related statistics that would appear on the printed statistics report. The number after the "#" is the count of samples of that category. The numbers at the end of the displayed line are the minimum, maximum, and average.|
||Displays whether debug tracing is enabled.
||Displays the available AX commands. The mix number of an Extract Worker can be used in the
||Terminates tracking. It causes the DBReadTranGroup entry point to exit at the next quiet point. Other copies of the DBEngine library, however, are still available for other Databridge Accessories. If DBReadTranGroup is waiting for more audit to become available, the
||Displays the same size-related statistics that would appear on the printed statistics report. The number after the "#" is the count of samples of that category. The numbers at the end of the displayed line.|
||Displays status information such as the current audit location being tracked, number of audit blocks, updates, and checkpoints. When the mix number of an Extract Worker is used in the AX STATUS command, the response shows the number of records extracted in the current data set, the number of data sets completed, and the total number of records extracted by the Extract Worker.|
This API provides access to DBEngine and the Support Library to retrieve structural information, layout information, and data from audit files and a DMSII database. All Databridge Accessories use the Databridge API. Following are examples of what you can do with the Databridge API at your site:
- EDP Auditors can develop Accessories to monitor changes being made to all or selected records.
- Applications can be developed to generate summary reports of changes made to the database (for example, net change to a dollar amount field).
- Applications can be developed to report history records of all changes made to the primary database.
Sample Accessories for the Databridge API include the following:
- SQLGen (ALGOL)
- COBOLGen (ALGOL)
- DASDLGen (ALGOL)
- AuditClose (ALGOL)
- ReadDoc (ALGOL)
- ExtractAddress (COBOL)
Source is available for each of these. Each serves as an example of how you can use the Databridge API. For more about the Databridge API, see the Databridge Host Programmer’s Reference.
Database Definition Retrieval
DBEngine provides several entry points for retrieving database definition information. These entry points are useful for programs that need to format data records or facilitate mapping from the DMSII database to the client database. The predefined formatting procedures in the Support Library use these entry points to load the arrays used for driving the formatting routines. For more information about entry points, refer to the Databridge Host Programmer’s Reference.