Skip to content

DMSII Reorganization and Rollbacks

This section lists changes that can occur to the DMSII database, how those changes affect the Databridge Flat File Client control and data files, and how to handle them in the Client. For instructions on handling a DMSII reorganization on the host, see Prepare for a DMSII Reorganization in Chapter 11 of the Databridge Host Administrator's Guide.

Initializations

A DMSII initialization occurs when a DMSII data set is purged of its records. When a data set is initialized, the Databridge Engine sends the Databridge Client a stateinfo record with a mode value of 4. The Databridge Client performs the actions described below after displaying the following message:

DataSet name[/rectype] has been purged

The Client deletes all of the files belonging to this data set and re-creates them, effectively purging the files of all records. When DBEngine is done sending stateinfo records, it sends a status of DBM_PURGE(21), causing the client to display the following message:

DataSets purged by DBEngine

The normal update processing will repopulate them.

Note

An initialization does not change the data set format level in the DMSII database.

Reorganizations

Although there are basically three types of DMSII database reorganizations (see DMSII Reorganizations and Rollbacks in the Databridge Host Administrator's Guide) -- record format conversion, file format conversions, and garbage collection reorganizations -- the types of reorganizations are not as important as whether the reorganization changes record layouts or record locations, as follows:

  • DMSII record format conversions change record layouts. When a data set is affected by a record format conversion, you must apply parallel changes to the flat file layouts. See Managing DMSII Changes to Record Layout in the Databridge Client Administrator's Guide.

  • DMSII file format conversions and garbage collection reorganizations change record locations. Only certain data sets require recloning in this case. See DMSII Changes to Record Locations in the Databridge Client Administrator's Guide.

Note

Filler substitutions are handled the same way record format reorganizations are. In a filler substitution, the item count changes.

Managing DMSII Changes to Record Layout

Use this procedure if a DMSII reorganization changes the layout of records. DMSII record layouts are changed in the following circumstances:

  • Record format conversion (also called structural reorganization in this section)

  • Filler substitutions

When the Databridge Engine notifies the Databridge Client that the layout of records have changed, the Databridge Client returns a message for each reorganized data set and then prompts you to run a redefine command. It then returns an exit_status value of 2 (DMSII reorganization). The redefine command can determine whether the layout for the data tables have been affected by the DMSII layout change and if the affected data sets need to be recloned. (For more information about what the redefine command does, refer to the Databridge Client Administrator's Guide's section About the redefine Command.

To run the redefine command

If you use the Client Configurator to customize the table layouts, skip steps 1 through 4 and run the Client Configurator instead. It will perform the same actions as the redefine command, but will also allow you to make customizations.

  1. If the DMSII changes are extensive or complex, we recommend that you back up the Client control tables first, before proceeding.

  2. Modify user scripts as required by the host layout changes.

  3. Run the redefine command as follows:

    dbutility redefine datasource

  4. If the redefine command results in errors because the user scripts were improperly updated, run a reload command using the unload file created in step 1 (or the unload file created by the redefine command itself, if you skipped step 1) to restore the control tables. Correct the user scripts and rerun the redefine command until no errors result.

  5. Unless you use the filename_mask parameter to make the file names unique, you will need to move the data files affected by the reorganization to a backup directory, as the new files will have different record layouts. When you are ready, switch the ds_mode column of any data set whose ds_mode is 31 back to 2. Unlike the relational database client there is no reorg command that does this.

    dbutility process datasource

The same principles apply to the user data files which include AA Values used as keys by the applications that process the files created by the Flat File Client. If the ds_mode column in the corresponding row in the DATASETS Client control table is 0, the data set will be recloned, which means that all data files associated with the data set will be repopulated with extracts. Refer to Recloning in Chapter 5 of the Databridge Client Administrator's Guide for details on when you need to reclone data sets.

Rollbacks

A DMSII "rollback" describes restoring the DMSII database to an earlier point in time to correct a problem. While restoring the DMSII database, replication stops. You must then restart the Client to resume replication.

If the Client has processed updates after the DMSII restore point, this replicated data will be wrong. Upon finding bad information in the stateinfo, the Databridge Engine typically returns a message stating that a rollback has occurred. To resolve this problem, the user data files must also be restored (to the DMSII restore point or earlier).

If the Client hasn't processed updates after the DMSII restore point, no action is required. This can often be the case as the Client tends to lag behind the DMSII database by several audit files.

The easiest way to achieve this is to periodically save off the unprocessed data files and keep track of the stateinfo in case you have to restore the files to their earlier state. You can save the stateinfo by unloading the Client control tables.

Recloning

Reasons for recloning include the following:

  • DMSII reorganization

  • DMSII rollback

  • An update is not possible (for example, because a table does not have a unique key)

You can use either the process or clone command to reclone data sets. The clone command lets you specify individual data sets on the command. The process command automatically reclones all data sets whose active column is 1 and whose ds_mode column is 0. Both commands perform fixups, tracking, and processing updates as needed (unless the defer_fixup_phase or the stop_after_fixups parameter is set to True). Refer to Recloning in the Databridge Client Administrator's Guide for more details on this topic.

Recloning Individual Data Sets

Use one of the following procedures to reclone data sets.

To reclone with a process command:

  1. Set the current directory to the one you created for the data source.

  2. Set the ds_mode column (in the DATASETS Client control table) to 0 for the data sets you want to clone by running a SQL command. If you are recloning all data sets, using the " -Y reclone_all" options eliminates the need to do this, as the client will update the DATASETS table automatically when this option is used.

  3. Run the process command with the -y option, as follows:

    dbutility process -y datasource

    The -y option forces any data sets whose ds_mode is set to 11 or 12 to be recloned, in addition to the recloning data sets whose ds_mode is set to 0. After the data extraction process is complete for the data sets being recloned, flat files whose active columns are set to 1 in their corresponding Client control table (and whose ds_mode is set to 1 or 2) are updated.

To reclone with a clone command:

  1. Set the current directory to the one you created for the data source.

  2. Set the parameter defer_fixup_phase to True to suspend audit file processing. If you don't do this, audit files will be processed twice, once for the data set you clone and once for all of the other data sets.

  3. Synchronize the tables by running a process command. Synchronization occurs when all data sets reach the same point in the audit trail.

Refer to Recloning Individual Data Sets in the Databridge Client Administrator's Guide for more details on this topic.

Adding a Data Set

Use this procedure to add a data set after you clone the DMSII database. You don't need to reclone the entire database.

  1. Run a relational database query tool and list the contents of the DATASETS Client control table with the following SQL command:

    select dataset_name, active, data_source from DATASETS

  2. Set the active column for the data set you want to add to the Databridge data tables to 1 (on), as follows:

    update DATASETS set active=1 where dataset_name='datasetname'

  3. Run a redefine command.

  4. Run one of the following commands to populate the new tables that correspond to the new data set

    dbutility process datasource

    --or--

    dbutility clone datasource datasetname

Note

If you run a process command, the Databridge data tables whose active columns are set to 1 in their corresponding Client control table are also updated.

After you complete this procedure, update your data set global mapping script (script.user_datasets.datasource) so that you do not lose this change the next time you run a define command.