Skip to content

Out of Order Processing

Occasionally you need to load a large amount of flat file data that has been archived, and the archived data is not available in the proper sequence. For example, you might have a set of backup tapes with SUMLOGs for several days and one set of tapes in the middle of the sequence is currently “on loan.”

The SUMLOG Reader library (and other Reader libraries) rely on the timestamp from StateInfo to locate the current file and the ABSN concatenated with INX to determine the record number.

In general you want to keep two audit locations—one for the “real” (relatively current) location, and another for the archive-catch-up location. The following is one approach to handling out-of-order processing. This procedure is for processing archived log files sequentially. Processing archived log files discontinuously requires additional techniques, which are not explained in this guide.

Before You Process

On the host, define a duplicate SOURCE in DBServer (call it SUMLOGARCHIVE) and add a STOP BEFORE timestamp condition to it, where timestamp is the timestamp of the oldest record that has already been loaded using the regular SUMLOG source. (This will prevent picking up the same files twice, including the active SUMLOG.) The SUMLOGs for this SOURCE could be in a different directory to avoid confusion.

On the client, you need the two audit locations and a way to switch them back and forth. Create a new table (AudLocSwitch) to hold the variable parts of the audit location. The variable parts are: AFN, ABSN, INX, and timestamp. Also define a text column in AudLocSwitch called “source_name.”

Create a row for each data set in AudLocSwitch with all zeros for the audit location and source_name = "SUMLOG". Create a similar row for each data set having source_name = "SUMLOGARCHIVE".

Process Archived SUMLOGs

To process archived SUMLOGs

  1. Copy the audit locations from the DATASETS table into the AudLocSwitch rows for source_name = "SUMLOG".

  2. Copy the AudLocSwitch rows for source_name = "SUMLOGARCHIVE" into the data sets table.

  3. Change data_source in the DATASOURCES table and the DATATABLES table to "SUMLOGARCHIVE".

  4. Run the following command: dbutility process SUMLOGARCHIVE

Process Current SUMLOGs

To process current SUMLOGs

  1. Copy the audit locations from the DATASETS table into the AudLocSwitch rows for source_name = "SUMLOGARCHIVE".

  2. Copy the AudLocSwitch rows for source_name = "SUMLOG" into the DATASETS table.

  3. Change data_source in the DATASOURCES table and the DATATABLES table to "SUMLOG".

  4. Run the following command:

    dbutility process SUMLOG

By maintaining separate audit locations and using the STOP BEFORE condition, you can continue to copy in old SUMLOGs, process them, and alternately switch back to processing the current SUMLOGs using the other SOURCE. If the old files are loaded out of order, before you can do the process command, you must manually remove the processed files and reset the audit location in the AudLocSwitch rows for "SUMLOGARCHIVE" to 0.