PreviousRebuilding Indexed Files

Chapter 15: File Conversion Utilities

This chapter describes the following utilities that Net Express  provides to help you move files between the mainframe and the PC:

File Conversion Utility
Use to:
DFCONV Convert downloaded data files to the required format on the PC.
VRECGEN Modify mainframe files into the format required for download from the mainframe to the PC.
VRECGEN2 Modify files uploaded from the PC into the format required on the mainframe.

15.1 DFCONV Batch File Conversion

The DFCONV utility, also known as the Data File Converter, converts data files. The utility provides a batch and called program interface to enable conversion of file formats, organizations and data formats and reorganization of indexed files using data file conversion profiles.

Use the Data File Converter to:


Notes:


15.1.1 Operation

The following sections describe the two Data File Converter operations:

15.1.1.1 Command Line Batch Interface

The command line is:

run dfconv profile-fname [input-fname] [output-fname]

where the parameters are:

profile-fname The name of the profile file that contains the details of the task you need to perform (see the section Defining File Conversion with a Profile File later in this chapter).
input-fname The name of the input file (optional).
output-fname The name of the output file (optional).

The input and output file-names are optional as they can be extracted from the profile file. If you specify file-names on the command line as well as in the profile file, those on the command line take precedence.

After using RUN DFCONV, you should manually update the English language messages by typing the following four lines at a new Net Express command prompt:

CD ..\BIN
RUN LIBRARY DFCV.LNG MFLANGDF.LBR = MFLANGDF.LBR
RUN LIBRARY DFCV.LNG MFLANG01.LBR = MFLANG01.LBR
RUN LIBRARY DFCV.LNG MFLANG05.LBR = MFLANG05.LBR

After conversion, the operating system error level (ERRORLEVEL) is set to non-zero if an error occurred during conversion. If you are running the conversion from a batch file, you can interrogate this to determine what action to take after the conversion.

15.1.1.2 Call Batch Interface

If a profile for a conversion exists, you can invoke the Data File Converter to do the conversion using the following interface:

call dfconv using dfconv-params

where dfconv-params comprises:

 01 dfconv-params.
     03 profile-filename    PIC X(65).
     03 input-filename      PIC X(65).
     03 output-filename     PIC X(65).

and where the parameters are:

profile-filename The name of the profile file containing the details of the data file conversion (see the section Defining File Conversion with a Profile File later in this chapter).
input-filename The name of the input file.
output-filename The name of the output file.

If you specify file-names in the parameter fields as well as in the profile file, those in the parameter fields take precedence.

On return from the Data File Converter, RETURN-CODE is set to non-zero if an error occurred, and the error number is returned in the profile-filename field.

15.1.2 Using the Data File Converter

The following sections describe how to use the Data File Converter to convert data files and rebuild file indexes.

15.1.2.1 File Format and Data Conversion

Converting files from one format to another enables data to be restructured from one format to another. For example, you can convert from sequential organization to indexed organization. You can also add extra secondary indices as the needs of an application change. When using the Data File Converter, you do not need to recreate data already produced or write special conversion programs.

Although many file transfer packages convert files from one format to another, they are restricted to converting the whole file to the other data format. Typical COBOL data files and database records have complex record structures with a mix of text and binary data that must be translated accordingly. The GUI version of the Data File Converter provides an easy method for data conversion using record layout descriptions. For more information on how to create a record layout file, see the section Creating a Record Layout File in the chapter Maintaining and Creating Data Files in your Getting Started.

The Data File Converter can also convert internal floating-point formats between IEEE format and S/370 format.

15.1.2.2 EBCDIC to ANSI Conversion

A mainframe file should be transferred to the PC in binary format. If, initially, this file is a variable-length indexed file, use the mainframe program VRECGEN to perform an initial reformat of the file on the mainframe. The Data File Converter can then download and process the file.

15.1.2.3 Mainframe to PC Print Conversion

A mainframe report format file is either a file downloaded to the PC from the mainframe or a file created by an application, downloaded to the PC and run on the PC.

A mainframe report format contains carriage-control characters which are converted by the Data File Converter to PC printable formats. When defining the mainframe report files, the record length includes the carriage-control character and the records must be fixed length. In PC Print files, the record length is solely the data length.

You can print the PC print format files on any standard ANSI printer capable of printing 132 characters per line. As some ANSI printers print only 80 characters per line, you might have to print your files in compressed mode to achieve this.

If you specify the input file as a mainframe report format file, but it was created on the PC using a downloaded mainframe application, you may receive error message 139 ("Input file not "real" mainframe report format"). This can occur if the printer output has been created in PC format rather than the mainframe report format. Check directives used to control the compilation and execution of the program.

The Data File Converter provides specific support for conversion between Mainframe Report files and PC Print files. Conversion to or from these formats to other formats is not supported.

15.1.2.4 Indexed File Reorganization

As indexed files are updated by adding, deleting and altering records, the index and data structures become disjointed, making processing less efficient. In addition, space left by deleted records is not always reused, making the file bigger than it needs to be. After a large number of changes have been made, it is worth reorganizing an indexed file. By ensuring that the input and output file settings are identical, the data and indices can be rebuilt in an ordered sequential manner, reclaiming any free space, and so providing optimum file performance and integrity.

15.1.2.5 Defining File Conversion with a Profile File

This section describes the entries required in the profile file to convert a file, or to restructure a file from one organization to another. For example, you might need to do one of the following:

DFS Profile File Version  V09.B01.07

Batch-Process: Convert-file
                   ***************************************************
                   *           (Twin)                  (Single)      *
                   * Options:  Convert-File            Rebuild-index *
                   *           Rebuild-index-and-data  None          *
                   ***************************************************
Strt-file:              drive:\path\filename.STR
Edit-Mode:     Quick
                   * Options:  Quick/Full                            *
--------------------------------------------------------------------------------
Input-File     drive:\path\inputfile.dat
   Format:         Micro-Focus
                       * Options:  Micro-Focus         IDXFORMAT(4)  *
                       *           Btrieve             C-ISAM        *
                       *           LEVEL-II            ESDS          *
   Organization:   VRECGEN
                       * Options:  Sequential  Line-Sequential       *
                       *           Indexed     Mainframe-Report-ANSI *
                       *           Relative    Mainframe-Report-mach *
                       *           VRECGEN     PC-Print              *
   Record-Format:  Variable
                       * Options:  Fixed/Variable                    *
   Character-Set:  EBCDIC
                       * Options:  ASCII/EBCDIC                      *
   Floating-Point: 370
                       * Options:  IEEE/370                          *
   Compression:    Off
                       * Options:  On/Off                            *
   Min-Rec-Length: 352
   Max-Rec-Length: 633
--------------------------------------------------------------------------------

Output-File    drive:\path\outputfile.dat
   Format:         Micro-Focus
   Organization:   Indexed
                       * Options:  Sequential  Line-Sequential       *
                       *           Indexed     Mainframe-Report-ANSI *
                       *           Relative    PC-Print              *
                       *           VRECGEN2                          *
   Record-Format:  Variable
   Character-Set:  EBCDIC
   Floating-Point: 370
   Compression:    Off
   Min-Rec-Length: 352
   Max-Rec-Length: 633
                       * Key to settings                             *
                       *       d=      allow duplicates              *
                       *       s=      sparse character set          *
                       *       Compression                           *
                       *           cd= on duplicates                 *
                       *           cl= on leading spaces             *
                       *           ct= on trailing spaces            *
   Prime-key :   1:1, 2:6

   Alt-Key  1:   8:1
                 d,cl
   Alt-Key  2:   9:1
                 d,cd,ct
   Alt-Key  3:   10:5

The first record in the file identifies the format of the profile file. The profiles used by DFCONV are the same format as used with earlier Workbench.

15.1.2.5.1 Specifying the Conversion Operation

The operation to be performed is chosen as a parameter to the Batch-Process option.

Batch-Process The operation to be performed on the defined file(s):

Convert-file
Rebuild-index
Rebuild-index-and-data

An Input-File is defined for all operations and an Output-File is defined for Convert-file and Rebuild-index-and-data. The Rebuild-index operation recreates the file index for the input data file so an Output-File description is not required.

Strt-file The file containing the Record Layout descriptions for the input file.

No-structure
location\filename.str

If the data file is being converted between EBCDIC and ANSI and the records contain non-text data items, a record layout description is required to define the data items requiring conversion and those to be left unconverted (for example, binary values). The record layouts are created with the Record Layout Editor and stored in files with an .str file extension. For more information on how to create a record layout file, see the section Creating a Record Layout File in the chapter Maintaining and Creating Data Files in your Getting Started.

Edit-Mode The edit mode if this profile is read by the Data File Editor

Quick
Full

This parameter is not used by DFCONV.

15.1.2.5.2 Specifying the Input File parameters

First of all, choose an input file:

Input-File The input file name for the defined operation.

Then, use parameters to describe the input file:

Parameter
Description
Format The file format for the input file.

Micro Focus
ESDS
Btrieve
IDXFORMAT(4)
C-ISAM
LEVEL-II

The default format is Micro Focus and this would normally be used for files downloaded from the host and recreated on the PC. If the chosen file is an emulation of the mainframe VSAM entry-sequenced data set, choose ESDS to provide the emulation of VSAM ESDS on the PC.

Organization The file organization for the input file.

Line sequential
Sequential
Indexed sequential
Relative
VRECGEN
ANSI Mainframe Report (The mainframe report with ANSI carriage control characters)
Machine Mainframe Report (The mainframe report with machine code carriage control characters)
PC Print.

Record-Format The record format for the input file.

Fixed
Variable

Note: Variable length Mainframe Report files are not supported.

Character-Set The character encoding used in the input file.

ASCII
EBCDIC

Note: The character encodings used in Net Express are EBCDIC and ANSI. ASCII is used in earlier Workbench but represents ANSI in this utility.

Floating-Point The type of floating-point format that is used in the input file.

IEEE
370

Compression An indication that data compression is used for the input file.

On
Off

Note: this should only be set to On if the file was created with data compression.

Min-Rec-Length The minimum record length for the input file.

If the file is fixed length this should be set to the same value as the maximum record length.

Max-Rec-Length The maximum record length for the input file.

If the file being created is indexed, the keys are defined using the following records.

Prime-key The start position (column) and length of the prime key.

position:length (for example, 1:12). A colon is used to separate the position and length fields.

A second record in the profile file sets the key flags required for the indexed file being created. The flags are separated by a comma (for example: d,cd).

d allow duplicate record keys
s sparse character - not supported
cd compress duplicate record keys
cl compress leading spaces on record keys
ct compress trailing spaces on record keys

15.1.2.5.3 Specifying the Output File

First of all, choose an output file:

Output-File The output file name for the defined operation.

Then, use parameters to describe the output file:

Parameter
Description
Format The file format for the output file.

Micro Focus
ESDS
Btrieve
IDXFORMAT(4)
C-ISAM
LEVEL-II.

The default format is Micro Focus and this would normally be used for files created on the PC. If the file is required to be an emulation of the mainframe VSAM entry-sequenced data set, select ESDS to provide the emulation of VSAM ESDS on the PC. For VSAM KSDS and RRDS, select Micro Focus format.

Organization The file organization for the output file.

Line sequential
Sequential
Indexed sequential
Relative
VRECGEN2
ANSI Mainframe Report (The mainframe report with ANSI carriage control characters)
PC Print.

Record-Format The record format for the output file.

Fixed
Variable

Note: Variable length Mainframe Report files are not supported.

Character-Set The character encoding to be used in the output file.

ASCII
EBCDIC

Note: The character encodings used in Net Express are EBCDIC and ANSI. ASCII is used in earlier Workbench but represents ANSI in this utility.

Floating-Point The type of floating-point format that is used in the output file.

IEEE
370

Compression An indication that data compression is used for the output file.

On
Off

Min-Rec-Length The minimum record length for the output file.

If the file is fixed length, this should be set to the same value as the maximum record length.

Max-Rec-Length The maximum record length for the output file.

If the file being created is indexed the keys are defined using the following records.

Prime-key The start position (column) and length of the prime key. Note that the start position (ie offset) begins at 1; whereas from the Key Information dialog box in the GUI version of the Data File Converter, key offsets start at zero.

position:length (for example, 1:12). A colon is used to separate the position and length fields.

A second record in the profile file sets the key flags required for the indexed file being created. The flags are separated by a comma (for example: d,cd).

d allow duplicate record keys
s sparse character - not supported
cd compress duplicate record keys
cl compress leading spaces on record keys
ct compress trailing spaces on record keys

Alternative keys are defined following the prime key by using "Alt-Key 1:" as the key word for the first alternate key and "Alt-Key 2:" for the second alternate key. The format of the two records in the profile file defining the alternate keys are as defined for the Prime key.

You are allowed to define up to 63 alternate record keys by specifying, for each, the POSITION and LENGTH in the record and whether duplicate values are permitted. Each key can be split and you can define overlapping keys.

Split keys are defined by including multiple POSITION and LENGTH definitions in the key definition record separated by comma and then a space (for example, 1:12, 20:5).

15.1.2.5.4 General Conversion Information

You should be aware of the following points when defining the input and output files:

15.1.2.6 File Rebuild

The interface to Rebuild from the Data File Converter supports the rebuilding of corrupt indices.

DFS Profile File Version  V08.U04.01

Batch-Process: Rebuild-index
                   ***************************************************
                   *           (Twin)                  (Single)      *
                   * Options:  Convert-File            Rebuild-index *
                   *           Rebuild-index-and-data  None          *
                   ***************************************************
Strt-file:     No-structure
Edit-Mode:     Quick
                   * Options:  Quick/Full                            *
--------------------------------------------------------------------------------
Input-File     e:\test\testidx.dat
   Format:         Micro-Focus
                       * Options:  Micro-Focus         IDXFORMAT(4)  *
                       *           Btrieve             C-ISAM        *
                       *           LEVEL-II            ESDS          *
   Organization:   Indexed
                       * Options:  Sequential  Line-Sequential       *
                       *           Indexed     Mainframe-Report-ANSI *
                       *           Relative    Mainframe-Report-mach *
                       *           VRECGEN     PC-Print              *
   Record-Format:  Variable
                       * Options:  Fixed/Variable                    *
   Character-Set:  ASCII
                       * Options:  ASCII/EBCDIC                      *
   Floating-Point: IEEE
                       * Options:  IEEE/370                          *
   Compression:    Off
                       * Options:  On/Off                            *
   Min-Rec-Length: 352
   Max-Rec-Length: 633
                       * Key to settings                             *
                       *       d=      allow duplicates              *
                       *       s=      sparse character set          *
                       *       Compression                           *
                       *           cd= on duplicates                 *
                       *           cl= on leading spaces             *
                       *           ct= on trailing spaces            *
   Prime-key :   1:1, 2:6

   Alt-Key  1:   8:1
                 d
   Alt-Key  2:   9:1
                 d

The Rebuild function can be used for rebuilding index sequential files only. To rebuild other file formats, use the file conversion function with identical settings for the input and output files.


Note: If you are using a profile file, as described above, any offsets you define in the Prime-key field start at 1. However, if you use the GUI version of the Data File Converter, key offsets start at zero from the Key Information dialog box. To find out more about the Data File Converter, see the section Converting Data Files in the chapter GUI Data Tools.


15.1.2.6.1 Recovery of Corrupt Indexed Files

Indexed files that have a corrupt index can be recovered using this option. Indices can become corrupt if, for example, a power loss occurs when a program has opened and changed an index file, but has not yet closed the file. A flag, indicating that this has occurred, is detected in the file and reported by subsequent accesses to that file. No further operations can be done on the file until its integrity has been recovered.

Rebuild rebuilds just the index, which is all that is required for the recovery. If the index is already in existence, Rebuild tries to recover the index using the information in that file.

15.1.2.6.2 General Rebuild Information

You should be aware of the following points when using Rebuild:

15.1.3 Data File Converter Error Messages

The error messages are described in the Net Express online help. (Click Help Topics on the Help menu. Then, on the Contents tab, click Reference, Error Messages, Data File Converter messages.)

15.2 VRECGEN

VRECGEN is a mainframe utility. Use VRECGEN to create a copy of a variable-length mainframe file. That copy embeds record length information in the data records for download from the mainframe to the PC. Micro Focus provides the source code for the VRECGEN utility (vrecgen.cbl). However, you may need to modify the source code before use, to specify options not catered for by the supplied version vrecgen.cbl.

The file created by VRECGEN has a two-byte record length field preceding each of the variable length data records. The record length field contains a value which is the two-byte record length field plus the length of the variable length data record. This is necessary because most file transfer software (for example, IBM SEND/RECEIVE) works with logical records and strips off the "Record Descriptor Word" which contains the length for each record. Without this information, it is impossible to separate the records to rebuild the data set on the PC.

The file created by VRECGEN is in a special download format. You should use the standard Transfer utility to transfer this file to the PC in binary format. The file can then be processed through the Data File Converter.


Note: The source code to VRECGEN (vrecgen.cbl) is provided by Micro Focus primarily as a model, rather than as a final utility that meets the needs of all sites without modification. You should edit your version of vrecgen.cbl so that it contains the specific values that meet your application or site requirements. A typical reason for editing vrecgen.cbl is to increase the maximum record length to be more than 32,000 bytes.


15.2.1 Installing VRECGEN

If you plan to download variable-length test files, you must first upload and compile the VRECGEN program in your mainframe environment. This program is provided as a sample COBOL program for converting mainframe variable-length format files.


Note: Each site should review the provided source code and determine maximum and minimum lengths for its variable-length files. These values (plus 4 for the MVS "Record Descriptor Word") should be used in the OCCURS DEPENDING ON clause of the record descriptions in the sample FDs.


To install VRECGEN:

  1. The source code file for VRECGEN is copied onto the PC as part of the installation procedure; the file name is vrecgen.cbl and it copies to the \Program Files\MERANT\Net Express\base\source directory

  2. After you install Net Express, upload the file vrecgen.cbl to the mainframe, specifying conversion from ANSI to EBCDIC. Verify that the file is in fixed file, 80-byte card image format.

  3. Compile VRECGEN on your mainframe using your COBOL Compiler. When compiling this program, specify the following options. Some are compiler options, some run-time system options, and some are link-edit options. Use only those applicable to your specific environment and compiler:


Note: If you use VS COBOL II, Release 3 or above, and receive a file status of 39, recompile VRECGEN using the CMPR2 compile option.


15.2.2 Running VRECGEN

You need to supply the appropriate JCL information before running the VRECGEN on the mainframe. Enter the following JCL skeleton pointing to the correct STEPLIB, input data set and output data set:

//MYJOB   JOB (ACCOUNTING INFORMATION)
//STP1    EXEC PGM=VRECGEN
//STEPLIB DD DSN=STEPLIB,DISP=SHR
//SYSOUT  DD SYSOUT=*
//INPDD   DD DSN=INPUT.OS.DATA.SET.DISP=SHR
//OUTDD   DD DSN=OUTPUT.OS.DATA.SET,
//        DISP=(NEW,CATLG,DELETE),
//        DCB=(RECFM=VB,LRECL=LLL,BLKSIZE=BBB),
//        SPACE=(TRK,(X,Y),,RLSE),
//        UNIT=SYSDA

Notes:


15.3 VRECGEN2

VRECGEN2 is a mainframe utility that converts files in Micro Focus variable-length record format to mainframe format. Thus it enables you to upload these files and use them on the mainframe.

15.3.1 Installing VRECGEN2

If you plan to convert files in Micro Focus variable-length record format to mainframe format, you must first upload and compile the VRECGEN2 program in your mainframe environment:

  1. The source code file for VRECGEN2 is copied onto the PC as part of the installation procedure. The file name is vrecgen2.cbl and it is copied to the directory \Program Files\MERANT\Net Express\base\source

  2. After you install Net Express, upload the file vrecgen2.cbl to the mainframe, in the same way as for VRECGEN, using CRLF and ANSI as parameters

15.3.2 Running VRECGEN2

The following steps show you how to upload a variable length data file whose records are between 100 and 400 bytes long:

  1. Convert your variable-length PC file to VRECGEN2 input format using the Data File Converter. Specify the minimum and maximum record lengths as 100 and 400 respectively. Note that because the Data File Converter prepends two bytes containing the record length, the actual minimum and maximum lengths are 102 and 402 respectively. If the file is in ANSI format, you should also convert it to EBCDIC format at the same time.

  2. On the mainframe, allocate a data file to receive the file from the PC. Use the following parameters:

    FORMAT = VB

    RECLEN = 406

    BLKLEN = 410

  3. Upload the file. Don't set CRLF insertion. If you converted to EBCDIC at step 1, don't specify ANSI.

  4. Change the source of vrecgen2.cbl on your mainframe so that the FD for the output file matches the format of the required mainframe data file

  5. Compile and link vrecgen2.cbl

  6. Run VRECGEN2. The JCL is the same as for VRECGEN, except for the following changes:

A valid variable length file should result as the output. The following shows the format of a VRECGEN2 format file, using a record length of 100 bytes as an example:

Bytes
Contents
1-4 Record Descriptor Word.
5-6 Record length.
7-106 Record data. The record is padded with null bytes if it is shorter than the maximum record size.


Note: The VRECGEN and VRECGEN2 file formats are incompatible in that VRECGEN format is used to download files from the mainframe and VRECGEN2 format is used to upload Micro Focus format files to the mainframe.



Copyright © 2000 MERANT International Limited. All rights reserved.
This document and the proprietary marks and names used herein are protected by international law.

PreviousRebuilding Indexed Files