PreviousDebugging Tutorials Library RoutinesNext

Chapter 6: FaultFinder

This chapter describes the FaultFinder utility.

6.1 Introduction

FaultFinder is an aid to debugging production applications. It enables you to to obtain a text-based report on the state of an application at the time an application terminates abnormally. The report is a snapshot of the application at the time of failure and identifies items such as open files, file buffers, loaded programs, and so on.

6.2 Operation

FaultFinder is controlled by run-time tunables. To turn on FaultFinder set the tunable faultfind_level to a positive value. Then, if the application terminates abnormally, the run-time system automatically invokes FaultFinder which in turn generates the report. For details of the report, see the section FaultFinder Report.

You can make a program start FaultFinder at any time by using the CBL_FFND_REPORT library routine in the progam. For details, see the chapter Library Routines.

You can also start FaultFinder using the command line. See the section Running from the Command Line for details.

The following run-time tunables control the operation and output of FaultFinder:

faultfind_level=n Turns on FaultFinder and enables tracing of events. n can be any positive value less than or equal to 1024 and enables the last n events to be traced. The default value is 0.
faultfind_outfile="[>>]filename" The file to which the analysis is written. If filename contains the string %p, this string is replaced by the process ID of the program that started FaultFinder. The default filename is ffnd_out.%p in the current working directory. If you specify >>, the output is appended to filename, if it already exists, otherwise a new file named filename is created.
faultfind_recsize="record_size" A numeric value that specifies the internal cache size used to store read/write data for the operation traceback log. All read/write data is stored and reported. By using this tunable you might improve performance. The default value is 64 and the maximum is 1024.

For detailed information on a tunable, click the tunable in the preceding table. For information on setting run-time tunables, see the chapter Run-time Configuration in your Server Express User's Guide.

set faultfind_level=5
set faultfind_outfile="applflt.log"
set faultfind_recsize=1024

6.2.1 FaultFinder Performance

Enabling FaultFinder event tracing (that is, setting faultfind_level to a value greater than 0) does have a performance impact, although it should not be noticeable in most cases. The number of events traced does not significantly impact performance; tracing one event is not significantly faster than tracing 1000. The most expensive operations to trace are file operations; the worst case for FaultFinder is a file handling benchmark. Even if this is the case, you would expect no more than a 5% performance degradation. Most applications which perform a mixture of operations should see minimal impact on performance.

6.2.2 Running from the Command Line

You can run FaultFinder from the command line to create a report on any COBOL program created using Server Express.

The command is:

cobffnd[_t] [options] pid1 [pid2 …]

where the parameters are:

_t Specifies the multi-threading version of the command. Use this if the program on which you want a report is a multi-threading program.
options Controls the FaultFinder reports.
pid1 [pid2 …] The process IDs for the programs on which you want FaultFinder reports.

options can be any combination of the following:

-m Produce a COBOL memory summary report.
-f fname Set the output filename to fname (fname is overwritten if it exists).
-F fname Set the output filename to fname (output is appended if fname exists).

If fname is - (a hyphen), the standard output is used. If fname contains the string %p, that string is replaced by the current process ID.

If the target COBOL process has the run-time tunable faultfind_level set to 0, you cannot obtain operation traceback or open file information in the report file.

6.3 FaultFinder Report

The FaultFinder report contains information on the:

If you run FaultFinder from the command line, you can also create a memory summary report. This information is contained in titled sections in the report. You can search for sections in the report by their titles.

For threaded programs (run with the multi-threading run-time system), additional information is reported; this is discussed in the relevant sections.


6.3.1 Environment

The Environment section of the report describes:

6.3.2 Call Stack

The Call Stack section of the report contains, for each thread running at the time of failure, a call stack traceback. Programs are listed with full pathname and extension (where available). Entry points are not listed. The information for each thread is listed in a reverse calling hierarchy, so that:

If the program contains Local-Storage and/or Linkage Sections, these sections are dumped with the corresponding call stack instance. For these sections, the following information is dumped:

For .int files, the current program-counter is also dumped.

6.3.3 Loaded Programs

The Loaded Programs section of the report contains all currently loaded programs, with their re-entrant and non-re-entrant data areas. The non-re-entrant data area has separate areas for Working-Storage Section data items and each instance of user-allocated data (using CBL_ALLOC_MEM, flag=0). A separate re-entrant data area exists for each thread executing that program and has separate areas for thread-local storage and thread-local allocated data (using CBL_ALLOC_MEM, flag=8). Each chunk of data states the hexadecimal address of the data item and the dump (in hexadecimal and ASCII with duplicate lines not repeated but replaced by a single * (asterisk)) of the data.

Note: The Working-Storage Section includes all of the non-user program specific data areas, as in the File-Control and File Sections, as well as system needed non-reentrant space. To better define the start of the program's actual Working-Storage Section, we recommend that you insert a special data item as the first item in the Working-Storage Section and give it a value that can easily be searched for.

6.3.4 Logically Canceled Programs

This section of the report contains a list of logically canceled programs, with full pathname and filename extensions.

Files that have been logically canceled are still known to the run-time system, so can be loaded and run directly without having to be loaded from disk. Files that are physically canceled release all their memory and no longer exist to the run-time system.

No other information is available in this section of the report.

6.3.5 Open Files

This section of the report contains information about all opened files, both local (non-external) and externally opened files. For each file the information listed is:

The entire last record and the file control descriptor (FCD) are both dumped (with address) in hexadecimal and ASCII.

The operation trace records the basic operation (READ, WRITE, OPEN and so on) as well as the:

The last two items are only recorded when applicable.

6.3.6 Trace Back

The Trace Back section of the report lists the last n events for each thread, where n is defined by the run-time tunable faultfind_level. For each thread, the operations traced are:

This section is broken down by thread; it records all operations for thread 1, followed by all operations recorded for thread 2, and so on. There are typically two events per operation: one as the operation begins, and another as it finishes. A start event is prefaced with a < while an end event is prefaced with a >.  The last event to have happened is printed first.

6.3.7 Memory Summary Report

The memory summary report is a condensed report that can help you pinpoint which COBOL programs are using memory.

Note: This report is for COBOL memory only; for example, working-storage. It does not include detail on memory at the process or run-time system level.

The following memory summary was generated using a running copy of the Tictac demonstration program.

Program Name                                            WKS      THR      DYN
ADISKEY2                                             2.73KB      0b      74b
ADISKEY                                              4.89KB      0b      73b
bsio                                                10.38KB      0b    2.44KB
ESDSSUB                                             16.68KB      0b     192b
mfini                                                3.32KB      0b   12.78KB
mFFH                                                27.79KB      0b    49.5KB
ADISINIT                                              7.3KB      0b      74b
ADIS                                                 4.35KB      0b      70b
/home/test/tictac.gnt                                2.24KB      0b     129b
Total                                               79.36KB      0b   65.29KB

Local Storage for thread 1308:
        No local storage found.

Total COBOL memory usage: 144.65KB

Each COBOL program found in the run-time system is listed, along with the amount of memory it has allocated to it.

Note: All the COBOL programs that the run-time system knows about are listed, even programs that are canceled but still have memory allocated.

The columns listed are:

WKS Working-Storage Section
THR Thread-Local Storage Section
DYN Dynamically Allocated (for example, by CBL_ALLOC_MEM or by the run-time system)

The CALL/PERFORM stack is examined to see if any active programs have a Local Storage Section. If so, this is reported.

At the end of the listing, the total memory used is shown.

6.4 FaultFinder Listing Generator

To help you analyze a FaultFinder report, you can take a snapshot of a program's source code, and its compiled characteristics, before shipping the program to your end-users. You can create a snapshot using the FaultFinder Listing Generator, which provides information in the form of a detailed listing file.

To create a listing file:

  1. Compile the program for animation

  2. Run the cobfflst command

6.4.1 Invoking cobfflst

To run the FaultFinder Listing Generator, use the command:

cobfflst options filename [listfile]

where the parameters are:

options Specifies the level of detail in the FaultFinder listing. options can be any combination of:
-s Suppresses the source listing
-d Suppresses the data map listing
-p Suppresses the procedure map listing
-x Suppresses the cross reference listing
-ln Specifies a page size of n lines. The default is 66 lines
-v Verbose output with version information
If you specify no options, the Listing Generator produces the full source, data map, procedure map and cross-reference listing. For details of these listings, see below.
filename The name of the .idy file to be processed. You do not have to specify the .idy file extension as this is the default file type.
listfile The name of the file to which the listing is saved. If you do not specify this parameter, the listing is directed to the screen.

Note: The FaultFinder Listing Generator always uses the .idy file, not the COBOL source file.

For example, if you have a program appfile.cbl, the following commands generate a full listing minus any cross-reference information in the listing file appfile.ffl:

cob -a appsource.cbl -C "int("
cobfflst -x appint.idy appsource.ffl

or, for a generated code file:

cob -xg appsource.cbl
cobfflst -x appsource.idy appsource.ffl

6.4.2 Understanding the FaultFinder Listing

The examples in this section were generated for the following program:

 program-id. 'funbar'.
 working-storage section.
 01 ws-var-1    pic x(255).

 thread-local-storage section.
 01 tls-var-1   pic x(1021).

 local-storage section.
 01 ls-var-1    pic x(511).

 linkage section.
 01 lnks-var-1  pic x(2011).
 01 lnks-var-1r pic a(1024) redefines lnks-var-1.
 01 lnks-var-2  pic x(3096).

 procedure division using lnks-var-1 lnks-var-2.
     move tls-var-1 to ws-var-1 
     if ws-var-1 = lnks-var-1 
         display ls-var-1 
     stop run.

This program was compiled to generated code, and the FaultFinder listing generated using the commands:

cob -xg fun.cbl
cobfflst -l19 fun.idy fun.lst Listing Header

The header to the listing provides useful information on the compilation of the program. An example header appears below, with additional notes describing the information presented below:

* * * * * * * * * * * * * COMPILATION SUMMARY * * * * * * * * * * *

        Micro Focus Server Express        V1.1 revision 000

        Product ID   : HXCTG/AA0/00000
        Compiler ref : GNR-071060000AD

        Date compiled: 05-Oct-99
        Time Compiled: 18:20

        Source File: fun.cbl
        .idy File  :   y
        Program ID : funbar
        Compiled to: Gnt/object with xref information

- - - - - - - - - - - - - - - - Sizes  - - - - - - - - - - - - - - -

 INT Code Proc:   000059  Data: 0001C4  Local: 000200 Thread: 0003FE
 GNT Code User:   000178  Full:   0002C8

- - - - - - - - - - - - -  02 Linkage Item(s) - - - - - - - - - - - -
 LNKS-VAR-1                 :   2011 bytes (h"000007DB")
 LNKS-VAR-1R                :   1024 bytes (h"00000400") (redef)
 LNKS-VAR-2                 :   3096 bytes (h"00000C18")

- - - - - - - - - - - -  Key Compilation Directives - - - - - - - - - 

 MF    noOSVS      noVSC2       CHECKDIV        PERFORM-TYPE(MF)
 noBOUND         TRUNC(ANSI)
 noSPZERO        noIBMCOMP

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 

The first section of the listing provides information on the Compiler and the sources compiled. For the Compiler, the product name, product version, product ID and compiler reference number are given. These values are often useful to MERANT support representatives. For the source code, the files compiled, the resulting .idy file, the main entry point of the program and target mode of the compilation are given.

The Sizes section of the report shows the sizes of the various segments defined in the program, in particular:

In the example listing above:

The Linkage Item(s) section of the report lists the Linkage Section data items defined in this program, and their sizes. As these sizes are significant in interprogram communication, they warrant their own section. In the example above, the item LNKS-VAR-1R is a redefinition of LNKS-VAR-1, so there are effectively only two distinct linkage items.

The Key Compilation Directives section of the report lists the Compiler directives that affect run-time behavior. Source Listings

There are two kinds of source listings that appear in the output, the Data Division listing and Procedure Division listing.

Data Division Listing

The Data Division listing looks like this:

fun.cbl, line 1
Line    Loc   :Size  ----+-*A-1-B--+----2----+----3----+----4----+---...-8
000001                        program-id. 'funbar'.
000002                        working-storage section.  
000003  000050:0000FF         01 ws-var-1     pic x(255).
000004  000005                thread-local-storage section.
000006  000000:0003FD         01  tls-var-1   pic x(1021).
000007  000008                local-storage section.
000009  000000:0001FF         01  ls-var-1    pic x(511).
000010  000011                linkage section.
000012  000000:0007DB         01  lnks-var-1  pic x(2011).
000013  000000:000400         01  lnks-var-1r pic a(1024) redefines lnks-var-1.
000014  000000:000C18         01  lnks-var-2  pic x(3096).
000015  000016               procedure division using lnks-var-1 lnks-var-2.

The listing contains a page header, not shown in the example above, that gives product and compilation information. This is followed by the name of the source module that the first line of page originates from and its sequence number relative to that module. For example, the first line of page 1 in the example source listing originates from the module fun.cbl and is line 1 in that module.

After the line showing the module is a column scale; this indicates the COBOL areas of the source line (that is, Area A, Area B, and so on). In the example above, this scale is compressed to allow notations. In this scale line, there is also a header for the sequence line of the source lines being displayed.

Following the column scale is the Data Division code. The first 80 characters of each source line that was compiled are displayed in the body of the source listing. If a source line exceeds 80 characters, the excess characters are not shown.

The header indicates that the location (Loc) and size (Size) of each data item definition is displayed (in hexadecimal) immediately before the actual source line that defines the data item. For example, on line 3, the data item ws-var-1 is defined and it has an offset from the start of statically allocated data of h"50" and a size of h"ff" (decimal 255). Similarly, at line 6 the variable tls-var-1 has an offset of h"00" and a size of h"3FD" (decimal 1021).

Procedure Division Listing

The Procedure Division Listing looks like this:

fun.cbl, line 17                          
IntAddr:GntAddr----+-*A-1-B--+----2----+----3----+----4----+--- … -8   Xref  
000017  00002e                 sec1 section.                          =000017
000018  000030                 par1.                                  =000018
000019  000031:00000CC             move tls-var-1 to ws-var-1          000006 >000003
000020  000039:00000F4             if ws-var-1 = lnks-var-1           ?000003 ?000012
000021  000042:0000160                display ls-var-1                 000009
000022  000049                     end-if
000023  00004B:000016C         stop run.

The listing contains a page header, not shown in the example above, that gives product and compilation information. This is followed by the name of the source module that the first line of the page originates from and its sequence number relative to that module. For example, the first line of page 2 in the listing originates from the module fun.cbl and is line 17.

After the line showing the module is a column scale that indicates the COBOL areas of the source line (that is, Area A, Area B, and so on). In the example listing above, this scale is compressed to allow notations. In the scale line there is also a header for the sequence line of the source lines being displayed.

The column scale contains the headers IntAddr and GntAddr, under which are shown the intermediate code address and the generated/object code address relative to the start of the module, of the actual source line from which the generated code was created. For example, for the syntax at line 19, the intermediate code address is h"31" and the generated/object code address is h"CC". The column scale aslo contains the header Xref, under which is shown the definition line of the data items that are manipulated in the statement, and any operations performed on the data items in the statement. The type of operations is indicated by a character:

= Definition
> Modification
? Test

For example, on line 18 of the listing above, par1 is defined; on line 19, there is a reference to tls-var-1 which is defined on line 6 and there is a modification to ws-var-1 which is defined on line 3. On line 20, there is a test of ws-var-1, which is defined on line 3, and of lnks-var-1, which is defined on line 12. Map Listings

There are two map listings, one for data items and one for procedural items.

Data Item Map Listing

The data item map listing looks like this:

* Data Map Listing  
* Line   Name                           Addr   Size   Sec  Usage   Picture string
  000003 WS-VAR-1                       000050 0000FF WS   Alnum   PIC X(255)
  000006 TLS-VAR-1.                     000000 0003FD TS   Alnum   PIC X(1021)
  000009 LS-VAR-1                       000000 0001FF Lcl  Alnum   PIC X(511)
  000012 LNKS-VAR-1                     000000 0007DB Lnk  Alnum   PIC X(2011)
  000013 LNKS-VAR-1R.                   000000 000400 Lnk  Alpha   PIC A(1024)
  000014 LNKS-VAR-2                     000000 000C18 Lnk  Alnum   PIC X(3096)
  000000 RETURN-CODE                    000150 000002 WS   Comp    PIC S9(4) COMP
  000000 SORT-RETURN                    000154 000002 WS   Comp    PIC S9(4) COMP

This listing shows:

Procedural Item Map Listing

The procedural item map listing looks like this:

*  Procedure Map Listing 
*  Line   Name                           Seg IntAddr  GntAddr  Sec/Par
   000017 SEC1                               000002E           Sec
   000018 PAR1                           
   0000030                                   00000CC           Par

This listing shows: Xref Listings

The Xref listing provides a sorted list of data names. It looks like this:

*  Xref Map Listing 
*  Def'd  Name                    References, ?Tests, >Modifications
   000012 LNKS-VAR-1              0000013 0000016 ?0000020
   000013 LNKS-VAR-1R.            
   000014 LNKS-VAR-2              0000016
   000009 LS-VAR-1                0000021
   000018 PAR1                    
   000017 SEC1
   000006 TLS-VAR-1               0000019
   000003 WS-VAR-1                >0000019 ?0000020
   000016 funbar*
   000001 funbar@

For each data item, the listing shows:

The header to the third column, References, ?Tests, >Modifications, shows the type of operations that are recognized, and the indicators for these operations. For example, lnks-var-1 is defined at line 12, referenced at lines 13 and 16, and tested at line 20, while ws-var-1 is defined at line 3, modified at line 19 and tested at line 20.

A name suffixed with an asterisk (*) indicates the line at which that program's Procedure Division begins, while a name suffixed with the at sign (@) indicates the line at which that program's program id is defined.

Copyright © 2000 MERANT International Limited. All rights reserved.
This document and the proprietary marks and names used herein are protected by international law.

PreviousDebugging Tutorials Library RoutinesNext