Introduction

Making an Inventory

Chapter 2: Verification and Remediation Process

This chapter describes how the tools within SmartFind Plus fit into a Year 2000 strategy, and suggests a procedure for using these tools.

2.1 Year 2000 Process

These processes can serve as a general Year 2000 guideline. Each must be treated as a separate project. There are two main Year 2000 processes:

• Verification - an audit to check that a previously fixed application has been correctly fixed

• Remediation - analysis and fixing of the problem

SmartFind Plus is an appropriate and effective tool for each of these processes. It can:

• Find code that needs fixing or further investigation and categorize for ease of reference

• Enable you examine and understand an application as well as all code involved in date handling

• Store final results in the worksheet (an industry-standard relational database) for ease of tailoring reports and statistics

• Automatically generate code for fixing date related source code

The tools within SmartFind Plus are designed to provide a systematic method to solving date problems in source code.

2.2 Year 2000 Verification

The aim of a verification process is to audit a previously remediated application to ensure that the code has been correctly and completely fixed. SmartFind Plus can verify code regardless of the methodologies or products used to fix the code.

The source code may have been fixed some years previously using relatively unsophisticated technology, or maintained since it had a Year 2000 fix. In either case, good business practice would demand a verification to confirm and document the Year 2000 compliance. Performing a verification prior to testing can reduce the overall testing effort by identifying problems early. This will give you more time to fix the problems.

• If only a few problems are found, then any fixes can be applied as part of the general maintenance process.

• If many problems are found you will have to start a remediation proces.

The Verification Process has five tasks:

• Inventory - The inventory task identifies all application source members and determines if any source code members are missing.

• Preparation - The preparation task is where you decide which categories will be assigned to the POIs for analysis purposes.

• Research - The research task is where you find all Year 2000 POIs, statements and data items, that will fail during or before the Year 2000.

• Analysis - In the analysis task you examine the POIs, categorize statements and data items and examine year types.

• Reporting - In the reporting task you prepare final reports on the verification process.

2.2.1 Inventory

The inventory task identifies all application source members and determines if any source code members are missing. You will need to determine which source code members are available, and which ones are missing. Ensure the source code is free of any significant syntax errors or warnings that might adversely affect the verification results.

1. Build a project using the supplied source code. See the Creating Projects chapter.

2. Check that the project is complete:
   1. In the Project menu select Complete.

   2. Highlight all items and click Expand One Level.

   3. Click Save to File. The results are saved as a text file.

   4. Review the results to check if any files are missing
3. Check any parsing problems:
   1. In the Tools menu select Scripts.

   2. Double-click Parsing Problems.

   3. In the list select List Errors, Warnings and Notices and click GO.

   4. Click Select All and then Ok.

   5. Click Save to File. The results are saved as a text file.

   6. Review the results to check if any significant errors or warnings have to be addressed.

2.2.2 Preparation

The preparation task is where you decide which categories will be assigned to the POIs for analysis purposes. You will configure the functions available in SmartFind Plus to find the POIs in the source code relevant to a verification process.

Note: If you do not see an analysis tool that is described here, it might be excluded from the palette. To access any excluded tools; in the Analysis Tools palette, click Options and then the Tool Palette Editor tab. You can add or remove tools from the tools palette.

Follow the steps below to prepare for a verification project.

1. Decide which final categories will be listed in the verification analysis final reports. If the categories supplied with SmartFind Plus do not meet your specific requirements, you can create your own custom categories. For example:
   • False positives
   • Windowing logic attempted
   • Failing statements
   • Categories that refer to site specific processes
   Make a list of the categories and define when each is applied and what description is attached to the category.

2. Update the default categories accordingly, using Edit Categories in the Administration menu . If this button is not visible, in the worksheet click Options, and in the Worksheet tab check Enable Administration tasks.

3. Review sample Verify reports to ensure they provide the information needed.

4. If you decide to run the year type consistency report check the year type assignments of data items in the worksheet first. Ensure that they identify how a date is stored in the field. The consistency report check will flag if two fields with different year types are in the same statement:
   • Statements such as MOVE CCYYMMDD TO MMDDYYCC
   will be flagged.
   • You may have to verify the year type assignment for each data item later in the process

5. Decide if you want data items on the Verify reports in addition to the failing statements.

6. Check if during source code remediation a list of known dates and/or false positives was produced. If so incorporate that information into Verify - false positives categorization in the Verify - all research tool.

7. You can customize site-specific non-date formats and names as well as non-year types. You can incorporate that information into the following tools:
   • Categorize non-year types extracts from the worksheet all data items whose year-type indicates that the field does not contain a year. The tools assigns category R-NonYearType both to the data items and the statements that contain them.
   • Categorize non-date names examines the data items in the worksheet and classifies them using the pattern list provided. It will assign the category R-NonDateName both to the data items that match the non-date pattern list and any statements that use them. The tool also uses a date-name pattern list to ensure that no obvious date fields are tagged as non-dates.
   • Categorize non-date formats examines data items whose picture string contains non-date elements (decimal points, debit/credit symbols, and currency symbols). Both the data items and the statements they occur in are marked with the R-BadFormat category.

8. There may be other site-specific information that has to be entered. For example, some single-step tools embedded in Verify - supplementary research in the Verify - all research tool, may need input directly related to the source code you are working on. Use the Configuration wizard tool to configure the relevant tool. Values should be in separate text files, so that you can import them easily. The three embedded tools to configure are:
   • Find valid site date routines, in the Verify - supplementary research tool for calls to programs that are valid date routines. You will need a list of valid program-names.
   • Find invalid site date routines, in the Verify - supplementary research tool for calls to date routines that are invalid or not supported. You will need a list of invalid program-names.
   • Find site system date usages, in the Verify - supplementary research tool for uses of application-level system dates, such as posting dates, or run-dates. You will need a list of date data names.

Note: If you customize an individual Analysis Tool, for example, Find valid site date routines, with the Configuration wizard, it will not affect the use of the tool Find valid site date routines embedded in Verify - all research. Similarly, if you configure a tool embedded in a composite tool it will not affect the use of the individual tool.

2.2.3 Research

The Research task is where you must find all Year 2000 POIs, statements and data items, that will fail during or before the Year 2000. All POIs must be moved to the worksheet. Only one tool is required to add code to the worksheet, the Verify - all research tool, which runs three other tools:

1. Verify - SmartFind research - This tool first finds the elements of code that represent dates, based on the size and type of data items, some specific literals, and some other criteria. It then focuses on the lines of code where these elements might cause a logic problem. For example, comparing two date data items across the century boundary causes a logic error, but moving one date data item to another does not.
   These potentially problematic data items and statements are then added to the worksheet with the category R-SF Wizard indicating that the SmartFind wizard found them.

2. Verify - supplementary research - This tool contains a whole range of embedded tools, each one looking for a specific type of date issue, such as truncation of data items containing dates, or negative additions that are in effect subtractions.
   Each embedded tool finds and adds the points of interest to the worksheet, assigning the tool's category to each one. The category indicates the reason why each points of interest is considered an issue. If a POI is found by more than one tool it is added only once, but additional categories are added, so that a POI can have several categories if it is found by several tools.

3. Verify - false positives categorizations - This comprises several tools that examine the POIs already added to the worksheet, to identify any that have some indication that they are not year related or are not a problem for other reason. These POIs are sometimes referred to as false positives.
   Each component tool examines the worksheet for false positives of one sort and assigns the tool's category to any that it finds.

Note: Verify - all research performs a number of extensive and sometimes complex analysis steps against the project database to uncover potential Year 2000 problems. It may run for a significant period of time against your project. We recommend running it overnight when possible.

2.2.3.1 Running the Verify - all research Tool

In this phase you generate an automatic search of the project's source code and add the POIs found to the worksheet.

1. From Analysis Tools run Verify - all research. You may have configured this tool in the preparation phase.

2. When the worksheet is displayed click Options, click the Worksheet tab and select the Find and Verify option.

3. You may know of other special cases that have to be checked for in the source code. Sometimes it is not be possible to check these special cases with the supplied Analysis Tools. We recommend that you use the functions in the Tools menu and add the results to the worksheet.
   • For example, some installations have windowing code that adds 50 to a YY field as part of renovation. To find these cases, you would search for literal 50 (YY), 5000 (YYMM) and 500000 (YYMMDD). You would use the Literals in the Browsers menu and add to the worksheet by right-clicking on the POIs and selecting Add to Worksheet.
4. During your examination of the source code, you may find other data items or statements that need to be added to the worksheet. Indicate in the Notes column why you have added the items and, if possible, assign a category as you add them.

2.2.4 Analysis

In the analysis task you examine the POIs, categorize statements and data items and examine year types.

1. Identify the obvious false positives so that genuine POIs are categorized.

2. Categorize each statement in the worksheet and, if required, any relevant data items.

3. If you have decided to run the year type consistency report, review the year type assigned to each data item in the worksheet and run the consistency report to identify any mismatched year types.

2.2.4.1 Review and Remove Obvious False Positives

1. In the Statements tab of the worksheet review the potential false positives. If at the same time you want to review false positive data items see step 2 below.
   • Use Display Filters in the worksheet to show only statements categorized as R-NonYearType, R-BadFormat or R-NonDateName.
   • If the statement involves only data items that are, in fact, not dates, assign your false positive category to it (for example, A-No, or V-False Pos). If you do not need to report on false positives, you can remove them from the worksheet: highlight the false positive POI and click
   Remove from list.
   • If the statement contains a date it needs investigation. Give it an appropriate category.
2. If you need false positive data items to be included in reports, you will also need to work through the Data Items tab. Assign your false positive category to the R-NonDateName and R-BadFormat items. If you do are going to run a year type consistency check, you should remove them from the worksheet: highlight the false positive POI and click Remove from list.

   You can do this at the same time you review the statements: right-mouse click on the statement and select Show contained data items to display the data items.

2.2.4.2 Categorize POIs in the Worksheet

In this phase you will configure the worksheet to hide statements and data items that have been assigned a final category. During your initial search of the source code with the Analysis Tools, POIs will have been categorized with a R (reason) category. You now assign a final category to the POIs, as determined in the preparation phase, usually a V or A category.

You may not want data items (as opposed to the statements that use them) contained in the final reports. If this is the case, assign all data items to a single category, for example A-Ignore, so that they can be excluded in reporting.

Using the method described below, the points of interest will be hidden from view as soon as they have been assigned a final category. This method provides a clearer view of the remaining POIs that still have to be assigned final categories.

1. In the worksheet click Display Filters.

2. In the Statements or Data Items tab click Hide items.

3. Click Categories and select those categories that have been determined as final categories.

4. In the worksheet click Options.

5. In the Worksheet tab check Enable active filtering.

6. Starting at the top of the worksheet, systematically assign a category to each point of interest. The categories you assign should be the ones decided on as final categories.

7. Use the Notes column to explain your decision or to indicate that the item needs further analysis.

8. If you have decided that you do not want to review or report on false positives, you can remove them from the worksheet, and not assign them a final category. We recommend that you keep the false positives.

2.2.4.3 Review the Year Types Assigned

In this task, you examine the data items that are used in the problem statements. You need to examine the data items with ambiguous year types, which are those with underscores, and assign the correct year types to them. You then need to check the statements to make sure that the data items within each statement have the same year types.

• Select the data items that have year types with underscores.

• Click Assign Year Types.

  This might assign year types to some of the data items that were originally assigned year types with underscores. This is because data items added early in the adding process might not receive full year types, which then become obvious when data items are added later in the process.

• Assign the correct year type to each data item that still has a year type with underscores, as follows:
  • Select the data item.
  • Determine the correct year type.
  • Click Select Year Type.
  • Choose the appropriate year type.

2.2.5 Reporting

In this task you will prepare final reports on the verification process. The reports should highlight any problems the source code may have in dealing with the Year 2000 problem.

Note: You need Microsoft Access Office 97 version to run the supplied Access report templates.

1. In the Worksheet click Reports.

2. Select Export worksheet details to Access.

3. If you want to run Access with the newly created database loaded, check Launch MS Access. If you want to generate the analysis history reports showing your activity for each item in the worksheet, check History for POIs.

4. In the SmartFind Plus Verify Reports window, click Category Options. Most of the categories will not be of interest. Uncheck Report Option to stop that category being reported on.

5. Run the Verify ... reports from the list of reports. You can access other formats for your reports by clicking on Export.

6. Check the reports. The majority of statements should be comparisons. There are typically a few hard coded literals of 19 and 20. Typically, they are leap year calculations. If the results are not fairly consistent with other work you know has been completed correctly, you should double-check your results.

2.2.6 Objectives of Verifying Code

This section describes the analysis objectives you should achieve when verifying remediated source code.

• Confirm data items already renovated by windowing

• Confirm data items expanded to include 4-digit year

• Identify data items missed in the previous renovation efforts

• Identify items requiring further review by a subject matter expert

• Identify modifications made in excess of windowing requirements

2.3 Year 2000 Remediation

The remediation process involves analyzing and fixing the source code. You have to examine the application in detail and determine which lines of code require changing. Then you need to apply the changes, confidence test and system test and finally put the application into production.

The remediation process has five tasks:

• Inventory - The inventory task identifies all application source members and determines if any source code members are missing. This is the same as for the verification process. See the section Inventory for details.

• Research - The research task is where you find all Year 2000 POIs, statements and data items, that will fail during or before the Year 2000. This is the same as for the verification process. See the section Research for details.

• Analysis - In the analysis task you examine the POIs to determine whether or not they are genuine problems and to categorize them accordingly. You also check and, if necessary, adjust the year types of the data items, so that the most appropriate fixes are automatically generated. This is broadly the same as for the verification process. See the section Analysis for details.

• Fixing - In the fixing task you examine the automatically generated fixes, and either approve them or adjust them as necessary. You then apply the fixes to a copy of the source files. See the section Fixing the Source Code for details.

• Reporting - In the reporting task you prepare final reports on the verification process. This is the same as for the verification process. See the section Reporting for details.

• Testing - In the testing task you confidence test the results, archive the project and carry out a full system test.

2.3.1 Fixing the Source Code

When you have identified all the statements that need to be fixed and have assigned year types to the relevant data items, you can start determining the fixes needed. Each statement in the worksheet has a fix automatically generated, and you now need to systematically check each fix and either approve or adjust it as necessary.

Examine and handle each statement as follows:

1. Select the statement and click SmartFix to display the automatically generated fix for the statement.

2. If the fix is correct, change its status to Fixed to indicate that you have checked and approved the fix.

3. If the fix is not correct, change its status to Fixed so that you can change the fix. You can then adjust the year types and macros used for each operand, so that you generate a different fix, or you can edit the fix code manually.

When you have a complete set of consistent fixes, click Apply Fixes to copy the source files that need fixing and to insert the fixes into those copies.

2.3.2 Milestones when Fixing Code

This section provides milestones you must complete when fixing source code. Each milestone helps you to make sure that the step you are working on is complete before moving onto the next step.

If you move on to the next step before completing all work in a milestone, you will uncover exceptions in the later steps and you will have to return to the earlier steps. This is time consuming and potentially error prone. In practice, you must decide your level of confidence by balancing time saved against the risk of possible exceptions.

Analysis work is generally assigned by program. You must take into account that the commands copy and include files can result in programs not being entirely self-contained. Therefore, the milestones naturally divide the analysis step into two stages: data item analysis and statement analysis.

2.4 Year 2000 Factory Considerations

If you are acting as a Year 2000 factory, and you take files from an external customer in increments, you cannot follow the recommended SmartFind Plus procedure. This procedure places two main constraints on the way you renovate source code:

• A complete inventory of all the files related to the application in the project must be available before starting the Year 2000 process.

• The original source files of the application and the associated Revolve database must remain unchanged during the whole procedure. Fixed files are always generated from these through the worksheet.

However, as a factory, you will be under pressure to complete the project due to the urgency of the problem. This can mean that you have to start some renovation work before all of the source code has become available. The only work that you should do is preparatory work and not start a full renovation or verification procedure until all the source code is available.

To save the results of any preparatory work:

1. Save the work by exporting worksheet details to an Access database.

2. Create a new project and worksheet with the complete source code.

3. Create appropriate queries on the Access database and list the results.

4. Reapply the results to the new worksheet. Some data can be entered manually by pointing and clicking. Other data, such as the names of date data items, can be entered by exporting the names to a text file and importing them.

Introduction

Making an Inventory