SetPrecision Function

Action

Sets a tolerance such that if two numbers differ by less than the precision (exclusive), they will be considered equal. Specifically, n1 == n2 if Abs (n1 - n2) < p, where p is the precision.

Syntax

SetPrecision (rPrecision)

Variable	Description
`rPrecision`	The tolerance to which real numbers are compared. REAL.

Notes

The default precision is 0.000001.
To compare two numbers significantly smaller than 1e-16 you need to normalize the difference comparison in some way. The reason is that the test for equality given above has been simplified. The complete test for equality between real numbers n1 and n2 is actually defined as:
```
n1 == n2 iff abs(n1-n2) + DBL_EPSILON <= precision
```
where DBL_EPSILON is the smallest number such that 1.0+DBL_EPSILON != 1.0. It is typically defined as: 2.2204460492503131e-016.
To normalize the comparison, apply a suitable multiplier to the precision and to the values being compared. The multiplier should be based on 1e-15, which is the first power of 10 greater than DBL_EPSILON. The example below demonstrates the use of such a multiplier.

Example

[ ] const MIN_PRECISION = 1e-15 // first power of 10 greater than DBL_EPSILON
[ ] 
[-] testcase Example () appstate none
	[-] LIST OF NUMBER ln1 = {...}
		[ ] 1
		[ ] .045
		[ ] .04
		[ ] 1.0e-20
		[ ] 1.0e-20
	[-] LIST OF NUMBER ln2 = {...}
		[ ] 4
		[ ] .04
		[ ] .05
		[ ] 1.1e-20
		[ ] 1.1e-20
	[-] LIST OF REAL lrPrecisions = {...}
		[ ] 10
		[ ] .01
		[ ] .01
		[ ] 0.00000000000000000001
		[ ] 1.0e-21
	[ ] REAL r1, r2, rPrecision, rMultiplier
	[ ] INTEGER i
	[ ] 
	[-] for i = 1 to ListCount (ln1)
		[-] if i <= 3
			[ ] Print ("Comparing {ln1[i]} to {ln2[i]} with precision {lrPrecisions[i]}")
		[-] else
			[ ] Printf ("Comparing %1.24f to %1.24f with precision %1.24f\n", ln1[i], ln2[i], lrPrecisions[i])
		[ ] r1 = ln1[i]  // will implicitly convert INTEGER to REAL
		[ ] r2 = ln2[i]  // will implicitly convert INTEGER to REAL
		[ ] rPrecision = lrPrecisions[i]
		[ ] // If necessary, normalize the comparison.
		[ ] // If omitted, the fourth comparison (1.0e-20 to 1.1e-20) will fail.
		[ ] rMultiplier = MIN_PRECISION / rPrecision
		[-] if rMultiplier > 1.0
			[ ] r1 *= rMultiplier
			[ ] r2 *= rMultiplier
			[ ] rPrecision *= rMultiplier
			[ ] SetPrecision (rPrecision)
			[ ] // The following verification should raise an exception
			[ ] // for the third comparison (.04 to .05 with precision .01)
			[ ] // and fifth comparison (1.0e-20 to 1.1e-20 with precision 1.0e-21)
		[-] do
			[ ] Verify (r1, r2, "the numbers match within the precision")
			[ ] // Could also use:
			[ ] // Verify (r1 == r2, TRUE, "the numbers match within the precision")
		[-] except
			[ ] LogError (ExceptData ())

The results file is:

Script OnLineHelpExample[1].t - 2 errors
Machine: (local)
Started: 03:49:33PM on 08-Nov-2006
Elapsed: 0:00:06
Passed: 0 tests (0%)
Failed: 1 test (100%)
Totals: 1 test, 2 errors, 0 warnings

Testcase Example - 2 errors
 Comparing 1 to 4 with precision 10.000000
 Comparing 0.045000 to 0.040000 with precision 0.010000
 Comparing 0.040000 to 0.050000 with precision 0.010000
 *** Error: Verify the numbers match within the precision failed - got 0.040000, expected 0.050000
 Comparing 0.000000000000000000010000 to 0.000000000000000000011000 with precision
0.000000000000000000010000
 Comparing 0.000000000000000000010000 to 0.000000000000000000011000 with precision
0.000000000000000000001000
 *** Error: Verify the numbers match within the precision failed - got 0.000000, expected 0.000000