Weird precision issue when subtracting two values
Sandbo
Hi,
I found that I cannot produce the expected results of the following subtraction whenever the difference is small (3 orders of magnitude smaller than the original number, e.g. 0.001 vs 1.001), tested in Igor 6 and also Igor 8:
//This is OK
•print/d (1.003-1.001)/2
0.001
//This is OK
•print/d (1.003-1.001)/4
0.0005
//This is NOT OK
•print/d (1.002-1.001)/2
0.000500000000000056
//Seems like something is not right with a certain subtraction
//This is NOT OK
•print/d 1.004-1.003
0.00100000000000011
•print/d 1.003-1.002
0.00099999999999989
//But then this is OK
•print/d 2.003-1.002
1.001
•print/d (1.003-1.001)/2
0.001
//This is OK
•print/d (1.003-1.001)/4
0.0005
//This is NOT OK
•print/d (1.002-1.001)/2
0.000500000000000056
//Seems like something is not right with a certain subtraction
//This is NOT OK
•print/d 1.004-1.003
0.00100000000000011
•print/d 1.003-1.002
0.00099999999999989
//But then this is OK
•print/d 2.003-1.002
1.001
Is this some limitations of floating point numbers?
Thanks.
This is not a bug, it's just how floating point arithmetic works. For more details (maybe more than you want to know), see https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
April 30, 2019 at 04:31 pm - Permalink
In reply to This is not a bug, it's just… by aclight
Thanks for clarifying, just realized we never noticed this before and it could account for some issues we have seen in the past.
April 30, 2019 at 06:33 pm - Permalink
Sure. I should have also pointed out the APMath operation which can be used whenever you need arbitrary precision.
May 1, 2019 at 08:28 am - Permalink