Create a Forum Topic by Logging In or Creating an Account

Weird precision issue when subtracting two values

Tue, 04/30/2019 - 03:06 pm

Hi,

I found that I cannot produce the expected results of the following subtraction whenever the difference is small (3 orders of magnitude smaller than the original number, e.g. 0.001 vs 1.001), tested in Igor 6 and also Igor 8:

//This is OK
•print/d (1.003-1.001)/2
  0.001
//This is OK
•print/d (1.003-1.001)/4
  0.0005
 
//This is NOT OK
•print/d (1.002-1.001)/2
  0.000500000000000056
 
//Seems like something is not right with a certain subtraction
//This is NOT OK
•print/d 1.004-1.003
  0.00100000000000011
•print/d 1.003-1.002
  0.00099999999999989
//But then this is OK
•print/d 2.003-1.002
  1.001

Is this some limitations of floating point numbers?

Thanks.

aclight

This is not a bug, it's just how floating point arithmetic works. For more details (maybe more than you want to know), see https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

April 30, 2019 at 04:31 pm - Permalink

Sandbo

Thanks for clarifying, just realized we never noticed this before and it could account for some issues we have seen in the past.

April 30, 2019 at 06:33 pm - Permalink

aclight

Sure. I should have also pointed out the APMath operation which can be used whenever you need arbitrary precision.

May 1, 2019 at 08:28 am - Permalink