error: The fitting function returned NaN for at least one X value.

Share icon

What might the above error message mean? I'm using the FuncFit operation.

This is my fit function:

function Rtot_line(w, x) : FitFunc
    wave w
    variable x
   
    variable val
    val = w[0] * (x - w[1]) + w[2]          //w[1] < 0
    return val
end


The x-wave is just a series of integers, 5,10,15,20... and when I print the coefficients w[0] w[1] and w[2] in the main body of the procedure that runs the FuncFit operation, I get an integer (not NaN). Wondering what exactly is the NaN which is being returned here?
It's possible that error is being returned erroneously instead of Singular Matrix error. Your fit function features a linear dependency. If you multiply out, you get
val = w[0] * x - w[0]*w[1] + w[2]

That means that you have a constant term composed of a combination of all three fit coefficents. Any change in one can be compensated for by a change in another, so there is not locally unique solution. You need to just fit a line.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

March 13, 2018 at 12:09 pm - Permalink

johnweeks wrote:
It's possible that error is being returned erroneously instead of Singular Matrix error. Your fit function features a linear dependency. If you multiply out, you get
val = w[0] * x - w[0]*w[1] + w[2]

That means that you have a constant term composed of a combination of all three fit coefficents. Any change in one can be compensated for by a change in another, so there is not locally unique solution. You need to just fit a line.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


Thanks John! The reason why the fit function was composed as such is because w[1] and w[2] are terms which I'd like to hold constant using the /H="011" flag; is there a way to get around this since I cannot use the standard "y=K0+K1x" fit function?

March 13, 2018 at 05:50 pm - Permalink

Ah. The technique of including constants in the fit function coefficients and then holding them is the easiest way to achieve your goal. If you hold w[1] and w[2] does the error go away?

The much more difficult way to do that is to define a structure fit function. The constants would be part of the "extra" members of the structure. This requires that you write a driver for the fit that sets up the structure.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

March 14, 2018 at 09:54 am - Permalink

johnweeks wrote:
Ah. The technique of including constants in the fit function coefficients and then holding them is the easiest way to achieve your goal. If you hold w[1] and w[2] does the error go away?

The much more difficult way to do that is to define a structure fit function. The constants would be part of the "extra" members of the structure. This requires that you write a driver for the fit that sets up the structure.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


Indeed, if I hold w[1] and w[2], the error is still present. Using the same set of numbers and attempting to fit them somewhere else with the expression w[0]*(x-w[1]) + w[2], where w[1] and w[2] were fixed and pre-determined by an earlier procedure in IGOR did not give rise to the singular matrix error, i.e. my data should fit when there is only one unknown fit coefficient. However when the FuncFit is run here, the error message as above still appears.

Could it mean something else other than the singular matrix error?

March 19, 2018 at 07:07 pm - Permalink

paperlovestory wrote:

Thanks John! The reason why the fit function was composed as such is because w[1] and w[2] are terms which I'd like to hold constant using the /H="011" flag; is there a way to get around this since I cannot use the standard "y=K0+K1x" fit function?


Since you hold the terms constant, you know their values in advance. Fit the straight line y = m*x + b. Extract the unknown w[0] by algebra using m, b and your known w[1] and w[2]. Use linear uncertainty propagation to obtain the uncertainty on w[0] from the uncertainties on m and b (since w[1] and w[2] are constant in this case, their uncertainties are zero).

Alternatively, define your fit function as w[0] * (x - alpha) + beta where alpha and beta are predefined (rather than held during the fitting procedure).

I am interested as well to know what might be behind the error when you hold the terms constant in advance. I would wonder what happens when you predefine them as above to avoid using a hold flag.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH

March 20, 2018 at 06:19 am - Permalink

Please post an Igor experiment file with a data set, fit function, initialization and FuncFit command that reproduces your problem. I will see if I can figure out what your problem is.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

March 20, 2018 at 09:42 am - Permalink

johnweeks wrote:
Please post an Igor experiment file with a data set, fit function, initialization and FuncFit command that reproduces your problem. I will see if I can figure out what your problem is.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com




I have attached a (simplified) experiment file here with a real data set and for which the error occurs.
20180321_lineFit_test.pxp (10.88 KB)

March 20, 2018 at 09:27 pm - Permalink

jjweimer wrote:

Alternatively, define your fit function as w[0] * (x - alpha) + beta where alpha and beta are predefined (rather than held during the fitting procedure).

I am interested as well to know what might be behind the error when you hold the terms constant in advance. I would wonder what happens when you predefine them as above to avoid using a hold flag.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


I have thought of this alternative to solve the problem described above as well, and this is what I've written:

for (aa = 0; aa < 13; aa += 1)
    D1D[] = D2D[p][aa]
    wavestats /M=1 /Q D1D
    make/o/n=1 W_coef
    W_coef = {1}            //"1" arbitrarily chosen, less likely to give error than "0"
    if (V_npnts > 2)
        FuncFit lineFit, kwcWave=W_coef, D1D /X=waveX
    else
        printf "%s\r", "Insufficient data points"
        continue
    endif
endfor

//--------------------------------------------------------------------------------

function lineFit(w, x) : FitFunc

    wave w
    variable x
   
    variable val, alpha, delta
    nvar X_ave = X_ave
    nvar Y_ave = Y_ave
    alpha = X_ave
    delta = Y_ave
   
    val = w[0] * (x - alpha) + delta            //w[1] < 0
 
    return val
   
end


Seems to get the job done, does not give an error (although I must admit my code seems quite poorly written?) but the set of coefficients generated using this, from the same data set, differs from the fitting done using another graphing software (manually) for the same X_ave and Y_ave.

March 21, 2018 at 01:52 am - Permalink

paperlovestory wrote:


I have thought of this alternative to solve the problem described above as well, and this is what I've written:

for (aa = 0; aa < 13; aa += 1)
    D1D[] = D2D[p][aa]
    wavestats /M=1 /Q D1D
    make/o/n=1 W_coef
    W_coef = {1}            //"1" arbitrarily chosen, less likely to give error than "0"
    if (V_npnts > 2)
        FuncFit lineFit, kwcWave=W_coef, D1D /X=waveX
    else
        printf "%s\r", "Insufficient data points"
        continue
    endif
endfor

//--------------------------------------------------------------------------------

function lineFit(w, x) : FitFunc

    wave w
    variable x
   
    variable val, alpha, delta
    nvar X_ave = X_ave
    nvar Y_ave = Y_ave
    alpha = X_ave
    delta = Y_ave
   
    val = w[0] * (x - alpha) + delta            //w[1] < 0
 
    return val
   
end


Seems to get the job done, does not give an error (although I must admit my code seems quite poorly written?) but the set of coefficients generated using this, from the same data set, differs from the fitting done using another graphing software (manually) for the same X_ave and Y_ave.


A more compact expression that avoids extra terms (local variables) and nomenclature issues (w and x terms in the call) is below.

Function lineFit(ww,xx)
     wave ww
     variable xx

     NVAR x_ave, y_ave

     return (w[0]*(xx- x_ave) + y_ave)
end


I really dislike the use of globals in the fit function. The only two ways around this are to pass parameters that are held constant or to use a structure fit function. I believe also that you can eliminate x_ave and y_ave as global inside the fit function in this way.

make/O/N=1 w_coeff
for (...)
    DID = D2D[p][aa]
    WaveStats/M=1/Q DID
    w_coef = 1
    DID += - y_ave
    waveX +=  x_ave
    FuncFit lineFitNew, kwcWave=w_coef, DID /X=waveX
end for

Function lineFitNew(ww,xx)
     wave ww
     variable xx

     return (w[0]*xx)
end


--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH

March 21, 2018 at 09:42 am - Permalink

paperlovestory wrote:
I have attached a (simplified) experiment file here with a real data set and for which the error occurs.

Thanks. You found a bug!
The results of one iteration of the fit were checking for NaN or Inf in elements of an internal matrix that corresponded to your fixed coefficients. It should only have checked elements that were actively being fit. The contents of those cells would be whatever was in memory when the matrix was allocated, so it usually didn't manifest. I don't know why that one fit had a problem; perhaps there were actual NaNs lying about in memory because the data set had several NaNs.

In any case, the problem is fixed for the next nightly build of Igor. Tomorrow, select Help->Igor Pro Nightly Builds to get the fix. Be sure the build number is at least 31485.

The bug has been there for many years. You had just the right combination to trigger it.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

March 21, 2018 at 10:32 am - Permalink

johnweeks wrote:
paperlovestory wrote:
I have attached a (simplified) experiment file here with a real data set and for which the error occurs.

Thanks. You found a bug!
The results of one iteration of the fit were checking for NaN or Inf in elements of an internal matrix that corresponded to your fixed coefficients. It should only have checked elements that were actively being fit. The contents of those cells would be whatever was in memory when the matrix was allocated, so it usually didn't manifest. I don't know why that one fit had a problem; perhaps there were actual NaNs lying about in memory because the data set had several NaNs.

In any case, the problem is fixed for the next nightly build of Igor. Tomorrow, select Help->Igor Pro Nightly Builds to get the fix. Be sure the build number is at least 31485.

The bug has been there for many years. You had just the right combination to trigger it.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


Thanks John.

After getting the fix, my build number was 31475. I'm not sure if it solved the problem because it caused many of my earlier codes to no longer work (or, are experiment files supposed to be built from scratch after replacing the files?). I have since reverted back to my original one (31118) and my earlier codes work again.

March 22, 2018 at 02:59 am - Permalink

paperlovestory wrote:
After getting the fix, my build number was 31475.

No, you didn't get a sufficiently recent build. Try again today, please.
Quote:
I'm not sure if it solved the problem because it caused many of my earlier codes to no longer work (or, are experiment files supposed to be built from scratch after replacing the files?). I have since reverted back to my original one (31118) and my earlier codes work again.

What went wrong? I don't think we have changed that much since 31118.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

March 22, 2018 at 09:46 am - Permalink

johnweeks wrote:
paperlovestory wrote:
After getting the fix, my build number was 31475.

No, you didn't get a sufficiently recent build. Try again today, please.
Quote:
I'm not sure if it solved the problem because it caused many of my earlier codes to no longer work (or, are experiment files supposed to be built from scratch after replacing the files?). I have since reverted back to my original one (31118) and my earlier codes work again.

What went wrong? I don't think we have changed that much since 31118.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


Thank you. It is 31485 right now and the earlier codes work again.

March 23, 2018 at 02:19 am - Permalink