DoNewGlobalFit in ipf

Share icon

Dear everybody, I am trying to fit globally about 20 experiments with the same input conditions.
Unfortunately, if I exchange the source data, all the information about fitting function, linking, initial guess and so on seem to be lost, even upon savin and restoring setups, it is only possible to retrieve the old source data. So I tried to work with the DoNewGlobalFit command in an ipf and looked into the Packages:NewGlobalFit-folder where I found a lot of files containing the information that I need. But I'm still not sure, which is the best way to proceed.
My data are organised in folders called xDLI_123, with data waves called xDLI_123_1_global to xDLI_123_24_global. "xDLI_"+num2str(var1)+"_"+num2str(var2)+"_global"

This yields me roughly a dozen of waves and columns therein, in which the corresponding entry is
root:xDLI_123:xDLI_123_1_global

The 123 is obviously the running number and needs to be changed by the for-loop. Am I right, that I need to assign manually (in a loop in the ipf) all the text wave columns with the corresponding strings, or is there a way to go through all the text waves in the NewGlobalFit folder and to replace the 123 by other var1-values? For some of the waves it seems to be possible to do it manually in one line per parameter, but e.g. for the NewGF_CoefficientNames, the entries are connected to the fitting parameter.

Cheers, Dominik
I take it you are trying to do a batch global fit, and you feel that the set-up and initial guesses for each are appropriate for all.

If you wish to do it programmatically, see DisplayHelpTopic "Global Fits From Your Own Code"

I think you could save a set-up using the Save button, then go into the data folder created that way and edit names as you suggest. Then use Restore Setup to get the control panel to reflect your edits. It may be difficult to make sure you have properly edited everything that needs to be changed.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

December 12, 2017 at 09:35 am - Permalink

Yeah, as data are normalised, initial guesses are appropriate for all, what makes it at least a bit easier.


            setdatafolder root:Packages:NewGlobalFit:
            wave/T NewGF_DataSetListWave, NewGF_DataSetsList, SelectedDataSetsListWave, NewGF_CoefficientNames, NewGF_CoefControlListWave
           
            for(bla=0;bla<=23;bla+=1)      // numbering due to having 24 waves, yielding 53 parameters after linking and 216 total curves
                NewGF_DataSetListWave[bla][0][0]=replacestring(num2str(num_old), NewGF_DataSetListWave[bla][0][0], num2str(num))
                NewGF_DataSetsList[bla][0]=replacestring(num2str(num_old), NewGF_DataSetsList[bla][0], num2str(num))
                SelectedDataSetsListWave[bla][0]=replacestring(num2str(num_old), SelectedDataSetsListWave[bla][0], num2str(num))
                NewGF_CoefficientNames[bla][0]=replacestring(num2str(num_old), NewGF_CoefficientNames[bla][0], num2str(num))
                NewGF_CoefControlListWave[bla][0]=replacestring(num2str(num_old), NewGF_CoefControlListWave[bla][0], num2str(num))
                NewGF_CoefControlListWave[bla][1]=replacestring(num2str(num_old), NewGF_CoefControlListWave[bla][1], num2str(num))
            endfor
            for(bla=24;bla<=52;bla+=1)
                NewGF_CoefficientNames[bla][0]=replacestring(num2str(num_old), NewGF_CoefficientNames[bla][0], num2str(num))
                NewGF_CoefControlListWave[bla][0]=replacestring(num2str(num_old), NewGF_CoefControlListWave[bla][0], num2str(num))
                NewGF_CoefControlListWave[bla][1]=replacestring(num2str(num_old), NewGF_CoefControlListWave[bla][1], num2str(num))
            endfor
            for(bla=53;bla<=215;bla+=1)
                NewGF_CoefControlListWave[bla][0]=replacestring(num2str(num_old), NewGF_CoefControlListWave[bla][0], num2str(num))
                NewGF_CoefControlListWave[bla][1]=replacestring(num2str(num_old), NewGF_CoefControlListWave[bla][1], num2str(num))
            endfor 
            print    NewGF_DataSetListWave[0][0][0]
            print    NewGF_CoefControlListWave[1][0]   
            print SelectedDataSetsListWave[0][0]   


Using the code above, I managed to replace the numbers in the six waves shown, but the batch GlobalFit doesn't work, and I could reproduce a certain behaviour:
When I run my ipf (without actually doing the fit, just preparing the data), I end up with SelectedDataSetsListWave[0][0] showing the specific term I expect and, upon editing this wave, showing the terms with the last experiment's term in the table. But, as soon as I want to do a global fit (manually), and click the "Add/Remove Waves" button, the table will change to some entries that come from the very first attempt to do a global fit. So somewhere in the background, these data are still present.

Using the debugger, I could find the place, where it happens:
In the Function BuildDataSetSelector(), there is an if-part:
    Wave/T DataSetListWave = root:Packages:NewGlobalFit:NewGF_DataSetListWave
    Variable nrows = DimSize(DataSetListWave, 0)
    if ( (nrows == 1) && AllFieldsAreBlank(DataSetListWave, 0))
        nrows = 0
    endif
    if (nrows > 0)
        Redimension/N=(nrows, 2) SelectedDataSetsListWave, SelectedDataSetsSelWave
            SelectedDataSetsListWave[][] = DataSetListWave[p][q][1]     // layer 1 contains full paths
    endif


In the line SelectedDataSetsList[][]=..., there something happens. Obviously, DataSetListWave[p][q][1] hides some older information, but I haven't found out, why and where this is stored.
Can you help me with that issue?

December 12, 2017 at 10:04 am - Permalink

DataSetListWave (which is the name of wave reference pointing to NewGF_DataSetListWave) is the text list wave for the ListBox control on the left side of the Data Sets and Functions tab of the control panel. That ListBox and its selwave are the storage for the names of the data sets.

SelectedDataSetsListWave is only for the list at the bottom of the Add/Remove Data Sets control panel that is displayed when you click the Add/Remove Waves button. That is just temporary storage while you manipulate the list of waves.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

December 12, 2017 at 01:20 pm - Permalink

Okay, when I just executed one global fit with output, I found one thing that might help to clarify the issue: The textwave DataSets, which is used in the DoNewGlobalFit command seems not to be updated when I update the other waves as shown above. The outcome of this is obvious:
        for (i = 0; i < numRows; i += 1)
            print "Data Set: ",DataSets[i][0]," vs ",DataSets[i][1],"; Function: ",FitFuncNames[privateLinkage[i][FuncPointerCol]]
            for (j = FirstCoefCol; j < (privateLinkage[i][NumFuncCoefsCol] + FirstCoefCol); j += 1)
                linkIndex = privateLinkage[i][j]
               
                FirstUseOfIndexInLinkMatrix(linkIndex, privateLinkage, firstUserow, firstUsecol)
                printf "\t%d\t%s\t%g +- %g", j-FirstCoefCol, CoefNames[privateLinkage[i][j]], MasterCoefs[privateLinkage[i][j]], W_sigma[privateLinkage[i][j]]
                if (hasHoldCol && CoefWave[privateLinkage[i][j]][%Hold])
                    printf " *HELD* "
                endif
                if ( (firstUserow != i) || (firstUseCol != j) )
                    printf " ** LINKED to data set %s coefficient %d: %s", DataSets[firstUserow][0], firstUseCol-FirstCoefCol, CoefNames[privateLinkage[i][j]]
                endif
                print "\r"
            endfor
        endfor

yields:
  Data Set:   root:xDLI_783:xDLI_783_1_global   vs   _calculated_  ; Function:   G400
[...]
0   A[G400][root:xDLI_971:xDLI_971_1_global]    0.159638 +- 1.05588 ** LINKED to data set root:xDLI_783:xDLI_783_1_global coefficient 0: A[G400][root:xDLI_971:xDLI_971_1_global]

The parameters called by CoefNames[...] are working, while the ones called by DataSets[*][*] aren't.

So I need to build the textwave DataSets before executing the DoNewGlobalFit(...) in the ipf - which I kinda thought so before, but first wanted to make sure that everything else works. ;)

When I want to build this wave, which format does it have to have? Comma- or semicolon-separated, with or without space between wavestrings?
Cheers, Dominik

December 13, 2017 at 01:07 am - Permalink

d_lenz wrote:

The parameters called by CoefNames[...] are working, while the ones called by DataSets[*][*] aren't.

So I need to build the textwave DataSets before executing the DoNewGlobalFit(...) in the ipf - which I kinda thought so before, but first wanted to make sure that everything else works. ;)

When I want to build this wave, which format does it have to have? Comma- or semicolon-separated, with or without space between wavestrings?
Cheers, Dominik


I have recently written an application specific Panel/GUI that plugs into Global Fit. My code makes the waves required by the DoNewGlobalFit function from scratch. I found the GlobalFit help file very useful for describing the format of each of these waves, and I also went through the Global Fit code to fully understand how it makes these waves from its Panel. It works very well, and is not too difficult to implement when taken one step at a time.

With respect to your specific question about the DataSets wave - it is a text wave with columns 0 for the names of the Y-data sets, 1 for the names of the X- datasets, and it may have extra named columns for weights and masks. The number of rows is the number of data sets that you want to include in your global fit. I.e. each row contains just one entry - the name of the data set (with path if appropriate). In my code I used something similar to DataSets[0] = GetWavesDataFolder(myFirstWave, 2). I actually put this in a loop to reference each dataset wave in turn and assign the (text) value in DataSets one at a time.
Note that the wave CoefDataSetLinkage should have the same number of rows as the DataSets wave.

Hope this helps,
Kurt

PS - As an aside, I should say a big thank you to Wavemetrics for the Global Fit package. It works really well and has made my task much easier than it could have been.



December 13, 2017 at 01:37 am - Permalink

Okay, then I will try to duplicate the already present NewGF_... waves into the ones that are required in the command.

Edit:
Well, I learnt something about Free Waves now.

December 13, 2017 at 02:21 am - Permalink

Okay, now I learnt that NewGF_DataSetListWave is a 3D-wave and how to see and change the layer. Yes, it is written in the comments, but...
Now it works much better: when I run my procedure and then click on "Add/remove waves", it will not jump back to older values and I was indeed able to do a global fit on the last experiment in the line.
Though, getting the global fit implemented into my procedure does not work, it will just run through the code and give a 'can't-overwrite-itself'-error message.
I guess, this is the point to stop it. Clicking 20 times the "Fit!" button after preparing the data just by choosing the folder (getdatafolder, sscanf and so on) and running a slightly modified version of my ipf before might be easier and might be also better to deal with singular matrix errors...


Edit:
Indeed: It is working now in principal, I just have to play around on the initial guesses. But I made it, when I choose a folder and run my ipf, then all the corresponding parameters are updated and I can just click "Fit!" to do it. :)

December 13, 2017 at 05:39 am - Permalink