Updating Global Fit with new parameters without fitting

Do I understand that right, that you want to have something like the Graph Now button of the Curve Fit dialog, but for Global Fit to preview and fine-tune your parameters? This might be possible to implement.

Otherwise, you could first fit your data individually to get an useful initial guess. This could even be automated. If your data and fit function is not too complex (i.e., just a bunch of Gaussians thrown together), then it might be not too difficult to find good initial guesses numerically from your data.

Hi chozo!

That is exactly what I am after... I actually started doing this myself, but realised that it was more of an undertaking than "just doing it in a few hours" - so now I am implementing a routine that does this only for my very specific data with my specific parameters and my specific fit-function - you get it, not for public use ever... However it should be possible to have a "Graph Now" button I think, and it would improve the usability of the function greatly (I think)...

My data is indeed a bunch of Gaussians, however they overlap to a great extent and fitting one spectrum only gives a hint of the final result... I will need to fit at the very least 5-6 waves to fit, and in total I only have 15-20 waves so I don't think I will win so much from that approach... In fact using the Global Fit I was able to discover some "hidden" features that otherwise would have been tricky to find (I think)

Best,

Johan

While I am suggesting new features it would be great to have a built-in function that shows the data-wave and the fit results in a dedicated window - and where it is easy to change what wave is being displayed... Perhaps this updates automatically when a new wav is selected in the "Coefficient control" tab? 

This I can easily implement myself, but it would be a natural (?) add-on to the existing code I think. 

Best,

Johan again

Hi Johan, regarding the Graph-Now type feature, this seems useful. I might discuss a possible update with the SI staff. It would probably boil down to grabbing the inserted coefficients from the list and feeding it into the user function for display. You already have something implemented yourself? Just to make sure, you can get the latest coefficients from the panel by accessing the third column (i.e., [2]) of

Note that this is a text wave, so you need to convert the column into numerical values, and then feed it into your fit function for display purposes.

Regarding the request for a dedicated window, I am not completely sure what you mean. The fit will be displayed in a dedicated graph as long as Fit Progress Graph is checked. Isn't this already what you want? And what do you mean by "where it is easy to change what wave is being displayed"? In the Coefficient Control tab you do not select any waves, unless you mean for loading presets.

chozo - did you make this feature - I really like it! Anyhow, let me try to reply

Grabbing the text-wave and converting to numbers is quite easy (as long as no error-checks are implemented at least) - so I was able to make a function that updates all fits based on the numbers I fed into the Coefficients (but I was somewhat surprised that this is a text-wave)...

Since I am not sure of how to use a fit-function to update a whole wave (I think I did this many years ago - so I think it can be done) I had to write a new function that updates the fit-waves. This in itself is trivial but I hope that there is a way around this for an advanced Igor programmer, otherwise the user would have to write two functions, one for the fit and one doing the exact same thing but that updates the fit (think Graph Now) and this will not be a good way in the end - I am 100% sure... I did it this way in the interest of saving time and solving a very specific problem hoping for a general solution later-on.

So, I have something that works for me for this particular case, but it is almost as far from a general solution as having nothing. But for my specific case it is solved.

---

As for a dedicated graph I mean something similar to MultiPeakFit... There one have a window showing the data as well as the fit. I was thinking of having a similar graph (this could also be toggled by a tick-button "Show data/fit" or something. In this window one shows the selected data and the fit corresponding to this data. As one selects a new dataset (and thus also new Coefficients) the new dataset and the new fit is shown in this graph.

The benefit of this is that it will be trivial to see how my parameters affects this particular dataset. This could (depending on how it is implemented) more or less replace the Graph Now feature. Then an "Update Fit with current parameters" button would be the thing to have instead.

Another (better?) option would be to have a panel where one can select what data-set (and corresponding fit) to show, this could also be relevant since the linked parameters are changed in one dataset, but I might want to look at a different dataset to see how this affects that dataset... In this panel one can have a dropdown menu where I can select "Show selected dataset;Show dataset DataSet1_NameHere;Show dataset DataSet2_NameHere" ... If "Show selected dataset" is selected then the graph (in the panel) updates whenever a new dataset is selected in the main panel. If I want to look at a particular dataset then I simply select that dataset in the dropdown menu and this is constantly shown in the graph. I have included a screenshot showing how I did something similar in my secret project JS_VERITAS. A panel with a graph that shows the dataset that I select in the dropdown menu, all that is missing is the option at top that should be "Show selected dataset"

I am well aware that I can have 20 open graphs and look at the relevant graph (or something like this I can also write myself I think) but at least how I use this fitting routine that would be an obvious add-on to the routine.

---

So, the above features I think are in general good for all who uses this and is probably quite fast to implement. Now over to something that might not be as important in general and that will generate a lot more data - but perhaps someone can see this as being useful. I write this since I realise that I will for sure implement this in the very end in order to show my fits. For my type of data this will be more or less a must-have. I can write my oen function that does this for my data - no worries, but maybe this would be relevant for the general audience.

Let's say I write a user-function that consists of three gaussians that are overlapping to a great extent. Then I think that it could often be useful to see each gaussian component in my fit - to make sure that nothing stupid happened... That one width ended up being near to 0, or that two are overlapping eachother. (This is exactly what is done in MultiPeakFit by the nature of how that feature works.)

I know that this can easily be seen in the parameters, but having access to a visual indication is probably very helpful at many times as well. At least I know it is for me, and I would use this a lot for setting parameters as well as for showing the final fits.

Then - in my user-function I could write a routine that created waves named in a specific fashion. In this example there are three Gaussians, then I could create the global waves "UserFunction_GFit_P1", "UserFunction_GFit_P2" and "UserFunction_GFit_P3" (this could be generalised to 1000 waves if needed - "UserFunction_GFit_P1000". Then the Global Fit routine would look for these waves, rename them for each dataset that is fitted (example: GFit_DataSetName_P1) and add this to the very same window. Then one could see each component and how it looks instead of only the overall result.

Again, I think that this is something that would benefit a lot of users - but it would

1. Create a lot of waves.
2. Probably take some more time to implement since it also requires re-writing the manual and really thinking of how to implement this in a robust way that does not make the fit-routine having to generate these waves for each iteration. Perhaps one can write something like
   if (GFit_UpdateFits == 1) //This is set to 1 when the fit routine has finished. Then the "Graph Now" command is automatically envoked to update all individual components
    
      ... Code that generates my individual components something like...
      UserFunction_GFit_P1 = Gauss(x,w[0],w[1])
   endif

   in one or several places in my fit routine - and then this would only be excecuted at the end of the fit or when I manually change a fit-parameter/click Graph Now... Again, I gave the implementation about 10 seconds of though so 

3. Lots of issues I did not even think of yet - I was simply dreaming when you asked the question...

Thanks a lot for reading!

Johan

Hi again...

It took me a while to understand what you ment by "The fit will be displayed in a dedicated graph as long as Fit Progress Graph is checked. Isn't this already what you want?"

That is 80% of what I want (see discussion about that this updates when I change parameters) - however this window looks like this for me

and this is quite useless to be honest... I guess this is due to the fact that I have many waves that I fit, but then the option that one would be shown individually is clearly prefered - especially with the discussion above.

The other option is to make sure that I have individual windows where the fit is (automatically) appended... But I personally think that the option I present is a cleaner and more efficient way of doing this...

I hope that cleared up what I ment...

Have a great weekend!

Johan

Hi Johan,

Thanks for your detailed reply. First, I would like to make clear that this tool was not written by me but by the WM / SI staff (mostly John and Jim, I believe). I just had the opportunity to look into a few minor things over the past years, so I know a bit how this stuff works. Yes, NewGF_CoefControlMasterWave is a text wave because it holds all relevant information such as parameter names as well, and was not meant for user interaction. It is just a tidy way to have everything in one place (it was a wave for displaying lists at one time).

I am not sure why you would have to write an extra function to display a preview, since your fit function should be of the format myfunc(w,x), where w is the coefficient wave. You would just have to prepare this wave with all the coefficients you want to preview and then write to an output, i.e., out_wave = myfunc(coef,x). No need for a second function here, unless I miss something.

I see your point with the graph / update feature request. I might put something simple together when I am back in the office (I am traveling currently) and then discuss the usefulness for the general public with the Igor masterminds.

As for the idea with showing 'individual components' as well, I see that this might be useful. But unlike Multipeak Fit, where you build up your total fit function from individual, well characterized function types, Global Fit has no way of telling what a given user function is made of. I don't see how one could write a general purpose feature here. I could imagine, that there might be an additional tool like a 'generic function builder' for Global fit, e.g., to put a few Gaussians together, as an add-on. But otherwise Global Fit is built to handle arbitrary fit functions, so this is a bit difficult.

By the way, why do you need Global Fit for your fitting project again. Is there some global parameter which you do not know and have to fit globally over the whole data range? Otherwise, Multipeak Fit would offer far more convenience for your if you are not really in need of the special purpose of Global Fit.

Hi Johan,

Your reply about the graph window just came in later. This indeed seems a bit useless, but mainly because there is no proper offset between the individual waves. A vertical offset would certainly help. You could use my Graph Tools to add a vertical offset to the graph, but I wonder if this should not be built in already, like a small control to set the offset to one's liking. Then you would already be much closer to your goal. Could you share one example analysis here, so that I can look into how to improve the visual clarity of the result graph? You can also send it via email if you like.

Hi chozo

I'll try to write a short reply this time.

View of fit-results

I can absolutely do what you suggest, or have different windows for all datasets. However, in my case where I have 15-20 datasets the "results view" becomes unpractical since that means zooming quite a lot in order to see the relevant features in each dataset. And picking the correct window between 15-20 graphs to view my dataset also becomes unpractical. 

I can easily write my own "add-on" that does what I suggest (I will since I feel I need it for my fits) but I think that it would improve the usability of Global Fit for many users - just a humble suggestion

Displaying a function from a fit-function

There is something I am missing here. I cannot figure out how to call the fit-function (myfunc(w,x)) and get out a wave from that... That is simply me being not proficient enough in Igor. I know it can be done since I did it many years ago, but now I fail. I think that I do not know what "x" should be, but after reading your reply perhaps I figured it out. Thank (in case it should be "x" and then the scaling of the output wave defines what "x" is).

Code that generates peaks for each "subpeak" in a userdefined function

I believe that this can quite easily be done. I would do it this way

Note that this code is 100% untested, it is written to show the main idea that it is possible to keep a track of all components in the fit by careful labeling of them. When calling this function as a normal fit-function then it just fits - however if one calls it with the parameters 

hen the code will generate each component (that I ask it to in my user-defined code) to generate. This will add many waves and it will add some programming to make it all smooth and error-free... It would also change what waves are displayed, but I think that it is quite easy to implement - but it will add size to the experiment.

But using this method I could see each component in the fit... I can off course write my own function that does just this - it does not have to be handeled by Igor (I will do this since I feel I need to see this - especially for my published results). However since I have seen so many (talented researchers) completely miss out on basic features in Igor (one example is when I say people adding or removing points in the beginning of a wave as "the only way to calibrate the x-scale" - SetScale was unheard of for them, as well as using the databrowser to edit the x-scaling - and these people were under 30 at the time) I think that this would simplify life for a lot of people.

I can e-mail you an example of why I want tu use Global fit - the main idea is that I cannot resolve all gaussians in any one dataset - but having many will overdetermine the fit and (hopefully) give better results.

Best wishes and really thanks for taking the time to read all my text... The main idea of writing this is to give suggestions for improvments that I think can benefit more than me... For my own case I can probably fix this in a quick-and-dirty hack...

If I have understood correctly, you use globalfit to fit a function UserFunc(w, xx) to fit many waves.

I imagine that the code to generate fit waves might look something like this:

This is just a guess, and I haven't tested to see if it works. To be more complete you should probably check whether global fit is fitting waveform data or XY wave pairs.

I see that the NewGF_CoefWave wave is not updated before you run a fit, so you can't use that to generate fit waves for your initial guesses. It looks like the guess values have to be extracted from column 2 of the text wave NewGF_CoefControlMasterWave.

With apologies for looking under the hood of GlobalFit.

The NewGF_CoefWave does not even exist before the first fit. The coefs only live inside the listboxfor most of the time (and are synchronized in the background with NewGF_CoefControlMasterWave). The NewGF_CoefWave is built upon that just before the fit happens. So, you have to grab all the relevant information from the master wave.

chozo wrote:

So, you have to grab all the relevant information from the master wave.

 Then something like this?

Johan,

I think it would be good to show the results in just one graph, and also possibly have the ability to offset and hide traces. This should not be too difficult to implement and maybe you have even started to write your own tool. I'll try to put something together when I have time as well. I did similar things already before.

Yes, the 'x' in the function is used as is, and feeds the wave's scaling into the function as values. Thus, you need to scale the output correctly before invoking out_wave = myfunc(coef,x). Since you want to fill the output with a fit to the source data, the scaling should be the same as the source, so you might as well duplicate the source to create your output.

As for your code that creates the sub-peaks, I cannot recommend to write such things inside the the fit function itself. Note that the fit function is called multiple times for every point of every data set during the fit. Your approach of creating waves and assigning values at each step would be a huge resource hog. Instead, write the relevant bits of code into a separate function, which you call once after the fit is done to generate your components.

I had the idea, that Global Fit could also support user code after the fit is done, exactly for these kind of post-processing tasks. I use this approach in my Super Quick Fit package. Users can define post-processing functions which get automatically executed after the fit. I can imagine having something similar for Global Fit would make things easier.

Thanks for giving some more details on why you want to use global fit. So the peaks you are searching for are heavily overlapping in one data set but might be clearer in another, so you link, e.g., their positions to get an overall better result? I see that Global Fit is your only option here. But I guess if you knew properties of these peaks (like fixed distances or peak ratios) you could also get a reasonable result with Multipeak Fit.

But anyway, very good suggestions, which should also not be too difficult to add to the package.

Here is a very dirty work-around. I publish this in case someone wants to do something similar... Note that this is far from perfect. I make use of two functions that should only need to be one - but my poor programming skills made this a quicker way to solve the problem... This is a quick-and-dirty fix for my fit-functions. You will need to change many parameters in order for this to work in your own fit-function.

1. Create a fit-function. Here is what I currently use - you will need to modify this for your own sake

2. Create a function that performs the same calculations as the fit-function using the same parameters but that saves data differently. Note that you will have to create the folder "GFit_Components" manually - this is not in the code (but can be easily added). A clever programmer should be able to incorporate this into the function above. Note that this is two functions and you need both. The function JS_Update_Global_Fit() is the function you run in order to update your fits based on the current parameters.

4. Display the fit and results and all components with commands like this

Again, this is a working solution, but the code can be heavily optimised and improved and made better. However, for anyone who wants to have what I needed this will at least get you 75% on the way (I hope)

Working scheme:

Modify the fit function and the MakeFitData function to fit your code - these are essentially the same but with some added function to create new (global) waves. You will need to keep the lines where I create the waves for each component in order for this to be useful...

Feed in parameters and fit at least once. My code needs the waves to exist. This can easily be improved, but since this is a quick-and-dirt fix I just did what I needed for this to work for me - sorry.

Run  JS_Update_Global_Fit() and the display code (4 above)

Fix all the bugs that I introduced. I take absolutely no responsibility for any errors in the code above, it is 100% up to you to make sure it does what you want it to...

Edit: I use far from all these peaks. Most are helt to 0, the reason for adding this many is that one time I needed one extra and had to fill in litteraly 100's of initial guesses from scratch - so in this code I heavily overdefined the number of peaks.

Sorry for replying without reading - I had an editor open and did not see this. I think that I approached this exactly like you suggested ;)

Johan

I confess I haven't read this entire thread in detail, so maybe this comment isn't really needed...

I am the original author of Global Fit. Jim Prouty has, indeed, touched the package occasionally to patch things up when he added features like control panel expansion or to fix my mistakes in laying out the control panels. chozo has gotten involved with Global Fit as a super-power user who actually uses the package, adding recent enhancements and bug fixes. I want to express my deep appreciation to him- he does it because he thinks it's fun! 

Perhaps I should point out that Global Fit has two fit functions: 

Those are the actual fit functions passed to FuncFit. They are both all-at-once style functions, but the one with "AllAtOnce" in its name knows how to call all-at-once user functions. The other does standard style fit functions.

Both functions know how to access the list of data sets and the linkage matrix in order to build a coefficient wave for each individual data set extracting coefficients from the master coefficient wave.

Because there are two separate fitting functions, you can't mix standard functions with all-at-once functions. No doubt Global Fit could be modified to remove that restriction. I can't recall why I didn't do that previously.

To see what you're up against to make a fit curve for one data set, see the Global Fit 2.ipf procedure file, starting at this line:

Your own function could be less complex depending on what you know about how your own fit functions work.

I will also note that if you don't link any coefficients, then there is no need for Global Fit. You would be better off using Batch Curve Fit. In fact, using Global Fit with no linkages won't get you quite the same solutions as you get with individual fits because the length of the data sets will alter the weight for each of the data sets. A short data set will contribute less to the chi-square value than a longer one, giving the long data set more say in the fit result.