multithreaded curvefitting

That would be because a polynomial fit is linear in the coefficients and uses totally separate code from the nonlinear, iterative fits. The linear fits (line fits, polynomial and poly2d) should still be faster than the equivalent nonlinear fit because they don't iterate.

@tony Can you provide a MWE for playing around? I also don't see a reason why poly order 5 doesn't benefit from multithreading.

OK, I misunderstood a bit. I was thinking of the automatic multithreading internal to the iterated nonlinear fits. But this is a question about doing multiple fits simultaneously via the Multithread keyword.

My guess is that the polynomial fits are sufficiently fast that for whatever number of fits you're doing, the threading overhead is swamping the benefit of multithreading. Just a guess. As Thomas suggests, some play would be in order.

Turns out that the slowdown happens only when a mask wave is involved. For poly fitfunc, not for others.

There is also an unexpected dependence on the length of waves to be fitted:

Other fit functions don't show this kind of behaviour.

Edit: add system info

  Model Name:    MacBook Pro
  Model Identifier:    MacBookPro15,2
  Processor Name:    Quad-Core Intel Core i5
  Processor Speed:    2.3 GHz
  Number of Processors:    1
  Total Number of Cores:    4
  L2 Cache (per Core):    256 KB
  L3 Cache:    6 MB
  Hyper-Threading Technology:    Enabled
  Memory:    16 GB

That is bizarre. Share the experiment so I can do some profiling?

did you take a look at the ipf linked above? is it a. reproducible for you and b. enough to do some profiling?

There must be more in this, my system (IP9.05 on MacOS ARM,  M1PRO)  shows very different results:

Now that is strange.

From my home computer, an older macbook pro:

compared with gaussian fits (poly order doesn't have any meaning here):

Hardware Overview:

  Model Name:    MacBook Pro
  Model Identifier:    MacBookPro14,2
  Processor Name:    Dual-Core Intel Core i5
  Processor Speed:    3.3 GHz
  Number of Processors:    1
  Total Number of Cores:    2
 

There's even more excitement in the Wintel world.  Gauss fits with any data length are fine, as are 4th order poly fits of any length.  However, with 5th order poly fits, if the datalength is greater than 300 or so, the multithread fitting causes Igor to essentially lock up and makes the computer super laggy.  The memory usage stays the same, but the CPU quickly goes to 100% and the process never completes.  I can abort the process, but when I try to close the experiment I often get a window saying that Igor has crashed.  4th order polynomial and Gauss fits seem fine, as do single threaded 5th order polynomial fits.  I've shipped the crash report off to WM support.

System info: Intel 12th Gen, i5-12600H
Windows 10 Enterprise (22H2), 10.0.19045
Igor 9.0.5.1

It's not reproducible:

MacBook Pro 16in, 2.4GHz i9, 32GB RAM

I am getting consistent times on the poly 5 1000.

Dare I propose that this is related to an i5 versus i7 versus i9 chip?

Interesting that the M1 chip does worse in multi-threading versus single threading in the poly 5 1000 test compared to the i9. Indeed, if I read this correctly, I should certainly not switch my i9 for an M1 as long as I am planning to need heavy processing with Igor Pro.

If instead of using CurveFit poly I fit a user-defined polynomial fitting function (FuncFit UserPoly), fitting is of course way slower, but I see the expected speed increase with multithreading. No surprise there.

It's clear that something is amiss with CurveFit poly. I am curious about how the M1pro fares with other fit functions. I would expect to see significant speed improvement with multithreading.

Another windows data point.  As KZarzana observed, poly 5 starts to get slow between 300 and 400 points and CPU usage remains at 100% until completion.  However, there was no problem with the computer responding to, say, mouse movement.  

Completion time for unmasked multithreaded poly 5 hit 2e8 for 400 and above points with approximately the same time beyond that number of points.  Single threaded times were on the order of 1e6.  Time for the Masked MT fit took a big jump some where between 500 and 1000 points.

Dell Precision 7760 - 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz; 32 GB RAM; 8 cores/16 logical processors

Macbook Pro, M1PRO (ARM), Sonoma 14.4.1, IP 9.0.5.1. I think it is working with Gauss as expected. 

Igor manages to load all cores. Since some cores are performance and some are efficiency, I wonder how that makes multithreading complicated as results from different cores must be arriving at very different speeds.  

KZarzana has sent us a report of an actual crash while doing the multithreaded poly fit test.

Given that the test uses enoise() to set the poly coefficients, it's highly likely that sometimes there is a pathological data set that causes unusual problems with the fitting.

The erratic results (see my test results above with three runs of N=500 above, where everthing looks mostly OK, but one test gives a result 1.5 orders of magnitude higher) plus KZarzana's crash, I'm betting on some sort of thread safety violation in the poly fit code.

That wouldn't be surprising given that the basic code was first written 30 years ago. At present, it is using a SVD solution based on Numerical Recipes edition 1.something, from before they tightened up their licensing restrictions. That would be suspect...

I'll let y'all know what I find.

Yeah, using enoise is a poor choice for making speed comparisons. But I doubt that a pathological data set could be created using this method. The fact that problems arise only under restricted combinations of inputs rather suggests otherwise. For this test, creating an input wave of zeroes would probably show the same trends.

I remember coding plenty of numerical recipes stuff in Igor in the days before we had so many built in methods available. I used to translate from the fortran version because that was the only code I understood.

Two more data points.  On an AMD Ryzen 3700X system (Win10, Igor 9.0.5.1):

There weren't any slow down issues on the Ryzen Windows system with the poly 5 fits.

And on an M2 Max (macOS Ventura 13.5.2):

I did some more testing, this time with a fixed dataset. Fitting a dataset that reproducibly ran into problems on Igor 9 worked fine on igor 8.

Igor 8 does, however, have issues if you try to multithread FuncFit rather than Curvefit.

The problem was thread contention between Igor's Multithread and the Intel MKL library's threading for LAPACK and BLAS calls. That has been fixed, but unfortunately the changes required were ones we didn't feel comfortable with adding to Igor 9. The fix was to make a call to MKL to tell it to not use threads.