Multitasking: Running multiple simulations at the same time

Hello,

My target is to run simulations with various initial conditions. I calculated and found that each simulation will take about 15 hours, so if I want to run 40 simulations, it will take almost 600 hours (which is so many days!). So I was wondering is it possible to run multiple simulations at the same time without losing simulation performance speed? Each simulation is using many do---while, for loops to use rejection methods and other things and the whole program is a bit lengthy- almost 4000 lines. I found that I can start another Igor Pro instant but will that decrease my simulation speed? Will the waves be overwritten and messed up with one another? 

I looked for multi tasking and went through thread safe functions etc, but I am new in using these. Do I need to use multiple cores/threads of the processor to run each simulation in a different thread for maintaining a good speed for each simulation? Notice that I am using a 2 GHz Dual-Core Intel Core i5 MacBook Pro machine.

Any suggestions will be much appreciated.

Thanks a lot.

Ann

First, since Igor's compiler doesn't do much in the way of optimization, you may be able to improve the performance of your code with some simple steps. Prefer wave assignments over loops, if you have a constant computation inside a loop, pull that out so you only do the computation once. MatrixOP will often give you a performance boost. It's a lot to ask a stranger to examine 4000 lines, but it might be that you should do that- have an experienced Igor user look over your code to look for opportunities to boost performance.

Second, since your model takes a very long time to run, running more than one instance of Igor is a very simple way to parallelize your jobs. Some operations in Igor are automatically multithreaded (at least in part) and it might be best to turn that off so that the jobs don't contend for the same CPUs. You can control a lot of that via the MultiThreadingControl operation.

If you are doing each simulation in a different experiment, Igor comes with code to run the same set of commands on multiple experiments in a row.  The Demo can be found here: File->Example Experiments->Programming->Multi-Experiment Process.  I made a version of the Multi-Experiment code that works with multiple Igor instances, so each instance of Igor can run multiple experiments in a row.  There are instructions in the comments at the top of the ipf.  I've used this in situations similar to yours, where the processing time for each experiment was long, so I didn't want to wait around to start the next experiment.  The code is a little kludgy, because I just needed something that worked, but so far it has done what I needed it to do.

A few thoughts ...

* An i5 is the low end in processing power, even for its time and certainly for now. Consider moving to an i9 processor on a (beg, borrowed, or stolen) Windows (or macOS) system or upgrading to the new M chips. Review especially the comparative benchmark specs. You may cut 15 hours significantly just by using a higher performance CPU.

* As John noted, for and do-while loops are labor intensive. Consider writing the code to transform explicit loops into implicit wave assignments.

Yes throwing better hardware at the problem will also help.

The obvious things to do to improve performance are:

- Convert hand-rolled loops to wave assignment statements, see "Waveform Arithmetic and Assignment"
and "Automatic Parallel Processing with MultiThread" for that.

An easy example is

Function Dostuff()

    variable i, numEntries, refTime

    Make/FREE/N=1e6 elements
    numEntries = DimSize(elements, ROWS)

    refTime = StopMSTimer(-2)

    for(i = 0; i < numEntries; i += 1)
        elements[i] = i
    endfor

    printf "elapsedTime: %g\r", (StopMSTimer(-2) - refTime) / 1e6

    refTime = StopMSTimer(-2)
    elements[] = p
    printf "elapsedTime: %g\r", (StopMSTimer(-2) - refTime) / 1e6

    refTime = StopMSTimer(-2)
    Multithread elements[] = p
    printf "elapsedTime: %g\r", (StopMSTimer(-2) - refTime) / 1e6
End

which gives

  elapsedTime: 0.0461154
  elapsedTime: 0.0206244
  elapsedTime: 0.003429

here. The last speedup with Multithread will not that good on your end with a dual core CPU, but still an improvement

- Always use the latest IP version, 9.05 is current

- Prefer to precalculate: So for example to to get the circumreference for n radiuses don't do

circumRef[] = radius[p] * 2 * pi

but instead

twoTimesPi = 2 * pi

circumRef[] = radius[p] * twoTimesPi

- Using stopmstimer(-2) for timing is an easy way to analyse the run time, a fancier approach would use "Profiling Igor Procedures"

- Always keep old versions of your code around so that you can go back in case some speedups broke something.
 

Looking into all these topics takes time, but should get the code much faster. Naively written code can easily be made ten times faster.