I have a large data set that for the time being I have divided into multiple pxps. Instead of running the same code in each file separately, I'd like to have a central pxp that calls each file, runs the code and copies the results only to the central pxp. Is this doable in igor? Or do I have to temporarily copy the waves from individual files and kill them after analysis? The data set in each individual file can be quite large, 1GB or more and I have 90 of them. I don't think I can run a single 90 GB file on this machine without it crashing. Any ideas on what would be most efficient?
You might want to look at using an unpacked experiment. This will generate a folder with he waves as separate files, as are the procedure files. They show up a normal directory. You can then load and unload the individual waves as necessary. I have used this approach when I built a metrology tool that generated individual data waves. I did not want to load all 6000+ sample runs to run the experiment. The other advantage was that since the files were saved individually if the program crashed I did not lose data even if the experiment had not been saved.
Or you can use multiple instances, one master Igor instance distributing the work and multiple Igor clients. For communicating between them you could use the ZeroMQ XOP [1].
You might want to look at using an unpacked experiment. This will generate a folder with he waves as separate files, as are the procedure files. They show up a normal directory. You can then load and unload the individual waves as necessary. I have used this approach when I built a metrology tool that generated individual data waves. I did not want to load all 6000+ sample runs to run the experiment. The other advantage was that since the files were saved individually if the program crashed I did not lose data even if the experiment had not been saved.
Andy
September 1, 2017 at 04:29 pm - Permalink
Disclaimer: I wrote the XOP.
[1]: https://github.com/AllenInstitute/ZeroMQ-XOP
September 2, 2017 at 04:30 am - Permalink