Low memory experiments by dynamic save/reopen in unpacked experiments?

Share icon

My (perhaps ridiculous) wish is for there to be an option (perhaps only available in unpacked experiments) where waves can be tagged in some way as "load only when in use". Igor would save to hard drive and kill these tagged waves whenever a function or procedure using them is complete, and bring them back when they are called again (or displayed). Ideally, when not in use/display, Igor would do some magic under the hood so that they would still be visible in the Data Browser and, perhaps, accessible by non-main-wave-data retrieval commands like DimDelta.

I imagine that even if that's possible, it's not going to be implemented any time soon, so I wonder whether there is something I might be able to write myself. In that regard, I wonder whether there is a way to get a "wave hook"-like function called whenever a wave is about to be accessed or modified? (I could see that hook slowing down processing quite a bit, but nevertheless...) if so, I could 1) check the wave name against the list of waves I want to "load only when in use" 2) reload matches if necessary, perhaps killing and replacing a placeholder before the access/modify begins.

If that type of more general background-running solution isn't possible, I will of course revert to more manually saving waves to hard drive, killing them, and reloading them before I need them.
May I ask how big your waves are in total?

You can implement something like the following yourself

Function/WAVE GetWavve()
// Load wave as ibw from disc
// convert to free wave
// return it
End


This function loads a wave from disc and converts it to a free wave. Always when you need that wave you use that function and after the wave assignment goes out of scope the memory is freed.

May 2, 2018 at 01:05 pm - Permalink

thomas_braun wrote:
May I ask how big your waves are in total?


johnweeks wrote:
Do you have a problem that can't be solved by using Igor64?


In the case that prompted me to write this, each original data file is 8Mb in an unpacked experiment. There are 4 of these per trial, 1-10 trials per experiment (4 on average), 20 experiment in an initial dataset and possible more to deal with down the road. The raw data from this dataset is therefore around 320 MB. The major problem is that I let my analysis get a bit out of hand (I am saving frequency-domain data out to a high nyquist frequency, and then saving differences of spectra and transformations), but I want to retain access to the raw data and the analysis results.

Here was the packed experiment size during analysis:
917 Mb before analysis
1.01 Gb after loading the raw data for one experiment (one set of trials)
1.53 Gb after analyzing that data

A few rounds of that and I am over the 32-bit limit. If I assume 600 Mb on average for the 20 experiments, I am nearing, though not at, the limit for Igor64 (which as I understand would be slightly less than the physical memory of my computer). Even then I could easily end up over the limit in the future.

That said, it is unfortunately far more convenient for me to work in 32-bit Igor, though I am always on a 64-bit computer, because I use an XOP that apparently will not be updated any time soon. I'm not particularly interesting in building my own workaround XOP. I will consider loading all the raw files and then reopening the Igor experiment file as 64-bit. Perhaps I might also be able to grab the files from an unpacked 32-bit experiment.

thomas_braun wrote:
Function/WAVE GetWavve()
// Load wave as ibw from disc
// convert to free wave
// return it
End
That's an interesting idea! Thanks!

johnweeks wrote:
I can think of many ways for such a scheme to go wrong.
I agree. Therefore, I would love to leave it to the experts (e.g., Wavemetrics)!

With solid state hard drives becoming the norm, I think it would be a really convenient addition to Igor. On the computer I am using right now, displaying one of those 8 Mb waves takes ~5-fold longer than saveing or loadwaveing it. There are definitely some reasonable options for workarounds, but this addition would let Igor users forget about memory (if necessary) and worry about other problems.

Thanks again for all the feedback!

May 2, 2018 at 02:10 pm - Permalink

Quote:
I think it would be a really convenient addition to Igor.


I agree.

It would require changes to our source code such that every access to a wave's data check to see if the data is already in memory or be guaranteed that such as check was already done. Although this is conceptually feasible, actually doing it and maintaining it in a foolproof manner is, I suspect, beyond our abilities.

(Perhaps it could be done using memory-mapped files but I suspect that would be quite difficult to implement flawlessly.)

Quote:
but I want to retain access to the raw data and the analysis results

Perhaps you could change this to retain a reference (i.e., a path to a file or folder on disk) to the raw data. You would use the reference to load the raw data, using the LoadData operation, as needed. You would unload it when no longer needed.

May 2, 2018 at 03:09 pm - Permalink

I see. It does sound tough to implement. I am not a particularly knowledgeable programmer, and I know nothing of error handling in built-in Igor functions; in spite of / because of that, an alternative that comes to mind would be to let a run-time error be thrown when a wave is missing and then implement code to deal with it. Or alternatively, allow the user to do so.

For a moment I thought I could do so with the user-defined hook for BeforeDebuggerOpensHook. However, in most cases (well, the useful non-literal cases) I fail to recover the name of the global wave that I was trying to declare/use locally. If it's possible to get that detail in all/many such error cases, then the user could handle it. Here's what I mean:

//example cmd:  displayExample("indeedIdont","andNeitherDoI") //in these examples, there are no globals called indeedIdont, andNeitherDoI
Function displayExample(waveRef0,waveRef1)
    String waveRef0,waveRef1
   
    WAVE iDontExistAndYouKnowItHere
    WAVE iMightNotButYouDontKnowMe=$waveRef0
    display/k=1 $waveref1
End

static Function BeforeDebuggerOpensHook(pathToErrorFunction,isUserBreakpoint)
    String pathToErrorFunction
    Variable isUserBreakpoint

    Variable rtErr= GetRTError(0)  
    String msg=GetErrMessage(rtErr,3) //could check for a wave name to recover in msg, then search some folder(s) for an ibw of the name
    Print "msg:",msg
    rtErr= GetRTError(1)
    return 1   
End


----EDIT: I should mention that this is what is returned when running that example--the identities of the literals can be recovered but not the globals (unless identical).
•displayExample("indeedIdont","andNeitherDoI")
  msg:  WAVE; WAVE reference to "iDontExistAndYouKnowItHere" failed.
  msg:  WAVE; WAVE reference to "iMightNotButYouDontKnowMe" failed.
  msg:  Display; expected wave name
----

A related possible alternative would be a user-defined hook function that is called on WAVE declarations (or at least their errors) and passes the name of the global that was sought.

This of course only handles the loading part, but the unloading part seems easier (e.g., user could just save and kill any large waves with a background function running every so often).

hrodstein wrote:
Perhaps you could change this to retain a reference (i.e., a path to a file or folder on disk) to the raw data. You would use the reference to load the raw data, using the LoadData operation, as needed. You would unload it when no longer needed.
Yes, I agree, and that's probably what I will end up doing. My initial code was (demonstrably) not the greatest, and I was looking for ways to minimize re-writing / additions.

May 3, 2018 at 05:18 am - Permalink

It would be difficult to code consistently, but you can use WAVE/Z to get at least some functionality. Something like:
Function displayExample(waveRef0,waveRef1)
    String waveRef0,waveRef1
 
    WAVE/Z iDontExistAndYouKnowItHere
    if (!WaveExists(iDontExistAndYouKnowItHere))
        WAVE iDontExistAndYouKnowItHere = Load_NonexistentWave(...)
    endif
    DoSomethingWithTheWave(iDontExistAndYouKnowItHere)
End

That function could be a GetPossiblyNonexistentWave() function that returns a wave reference to the now-loaded wave.

But it would be easier to use Igor64. What XOP is it that you need?

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

May 3, 2018 at 11:21 am - Permalink

johnweeks wrote:
It would be difficult to code consistently, but you can use WAVE/Z to get at least some functionality
I agree. Hopefully it will be mostly as simple as a function like that (coupled with the previous suggestions of tracking where waves were stored to the hard drive), followed by some CTRL+F for WAVE. My habit of de-referencing global waves e.g. Display $waveRef instead of declaring them locally is probably going to add a little headache.

johnweeks wrote:
But it would be easier to use Igor64. What XOP is it that you need?
The raw data files are Axon Instruments/Molecular Devices ABF files, and I use Bruxton's DataAccess XOP (https://www.bruxton.com/DataAccess/index.html). I have asked Bruxton about it, and at least at the time they did not have plans to add 64-bit compatibility. Adam Light once mentioned to me that it may in fact be a limitation of the tools provided by Molecular Devices for ABF files (along the lines discussed here).

As one potential workaround, I have wondered whether it might be possible to run one instance of 32-bit and one instance of 64-bit Igor side by side (as unpacked experiments if necessary) with some way of dynamically requesting that the 32 bit retrieve the waves and save them as ibws so the 64-bit instance can open them in that form. The 32-bit instance would be something of a slave/intermediary between the 64-bit instance and the XOP. Might there be an examples of packages that others have written that move files between two instances of Igor?

Thanks again for all the input.

May 3, 2018 at 12:33 pm - Permalink

Quote:
Might there be an examples of packages that others have written that move files between two instances of Igor?


For a low-tech approach that may work, choose File->Example Experiments->Programming->Watch Folder.

The master IGOR64 would write commands to a text file in a folder watched by the slave IGOR32. The slave would execute the commands and return results via another folder.

Or simpler yet, you could manually execute commands in IGOR32 and have IGOR64 watch for results. I would definitely start with this simpler approach.

Quote:
If I assume 600 Mb on average for the 20 experiments, I am nearing, though not at, the limit for Igor64 (which as I understand would be slightly less than the physical memory of my computer).


I think in your scenario you might do better than that because of virtual memory. If the experiment contains a lot of data that you don't access after loading the experiment, then the physical memory for that data will be used for other data that you do access. That said, physical memory inexpensive and is a good investment when dealing with a lot of data.

May 3, 2018 at 12:58 pm - Permalink

aoa wrote:

As one potential workaround, I have wondered whether it might be possible to run one instance of 32-bit and one instance of 64-bit Igor side by side (as unpacked experiments if necessary) with some way of dynamically requesting that the 32 bit retrieve the waves and save them as ibws so the 64-bit instance can open them in that form. The 32-bit instance would be something of a slave/intermediary between the 64-bit instance and the XOP. Might there be an examples of packages that others have written that move files between two instances of Igor?
.


The example Howard pointed out is a good place to start. Another approach that should work would be to use an ExecuteScriptText command from Igor64 to call the 32-bit Igor.exe with the /X flag. The command you would pass to the \X flag would be something like
[full path to Igor.exe] /X "ConvertABFToWave(\"C:\\Users\\me\\Documents\\data1.abf\")"

Getting the escaping of the quote marks and backslashes right might take some work.

You can use SpecialDirPath with the "Igor Application" selector to get the Igor installation directory. Then you'd append "\\IgorBinaries_Win32\\Igor.exe" to the end of that to get the full path to the 32-bit Igor executable.

ConvertABFToWave() would be a function defined in an Igor procedure file that's contained in your Igor Pro 7 User Files\Igor Procedures directory. Since the file would also be loaded by Igor64, you would want to conditionally compile that function based on the absence of the IGOR64 symbol. See DisplayHelpTopic "Predefined Global Symbols" for details on that.

ConvertABFToWave would do the conversion from ABF to IFW and save the output file in the same directory as the source directory (or something else, if that works better for you). You have the option of using the /W flag with ExecuteScriptText on windows to force Igor to wait until the process returns, but that won't be of much use if you're calling Igor.exe, since it's a GUI program and ExecuteScriptText will return once Igor.exe starts processing messages. So you'll need to have a loop after the call to ExecuteScriptText that waits until the output file is written.

For an ExecuteScriptText example, see http://www.igorexchange.com/node/1243.

See also:
DisplayHelpTopic "Calling Igor from Scripts"

May 3, 2018 at 02:42 pm - Permalink

In case one want to spend an enourmous amount of time on a edge case project, I would suggest to make 32bit XOPs run with 64bit IP instead of implementing on-demand wave loading. 64bit is here to stay for the forseaable future and having something like 64GB RAM on a normal computer is nothing impossible.

May 4, 2018 at 06:19 am - Permalink

Thanks for all the great suggestions! I was able to run the analysis in 64-bit Igor, and, indeed, it did not run out of memory. I expect to be able to use Igor64 whenever I would like to in the future, which is great.

For anyone else that hasn't done something like this before and might like to try, the approach I settled on was a windows command-line back and forth between 32-bit and 64-bit. I was surprised to find that it was possible to avoid background tasks because script-initiated commands conveniently appear to run in parallel with on-going execution.

  1. A text wave loaded on both Igor32 and Igor64 held wave names and the path to their ABFs

  2. My analysis function, in Igor64, requested the wave that was needed by passing the name to function A, after which the analysis function went into a loop that broke when the requested wave existed (or until some timeout time).

  3. Still in Igor64, function A found the wave's index, stored the wave name in a global, and called function B (in 32-bit Igor), which took the index as a parameter. Calling functions from one instance of Igor to another used the executeScriptIn32Or64 function below.

  4. In 32-bit Igor, function B loaded the wave from the abf file associated with that index, saved it as ibw to a path that was known to both instances of Igor, called function C in Igor64 (and then killed the wave)

  5. In Igor64, function C used the global that function A had stored to determine what wave was sought, then loaded the wave from the ibws in the shared path.

  6. Still in Igor64, the presence of the wave broke the loop in the main analysis function, which then proceeded.


One key to setting this up in a timely manner was giving up on nested strings in /x script commands from the windows command line. As predicted, it seemed like it would really take some work.

Here's the function that acted as intermediary between instances of Igor (not mac compatible):
strconstant ksExecutable32="IgorBinaries_Win32\Igor.exe"
strconstant ksExecutable64="IgorBinaries_x64\Igor64.exe"
function executeScriptIn32Or64(cmdStr,igorVers)
    String cmdStr           //command to execute in other instance of igor e.g., "funcB(7223)"
    Variable igorVers       //version of Igor to command
   
    String windowsIgorPath = SpecialDirPath("Igor Application",0,1,0)
    string execPath
    switch (igorVers)
        case 32:
            execPath =  "\""+ windowsIgorPath  + ksExecutable32 +"\""
            break
        case 64:
            execPath =  "\""+ windowsIgorPath  + ksExecutable64 +"\""
            break
        default:
            return 0        //not a valid igor version
    endswitch

    String scriptText = execPath +  " /x \"" + cmdStr + "\""
    ExecuteScriptText  scriptText
end


thomas_braun wrote:
In case one want to spend an enourmous amount of time on a edge case project, I would suggest to make 32bit XOPs run with 64bit IP instead of implementing on-demand wave loading.
That would be great. For better or worse, at the moment I am stuck on other time-consuming edge case projects, such as wrapping up my dissertation research!

May 5, 2018 at 07:35 am - Permalink

Quote:
For anyone else that hasn't done something like this before and might like to try, the approach I settled on was a windows command-line back and forth between 32-bit and 64-bit.


Nice!

May 5, 2018 at 12:43 pm - Permalink