Better way to handle temporary data and panels

I'm no expert on Hook Functions, but couldn't you simply run your cleanup function every time before an experiment is saved using BeforeExperimentSaveHook?

I assume the DoCleanup() is the function to dump the trash in a manner of speaking. Have you considered the benefits to have the cache files stored outside of the experiment? The intent is to avoid any need to have to clean up the experiment (e.g. if it crashes). The one added benefit is that you can store a default cache file that can always be restored regardless of how the experiment "went down" (again in a manner of speaking). With some clever programming, you could even sense whether the user had stored a favorite template set that should be restored in place of the default. The downsides are that you move the overhead (time) now taken by the DoCleanup() function to a set of functions akin to WriteToCache() and ReadFromCache() and that you have to decide where the cache file is going to be stored for your package.

As for saving the experiment without the informed consent of the user ... Does the if(!modifiedBefore ...) segment support a DoPrompt call with the Yes/No request? In other words

if (...)
   ...
   DoPrompt ... "Do you want to save this cleaned-up experiment?"
   ...
endif

Alternatively, I might think that you could check whether cleanup is needed (e.g. have a CleanUpNeeded() function that returns 0 or 1) and then ask the user for permission. Finally, you could invert the the cmpstr(...) portion of your test and only do the unbidden save if a user had already saved the experiment with a real name. In this case, return 0 from the hook functions and, if I am not mistaken, Igor will also prompt the user whether to save an "Untitled" experiment or not. Otherwise, return 1 to indicate that your function is taking over what Igor normally does.

Thanks both for your replies.

@olelytken

> I'm no expert on Hook Functions, but couldn't you simply run your cleanup function every time before an experiment is saved 
> using BeforeExperimentSaveHook?

If I do that I would make it hard to continue measuring after saving, currently you can also save during data acquisition and this works.

@jjweimer

> External cache

If I make the cache external of the current experiment, I need a way to access that and would end up with waves from that cache inside the experiment which I would then need to delete again.

>  The downsides are that you move the overhead (time) now taken by the DoCleanup() function to a set of functions akin to WriteToCache() and ReadFromCache() and that you have to decide where the cache file is going to be stored for your package.

The cache has to be very fast so making it a bit slower to solve the issue here would not really be an option.

I would also like to avoid asking the user. Most users are not very firm with Igor Pro and I like to avoid any possible friction.

> If I make the cache external of the current experiment, I need a way to access that and would end up with waves from that cache inside the experiment which I would then need to delete again.

Yes. Certainly. I was for some reason under the mistaken belief that one could use an external cache as a disposable storage in its own right.

My thought is equivalent to being able to use an #include pragma but for data ... #withdatafile "MyDataFile.dat".

Would it possible to do some sort of weird gymnastics where your procedure converts your acquisition datafolder to a /free datafolder, saves the experiment, and then converts the datafolder back to a normal datafolder? Would a free datafolder be saved?

I always have to think carefully about free anything...

In order to avoid having the free data folder be destroyed, you have to keep a reference to it. Since the reference has to exist outside of a running function (unless I'm not quite getting your solution) then it would have to be held in a DFREF wave. A non-free DFREF wave. That wave would be saved...

Igor, in fact, goes to great lengths to save and restore the free contents of DFREF and WAVE waves.

I guess a robust solution would be a new specialized 'cache' or 'temporary' folder and panel type, which reside in memory when Igor is running but are never ever saved to disc. If the program is closed, they are gone. I would probably make use of this in my projects. No more worrying about users haphazardly leaving old (=version) and stale panels open when saving experiments would be great!

This seems to work for me. Test1() creates a big wave and saves the experiment with the big wave in it. Test2() saves the experiment without the big wave, but also without deleting the big wave from the active experiment. The big wave is still there after the function call.

I think this will not work if BigWave is displayed in a graph or otherwise used (KillWaves will fail).

True, you will have to kill the graph before saving the experiment, and then recreate it after, but it can be done with a little work

Yes, if you keep track of everything you do not want to have saved and hide it from the user (so they cannot mess with the data) then you could pull it off by silently shifting all data into free waves and close/rebuild all graphs or panels which should not be saved. This would need to be done in BeforeExperimentSaveHook(). I wonder how difficult it would be to save a recreation macro of a panel / graph in a free text wave and then have it closed / reopened. I guess this would at least lead to a flicker or bring the panel to the front. Might be OK, though.

EDIT: Thinking a bit longer about it, I wouldn't even know how to pull this off using BeforeExperimentSaveHook(), since I guess the function terminates before the experiment is saved.

Maybe johnweeks' suggestion of a normal DREF wave holding a reference to a free wave would keep the free wave alive...

On second thought. I would end BeforeExperimentSaveHook() with a SaveExperiment command and then cancel the whole ExperimentSaveHook chain. That should be possible, or?

Yes, but I have no idea how to cancel the save process from within BeforeExperimentSaveHook(). The help does not say anything in this regard. I haven't tested anything in this regard, though.

I believe that olelytken's approach would work to save the experiment without also saving the free wave, but I think it is playing with fire (though very clever).

In IP9, you would probably be better off using MoveWave instead of Duplicate + KillWaves so that the wave doesn't actually need to be copied. I'm not sure if MoveWave can do this in IP8---based on the documentation I don't think it can.

Anyone using this technique must keep in mind the fact that if there is an error that stops function execution, or if the user stops execution (either by aborting or perhaps in the debugger), the BigFreeWave reference count will drop to 0 and it will automatically be killed. So make sure that whatever is in BigFreeWave is completely expendable. It sounds like in Thomas's case it is.

Igor does have an internal concept called a homeless wave that is essentially what Thomas is requesting. This is a wave (it can even be displayed in a graph) that is not connected to the global hierarchy in any way and which is therefore not saved with the experiment. I believe that contour plots use homeless waves, and they are also used in the Data Browser in some situations. There is currently no way for user code to create or otherwise interact with homeless waves. The fact that they can be displayed in graphs makes them dangerous because if the wave is killed without the graph being killed first, Igor will crash. So exposing homeless waves to user code is something we would be very hesitant to do. But we would be interested to hear how users might use such a feature if it existed.

but fire is so irresistible :)

I tried a version with a wave reference wave pointing to a free wave, see the example below, but if I save this experiment the free wave is saved with the experiment.

olelytken wrote:

but fire is so irresistible :)

I tried a version with a wave reference wave pointing to a free wave, see the example below, but if I save this experiment the free wave is saved with the experiment.

Function Test3()

    //  Creates a big data wave
    Make/O/FREE/N=1e8 BigData=enoise(1)
   
    //  Creates a normal wave reference wave pointing to the big data wave
    Make/O/WAVE root:BigDataReference={BigData}
end

 

Right, that's the expected behavior. The global root:BigDataReference wave holds a reference to the free BigData wave, so it's saved in the experiment. In your earlier example the wave reference to the free wave existed only within the Igor function itself, but there was no reference to that free wave from anything in the global space, so it's not saved with the experiment.

Another potential problem with relying on this behavior is that it may not actually be intentional on our part, so it's possible this could change in the future.

I rather would be happy about a method to keep panels from being saved, preferably without killing them during the save operation. Would it be possible to introduce a flag to mark graphs, panels (and waves) as locked / transient for the saving process?

I just reread the documentation for "BeforeExperimentSaveHook" and played around with it some more. And I think I misunderstood how it works. You don't need to call SaveExperiment in it, as you are called *before* the experiment is saved. And the user already choose to save as well.

Being able to mark things as transient sounds like a good idea. This would probably need to be inheritable, meaning a graph displaying a transient wave would also be transient. And transient datafolder would be a bon as well.

Or maybe we need another set of hooks which allow to mark something as transient just before saving?

Something like

but that would need to be called for every object in IP. And would probably be dead slow.

I want to draw renewed attention to my posting a ways back:

In order to avoid having the free data folder be destroyed, you have to keep a reference to it. Since the reference has to exist outside of a running function (unless I'm not quite getting your solution) then it would have to be held in a DFREF wave. A non-free DFREF wave. That wave would be saved...

Igor, in fact, goes to great lengths to save and restore the free contents of DFREF and WAVE waves.

Yes, saving free waves is deliberate. It is driven by the fact that a non-free DFREF or WAVE wave will keep free waves alive. If we didn't do this, we would have to go to some great lengths to avoid crashing on loading a wave full of pointers to nonexistent waves.

Adam can look at the convoluted code in Igor that saves DFREF and WAVE waves!

> But we would be interested to hear how users might use such a feature if it existed.

Homeless sounds a bit forlorn. How about expendable or trashable or purgeable.

make/N=.../EXPEND
make/N=.../TRASH
make/N=.../PURGE

Such operations would create (globally accessible) waves that would reside in their own "sandbox", perhaps even in their own data folder directory spaces. These waves would disappear when the experiment is closed or quit (i.e. be purged). The programmer would be responsible to check for their existence or to transfer their contents to permanent storage when needed.

They could be the meta-goal of all free waves ... be resident not just for the life of a function but for the life of the open experiment.

I could live with any requirements that might be needed to make clean-up easier. For example, I would have no problems when such waves could not be graphed or used in dependencies.

How would I use them? Depending on the performance improvements or the ease in administrative oversight, I might have one master Init function that would create these waves rather than creating the same kinds of /FREE waves in a host of comparable function calls. It is the equivalent of knowing that you have a second desktop where you can scribble around and when you come back the next morning to start again, you have a fresh desktop.

A big part of the reluctance we might have for "trashable" waves is the need to restrict their use. Since free waves were introduced, we have been closing the loopholes in how they can be used. When they were new, we had crashes reported from making graph traces with free waves. Just recently we closed a loophole in which a free data folder was made the current data folder (perfectly OK), but then a new folder was made inside it and made to be the current data folder. That released the only DFREF and the folder died. Igor crashed trying to restore the current data folder.

Think of the many ways a wave can be used that require persistence. Like graphs, dependencies, use as a value for a control, many other uses. All of which need to be checked.

@jweimer We use an int32 for operation flags, so they have to be four characters or less.

Make/N=.../XPND

Make/N=.../TRSH

Make/N=.../PURG

We try to avoid actual four-letter words! :)

> Since free waves were introduced, we have been closing the loopholes in how they can be used.

I appreciate the effort.

> We use an int32 for operation flags, so they have to be four characters or less. ... We try to avoid actual four-letter words! :)

Oh, I see a Jeopardy box here. ... A four letter phrase that is never a four letter word. What is a flag in Igor Pro?

(This in itself gives thoughts to a rather interesting adventure ... Let's create Igor Pro Jeopardy game).

I like XPND as eXPeNDable.

jjweimer wrote:

I like XPND as eXPeNDable.

Well, to me that looks like eXPaND. On the other hand, I can't find 'XPND' in our source code, so at least it wouldn't be an inconsistent use. You must have noted that Igor is a model of consistency (sarcasm over with) :)

But this is all theoretical unless we actually implement something like this!