Loading an experiment file without loading the contained Igor Wave files

Share icon

Hi,

These days I am working on analysing a bunch of data contained by 1000 or more .ibw files.
I wrote a script which load them without copying, such that I can keep them in RAM and .pxp file is small.

However, if I move the drive containing the .ibw files away, the experiment file (.pxp) can't be open unless I skipped over all the file.
If I abort the loading, I can't even access the command history which I often need to see.
Is there a way to skip this loading easily (apart from skipping them one by one)?

 

Thanks.

Hi,

 

One idea that I use to copy procedures from one experiment to another.  Open the file in a text editor, I use BBEdit on the Mac. Scroll to the bottom and you see all the procedures in clear text.  I copy and them and then open a new experiment and paste.  I them browse the original experiment from the data browser and copy in the need waves, variables, and strings...

 

One Idea

 

Andy

October 24, 2018 at 12:44 pm - Permalink

Quote:
Is there a way to skip this loading easily (apart from skipping them one by one)?

No.

What you could do is save the original experiment on the original machine using a different file name, kill the waves in the new experiment and save it, and then transfer the new experiment to the other machine.

 

 

October 24, 2018 at 01:33 pm - Permalink

hrodstein wrote:

 

Quote:

Is there a way to skip this loading easily (apart from skipping them one by one)?

 

No.

What you could do is save the original experiment on the original machine using a different file name, kill the waves in the new experiment and save it, and then transfer the new experiment to the other machine.

Is there a hook which can be bound to the event raised upon LoadWave?

 

best,

_sk

October 25, 2018 at 01:12 am - Permalink

hrodstein wrote:

 

Quote:

Is there a way to skip this loading easily (apart from skipping them one by one)?

 

No.

What you could do is save the original experiment on the original machine using a different file name, kill the waves in the new experiment and save it, and then transfer the new experiment to the other machine.

 

 

This is a preventive measure, but it won't help the files already saved that way. I can still get what's inside them from browsing, but then other things like opened graph or commands history won't be accessible.

From the fact that I can manually skip all the attached waves (let's say 100 of them) and then still open the experiment file, this is something solvable by allowing an option to "skip all attached waves" or equivalent, instead of one by one, maybe considered as a feature later?

October 25, 2018 at 11:09 am - Permalink

Quote:
this is something solvable by allowing an option to "skip all attached waves" or equivalent, instead of one by one, maybe considered as a feature later?

I will keep it in mind. As far as I can recall, you are the first to request it.

In most cases skipping waves leads to errors later, such as when graphs or tables are created.

 

 

October 25, 2018 at 12:09 pm - Permalink

 I wrote a script which load them without copying, such that I can keep them in RAM and .pxp file is small.

Why? Hard drive storage is ... cheap. This approach also seems to break common sense rules to assure that you have a bullet-proof future review of all of your experimental protocols.

Graduate Student: I was trying to re-analyze the data in the Igor Pro experiment to prove our Nobel Prize Results and cannot seem to find the off-line hard drive to get the raw data back.

You: Ooops! I think that hard drive just died yesterday.

October 25, 2018 at 12:46 pm - Permalink

jjweimer wrote:

 I wrote a script which load them without copying, such that I can keep them in RAM and .pxp file is small.

Why? Hard drive storage is ... cheap. This approach also seems to break common sense rules to assure that you have a bullet-proof future review of all of your experimental protocols.

Graduate Student: I was trying to re-analyze the data in the Igor Pro experiment to prove our Nobel Prize Results and cannot seem to find the off-line hard drive to get the raw data back.

You: Ooops! I think that hard drive just died yesterday.

What you say is true, unfortunately in our lab the computing resources are shared so I can only stick to a computer that is available at that moment to start the analysis.Very often what happens is I need to run an analysis over 100 GB of data away from the measurement computer, then I will just use a USB stick to move them. As a result, copying back to harddisk takes time and can really fill it up quick.

Indeed, just in case I am getting Nobel from it, I have ALL data backed up in multiple physical location, plus one being a RAID 6 array.
Still it doesn't solve the problem where I need to quickly go from one computer to another.
Maybe in the future we can consider a network drive.

October 26, 2018 at 12:04 am - Permalink

hrodstein wrote:

 

Quote:

this is something solvable by allowing an option to "skip all attached waves" or equivalent, instead of one by one, maybe considered as a feature later?

 

I will keep it in mind. As far as I can recall, you are the first to request it.

In most cases skipping waves leads to errors later, such as when graphs or tables are created.

 

 

It's true that it's not an elegant way, but it helps when I really just want to read some command history or to show a graph which was formatted. Users should be aware of this not being a safe way to work on that experiment file.

October 26, 2018 at 12:06 am - Permalink

Sandbo,

The easiest way is what hegedus recommended, open the experiment file and look at the very end for the records: experiment recreation, history, procglobal procedures, etc.

If you want to make it nice, you could use the experiment technical note (PTN003 Writing Packed Files, written by hrodstein) and find the regions programmatically. This could be done from within Igor also.

best,

_sk

October 26, 2018 at 12:44 am - Permalink

Very often what happens is I need to run an analysis over 100 GB of data away from the measurement computer, then I will just use a USB stick to move them. As a result, copying back to harddisk takes time and can really fill it up quick.

I might suggest a different approach.

Purchase a dedicated USB 3 external hard drive. Create one location on that drive where the files are to be stored from the measurement computer. Rather than put the data from the instrument computer on a USB stick, put it into the storage location on the hard drive. Re-write your Igor Pro experiment. Have it first check whether that drive is mounted and the location is available. Then, only when the drive is mounted and the location is available, have it load any (new) data (without copying).

You could force the Igor Pro experiment to work always by putting the Igor Pro experiment ONLY on the external hard drive and NEVER on a local computer. Buy a hard drive for each member of your lab group. You then have not need to consider setting up a network drive. And, you can do the data analysis while sitting directly at the measurement computer (in case your analysis tells you that need to pull in other data).

Also, I might suggest that you consider using IgorBeforeQuitHook to unload the data waves from the experiment that you are quitting. 

 

 

October 26, 2018 at 07:39 am - Permalink

hrodstein wrote:

 

Quote:

this is something solvable by allowing an option to "skip all attached waves" or equivalent, instead of one by one, maybe considered as a feature later?

 

I will keep it in mind. As far as I can recall, you are the first to request it.

In most cases skipping waves leads to errors later, such as when graphs or tables are created.

 

 

 

Allow me to be (at least) the second person to request this option. I just ran into the issue in question when trying to re-opening a pxp file that I have loaded ~12,000 waves into. I'd loaded the waves with calls to loadwave (no /H flag), so the waves accidentally went un-adopted. Those waves aren't available to me at the moment. (The lack of the /H flag was a coding error on my part that I later remedied, without remembering that this pxp was compromised. In the meantime, I deleted the folders and files that the pxp is looking for and switched computers.) It would be great to be able to skip them all the missing waves with one click or option so that I can get to some unrelated data in the pxp file. 

It's also annoying that even if I restore one of the waves (at a new location, different computer) and the others are saved in the same (new) folder, Igor doesn't automatically realize that they are all available in that (new) location (path). If I find the first, it asks for the second, and so on.

I will probably be able to store the files at the original location and reopen the pxp on the original computer, but it would have been easier to just tell Igor to skip all.

Tangentially, I tried Andy's recommendation to open the pxp in a text editor and get the relevant commands out. Unfortunately, the pxp is 6 gb and apparently too large for the versions of notepad and notepad++ I have. (Would be curious to know if there are text editors better equipped for this; maybe I could read in portions of the text in Igor.) 

October 28, 2018 at 07:01 pm - Permalink

aoa wrote:

Tangentially, I tried Andy's recommendation to open the pxp in a text editor and get the relevant commands out. Unfortunately, the pxp is 6 gb and apparently too large for the versions of notepad and notepad++ I have. (Would be curious to know if there are text editors better equipped for this; maybe I could read in portions of the text in Igor.) 

 

You can try something like tail. In *nix systems this will display the last portion of a file using ansi or whatever encoding.

In windows, something like baretail(http://www.baremetalsoft.com/baretail/) may work, but I have not tested it.

 

best,

_sk

October 28, 2018 at 11:57 pm - Permalink

Quote:
It's also annoying that even if I restore one of the waves (at a new location, different computer) and the others are saved in the same (new) folder, Igor doesn't automatically realize that they are all available in that (new) location (path).

I have added this to our wish list for Igor9. I don't know if it will rise to the top of the to-do list.

Use symbolic paths with LoadWave (/P=<symbolic path name>). If you use a symbolic path, then, when Igor loads the experiment, if the folder that the symbolic path points to is missing, Igor asks you to locate the folder. Then all subsequent references to the symbolic path point to the folder you have identified.

I also added skipping all subsequent waves to the to-do list for Igor9.

October 29, 2018 at 09:19 am - Permalink