Support for loading zipped (single or even multiple) ibw / pxp files

Share icon

Inspired by the recent (un)zip support in Igor 9, and the nice code snippets from Tony, I began to dream about working with zipped ibw / pxp (the former much more important for me). As disk space is not limitless even in 2021 (I am working from a notebook with a not-so-big SSD here), while advanced experiments have an ever increasing data output, I though it would be nice to offset this dilemma by working with zipped files. Before I try to cook something up on my own, I might as well ask how realistic official support for such a feature would be. What I am asking for is an unzip-load combination which natively eats zip files and in the same step loads the contained data as waves in an Igor experiment. In my case, the difference would be as extreme as having ~10 GB of raw data or zipped files of just several 10 MB total size. Of course, there will be a small time penality, but I would be more than happy to take this over time wasted organizing data on / copying data to external drives (other than for backup purposes of course). 

I don't know of any plans to implement what you suggested.

But there are some other features that exist already or are in IP9 beta that might be of some use to you.

First, for years ImageTransform compress/decompress can be used to compress or decompress a wave. If I recall correctly, the operation outputs data that contains the entire wave compressed, not just the wave's data.

Also, starting with IP9.00B03, there is some support for compression of HDF5 experiment files. For more information, execute:

DisplayHelpTopic "HDF5 Compression"

 

January 20, 2021 at 10:58 am - Permalink

Thank you for the info. Very interesting. However, my use case would be to have a compressed zip file of my data on disc (i.e., compressing all my ibw data with a simple zip program) and ideally get them to load into Igor by drag-and-drop as if they were uncompressed files. For example, GIMP has the ability to load / save gz-zipped files without any hiccups. Compressing ibw files upon saving would also be nice, but I guess this can be quickly done with available scripts.

January 21, 2021 at 04:05 am - Permalink

@chozo: So you want to store the data outside of Igor experiments but compressed? Why not use compressed HDF5 files?

If you want to stick with ibw you can for the loading part use an AfterFileOpenHook which unzips the file on disc using IP9s UnzipFile and then load the extracted ibw. The writing part would involve either some OS routines or a self-written XOP doing the compression. Or adjust https://www.wavemetrics.com/code-snippet/extracting-compressed-archives… for writing.

January 21, 2021 at 04:45 am - Permalink

Thank you for your linking your code example. Yes, in the end I want to look into writing my own extract code using AfterFileOpenHook for this purpose. I just thought it can't hurt to ask if there is a chance for some 'official' solution. After all, if there is an UnZip function now, it may take just some additional code to have the data extract into Igor's memory instead of the disk. 

Actually, the measurement software I use spits out ibw files, so that's where I start. I can use a zip program or even the built-in OS functionality to go from ibw to zip and back. Bu it's an extra step. So my request is mostly for increased convenience in working with files. I wondered if other users would find this useful as well. Maybe there could be a zipped file format (izw anyone?) in the future. As written in the initial post, the data I work with has an insane compression ratio and I'd rather not waste my disk space with huge chunks of zeros.

So yes, I will go for the scripted unzip-and-then-load approach for now.

January 21, 2021 at 07:23 am - Permalink

As far as I know, all of our internal routines that do the work of loading data work only on files and not streams (or memory buffers). So while what you suggest would be technically possible to implement, it would require a substantial amount of work to rearchitect the data loading code to be able to support loading from something other than a file.

It might be feasible for us to modify the LoadData operation to accept a .zip file, extract the .zip file into a temporary directory on disk, and then to act as if you had provided that temporary folder directly to LoadData.

If we made that change then you could call LoadData to load your .zip file. But would that be sufficient for your needs? Or would you want to be able to do other things with the file such as open it from the File menu, drag it into Igor, etc?

January 21, 2021 at 08:22 am - Permalink

@chozo: Is your workflow heavily depending on the files being ibw? If not I would write some IP code to convert ibw files on disc to compressed hdf5 files and continue from there.

January 21, 2021 at 09:25 am - Permalink

Thank you for all your input. I see that it is difficult to easily implement loading zip files directly into (Igor's) memory. Having LoadData accept zip would help (this could be invoked in BeforeFileOpenHook), but then the question is what happens to the extracted temporary file? This already comes very close to just have a procedure handle the extraction in the first place. So I won't waste more of your time and try to write something. Thanks again.

January 21, 2021 at 05:21 pm - Permalink