Is the size of an Igor binary wave file format limited to 2GB in Igor Pro v8.04?

The .ibw format is limited to 2GB. Igor can not write waves with data >= 2GB to a .ibw file.

You can save larger in a packed experiment (.pxp) file. If the wave data is < 2GB, Igor writes it in a .ibw format record in the packed experiment file. If the wave data is >= 2GB, Igor writes it in a special 64-bit format record in the packed experiment file.

You can write a wave or data folder to a .pxp file using SaveData operation. You can also do it using the Data Browser - select a wave or waves and click Save Copy. Igor writes the wave or waves to the .pxp file in .ibw format or in 64-bit format depending on the size of the wave.

You can write waves or data folders to an HDF5 file using the HDF5SaveData and HDF5SaveGroup operations.

In Igor Pro 9, you can write an entire experiment to an HDF5 file using File->Save Experiment or the SaveExperiment operation.

Hello hrodstein,
Thank you for clarifying. I don't think writing to .pxp files will be viable for my use case, I am more concerned with writing to individual wave files independent of experiments. Also, since I am creating waves externally in c++, I am not utilizing the data browser or SaveData operations.

Writing to HDF5 files seems like the most viable option in my case; I'll look further into writing waves to that filetype. Will the HDF5 package be supported in Igor long-term? 

Quote:

I am creating waves externally in c++

The easiest way to export wave data to Igor from C++ is to write plain binary data files containing the wave data only and load that into Igor using GBLoadWave.

If you need to include wave properties, such as units, you could write to a plain binary data file and then create one Igor Text file that loads the binary data files and sets the properties. One Igor Text file could contain commands to load any number of plain binary data files.

mkuhler wrote:

Will the HDF5 package be supported in Igor long-term?

That is my expectation.

Writing HDF5 files means you need to learn the basics of HDF5 and link with an HDF5 library. This will take some effort. It is worthwhile if you have a general need for a flexible and open data format, but it will be more work than writing plain binary data files.

Another option for you, since you apparently are already able to write .ibw files, is to write a single large wave to multiple .ibw files. You could then use an Igor Text file to put the pieces back together.

If you plan to use HDF5 in C++ I would suggest to not use the "official" HDF5 C++ API as this is not a nice API. https://github.com/ess-dmsc/h5cpp is nicer to use.

Just out of curiosity: Is your writing code based on the example code from "TN003 Igor binary format"? Can you attach a zipped ibw file which you created from C++ with more than 2GB? If you only add zeros as data it will compress well.

Since I maintain some tools that write Igor binary wave files, I'm curious, too.

The main problem I see is that the wfmSize field of the BinHeader5 structure is a signed 32bit integer, making it virtually impossible to store the length for data >2GB in this field. Trying to do that might even result in a negative value.

I guess Igor would report an error when trying to load a wave with wfmSize<0, wouldn't it?

Quote:

I guess Igor would report an error when trying to load a wave with wfmSize<0, wouldn't it?

I'm not sure what error it would report.

If you are writing Igor binary wave files, I recommend testing for size>2GB and reporting an error on writing the file.

> I'm not sure what error it would report.

How about something like "invalid IBW format". I would guess the routine for reading IBW files already has to do some sanity checks.

hrodstein wrote:

Quote:

I guess Igor would report an error when trying to load a wave with wfmSize<0, wouldn't it?

I'm not sure what error it would report.

If you are writing Igor binary wave files, I recommend testing for size>2GB and reporting an error on writing the file.

Out of curiosity I tested this. There was no error reported on loading and in Igor the wave looked very much like what mkuhler saw (lots of zeros at the end of the wave).

@christian.halaszovich

Can you attach a compressed ibw file showing the problem?

thomas_braun wrote:

@christian.halaszovich

Can you attach a compressed ibw file showing the problem?

Yes, no problem.

I attached a zip archive containing two ibw files (created with the tool PMbrowser [https://github.com/ChrisHal/PMbrowser] from a HEKA PatchMaster file). The waves are not big (around 29 KiB each).

The difference between both files is that in the file "testNegSize_1_2_5_Imon.ibw" the field wfmSize in the BinHeader5 structure has the sign reversed, which confuses Igor. (Trying to store the size of a wave >=2GiB will also result in a negative value for this field (and possibly other fields, too).)

Note that also the wave-note ends up as garbage if the corrupted file is read. Since the wave note is stored right after the wave data this is probably not surprising (I tested this with IgorPro 8.04 64 bit.)

P.S.

We might be going slightly of topic for this thread. I apologize for that.