Out of memory error during Save operation

Share icon

I'm getting an out of memory error when saving a text wave to disk using the Save operation. The text wave is 1747 rows by 64 columns; the data browser shows it as having 701, 811 bytes. Despite the error, most of the file is saved; the saved file has 1678 rows and 64 columns and is 752 KB in size. The OS is 32 bit Win7 and Igor version 6.22A with Asylum MFP3D version 101010 loaded on top of Igor.

This occurs for the following, executed from the Save Delimited Text dialog
Save/J/U={0,0,1,0} wRunLog0 as "wRunLog0.txt"

Or the following, executed from a function:
Save/J/U={0,0,1,0}/O wWaveInTable as S_fileName

However, if I load a second instance of Igor, version 6.32A, the entire file is saved successfully, with no error. The complete file on disk is 796 KB in size.

A quick search of IgorExchange doesn't really show any problems of this type and a review of the Change File for updates since 6.30 (WaveMetrics web site) doesn't show any changes related to Save.

I will try setting up Igor 6.32A with the 3rd party software, but thought I would send this note while I work on that.

Any ideas on other things to try or explanations would be appreciated.
I can't reproduce this using Igor Pro 6.22A on Windows 7 with the Asylum code loaded but no user waves created. Here is my test function:
Function Test()
    Make/O/T/N=(1747,64) textWave0 = "Row=" + num2istr(p) + ", column=" + num2istr(q)
    Variable column
    for(column=0; column<64; column+=1)
        String dimLabel
        sprintf dimLabel, "Column%d", column
        SetDimLabel 1,column,$dimLabel,textWave0
    endfor 
    Save /O/J/U={0,0,1,0} textWave0 as "textWave0.txt"
End


This creates a roughly 2MB file. The function runs without error.

If you come up with a recipe to reproduce the error please let me know.

It's conceivable that Igor is actually running out of memory because you are right on the edge but it seems unlikely. If that is the problem then killing some waves would allow it to run without error.

August 1, 2013 at 06:07 pm - Permalink

Howard,

Thanks for looking into this. It sounds like there are no known problems with Save, that is very helpful to know. I will try a few things and see if I can create a recipe, otherwise, I can send my experiment. I did try rebooting the computer and, after loading the same experiment, the problem persisted.

Does it make sense to have the Igor 6.22A/Asylum combination in memory along with just Igor 6.32A in memory at the same time and have the former report the error while the latter saves the same file without any problem?

August 1, 2013 at 06:51 pm - Permalink

Quote:
Does it make sense to have the Igor 6.22A/Asylum combination in memory along with just Igor 6.32A in memory at the same time and have the former report the error while the latter saves the same file without any problem?


They each have their own virtual memory space (4GB on a 64-bit OS) so one could be out of memory without affecting the other.

Since this operation does not require a lot of memory it seems unlikely that the 6.22A is running out of memory but it is possible and that is the most straight-forward explanation. Another possibility is an uninitialized variable bug. My guess is that the cause is the third possibility: something I haven't though of :)

August 1, 2013 at 07:01 pm - Permalink

Your test ran without problem in the same experiment where my log file encountered the out of memory error during the Save operation. Therefore, it seems unlikely that the size of my log file is the problem. In fact, my log file is smaller than your test file.

After some testing, I believe that I have isolated the problem to a few elements in my log wave that contain many (~1800) values of 0 (zero) following actual text characters. If these few elements have their values cleared or replaced with just the text, then the file is saved without encountering an error from Igor 6.22A. (This is the experiment with I6.22A and Asylum 101010 software loaded.) As an example, strlen shows one of these elements to be 1854 characters long, while printing the element to the history window prints only 47 characters. I can send my file to you, but this may be enough for you to reproduce the problem.

Some additional testing showed that Igor 6.32A by itself saves the problem elements in delemited text format with the 0's intact. (This was determined by examining the delimited text file in a hex editor.) However, both 6.22A and 6.32A will load the file saved by 6.32A and will remove the 0's.

The data in my log file is pulled from info that the Asylum code saves with experiment results. The problem elements come from the wave note saved with this data. If the data file is viewed in the Igor Data Browser, then the wave note displayed in the Data Browser will stop displaying at the text item where I've run into the Save error. (Hope that makes sense.) Lastly, this seems to be a problem with just a few data sets; I've successfully loaded several thousand and only encountered problems with 3 consecutive files.

August 2, 2013 at 08:11 am - Permalink

Bytes with 0 in text cause problems whenever we treat the text as a C string since 0 means "end of string" in a C string. This will occur, for example, when printing a text value containing a 0 byte.

I don't recall any changes that explain the difference between 6.32A and 6.22A.

Please save the wave to an Igor binary file (Data->Save Waves->Save Igor Binary) and post the file here or send it to WaveMetrics support.

Thanks.

August 2, 2013 at 10:02 am - Permalink

hrodstein wrote:
I don't recall any changes that explain the difference between 6.32A and 6.22A.

Please save the wave to an Igor binary file (Data->Save Waves->Save Igor Binary) and post the file here or send it to WaveMetrics support.


I have sent an abbreviated version of the problem wave as an .ibw to Wavemetrics support.

I did just try saving the problem wave from a copy of Igor 6.22A without the Asylum add on; the out of memory error still occurs. As far as I can tell with a quick test, the problem is the same.

August 2, 2013 at 10:58 am - Permalink

I traced this to a bug that was fixed on 2013-01-15 in a utility routine that is called during handling of text. It incorrectly returned out-of-memory under some circumstances if a handle containing text ended with 0.

This fix was first published in 6.30B03 in February.

You can sidestep the problem in 6.22A by adding:
        /E=0
to your Save command. See the documentation for Save for an explanation of what this does.

August 2, 2013 at 11:48 am - Permalink

Much thanks. This will work until I have a chance to update my software.

August 2, 2013 at 08:57 pm - Permalink

The following check was added to the function responsible for pulling the offending strings out of a wave note:

    vPos = strsearch( sString, num2char( 0 ), 0 )
    if( vPos > -1 )
        sString = sString[ 0, vPos - 1 ]
    endif


If a null character (numeric value of 0) is found in the string only the characters to the left of the first null are exported to the output string.

August 7, 2013 at 05:15 am - Permalink