Writing to notebook crash

Have you considered writing to a text string or text wave instead of a notebook?

There is a limit of about 16 million paragraphs where "paragraph" means text plus carriage-return. You may be hitting this but only if your lines are short on average - e.g., 10 bytes or so.

What does the typical line of text look like?

I would not write 150MB to a notebook. I would write it to a text file using Open, fprintf, and Close. You could also accumulate the text in a string variable and write the variable to a file at the end of the process.

If you can create a simplified self-contained example that reproduces the crash, I will investigate.

Thanks jjwiemer, hrostein

I'm not hitting the paragraph limit it seems then - I write out numerical data in 4096 number blocks - typically those numbers are floating point but could be byte or unsigned. 

I've just modified my code based on jjwiemers' suggestion but haven't thought of a good way to get the 16446330 character test string into a text file easily (suggestions welcome of course).

hrodstein - I changed my code originally from Open, fprintf, Close method to writing to memory as this gave me ~10x improvement in performance e.g. writing a turbulence data netCDF of ~110mb takes 1 minute instead of 10 - however I didn't work too hard on removing the inefficiency in the fprintf method as this also has the occasional (rare) habit of corrupting while being written to. 

How simplified would be acceptable - there is necessarily quite a bit of data and a lot going on before the CDL is written but to run the code is a single button click.      

cpr wrote:

How simplified would be acceptable - there is necessarily quite a bit of data and a lot going on before the CDL is written but to run the code is a single button click.

If I can reproduce the crash then I can investigate whether you can simplify or not.

If possible, remove extraneous stuff.

Thanks hrodstein

Does WM have an FTP drop site I can use - the .pxp is necessarily large (~350mb) to induce a crash even with everything extraneous removed. Otherwise I can post the code (though that is also long) and you can - I'm sure - create your own junk waves. The part of the loop that is writing the actual data - rather than metadata is attached. 

Edit: In my case I have ~100 waves (single precision or less) all 27470 points long. Maybe 10% of those ar 2D 30 or 64 points wide. To give you an idea about the number of actaul points. 

>Does WM have an FTP drop site I can use

Yes. Send an email to WaveMetrics support and we will send you FTP upload information.

The best way to send an email to WaveMetrics support in Igor7 and Igor8 is to choose Help->Contact Support. That sends us information about your system that we sometimes need.

@cpr: What type of netCDF files are you creating? HDF5 based ones or classic? Do you know about wfprintf? This allows you to write a wave straight to file without having to manually handle the looping. Maybe that would also help in making the code faster.

@thomas_braun: both netCDF-3 and -4 are handled with a correctly formatted CDL file and the correct argument for ncgen. I produce both - with mostly the same data. netCDF-3 for my oldskool users and their ancient IDL/Fortran code, netCDF-4 for everyone else.   

I've actually tried your suggestion of writing directly to file with wfprintf, but this only marginally improved things - I guess because the wave isn't written all at once, rather the buffering is handled 'behind the scenes' as it were perhaps. That's how I ended up with using wfprintf to write the file into memory which works a treat until I've reached some file size limitation above, say, 150mb for a notebook which induces a hang or crash. 

I received a test experiment from cpr. Here is what I found.

The crash is due to a limit that I probably knew in 1989 when I wrote the initial code but forgot long ago. There is a limit of about 268 million bytes of modified notebook data (data in memory not yet written to disk).

(At the time I wrote the code, my machine had 8MB of RAM, a 40MB hard drive, and a CPU running at 16MHz. According to Wikipedia, it cost the equivalent of $12,373 in 2019 dollars.)

I have changed Igor Pro 8.05 to detect this situation and return an error rather than crash. We are not able to make changes to Igor6 using today's operating systems and development systems.

I then rewrote cpr's procedure using a string variable to collect output instead of a notebook. This worked without crashing. It produces a file consisting of 311,130,745 bytes in 32,527 lines.

It runs reasonably quickly in Igor6/7/8 *except* for Igor6 on Windows where, for technical reasons that we mostly understand, it is extremely slow for this amount of data.

It runs quickly in Igor9 on both Macintosh and Windows because of improvements we have made. (We expect beta testing of Igor9 to start some time in the next couple of months.)

Many thanks to Howard

My switch to Igor 8 has proved worthwhile with minimal tweaks needed and of course with your help. I'm excited to see what Igor 9 can has to offer - or if I can find new and interesting ways to break it! :-)