Native netCDF creation

netCDF-4 is based on HDF5. So in principle this already works as you can create HDF5 files from scratch in IP. For netCDF-3 only read support is currently there.

There is an example of how to create netCDF-3 files at:

https://www.wavemetrics.com/project/NCGEN

But it does require the installation of other tools (noted at the top of the procedure window) depending on which platform you are using.

thomas_braun

The HDF-5 tools in Igor will not allow you to make a CF compliant netCDF-4.

kaikin

This tool from my colleague I know all about and have my own modified version to support the complex file structure and extra primitive data types available in netCDF-4 (which people are welcome to FYI). It was the author himself lamenting the lack of proper netCDF support that prompted me to post - I was twisting his arm about updating a 32-bit XOP to 64-bit - he was asking why he would move from IP6.37 for no change in useability in his circumstances. Especially with a 30 seat license and the inevitable "why aren't you doing it 'in python?", it's free! 

Forgive me, I might be reading your user name completely wrong, but in my head it parses to a name I recognise from a certain series of GeneralMacros out of NOAA I've been using versions of since the dark ages. If so I probably owe you!

A native tool would be a great addition to IP I think, even if currently if isn't holding me back personally. 

@cpr

> The HDF-5 tools in Igor will not allow you to make a CF compliant netCDF-4.

Do you know what is missing in IP for being able to write netCDF-4 files as plain HDF5 files?

You are also speaking about an XOP. Is that available somewhere?

thomas_braun

I don't recall the specific issue, and this might have changed since IP6. What I should have written is:

>"The HDF-5 tools in Igor did not seem to allow me to make a CF compliant netCDF-4." 

The XOP I was referring to isn't really related to netCDF, other than being used to generate some data that is output as netCDF. It isn't openly available and is unlikely to be of interest to anyone who doesn't also have an atmospheric research aircraft, or perhaps a cloud observatory on a mountain top. 

Agreed, a native tool would be very useful and may help get Igor wider recognition and use since netCDF files are more easily dealt with in other code as you mention.

Yes, you are correct about the reference to General Macros code from NOAA! Glad you've found it to be useful.

I (re-)discovered the limitation with the Igor HDF-5 writing tools which do not permit direct creation of netCDF-4 (~ish) files.

There is no way to output the dimensional dependencies that netCDF has. Presumably this is netCDF specific and implemented as an extension to HDF5 in netCDF-4. Everything else seems possible, albeit you must be very explicit in writing internal attributes compared with using ncgen which handles this i.e. _Netcdf4Dimid, _Netcdf4Coordinates, _nc3_strict, _NCProperties. 

Maybe the ability to write the dimensional dependencies could be added? As an extra flag to HDF5SaveData perhaps much like global and group attributes are handled? Please? 

For reference if you create a HDF-5 file with all the requisite netCDF attributes but without the dimensional dependencies and use nccopy to convert it to a netCDF-4 a whole menagerie of dummy dimensions are created; one for each variables' dimension. With just the ability to deal with netCDF-4 properly you can easily convert between nc3 and nc4 or hdf5 with nccopy if required and avoid having to load whole files into memory and write them back out again.  

> Presumably this is netCDF specific and implemented as an extension to HDF5 in netCDF-4.

Do you mean that netCDF-4 files are HDF5 files with some extra stuff? That would really suprise me.

Can you give an example of a netCDF-4 file which you can not create with the HDF5 tools in IP?

>Do you mean that netCDF-4 files are HDF5 files with some extra stuff? 

That's precisely what it is, see:

https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_introduction.h…

Edit: This rather comically written note on netCDF-4 / HDF-5 interoperability sums it up nicely:

https://www.unidata.ucar.edu/software/netcdf/docs/interoperability_hdf5…

Thanks for the pointers.

I've also found https://www.unidata.ucar.edu/blogs/developer/en/entry/dimensions_scales which is a 4 part series which explains the details about HDF5 wrt netCDF4 dimension scales.

Playing around with the attached cdl files converted to netcdf4 and loading them into IP reveals that it uses quite complicated HDF5 data types like compounds with object references in them. And I don't think you can write those from within IP. There is also no support for the HDF5 dimension scales in IP which avoids requiring you to manage these low level data types.

But this is just me guessing, I propose to reach out to WM support.

I still think though that netCDF4 files are 100% standard HDF5.

@thomas_braun

Yes, it's the dimension scales that are the sticking point. I think the datatypes that igor can't produce are not allowed in netCDF anyway, so that solves that problem. Writing netCDFs is possible indirectly via cdl as you have done, and I do this regularly. In that case ncgen handles the niceties of dimensionality. As dimension scales are a part of HDF5 it would be nice to be able to use them in Igor and by extension be able to make netCDF-4 files directly, and also to modify then directly without having to load the whole thing into memory. 

Reading netCDFs isn't a problem FYI, I haven't found one that I couldn't nicely parse using some combination of HDF5.xop, netCDF.xop or nc_load.xop.

> Reading netCDFs isn't a problem FYI, I haven't found one that I couldn't nicely parse using some combination of HDF5.xop, netCDF.xop or nc_load.xop.

But how do you read the dimension scales? The HDF5 xop AFAIU is not able to read them.

> The HDF5 xop AFAIU is not able to read them.

Much to my chagrin... you could parse them out of the HDF5dump it has just (re-)occurred to me,  I had some horrible kluge to get all the metadata this way but retired it - maybe I'll look again.

EDIT: get dimensions with HDF5dump is slooooow.

If the netCDF was made with a netCDF API of some description there will be hidden/packed internal attributes _Netcdf4Dimid and _Netcdf4Coordinates which contain all the information you need to get the dimensions. Spat out of python code though, not so much.

The old nc_load.xop doesn't return enough info to get the dimensions if you have more than 1! That was written in 1999 though and I don't use it any longer but have written a basic parser that will use it if you're using Igor <8 for users of my data.  

Something like this hastily hacked together example:

Igor can interpret the dimensions from HDF5 in some way as it dumps them in the HDF5 browser, but no tool is provided to load them directly and it isn't clear why - something about the datatype being VLEN, though I stopped reading at the point in the docs where it said it wasn't possible.

This is way off track now....   

Update:

hrodstein has added a HDF5DimensionScale operation to the Igor Pro 9 beta which allows you to query and write HDF5 dimension scales. This by extension will allow you to make fully fledged netCDF-4 files directly from Igor 9 now easily.

Many thanks to Howard.  

EDIT: below is a simple example that creates a file with most of the netCDF-4 features utilised.