HDF5_Inquire
cpr
Following on from this thread on netCDF creation it fell out that it would be great to be able to get the dimension information of a dataset in HDF5 (netCDF-4) in the same way as the NC_Inquire.
Getting dimensional/dependency information isn't currently possible to do reliably. Some operation that interrogates the "DIMENSION_LIST" attribute would get what I want. Note it is possible to use HDF5Dump to try and get the dimension info but it isn't reliable and certainly isn't quick.
I would like this to be able to treat netCDF-4 files in the same way as netCDF-3.
Following your post in the other thread, I spent several days familiarizing myself with HDF5 dimension scales, mostly relying on HDF5 Dimension Scale Specification, which is from 2005.
Dimension scales use HDF5 data types that don't map easily to Igor wave data types.
The DIMENSION_LIST attribute data type is variable-length array of references (H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT }}) and variable-length HDF5 data types map to a collection of Igor waves, not to a single Igor wave like most HDF5 data types, which is why supporting them is difficult.
It may be possible for Igor to support loading that data type. I will give it some thought. But that would only go so far toward supporting netCDF coordinate variables (implemented in netCDF-4 as HDF5 dimension scales). A comprehensive solution would require extending our netCDF XOP to netCDF-4 (it currently supports only netCDF-3) and to writing (it currently supports only reading), both of which are large projects.
For my edification, if possible, please send to WaveMetrics support a typical netCDF-4 file and, if possible, the corresponding netCDF-3 file.
February 22, 2021 at 04:43 am - Permalink
Hi Howard,
Thanks for looking into this. My Basic need is just the names of the dimensions attributable to each wave like how NC_Inquire returns S_names, as a string list. The rest I can handle pretty easily with already available in-built operations or my own functions so that all the dimension, scale etc. information stays with the loaded wave (in the note mostly).
I can certainly send a couple of netCDFs through and an example of what I'm trying to do. EDIT: emailed some netCDFs 3/4/classic/enhanced metadata and the prodecure I use to interact with them.
Writing NCs isn't such a big deal, just nice to have.
February 22, 2021 at 05:04 am - Permalink
Update:
Howard (hrodstein) has very kindly and incredibly quickly provided a solution which is rolled into the Igor 9 beta. The dimensions of netCDF-4/HDF5 files can now be easily found e.g. as a string list with something like this:
// groupID returned by HDF5OpenGroup
// WvStr is the name of the variable as a string
Function/S getHDF5Dimensions(Variable groupID, String WvStr)
// load the dimension list attribute
HDF5LoadData/O/A="DIMENSION_LIST"/TYPE=2 /N=temp_DIMENSION_LIST /Q /Z groupID, WvStr
Variable i ; String rtnStr=""
// get a list of all possible dimension list waves created by HDF5LoadData
// e.g. temp_DIMENSIONLIST,temp_DIMENSIONLIST0,temp_DIMENSIONLIST1,...
String dimWvLst = WaveList("temp_DIMENSION_LIST*",";","")
if (V_flag==0)
for (i=0;i<ItemsInList(dimWvLst);i+=1)
Wave/T temp_DimLstWv = $StringFromList(i,dimWvLst)
// add dimension name, without path, to the string list in the correct order
rtnStr+=ParseFilePath(0,temp_DimLstWV[0],"/",1,0)+";"
KillWaves temp_DimLstWv
endfor
else
rtnStr=WvStr // if a variable has no dimensions it's because it is a dimension
endif
return rtnStr // list of dimensions e.g. dimStr0;dimStr1;dimStr2;dimStr3;
end
You can then pass the list of dimensions to a function that sets the DimLabels of the wave, or/and as I do write them to the WaveNote in CDL syntax along with all the other netCDF attributes like _fillValue etc so that they stay with the wave. The NC_Inquire operation provides analogous functionality for netCDF-3 files with its S_names output.
This allows you to keep netCDF metadata with the loaded wave which is useful if you want to write that wave out again, or for automatically making plots without knowing what the X,Y,Z waves are in advance as an example.
February 23, 2021 at 10:20 am - Permalink