interpolate missing data
hello,
I have a 2D matrix with a number of missing points. I'd like to fill these in using a simple interpolation--bilinear would be fine, or a spline so long as it doesn't change existing values. I don't have a deep understanding of the ImageInterpolate function, but it seems this ought to do the trick. However I've tried
ImageInterpolate/U=2 bilinear data
which appeared to do nothing to the empty data, and
Imageinterpolate/D=2 spline data
which gave an "insufficient input" error.
Probably both of these are way off base, but, a lot of playing around and searching the forum hasn't led me to an answer yet. Possibly a problem is the "missing" data points are just empty, not NaNs....? Surely there's a (more or less straightforward) way to interpolate missing points from data nearby in a 2D matrix?
Thank you!
The first question is: What do you mean by 'missing points'? When they are not NaNs what are they? Zero or some other number? I am not sure, but I do not think ImageInterpolate will help you here, since this function is meant to stretch out (or compress) images to larger/smaller dimensions and fill in the gaps in the process. What you are looking for instead is a way to just fill gaps which are already present in the first place. Do we talk about 'missing places' which are completely surrounded by useful values? Or are there more or less large patches? Is this 'missing data' always the same (i.e., easily identifiable)? The problem changes depending on the type of data missing, the type of surrounding data, and the size of the missing chunk. Assuming we are talking about single places with zero surrounded by useful data, I would think it is straight-forwards to write your own 'bilinear interpoation' loop by searching through the data for these gaps and then filling it with an averaged sum of the surrounding. Do you have some example data which you could post, if it is not too big? Maybe others have a better idea.
September 6, 2018 at 11:15 pm - Permalink
The data is, for arguments' sake, a 100 x 100 matrix with, say, 50 empty bins--empty, not zero, not NaN, just no number assigned. Mostly these occur as individual empty bins, though occasionally there are two next to each other. They are otherwise surrounded by 'good' numbers, e.g. useful values. The empty spots arise from an algorithm that turns three 1D waves--call them X, Y, and Z--into a 2D matrix of evenly spaced X and Y values that are binned into an NX x NY lattice, NX and NY being the number of bins in the X and Y axes. It's simply the case that sometimes there are no data points for a given bin; and sometimes there are multiple points in any one bin, which get averaged. Now I could reduce the values of NX and/or NY, making larger bins that have a larger chance of containing a value and ultimately I do get a completely filled matrix; but this also reduces the overall resolution. I'd rather keep higher NX and NY, and interpolate to fill in the few missing data points. In the end this is so I can take a smooth(er) derivative along the X or Y directions.
Yes, I thought about writing my own loop as you suggest, but was hoping there'd be some built-in function that might be faster/more flexible. As usual if I'd done that from the start I'd probably long since be done with this! ;) but nice to try and learn more about built-in functions too.
September 7, 2018 at 06:38 am - Permalink
If by "empty bin" you mean an empty cell in an Igor table, that is a NaN. Igor displays NaN as an empty cell in a table.
You might try Data->Packages->XYZ to Matrix
September 7, 2018 at 09:35 am - Permalink
The real issue is when you have multiple NaNs that are contiguous. Single NaNs (that have 8 non NaN neighbors) can be handled by simple bilinear interpolation. Two touching NaNs would have to be handled similarly with some care. If you have much more than that you would need to look into algorithms for textured fill.
A.G.
September 10, 2018 at 12:26 pm - Permalink