Replace NaN in data with the average of the two nearest data points

The Interpolate function can do this for you. Go to Analysis => Interpolate. Then set the Destination Points to the same size as your input. NaNs will then be replaced with the average.

Hi,

I have similar question.  I have data where every ~5 data point is missing - bad acquisition system.  I would like to have original data points and interpolated values only where NaN exists.  In using interpolate2, it also interpolates the original points which I would like to keep as original.

Is there a way within interpolate to do this or do I need to write a custom function?

Andy

Suppose your source data with NaN is srcwave. Suppose your interpolated wave is intwave. Generate a final wave fwave by duplicating the source. This line will replace the final data with interpolated points only where the values were NaN in the source wave.

Hi,

jjweimer wrote:

Suppose your source data with NaN is srcwave. Suppose your interpolated wave is intwave. Generate a final wave fwave by duplicating the source. This line will replace the final data with interpolated points only where the values were NaN in the source wave.

fwave = (numtype(srcwave[p]) == 2) ? intwave[p] ? srcwave[p]

 

Close but no "Wee Oh".  If you look at the specifics of the interpolated wave (see graph) , the original points are shifted so the interpolation has been done of modified original points.  The difference maybe small and inconsequential - until it isn't.  So before I go down your route, I would like to know if there is an option with higher fidelity.

Andy

Oops! Yes, I see the displacement. Presumably from the smoothing during interpolation.

I'm curious now whether interpolate can be done in a piece-wise manner simply to fill in gaps.

And I'm waiting to learn the objective meaning behind "wee oh" so that I can use it in my next scientific conference. :-)

May I ask why you want to fill the gaps with interpolated points? If it is just for display purposes, then why not simply switch off gaps in the graph and let Igor connect points with a line? Simply go to the Modify Traces dialog and uncheck the Gaps option.

Hi Chozo,

chozo wrote:

May I ask why you want to fill the gaps with interpolated points? If it is just for display purposes, then why not simply switch off gaps in the graph and let Igor connect points with a line? Simply go to the Modify Traces dialog and uncheck the Gaps option.

I am starting an effort with a client looking at time series data from a chemical reactor and this is sensor data and I guess they used the lowest bidder for their data acquisition system as it messes up and misses recording actually pretty frequently. For display purposes gaps are fine, but for analysis the NaNs will give no end problems.  If I just zapNaNs with wave transform, I mess up the timing information.  I can workaround that by creating a time wave but then I need to carry that around everywhere and lose the benefit of wave scaling. In looking at the data if the NaNs were more consistent, say always just a single one, I could easily envision a function to do a local interpolation, but I might need to get more elaborate since sometimes there are two in a row. At this stage I am looking to get an easily workable data set with the minimum of added distortion. I will do checks to validate, but at this stage I want to preserve the wave scaling feature in the early stages of looking at the data and finding major patterns.

Andy

hegedus wrote:

  In using interpolate2, it also interpolates the original points which I would like to keep as original.

if the number of points is the same in input and output waves, and you are not using a smoothing spline or preaveraging, there should be no offset. something is amiss in your plot. try duplicating the source wave and choose Dest X Coordinates = From Dest Wave in the interpolate dialog.

Hi,

Using Tony's advice and double checking the data first, it seems to work with interpolate2. The issue that was tripping me up was that the first point was a NaN which threw things off.  Removed it and it is working as expected.

Andy

Why not use 3-point median smoothing with the median criteria set to NaN:

From the Smooth help:

/M = threshold

Invokes running-median smoothing and specifies an absolute numeric threshold used to optionally replace "outliers". Points that differ from the central median by an amount exceeding threshold  are replaced, either with the replacement  value specified by /R, or otherwise with the median value.

Special threshold  values are:
    0:    Replace all values with running-median values or the replacement  value.
    NaN:    Replace only NaN input values with running-median values or the replacement  value.
    The smoothing factor num  is the number of points in the smoothing window used to compute each median.