Binning time data
tooprock
I have a dataset which contains information for several parameters like "fluorescence intensity", "particle size", etc. in time. However, the time values are not uniformly distributed.
17.10.2012 17:23:53
17.10.2012 17:23:54
17.10.2012 17:23:55
17.10.2012 17:23:55
17.10.2012 17:23:56
17.10.2012 17:23:57
17.10.2012 17:23:57
17.10.2012 17:33:01
17.10.2012 17:33:02
My goal is defining a function which lets the user enter the time size (in minutes) and averages the data. Since there are gaps in the data, I need to find those gaps and add missing time values (of course there will be no data point for those lines). Would you suggest converting the data in 2D form because there are parameters connected to the time values. If I avarage the time values, I need to average also the corresponding data values. If we assume that we average for every 5 minutes, I expect to end up with something like that:
17.10.2012 17:23:00
17.10.2012 17:28:00
17.10.2010 17:33:00
What I managed to write is here: Do you have any suggestions?
Function BinWIBSData(timeinterval) Variable timeinterval // timeinterval will be assigned by the user from the front panel "Set Time Resolution" Variable timeincrease = timeinterval * 60 // Convert the value given by the user to seconds (user gives always values in minutes) Variable startTime, endTime, numofLines, sizeoftimewave Duplicate/O DateTimeWave, TimeWave // Protect the original wave and work on the copy startTime = ceil(Timewave[0]) // startTime is the time where we start binning endTime = ceil(Wavemax(TimeWave)) // endTime is the time where we stop binning numofLines = ceil((endTime - startTime) / timeincrease) + 1 // numofLines is required to generate an empty cell to save binned time values Make/D/O/N=(numoflines-1), BinTimeWave SetScale d, 0, 0, "dat", BintimeWave // We make the destination wave. Destination wave will contain too many data points (missing data or gap between data points) //print BinTimeWave Variable k = 1 Variable l = 0 Variable n BinTimeWave[0] = startTime sizeoftimewave = numpnts(TimeWave) for(k=1; k<= sizeoftimewave-1;) if(TimeWave[k] - startTime == timeincrease) // If the difference between two lines equals to the timeincrease, it means that we found the latest member of this bin. Increase l and save this in Bintimewave l += 1 BinTimeWave[l] = TimeWave[k] startTime = BinTimeWave[l] //print BinTimeWave elseif(TimeWave[k] - TimeWave[k-1] > timeincrease) // If the difference between two lines is larger than the given timeincrease, it means that there is a gap. We must find out how big this gap and include missing // data points. Variable nummislines = ceil((TimeWave[k] - TimeWave[k-1]) / timeincrease) // Calculate the number of missing lines between two data points. Make/D/O/N = (nummislines+1), temp_wave1 // Make a new temporary wave to generate and store the missing date values SetScale d, 0, 0, "dat", temp_wave1 temp_wave1[0] = ceil(TimeWave[k-1]) // Calculate missing points. for(n=1; n<= nummislines; n+= 1) temp_wave1[n] = temp_wave1[n-1] + timeincrease endfor BinTimeWave[l,] = temp_wave1[p-l] // Move missing data points to the BinTimeWave k += 1 else k += 1 // If difference between two lines is smaller than the given timeincrease go on searching endif endfor return 0 End
I work with time-stamped data with many (small) gaps. I first use a routine I wrote called FillGaps,
which sets all missing datapoints to NaN, then makes it a timescaled wave.
Next I use AverageScaled, which allows me to average with any interval.
Available on request.
FillGaps (NominalDeltaTime, TimeWave, WaveList)
AverageScaled (WaveToAverage, Number0fPoints, Point_Number_to_Start_With, OutputWave, MinimumNumberOfPointsInAn_Average)
Roger Pyle
March 6, 2013 at 05:03 pm - Permalink
It sounds great. Can I have this routine? I am not sure if it will work for my dataset. It would be great to test it.
Cheers,
Emre
March 7, 2013 at 01:37 am - Permalink
convert a wave and its associated time wave to a time-scaled wave, with gaps indicated as NaNs.
average a time-scaled wave (ignoring NaNs).
March 7, 2013 at 07:05 am - Permalink
Cheers,
Emre
March 7, 2013 at 07:34 am - Permalink