data categorization

Share icon

Hi,

I have time series of measurements (9 columns) which include wind directions. I want to categorize the measurements by wind direction, [300,360), [0, 165), [165, 180), [180, 210), [210, 220), [220, 255), [255, 300).

What I used to do is to make a new wave for each wave that meets the wind direction criteria under each category and redimension them in loop, but considering there are quite a few bins and number of concurrent measurements, this approach seems to be cumbersome. Is there an efficient way to do it?

Thank you! Your help is very very much appreciated!
It is a bit difficult to understand what exactly you want to optimize. An minimal experiment file with example data, also if possible optimally processed data(i.e., to see what you want to achieve), and an explanation of extraction criteria would really help to be able to help. ;) Just to give a shot at an answer: You could use 'Sort' to sort your data with regard to wind direction and then just split everything up at direction transitions. This way you would not need to pick the data by case. Another approach would be to use 'Extract' and use the wind direction as test parameter. This way you would bin the data but would not be able to include your wind information together easily.

June 1, 2015 at 10:42 pm - Permalink

chozo wrote:
It is a bit difficult to understand what exactly you want to optimize. An minimal experiment file with example data, also if possible optimally processed data(i.e., to see what you want to achieve), and an explanation of extraction criteria would really help to be able to help. ;) Just to give a shot at an answer: You could use 'Sort' to sort your data with regard to wind direction and then just split everything up at direction transitions. This way you would not need to pick the data by case. Another approach would be to use 'Extract' and use the wind direction as test parameter. This way you would bin the data but would not be able to include your wind information together easily.


Thank you chozo! I attached my dataset and the procedure I put together. My biggest concern is this function I have to execute multiple times to be able to process all the waves like temperature, ppfd, swrad, etc, and have to rename the waves every time they are output. If there are more categories, this becomes very inconvenient, and I have a similar task that is to categorize all the data by day and night, so I was wondering about a solution to this type of task in general.

Does anyone know a snippet that would achieve this? Thank you!
igorexchange.pxp (55.01 KB)

June 2, 2015 at 03:16 pm - Permalink

I have created a function which does (hopefully) what you want. I have attached the procedure file to this post. Load the file into your Igor session and execute CatWind(thewinddirectionwave). I have used the sort and split approach, so the data ends up in different folders for different bins, keeping the original names intact. This all may not be the most effective approach, but it works without any manual labor. In fact, there is a lot of redundant code in there, which could probably be streamlined, but I also want to keep stuff readable.
Some notes:
- there is no backup functionality and the original data will be altered. Make a copy if you want to keep the data intact (could also be added in the code if needed)
- this function will act on ALL waves in the current folder, so keep stuff which you don't want to be processed separate
- after the processing the data is still sorted according to wind direction. It is possible to revert to time sorting in an additional step, if you need that (not necessary for, e.g., plotting)
- if you want to add extra bins, just edit the third line with the numbers {165,180,210,220,255,300} (note that the last bin is always 'rest', and you cannot add bins which are greater than the highest value)

I also add the code here for everyone to look at:
Function CatWind(windwave)
    Wave windwave
    Make/FREE binlist = {165,180,210,220,255,300}   // the list of bins, i.e., 0 to 165,  165 to 180 etc. ... data above the last item will be stored in 'the rest'
    // +++ first step: sort the waves after wind direction +++
    String Datalist = wavelist("!"+nameofwave(windwave),";","")+nameofwave(windwave)    // make a list of all waves, make sure the wind data comes last!
    Variable i, pointnum
    String currentname
    for (i = 0; i < itemsinlist(Datalist); i +=1)   // for every data wave sort after wind direction (including wind direction itself)
        currentname = Stringfromlist(i,Datalist)    // extract the name of the current wave from the list
        Wave current = $currentname // create a wave reference to work with
        Sort windwave, current  // sort
    endfor
    // +++ second step: get rid of points with no valid wind data (i.e., nans) +++
    FindValue /V=(Wavemax(windwave)) windwave   // find the last point with valid data
    pointnum = V_Value
    for (i = 0; i < itemsinlist(Datalist); i +=1)
        currentname = Stringfromlist(i,Datalist)
        Wave current = $currentname
        DeletePoints pointnum+1, inf, current   // delete points after valid data
    endfor
    // +++ third step: store the data in separate folders depending on binning criteria +++ 
    Variable currbin
    for (currbin = 0; currbin < numpnts(binlist); currbin +=1)
        FindLevel/Q windwave, binlist[currbin]  // find the crossing point to the next bin in the data
        pointnum = ceil(V_LevelX)
        NewDataFolder/O $("binned_"+num2istr(binlist[currbin])) // create a data folder
        DFREF folder =root:$("binned_"+num2istr(binlist[currbin]))  // refer the data folder to work with
        for (i = 0; i < itemsinlist(Datalist); i +=1)
            currentname = Stringfromlist(i,Datalist)
            Duplicate/O $currentname, folder:$(currentname) // duplicate into data folder
            Wave current = $(currentname)
            Wave copy = folder:$(currentname)
            DeletePoints 0, pointnum, current   // delete points of current bin in main waves
            DeletePoints pointnum, inf, copy    // delete points OUTSIDE current bin in copied wave
        endfor
    endfor
    
    NewDataFolder/O $("binned_rest")    // the data above the last bin will be stored in 'rest'
    DFREF folder =root:$("binned_rest")
    for (i = 0; i < itemsinlist(Datalist); i +=1)
        currentname = Stringfromlist(i,Datalist)
        Duplicate/O $currentname, folder:$(currentname)
        Wave current = $(currentname)
        KillWaves/Z current // kill the main waves if possible (keeps stuff tidy)
    endfor
End
Categorize.ipf (2.53 KB)

June 2, 2015 at 08:39 pm - Permalink

Thank you chozo, this exactly does what I want, thank you thank you!!

June 4, 2015 at 01:50 pm - Permalink