Load several data files

Share icon

Hi all,

I have some data with irregular format. What I'm trying to do is writing some functions to read only selected files into separate data folders. I put an example below and this format repeats itself in every file. Attached you can see an example data file. I have so far:

- First row is the Date/Time
- Second, third and fourth rows are data points for three main groups (each one has 10 parameters). My final wave will have the following dimensions (Rows = X; Columns = 10; Layers = 3)
- Fifth row has different number of columns therefore I thought they can be saved in a 2D wave (Rows = X; Columns = 8)
- Sixth row has also different number of columns than other five rows. So again a 2D wave (Rows = X; Columns = 9)

Example Data:
-----------------------

T,2011,06,09,10,26,11
B,319927,13990,111,689,162065,6511,113,690,1003.6,303.2
G,572177,12878,240,689,295490,4997,241,690,1003.6,303.2
R,699138,21745,1437,689,366300,10369,1350,690,1003.6,303.2
D,BBXX,234,2.890e-5,1.771e-5,1.705e-5,1.291e-5,6.409e-6,6.479e-6
Y,168092,1003.6,303.2,298.2,62.6,9.3,7.9,0,0000
T,
B,
G,
R,
D,
Y,
...

I thought I may write two functions. One to read data files in order and save the data in a separate data folder. A second function to do the main job and read/load individual data points. 

 Function LoadSelectedNephFiles()
    Variable refNum
    String message = "Select one or more .dat files"
    String outputPaths
    String fileFilters = "Data Files (*.dat):.dat;"
    SVAR workdir = root:workdir
    fileFilters += "All Files:.*;"
 
    Open /D /R /MULT=1 /F=fileFilters /M=message refNum
    outputPaths = S_fileName
    
    if (strlen(outputPaths) == 0)
        Print "Cancelled"
        return -1
    endif
    
    Variable numFilesSelected = ItemsInList(outputPaths, "\r")
    NVAR numberofAnalFiles = root:numberofAnalFiles
    numberofAnalFiles = numFilesSelected
    Variable i
    for(i=0; i<numFilesSelected; i+=1)
        String path = StringFromList(i, outputPaths, "\r")
        // Printf "%d: %s\r", i, path               // For debugging only
        Variable err = LoadNephFile("", path)       // It is OK to pass "" for pathName because we are passing a full path for fileName
        if (err != 0)
            return err
        endif
    endfor
    
    return 0
                // Success
End


Second function is still not clear to me and at this point I would appreciate any helps/suggestions. Some initial thoughts:

- Can I use LoadWave function? If yes, how should I design it work with this non-ideal data structure?
- How should I deal with the first block and put that in a loop?

Cheers,
Emre NL110609.zip (16.57 KB)
Quote:
How should I deal with the first block and put that in a loop?


I think you mean "How can I loop through all of the files in a folder". Use IndexedFile. For and example, see http://www.igorexchange.com/node/5843

Quote:
Can I use LoadWave function?


LoadWave can only load a regular block of numbers. You can tell LoadWave to load a block of lines of text using the /L flag. So you can tell LoadWave to load lines 1, 2 and 3 (zero-based). You can then parcel the data into your 3D output wave and repeat for the next block of RGB lines. To do this you need to be able to calculate the line numbers in each RGB block. If the files are regular, that is doable. However you also need to know when to stop. For this you need to know the total number of lines in the file, which can note be determined except by examining the file line-by-line.

Because of this last issue, I think I would dispense with LoadWave and use FReadLine instead. There is an example at http://www.igorexchange.com/node/880. You would then use sscanf to parse a given line and store the values in the appropriate place in the output wave.

March 31, 2015 at 10:16 am - Permalink

Thanks for the comment. I think I did exactly what you suggested. This function can extract the Date/Time for each parcel. I am now thinking of applying the same for RGB data.

 Function LoadNephFile(pathName, fileName)
    String pathName             // Igor symbolic path name or "" for dialog.
    String fileName             // file name, partial path and file name, or full path or "" for dialog.
    
    Variable err
 
    // This puts up a dialog if the pathName and fileName do not specify a file.
    String message = "Select a data file"
    Variable refNum
    String fileFilters = "Data Files (*.dat):.dat;"
    Open /R /Z=2 /P=$pathName /F=fileFilters /M=message refNum as fileName
    
    // Save outputs from Open in a safe place.
    err = V_Flag
    String fullPath = S_fileName
 
    SVAR workdir = root:workdir
    workdir = S_fileName
    if (err != 0)
        return err          // -1 means user canceled.
    endif
    
    String fileNameFromFile     // File name as read from the first line of the file
    
    // Create data folder
    String dfName = ParseFilePath(3, fullPath, ":", 0, 0)   // Get file name without extension
    dfName = CleanupName(dfName, 0)
    NewDataFolder /O /S $dfName
    
    String targetText
    String format
    Variable year, month, day, hour, minute, second
    Variable foundDateTime
    Make/D/O/N=0 TimeWave
    Variable thRow = 0
    
    do
        FReadLine refNum, targetText
        if(stringmatch(targetText,"T,*"))
            format = "T,%d,%d,%d,%d,%d,%d"      
            sscanf targetText, format, year, month, day, hour, minute, second
            foundDateTime = Date2Secs(year,month,day)       // Convert year. month and day into seconds
            foundDateTime += second + 60*minute + 60*60*hour
            InsertPoints thRow, 1, TimeWave
            TimeWave[thRow] = foundDateTime
            thRow += 1
        endif   
    
    while(strlen(targetText)!=0)
    
    SetScale d, 0, 0, "dat", TimeWave
    Close refNum
    
    SetDataFolder ::
    return 0
End


Cheers,
Emre

March 31, 2015 at 10:28 am - Permalink