Loading multiple files and calculating average hourly

Hi all,
I am new to IGOR. I just started to learn Igor instead of Origin.

I have data (tons!!) of 1 sec concentration of chemical species (CO2 and CH4) over a couple of year.. And I would like to calculate mean and standart deviation for each hour over the length of my data.

I have tons of files (*dat), but I can't load all using Igor. I tried the function LoadAndGraphAll as suggested in manual, but I failed. I think that the problem is because I have a lot of rows and columns!! There are a lot of columns, but I need load just DATE TIME CH4_dry CO2_dry

Any help would be greatly appreciated as I am struggling with this.

CO2example.txt (462.16 KB)
Quote:
I am new to IGOR.

If you have not already done it, do the guided tour. It is essential. Choose Help->Getting Started.

Your file is a FORTRAN-style fixed field file with 23 columns in fields of 26 characters each. Therefore we need to use LoadWave/F.

After you do the guided tour you will need to learn about Igor data folders. Execute:
DisplayHelpTopic "Data Folders"


I have pasted below an Igor procedure that will load your file and another that will load all .txt files in a given folder on disk into Igor data folders. Execute this to learn about Igor procedure files:

DisplayHelpTopic "Procedure Windows"


You should paste the procedure text below into an Igor procedure file and save the file in your "Igor Procedures" folder. The "Igor Procedures" folder is explained in the help for procedure windows.

The procedures add items to the Load Waves submenu of the Data menu.

#pragma rtGlobals=3     // Use modern global access method and strict wave access.

Menu "Load Waves"
    "Load Nogueira File...", LoadNogueiraFile("", "")
    "Load All Nogueira Files in Folder...", LoadAllNogueiraFilesInFolder("")
End

Function LoadNogueiraFile(pathName, fileName)
    String pathName     // Name of an Igor symbolic path or "".
    String fileName         // Name of file or full path to file.
 
    // First get a valid reference to a file.
    if ((strlen(pathName)==0) || (strlen(fileName)==0))
        // Display dialog looking for file.
        Variable refNum
        Open/D/R/F="*.txt"/P=$pathName refNum as fileName
        fileName = S_fileName           // S_fileName is set by Open/D
        if (strlen(fileName) == 0)      // User cancelled?
            return -1
        endif
    endif
 
    // Now load the data.
    // The file uses FORTRAN fixed-field style with 23 columns in fields of 26 characters each.
    // We are interested in the DATE, TIME, CH4_dry, and CO2_dry columns only.

    // "Date" and "Time" are not available as wave names because they are Igor function names
    String columnInfoStr = " "
    columnInfoStr += "N='DateTimeW',F=6;"       // Load DATE column - will become date/time wave
    columnInfoStr += "N='TimeW',F=7;"           // Load TIME column
    columnInfoStr += "N='_skip_';"              // Skip FRAC_DAYS_SINCE_JAN1
    columnInfoStr += "N='_skip_';"              // Skip FRAC_HRS_SINCE_JAN1
    columnInfoStr += "N='_skip_';"              // Skip EPOCH_TIME
    columnInfoStr += "N='_skip_';"              // Skip ALARM_STATUS
    columnInfoStr += "N='_skip_';"              // Skip AmbientPressure
    columnInfoStr += "N='_skip_';"              // Skip CH4
    columnInfoStr += "N='CH4_dry';"         // Load CH4_dry
    columnInfoStr += "N='_skip_';"              // Skip CO2
    columnInfoStr += "N='CO2_dry';"         // Load CO2_dry
    columnInfoStr += "N='_skip_';"              // Skip CavityPressure
    columnInfoStr += "N='_skip_';"              // Skip CavityTemp
    columnInfoStr += "N='_skip_';"              // Skip DasTemp
    columnInfoStr += "N='_skip_';"              // Skip EtalonTemp
    columnInfoStr += "N='_skip_';"              // Skip H2O
    columnInfoStr += "N='_skip_';"              // Skip MPVPosition
    columnInfoStr += "N='_skip_';"              // Skip OutletValve
    columnInfoStr += "N='_skip_';"              // Skip WarmBoxTemp
    columnInfoStr += "N='_skip_';"              // Skip cavity_pressure
    columnInfoStr += "N='_skip_';"              // Skip cavity_temperature
    columnInfoStr += "N='_skip_';"              // Skip solenoid_valves
    columnInfoStr += "N='_skip_';"              // Skip species
    columnInfoStr += "N='_skip_';"              // Skip time
    LoadWave /F={23,26,0} /L={0,1,0,0,0} /D /K=0 /A /B=columnInfoStr /Q /R={English, 1, 2, 1, 1, "Year-Month-DayofMonth", 40} /P=$pathName fileName
    Variable numWavesLoaded = V_flag            // V_flag is set by LoadWave
    if (numWavesLoaded != 4)
        Print "Error loading file - wrong number of waves loaded"
        return -1
    endif
 
    Wave DateTimeW,TimeW            // Create reference to waves created by LoadWave
    DateTimeW += TimeW              // Add time to date to obtain date/time
    KillWaves/Z TimeW                   // This is no longer needed

    return 0                            // Success
End

Function LoadAllNogueiraFilesInFolder(pathName) // Loads all .txt files in specified folder
    String pathName                 // Name of symbolic path or "" to get dialog
    String fileName
    Variable index=0

    if (strlen(pathName)==0)            // If no path specified, create one
        NewPath/O temporaryPath     // This will put up a dialog
        if (V_flag != 0)
            return -1                   // User cancelled
        endif
        pathName = "temporaryPath"
    endif

    Variable result
    do          // Loop through each file in folder
        fileName = IndexedFile($pathName, index, ".txt")
        if (strlen(fileName) == 0)          // No more files?
            break                                   // Break out of loop
        endif

        // Create data folder from file name
        String dfName = ParseFilePath(0, fileName, ":", 1, 0)
        dfName = RemoveEnding(dfName, ".txt")  
        if (DataFolderExists(dfName))
            String prompt
            sprintf prompt, "Data folder named '%s' exists. Click Yes to overwrite the existing data, No to cancel.", dfName
            DoAlert 1, prompt
            if (V_flag == 2)
                Print "Load cancelled. You can rename the existing data folder and try again."
                return -1           // Cancelled
            endif
            SetDataFolder $dfName
        else
            NewDataFolder/S $dfName
        endif
       
        result = LoadNogueiraFile(pathName, fileName)

        SetDataFolder ::                // Back to parent data folder

        index += 1
    while (1)

    if (Exists("temporaryPath"))        // Kill temp path if it exists
        KillPath temporaryPath
    endif

    return 0                        // Signifies success.
End
Many thanks hrodstein!!

This is a great start. I've made the tour#1 and #2... but everything is new for me.

Best,