Loading multiple files and calculating average hourly
thiago.nogueira
Hi all,
I am new to IGOR. I just started to learn Igor instead of Origin.
I have data (tons!!) of 1 sec concentration of chemical species (CO2 and CH4) over a couple of year.. And I would like to calculate mean and standart deviation for each hour over the length of my data.
I have tons of files (*dat), but I can't load all using Igor. I tried the function LoadAndGraphAll as suggested in manual, but I failed. I think that the problem is because I have a lot of rows and columns!! There are a lot of columns, but I need load just DATE TIME CH4_dry CO2_dry
Any help would be greatly appreciated as I am struggling with this.
If you have not already done it, do the guided tour. It is essential. Choose Help->Getting Started.
Your file is a FORTRAN-style fixed field file with 23 columns in fields of 26 characters each. Therefore we need to use LoadWave/F.
After you do the guided tour you will need to learn about Igor data folders. Execute:
I have pasted below an Igor procedure that will load your file and another that will load all .txt files in a given folder on disk into Igor data folders. Execute this to learn about Igor procedure files:
You should paste the procedure text below into an Igor procedure file and save the file in your "Igor Procedures" folder. The "Igor Procedures" folder is explained in the help for procedure windows.
The procedures add items to the Load Waves submenu of the Data menu.
Menu "Load Waves"
"Load Nogueira File...", LoadNogueiraFile("", "")
"Load All Nogueira Files in Folder...", LoadAllNogueiraFilesInFolder("")
End
Function LoadNogueiraFile(pathName, fileName)
String pathName // Name of an Igor symbolic path or "".
String fileName // Name of file or full path to file.
// First get a valid reference to a file.
if ((strlen(pathName)==0) || (strlen(fileName)==0))
// Display dialog looking for file.
Variable refNum
Open/D/R/F="*.txt"/P=$pathName refNum as fileName
fileName = S_fileName // S_fileName is set by Open/D
if (strlen(fileName) == 0) // User cancelled?
return -1
endif
endif
// Now load the data.
// The file uses FORTRAN fixed-field style with 23 columns in fields of 26 characters each.
// We are interested in the DATE, TIME, CH4_dry, and CO2_dry columns only.
// "Date" and "Time" are not available as wave names because they are Igor function names
String columnInfoStr = " "
columnInfoStr += "N='DateTimeW',F=6;" // Load DATE column - will become date/time wave
columnInfoStr += "N='TimeW',F=7;" // Load TIME column
columnInfoStr += "N='_skip_';" // Skip FRAC_DAYS_SINCE_JAN1
columnInfoStr += "N='_skip_';" // Skip FRAC_HRS_SINCE_JAN1
columnInfoStr += "N='_skip_';" // Skip EPOCH_TIME
columnInfoStr += "N='_skip_';" // Skip ALARM_STATUS
columnInfoStr += "N='_skip_';" // Skip AmbientPressure
columnInfoStr += "N='_skip_';" // Skip CH4
columnInfoStr += "N='CH4_dry';" // Load CH4_dry
columnInfoStr += "N='_skip_';" // Skip CO2
columnInfoStr += "N='CO2_dry';" // Load CO2_dry
columnInfoStr += "N='_skip_';" // Skip CavityPressure
columnInfoStr += "N='_skip_';" // Skip CavityTemp
columnInfoStr += "N='_skip_';" // Skip DasTemp
columnInfoStr += "N='_skip_';" // Skip EtalonTemp
columnInfoStr += "N='_skip_';" // Skip H2O
columnInfoStr += "N='_skip_';" // Skip MPVPosition
columnInfoStr += "N='_skip_';" // Skip OutletValve
columnInfoStr += "N='_skip_';" // Skip WarmBoxTemp
columnInfoStr += "N='_skip_';" // Skip cavity_pressure
columnInfoStr += "N='_skip_';" // Skip cavity_temperature
columnInfoStr += "N='_skip_';" // Skip solenoid_valves
columnInfoStr += "N='_skip_';" // Skip species
columnInfoStr += "N='_skip_';" // Skip time
LoadWave /F={23,26,0} /L={0,1,0,0,0} /D /K=0 /A /B=columnInfoStr /Q /R={English, 1, 2, 1, 1, "Year-Month-DayofMonth", 40} /P=$pathName fileName
Variable numWavesLoaded = V_flag // V_flag is set by LoadWave
if (numWavesLoaded != 4)
Print "Error loading file - wrong number of waves loaded"
return -1
endif
Wave DateTimeW,TimeW // Create reference to waves created by LoadWave
DateTimeW += TimeW // Add time to date to obtain date/time
KillWaves/Z TimeW // This is no longer needed
return 0 // Success
End
Function LoadAllNogueiraFilesInFolder(pathName) // Loads all .txt files in specified folder
String pathName // Name of symbolic path or "" to get dialog
String fileName
Variable index=0
if (strlen(pathName)==0) // If no path specified, create one
NewPath/O temporaryPath // This will put up a dialog
if (V_flag != 0)
return -1 // User cancelled
endif
pathName = "temporaryPath"
endif
Variable result
do // Loop through each file in folder
fileName = IndexedFile($pathName, index, ".txt")
if (strlen(fileName) == 0) // No more files?
break // Break out of loop
endif
// Create data folder from file name
String dfName = ParseFilePath(0, fileName, ":", 1, 0)
dfName = RemoveEnding(dfName, ".txt")
if (DataFolderExists(dfName))
String prompt
sprintf prompt, "Data folder named '%s' exists. Click Yes to overwrite the existing data, No to cancel.", dfName
DoAlert 1, prompt
if (V_flag == 2)
Print "Load cancelled. You can rename the existing data folder and try again."
return -1 // Cancelled
endif
SetDataFolder $dfName
else
NewDataFolder/S $dfName
endif
result = LoadNogueiraFile(pathName, fileName)
SetDataFolder :: // Back to parent data folder
index += 1
while (1)
if (Exists("temporaryPath")) // Kill temp path if it exists
KillPath temporaryPath
endif
return 0 // Signifies success.
End
March 5, 2015 at 08:49 am - Permalink
This is a great start. I've made the tour#1 and #2... but everything is new for me.
Best,
March 5, 2015 at 01:10 pm - Permalink