Loading a text file with space and tabs
LaMP
Hello,
I would like to load a series of .csv files that unfortunately contain a mixture of "tabs" and spaces. The procedure that I have to load the data works well for the data that is separated with tabs but not at all for the spaces. I have included an example of my data files plus the IGOR procedure used for loading.
Any help is appreciated
Function Loadsmps() // Loads smps data
// Prompts to select folder into which data is loaded
String PathToFldr="",CtrlName="",PromptStr="Select folder to load smps data into:",CurrentFldr=getdatafolder(1) // variables for Browse4Folder
Browse4Folder(PathToFldr,CtrlName,PromptStr,CurrentFldr) // prompts user to select an experiment
// Get folder with data in it
NewPath/M="Select folder containing smps data:"/O/Q pathname // prompts user to select path to data folder
PathInfo pathname
// Abort if cancel is pressed
If(V_flag==0)
Abort "proceedure cancelled"
EndIf
// Make matrix for smps data
Make/o/n=(0,175) smpsDataMatrix
// Strings and Variables
Variable numfolders,n, subdirs, numfiles, filenum, refnum // variables
String folderlist, filename, name, currentfolder // strings
// Counts number of files in folder
NumFiles=ItemsInList(IndexedFile(pathname,-1,".csv")) // # files in folder , specify extensions e.g. ".txt"
For(filenum=0;filenum<numfiles;filenum+=1) // loops through each file name
filename=s_path+StringFromList(filenum,IndexedFile(pathname,-1,".csv")) // get full path to data file , specify extensions e.g. ".txt"
If(Stringmatch(FileName,"*dmps*")==1)
Open /R/P=pathname/Z refNum as filename // open
FStatus refNum // check there is data in file
If(V_logEOF>10) // specify minimum file size to load e.g. >10bytes
Load_smps_File(filename, refnum) // calls function to load file // unique function for specific file format
Wave Temp_smpsDataMatrix0
// add loaded waves to loaded data
InsertPoints DimSize(smpsDataMatrix,0),DimSize(Temp_smpsDataMatrix0,0), smpsDataMatrix
smpsDataMatrix[DimSize(smpsDataMatrix,0)-DimSize(Temp_smpsDataMatrix0,0),*][]=Temp_smpsDataMatrix0[p-DimSize(smpsDataMatrix,0)+DimSize(Temp_smpsDataMatrix0,0)][q]
currentfolder=GetDataFolder(1) // root folder for experiment
Endif // end data to folder allocation
EndIf
EndFor // end loop through files of a dp
Close/A // close open files
Killwaves Temp_smpsDataMatrix0
// Do additional analytsis on smps data
Processsmps() // adds a time wave
End
Function Load_smps_File(filename, refnum) // load single smps file
String filename
Variable refnum
print "Loading: " + filename
LoadWave/V={"\t, ","",0,0}/q/a/n=Temp_smpsDataMatrix/m/o/j/k=1/L={0,3,0,0,0} filename
End
Function Processsmps() // Additional processing of smps data
Wave smpsDataMatrix
Make/o/d/n=(DimSize(smpsDataMatrix,0)) smpsTime
Make/o/d/n=(DimSize(smpsDataMatrix,0)) smpsdate
SetScale d,0,0,"dat" smpsTime
SetScale d,0,0,"dat" smpsdate
smpsDate=Date2Secs(smpsDataMatrix[p][0],smpsDataMatrix[p][1],smpsDataMatrix[p][2])+smpsDataMatrix[p][3]*3600+smpsDataMatrix[p][4]*60+smpsDataMatrix[p][5]
smpsTime=smpsDataMatrix[p][3]*3600+smpsDataMatrix[p][4]*60+smpsDataMatrix[p][5]
End
End
e#pragma rtGlobals=1 // Use modern global access method.
Function Browse4Folder(PathToFldr,CtrlName,PromptStr,CurrentFldr) // based on mg code from htdma toolkit
String PathToFldr // path to data folder
String PromptStr // Prompt text in data browser
String CurrentFldr // current data folder
String CtrlName
variable exit=0 // stops trying when exit reaches 6
do
string/g BrowserList
string /g TempFldr=CurrentFldr
variable/g Flag
execute "CreateBrowser prompt=\""+PromptStr+"\", select=tempfldr" // prompt for file
killstrings/z tempfldr
svar BrowserList=s_browserlist
nvar Flag=V_Flag
// print browser_list
if (!Flag)
PathToFldr="cancelled"
break
endif
if (itemsinlist(BrowserList)==1)
if (stringmatch("root:ROOT;", BrowserList)) //data browser returns strange path if root is selected
BrowserList="root"
endif
PathToFldr=stringfromlist(0,BrowserList)+":"
if(DataFolderExists(PathToFldr))
break
else
doalert 0, "you must select a data folder, not a wave, string or variable! => try again"
endif
else
doalert 0, "none or multiple data folders were selected => try again"
endif
exit+=1
while (exit<6) //maximum six attempts
End
// Prompts to select folder into which data is loaded
String PathToFldr="",CtrlName="",PromptStr="Select folder to load smps data into:",CurrentFldr=getdatafolder(1) // variables for Browse4Folder
Browse4Folder(PathToFldr,CtrlName,PromptStr,CurrentFldr) // prompts user to select an experiment
// Get folder with data in it
NewPath/M="Select folder containing smps data:"/O/Q pathname // prompts user to select path to data folder
PathInfo pathname
// Abort if cancel is pressed
If(V_flag==0)
Abort "proceedure cancelled"
EndIf
// Make matrix for smps data
Make/o/n=(0,175) smpsDataMatrix
// Strings and Variables
Variable numfolders,n, subdirs, numfiles, filenum, refnum // variables
String folderlist, filename, name, currentfolder // strings
// Counts number of files in folder
NumFiles=ItemsInList(IndexedFile(pathname,-1,".csv")) // # files in folder , specify extensions e.g. ".txt"
For(filenum=0;filenum<numfiles;filenum+=1) // loops through each file name
filename=s_path+StringFromList(filenum,IndexedFile(pathname,-1,".csv")) // get full path to data file , specify extensions e.g. ".txt"
If(Stringmatch(FileName,"*dmps*")==1)
Open /R/P=pathname/Z refNum as filename // open
FStatus refNum // check there is data in file
If(V_logEOF>10) // specify minimum file size to load e.g. >10bytes
Load_smps_File(filename, refnum) // calls function to load file // unique function for specific file format
Wave Temp_smpsDataMatrix0
// add loaded waves to loaded data
InsertPoints DimSize(smpsDataMatrix,0),DimSize(Temp_smpsDataMatrix0,0), smpsDataMatrix
smpsDataMatrix[DimSize(smpsDataMatrix,0)-DimSize(Temp_smpsDataMatrix0,0),*][]=Temp_smpsDataMatrix0[p-DimSize(smpsDataMatrix,0)+DimSize(Temp_smpsDataMatrix0,0)][q]
currentfolder=GetDataFolder(1) // root folder for experiment
Endif // end data to folder allocation
EndIf
EndFor // end loop through files of a dp
Close/A // close open files
Killwaves Temp_smpsDataMatrix0
// Do additional analytsis on smps data
Processsmps() // adds a time wave
End
Function Load_smps_File(filename, refnum) // load single smps file
String filename
Variable refnum
print "Loading: " + filename
LoadWave/V={"\t, ","",0,0}/q/a/n=Temp_smpsDataMatrix/m/o/j/k=1/L={0,3,0,0,0} filename
End
Function Processsmps() // Additional processing of smps data
Wave smpsDataMatrix
Make/o/d/n=(DimSize(smpsDataMatrix,0)) smpsTime
Make/o/d/n=(DimSize(smpsDataMatrix,0)) smpsdate
SetScale d,0,0,"dat" smpsTime
SetScale d,0,0,"dat" smpsdate
smpsDate=Date2Secs(smpsDataMatrix[p][0],smpsDataMatrix[p][1],smpsDataMatrix[p][2])+smpsDataMatrix[p][3]*3600+smpsDataMatrix[p][4]*60+smpsDataMatrix[p][5]
smpsTime=smpsDataMatrix[p][3]*3600+smpsDataMatrix[p][4]*60+smpsDataMatrix[p][5]
End
End
e#pragma rtGlobals=1 // Use modern global access method.
Function Browse4Folder(PathToFldr,CtrlName,PromptStr,CurrentFldr) // based on mg code from htdma toolkit
String PathToFldr // path to data folder
String PromptStr // Prompt text in data browser
String CurrentFldr // current data folder
String CtrlName
variable exit=0 // stops trying when exit reaches 6
do
string/g BrowserList
string /g TempFldr=CurrentFldr
variable/g Flag
execute "CreateBrowser prompt=\""+PromptStr+"\", select=tempfldr" // prompt for file
killstrings/z tempfldr
svar BrowserList=s_browserlist
nvar Flag=V_Flag
// print browser_list
if (!Flag)
PathToFldr="cancelled"
break
endif
if (itemsinlist(BrowserList)==1)
if (stringmatch("root:ROOT;", BrowserList)) //data browser returns strange path if root is selected
BrowserList="root"
endif
PathToFldr=stringfromlist(0,BrowserList)+":"
if(DataFolderExists(PathToFldr))
break
else
doalert 0, "you must select a data folder, not a wave, string or variable! => try again"
endif
else
doalert 0, "none or multiple data folders were selected => try again"
endif
exit+=1
while (exit<6) //maximum six attempts
End
Forum
Support
Gallery
Igor Pro 9
Learn More
Igor XOP Toolkit
Learn More
Igor NIDAQ Tools MX
Learn More
Since your file does not contain string date or date/time data, and since it does not contain blank data as far as I can tell, you can use LoadWave/G (Load General Text) instead of LoadWave/J (Load Delimited Text).
Load General Text automatically accepts spaces or a tab as a delimiter. It also automatically skips the header to find the numeric data, so you don't need the /L or /K flags.
This worked for me to load the files you attached:
For more information, execute:
DisplayHelpTopic "Loading General Text Files"
DisplayHelpTopic "Loading Delimited Text Files"
October 22, 2019 at 02:31 pm - Permalink
Hello hrodstein,
Thank you very much for getting back to me and apologies for the delay in my reply. However, I am having the same problem. The first 14 columns load correctly and then the 14th column is copied for the remainder of the matrix.
When I check the raw files it seems that there are tabs at the start and then these change to spaces afterwards.
Thanks again for your help.
October 25, 2019 at 01:06 pm - Permalink
In reply to Hello hrodstein, Thank you… by LaMP
I don't see that. For me, the waves load correctly using the command the I listed above.
I am using Igor Pro 8.04.
I will attach a zip archive containing my test experiment and the result of executing the LoadWave command above as well as the files that I loaded. The files are your files renamed for simplicity.
If you need further help, please specify your Igor version and include a self-contained example like the one I am attaching that reproduces the problem.
October 25, 2019 at 02:56 pm - Permalink
Hello again,
Problem solved. I had copied the new code snipet into the wrong load procedure :/, so that is why I didnt see any change! Thanks again for your help.
For info, I am still working in IGOR 6.
October 25, 2019 at 11:17 pm - Permalink