Loading .csv
LaMP
Hello,
I am trying to load a .csv file. This is a straight forward file with only numbers and it appears to be tab delimited. The load is working well until the 16 column of the data matrix. At this point the load just copies the data in column 16 to the end of the matrix load (n= 175).
I have copied an example file as well as the code that I am using.
I have been playing around with the load wave function but I can not seem to get it to work.
Any suggestions are very welcome.
Thank you.
Function Loadsmps() // Loads smps data
// Prompts to select folder into which data is loaded
String PathToFldr="",CtrlName="",PromptStr="Select folder to load smps data into:",CurrentFldr=getdatafolder(1) // variables for Browse4Folder
Browse4Folder(PathToFldr,CtrlName,PromptStr,CurrentFldr) // prompts user to select an experiment
// Get folder with data in it
NewPath/M="Select folder containing smps data:"/O/Q pathname // prompts user to select path to data folder
PathInfo pathname
// Abort if cancel is pressed
If(V_flag==0)
Abort "proceedure cancelled"
EndIf
// Make matrix for smps data
Make/o/n=(0,175) smpsDataMatrix
// Strings and Variables
Variable numfolders,n, subdirs, numfiles, filenum, refnum // variables
String folderlist, filename, name, currentfolder // strings
// Counts number of files in folder
NumFiles=ItemsInList(IndexedFile(pathname,-1,".csv")) // # files in folder , specify extensions e.g. ".txt"
For(filenum=0;filenum<numfiles;filenum+=1) // loops through each file name
filename=s_path+StringFromList(filenum,IndexedFile(pathname,-1,".csv")) // get full path to data file , specify extensions e.g. ".txt"
If(Stringmatch(FileName,"*pdd*")==1)
Open /R/P=pathname/Z refNum as filename // open
FStatus refNum // check there is data in file
If(V_logEOF>10) // specify minimum file size to load e.g. >10bytes
Load_smps_File(filename, refnum) // calls function to load file // unique function for specific file format
Wave Temp_smpsDataMatrix0
// add loaded waves to loaded data
InsertPoints DimSize(smpsDataMatrix,0),DimSize(Temp_smpsDataMatrix0,0), smpsDataMatrix
smpsDataMatrix[DimSize(smpsDataMatrix,0)-DimSize(Temp_smpsDataMatrix0,0),*][]=Temp_smpsDataMatrix0[p-DimSize(smpsDataMatrix,0)+DimSize(Temp_smpsDataMatrix0,0)][q]
currentfolder=GetDataFolder(1) // root folder for experiment
Endif // end data to folder allocation
EndIf
EndFor // end loop through files of a dp
Close/A // close open files
Killwaves Temp_smpsDataMatrix0
// Do additional analytsis on smps data
Processsmps() // adds a time wave
End
Function Load_smps_File(filename, refnum) // load single smps file
String filename
Variable refnum
print "Loading: " + filename
loadwave/v={"\t ","",0,1}/q/a/n=Temp_smpsDataMatrix/m/o/j/k=1/l={0,4,0,0,0} filename // loads smps data as a matrix
End
Function Browse4Folder(PathToFldr,CtrlName,PromptStr,CurrentFldr) // based on mg code from htdma toolkit
String PathToFldr // path to data folder
String PromptStr // Prompt text in data browser
String CurrentFldr // current data folder
String CtrlName
variable exit=0 // stops trying when exit reaches 6
do
string/g BrowserList
string /g TempFldr=CurrentFldr
variable/g Flag
execute "CreateBrowser prompt=\""+PromptStr+"\", select=tempfldr" // prompt for file
killstrings/z tempfldr
svar BrowserList=s_browserlist
nvar Flag=V_Flag
// print browser_list
if (!Flag)
PathToFldr="cancelled"
break
endif
if (itemsinlist(BrowserList)==1)
if (stringmatch("root:ROOT;", BrowserList)) //data browser returns strange path if root is selected
BrowserList="root"
endif
PathToFldr=stringfromlist(0,BrowserList)+":"
if(DataFolderExists(PathToFldr))
break
else
doalert 0, "you must select a data folder, not a wave, string or variable! => try again"
endif
else
doalert 0, "none or multiple data folders were selected => try again"
endif
exit+=1
while (exit<6) //maximum six attempts
End
// Prompts to select folder into which data is loaded
String PathToFldr="",CtrlName="",PromptStr="Select folder to load smps data into:",CurrentFldr=getdatafolder(1) // variables for Browse4Folder
Browse4Folder(PathToFldr,CtrlName,PromptStr,CurrentFldr) // prompts user to select an experiment
// Get folder with data in it
NewPath/M="Select folder containing smps data:"/O/Q pathname // prompts user to select path to data folder
PathInfo pathname
// Abort if cancel is pressed
If(V_flag==0)
Abort "proceedure cancelled"
EndIf
// Make matrix for smps data
Make/o/n=(0,175) smpsDataMatrix
// Strings and Variables
Variable numfolders,n, subdirs, numfiles, filenum, refnum // variables
String folderlist, filename, name, currentfolder // strings
// Counts number of files in folder
NumFiles=ItemsInList(IndexedFile(pathname,-1,".csv")) // # files in folder , specify extensions e.g. ".txt"
For(filenum=0;filenum<numfiles;filenum+=1) // loops through each file name
filename=s_path+StringFromList(filenum,IndexedFile(pathname,-1,".csv")) // get full path to data file , specify extensions e.g. ".txt"
If(Stringmatch(FileName,"*pdd*")==1)
Open /R/P=pathname/Z refNum as filename // open
FStatus refNum // check there is data in file
If(V_logEOF>10) // specify minimum file size to load e.g. >10bytes
Load_smps_File(filename, refnum) // calls function to load file // unique function for specific file format
Wave Temp_smpsDataMatrix0
// add loaded waves to loaded data
InsertPoints DimSize(smpsDataMatrix,0),DimSize(Temp_smpsDataMatrix0,0), smpsDataMatrix
smpsDataMatrix[DimSize(smpsDataMatrix,0)-DimSize(Temp_smpsDataMatrix0,0),*][]=Temp_smpsDataMatrix0[p-DimSize(smpsDataMatrix,0)+DimSize(Temp_smpsDataMatrix0,0)][q]
currentfolder=GetDataFolder(1) // root folder for experiment
Endif // end data to folder allocation
EndIf
EndFor // end loop through files of a dp
Close/A // close open files
Killwaves Temp_smpsDataMatrix0
// Do additional analytsis on smps data
Processsmps() // adds a time wave
End
Function Load_smps_File(filename, refnum) // load single smps file
String filename
Variable refnum
print "Loading: " + filename
loadwave/v={"\t ","",0,1}/q/a/n=Temp_smpsDataMatrix/m/o/j/k=1/l={0,4,0,0,0} filename // loads smps data as a matrix
End
Function Browse4Folder(PathToFldr,CtrlName,PromptStr,CurrentFldr) // based on mg code from htdma toolkit
String PathToFldr // path to data folder
String PromptStr // Prompt text in data browser
String CurrentFldr // current data folder
String CtrlName
variable exit=0 // stops trying when exit reaches 6
do
string/g BrowserList
string /g TempFldr=CurrentFldr
variable/g Flag
execute "CreateBrowser prompt=\""+PromptStr+"\", select=tempfldr" // prompt for file
killstrings/z tempfldr
svar BrowserList=s_browserlist
nvar Flag=V_Flag
// print browser_list
if (!Flag)
PathToFldr="cancelled"
break
endif
if (itemsinlist(BrowserList)==1)
if (stringmatch("root:ROOT;", BrowserList)) //data browser returns strange path if root is selected
BrowserList="root"
endif
PathToFldr=stringfromlist(0,BrowserList)+":"
if(DataFolderExists(PathToFldr))
break
else
doalert 0, "you must select a data folder, not a wave, string or variable! => try again"
endif
else
doalert 0, "none or multiple data folders were selected => try again"
endif
exit+=1
while (exit<6) //maximum six attempts
End
>I have copied an example file
I don't see any attachment.
October 18, 2018 at 08:31 pm - Permalink
Apologies...here is the file
October 18, 2018 at 09:52 pm - Permalink
Hello LaMP,
I was able to read your example file with the following LoadWave command (Igor 8.02, 64bit, Windows 10):
Delimited text load from "181002Pdd_0.csv"
Data length: 575, waves: wave0...wave177
It appears that some of the columns are tab delimited while others are space delimited.
Hope this helps,
Klaus
October 19, 2018 at 01:01 am - Permalink
The file's data starts on line 5 (zero-based).
The file is tab-delimited, but columns 8 and 11 (zero-based) have a space between the preceding tab and the number.
To make it easier to see if I was getting the right results, I created a version of the file where the first data line contains zero-based column numbers from 0 to 175 for a total of 176 columns. I will attach my version of the file which I called Test.csv.
This command:
LoadWave/J/D/E=1/K=0/V={"\t, "," $",0,0}/L={0,6,0,0,0}
has two errors. The 6 should be 5 because the data starts on line 5 (zero-based).
Also, including space as a delimiter (/V flag) is incorrect because it causes LoadWave to create two extra columns.
I get the right result with this command:
LoadWave/J/D/E=1/K=0/L={0,5,0,0,0}/A/P=Data "Test.csv"
The preceding command uses a symbolic path named Data which you must create. To learn about symbolic paths, execute:
Since the file contains just numbers (no strings, dates, or times), you can use the simpler LoadWave/G instead of LoadWave/J. I get the correct results with this command:
LoadWave/G/D/E=1/A/P=Data "Test.csv"
I recommend using LoadWave/G. To learn about it, execute:
DisplayHelpTopic "Loading General Text Files"
October 19, 2018 at 08:32 am - Permalink