Load waves from my csv file
Hi, I am trying to load the wave header to get the Start and Increment of my csv file.
// Here is the beginning of an example file:
// X, CH2, CH3, Start, Increment
// Sequence, VOLT, VOLT, -6.00E -6,1.00E-8
// 0, -8.00E -2,8.00E-2
// 1, 8.00E-2, 0.00E+0
// 2, -8.00E-2, 0.00E+0
// 3, 8.00E-2, 0.00E+0
// 4, -8.00E-2, 0.00E+0
I am trying to use
LoadWave/D/J/O/L={0,1,0,3,0}/A=dummy /p=my_path fileName
if the number of channels change then my L ={} will change correspondingly
I think but there should be a way for all possible # of channels. So is there a way to load the waves and get the number of columns first, lets call that ncolums, then L will be L={0,1,0,ncolums-2,0}, so that I could always load the wave header of last two columns of my data. Thanks
I have cleaned up (I think) the data you provided. I think "-6,1.00E-8" should be "1.00E-8".
Also "-6.00E" should be "-6.00E+0" and there was an extraneous "2," in the first line of data.
This gives the following:
Sequence, VOLT, VOLT, -6.00E+0, 1.00E-8
0, -8.00E, 8.00E-2
1, 8.00E-2, 0.00E+0
2, -8.00E-2, 0.00E+0
3, 8.00E-2, 0.00E+0
4, -8.00E-2, 0.00E+0
Given that data, the following code determines the number of data columns to be loaded and loads them:
String pathName // Name of an Igor symbolic path or ""
String fileName // Name of file or full path to file
Variable refNum
Open/R/P=$pathName refNum as fileName
String buffer, text
Variable line = 0
do
FReadLine refNum, buffer
if (strlen(buffer) == 0)
Close refNum
return -1 // Error: No more data
endif
text = buffer[0,7]
if (CmpStr(text,"Sequence") == 0)
// This line tells us the number of columns
int numberOfColumns = ItemsInList(buffer,",")
int numberOfDataColumns = numberOfColumns - 2 // We want to skip the last two columns
Close refNum
return numberOfDataColumns
endif
line += 1
while(1)
return -1 // We will never get here
End
Function FindNumberOfDataColumnsAndLoad(pathName, fileName)
String pathName // Name of an Igor symbolic path or "" for dialog
String fileName // Name of file or full path to file or "" for dialog
// First get a valid reference to a file
if ((strlen(pathName)==0) || (strlen(fileName)==0))
// Display dialog looking for file.
Variable refNum
Open/D/R/P=$pathName refNum as fileName
fileName = S_fileName // S_fileName is set by Open/D
if (strlen(fileName) == 0) // User cancelled?
return -2
endif
endif
// Now find the number of data columns in the file
Variable numberOfDataColumns = FindNumberOfDataColumns(pathName, fileName)
if (numberOfDataColumns <= 0)
Print "FindNumberOfDataColumnsAndLoad Error - Could not find number of data columns"
return -1
endif
// Now load the data
int firstDataLine = 2
LoadWave/Q/J/D/P=$pathName/E=1/K=0/L={0,firstDataLine,0,0,numberOfDataColumns}/N=temp fileName
return 0
End
There are some improvements that I would do:
1. Skip the first column which is of no use assuming that it is always a sequence starting from 0 and stepping by 1.
2. Provide better wave names using the /B flag.
3. Assuming that Start and Increment refer to the starting X value and delta X value, use these to set the X scaling of the loaded waves.
I can help you with those tasks but I'd prefer to work on an actual data file which you can upload as an attachment.
July 26, 2024 at 11:54 am - Permalink
Yes, what I am trying to do is to write a Macro that could load the waves. And the first thing to do is get the starting X value and delta X value, so I could use these to set the X scaling of the loaded waves. But I would like to write a macro that could work for all possible # of channels instead of only two. That is why I am asking how to get the header of last two columns. Thanks
July 26, 2024 at 04:35 pm - Permalink
The code above does load all columns of data regardless of how many there are.
Here is an updated version that:
1. Skips the first column which is of no use assuming that it is always a sequence starting from 0 and stepping by 1.
2. Names the output waves CH0, CH1,...
3. Sets the X scaling of each output wave based on the last two numbers in the "Sequence" line.
This code works assuming that the format of the text in the file matches the format of my cleaned-up text in the post above.
If you attach an actual data file, I will test to make sure this code works with your actual data file.
String buffer // e.g., "Sequence, VOLT, VOLT, -6.00E, 1.00E-8"
double& x0 // Output
double& dx // Output
x0 = 123
dx = 456
String list = ReplaceString(" ", buffer, "") // Remove spaces
int numItems = ItemsInList(list, ",")
String x0Str = StringFromList(numItems-2, list, ",")
x0 = str2num(x0Str)
String dxStr = StringFromList(numItems-1, list, ",")
dx = str2num(dxStr)
String expr = ""
return 0
End
Function SetWaveScaling(listOfWaveNames, x0, dx)
String listOfWaveNames
double x0, dx
int numItems = ItemsInList(listOfWaveNames)
int itemNumber
for(itemNumber=0; itemNumber<numItems; itemNumber+=1)
String name = StringFromList(itemNumber, listOfWaveNames)
WAVE w = $name
SetScale/P x, x0, dx, w
endfor
End
Function FindNumberOfDataColumnsAndScaling(pathName, fileName, x0, dx)
String pathName // Name of an Igor symbolic path or ""
String fileName // Name of file or full path to file
double& x0 // Output
double& dx // Output
Variable refNum
Open/R/P=$pathName refNum as fileName
String buffer, text
Variable line = 0
do
FReadLine refNum, buffer
if (strlen(buffer) == 0)
Close refNum
return -1 // Error: No more data
endif
text = buffer[0,7]
if (CmpStr(text,"Sequence") == 0)
// This line tells us the number of columns
int numberOfColumns = ItemsInList(buffer,",")
int numberOfDataColumns = numberOfColumns - 2 // We want to skip the last two columns
Close refNum
ExtractScaling(buffer, x0, dx)
return numberOfDataColumns
endif
line += 1
while(1)
return -1 // We will never get here
End
Function FindNumberOfDataColumnsAndLoad(pathName, fileName)
String pathName // Name of an Igor symbolic path or "" for dialog
String fileName // Name of file or full path to file or "" for dialog
// First get a valid reference to a file
if ((strlen(pathName)==0) || (strlen(fileName)==0))
// Display dialog looking for file.
Variable refNum
Open/D/R/P=$pathName refNum as fileName
fileName = S_fileName // S_fileName is set by Open/D
if (strlen(fileName) == 0) // User cancelled?
return -2
endif
endif
// Now find the number of data columns in the file
double x0, dx
Variable numberOfDataColumns = FindNumberOfDataColumnsAndScaling(pathName, fileName, x0, dx)
if (numberOfDataColumns <= 0)
Print "FindNumberOfDataColumnsAndLoad Error - Could not find number of data columns"
return -1
endif
// Now load the data
int firstDataLine = 2
int firstColumnToLoad = 1 // Skip column 0 which is just 0,1,2,...
LoadWave/Q/J/D/P=$pathName/K=0/O/A=CH/E=1/L={0,firstDataLine,0,firstColumnToLoad,numberOfDataColumns}/N=temp fileName
SetWaveScaling(S_waveNames, x0, dx)
return 0
End
July 26, 2024 at 08:11 pm - Permalink
Much appreciated!
Here is the Macro that I have so far that only works for two channels, could you try to modify that, so it could work for al possible number of channels? I am hesitating to give you my actual data, since it is important.
//7/22/2024 This macro loads a Rigol scope trace.
variable/d tracenum // sets naming of waves
String fileName // name of file to load or "" to get dialog
String Pathname // name of path or "" get dialog
variable/d num_waves_loaded
String listItem
variable/d start,increment
String startwavename, incrementwavename
// Silent 1; PauseUpdate
String w0, w1, w2, w3, w4 // Wavenames of loaded waves.
// can have up to 4 traces, and time is first column
String newname0,newname1,newname2,newname3,newname4 // New wave names
// Here is the beginning of an example Rigol file:
// X,CH2,CH3,Start,Increment
// Sequence,VOLT,VOLT,-6.00E-6,1.00E-8
// 0,-8.00E-2,8.00E-2
// 1,8.00E-2,0.00E+0
// 2,-8.00E-2,0.00E+0
// 3,8.00E-2,0.00E+0
// 4,-8.00E-2,0.00E+0
// First we have to load the wave header to get the time scale since that is how Rigol formats their csv files:
//LoadWave/D/J/O/L={0,1,0,3,0}/A=dummy /p=$Pathname fileName
LoadWave/D/J/O/L={0,1,0,3,0}/A=dummy /p=my_path fileName
// /L={nameLine, firstLine, numLines, firstColumn, numColumns}
print("Loaded S_fileName=")
print(S_fileName)
print("Loaded S_fileName=")
print(S_path)
print(S_waveNames)
startwavename=StringFromList(0, S_waveNames)
incrementwavename=StringFromList(1, S_waveNames)
Pathname=S_path // for next load command, if user selected using dialog
fileName=S_fileName// for next load command
// start = ....
start=$startwavename[0]
// increment = ...
increment=$incrementwavename[0]
print("start=")
print(start)
print("increment=")
print(increment)
killwaves $startwavename
killwaves $incrementwavename
// Next, load in waves.
print ("loading filanemame=")
print filename
//LoadWave/A=dummy/Q/D/G/O/p=my_path filename
LoadWave/D/J/O/L={0,2,0,0,3}/W/A/Q /p=my_path fileName
if (V_flag==0) // Now waves loaded. Perhaps user cancelled.
return
endif
print("Loaded S_fileName=")
print(S_fileName)
print("Loaded S_fileName=")
print(S_path)
print("S_waveNames=")
print(S_waveNames)
//Print ItemsInList(stringList, ",")
print("num_waves_loaded=V_flag=")
print(V_flag)
print("done printing....")
w0= StringFromList(0,S_waveNames) // this is the point number, we don't want it
w1= StringFromList(1,S_waveNames)
w2= StringFromList(2,S_waveNames)
w3= StringFromList(3,S_waveNames)
w4= StringFromList(4,S_waveNames)
// put the name of waves into string variables
if (strlen(w0) != 0) // w0 exists
print("w0 ok!")
w0= StringFromList(0,S_waveNames)
killwaves $w0 // this is the point number, we don't want it
endif
if (strlen(w1) != 0) // w1 exists
print("w1 ok!")
w1= StringFromList(1,S_waveNames)
newname1=w1+"_" + num2str(tracenum)
//newname1="data_1_" + num2str(tracenum)
duplicate/o $w1, $newname1
killwaves $w1
// set scale to volts
SetScale/P x start,increment,"s", $newname1
SetScale d 0,0,"V", $newname1
endif
if (strlen(w2) != 0) // w2 exists
print("w2 ok!")
w2= StringFromList(2,S_waveNames)
newname2=w2+"_" + num2str(tracenum)
//newname2="data_2_" + num2str(tracenum)
duplicate/o $w2, $newname2
killwaves $w2
// set scale to volts
SetScale/P x start,increment,"s", $newname2
SetScale d 0,0,"V", $newname2
endif
if (strlen(w3) != 0) // w3 exists
print("w3 ok!")
w3= StringFromList(3,S_waveNames)
newname3=w3+"_" + num2str(tracenum)
//newname3="data_3_" + num2str(tracenum)
duplicate/o $w3, $newname3
killwaves $w3
// set scale to volts
SetScale/P x start,increment,"s", $newname3
SetScale d 0,0,"V", $newname3
endif
if (strlen(w4) != 0) // w4 exists
print("w4 ok!")
w4= StringFromList(4,S_waveNames)
//newname4="data_4_" + num2str(tracenum)
newname4=w4+"_" + num2str(tracenum)
duplicate/o $w4, $newname4
killwaves $w4
SetScale/P x start,increment,"s", $newname4
SetScale d 0,0,"V", $newname4
// set scale to volts
endif
EndMacro
July 26, 2024 at 09:39 pm - Permalink
I wonder, did you look at what hrodstein has provided? Did it not work for you? It seems a very neat solution to me... Also, macros are very old tech and should be avoided in favor of functions.
Anyway, here is a code fragment which wraps your manual assignment into a simple loop. Things could be done much more neatly (see Howards solution), but if you just want to quickly modify what you already have...
LoadWave/D/J/O/L={0,2,0,0,3}/W/A/Q /p=my_path fileName
if (V_flag==0)
return
endif
Variable i
for (i=1; i<ItemsInList(S_waveNames);i+=1)
string currW = StringFromList(i,S_waveNames)
string Wname = currW + "_" + num2str(tracenum)
Duplicate/o $currW, $Wname
KillWaves $currW
SetScale/P x start,increment,"s", $Wname
SetScale d 0,0,"V", $Wname
endfor
... whatever else you want to do ...
July 27, 2024 at 09:15 pm - Permalink
In reply to Much appreciated! Here is… by minghaoj
Agree to chozo's take. It really comes down to the /L={0,2,0,0,3}. The 3 dictates how many columns you are loading (in your case the X, CH2, and CH3). You can increase this number accordingly if you have additional channels of data. Since your meta data (Start and Increment) are in the first rows, you won't have issue loading your data.
I am guessing your question is how to have your function be versatile to handle things beyond (the repetitive chunks of code for) w0 to w4??
Personally, for data like this, I prefer to load them into a 2D wave matrix by adding /M to LoadWave and go from there.
Another option would be creating a function that does your post loading processing and then feed the waves generated to that function in a loop. I see you already found the list of names of waves are stored in S_waveNames. You can use ItemsInList to determine the number of channels/waves loaded. Then use a For loop to repeat the process.
string wvName, newWvName
numWaves = itemsinlist(S_waveNames)
for(i =0; i< numWaves;i++)
wvName= StringFromList(i,S_waveNames)
newWvName = wvName + "_" + num2str(tracenum)
duplicate/o $wvName, $newWvName
killwaves $wvName
// set scale to volts
SetScale/P x start,increment,"s", $newWvName
SetScale d 0,0,"V", $newWvName
endfor
Personally, I would change the order of operation as follow and use name the waves differently (you can also use the LoadWave/Name added in Igor Pro 9 or later to handle your wave name changing. Note to myself, start using this).
string loadedWvName, wvName
numWaves = itemsinlist(S_waveNames)
for(i =0; i< numWaves;i++)
loadedWvName= StringFromList(i,S_waveNames)
// set scale to volts
SetScale/P x start,increment,"s", $loadedWvName
SetScale d 0,0,"V", $loadedWvName
//Chose one below
//If you prefer the same channel of data from different files are grouped together in Data Browser use this line
wvName = "CH" + num2str(i+1) + "_" + num2str(tracenum)
//If you prefer data of different channels from the same data file are grouped together in Data Browser use this line.
wvName = "_" + num2str(tracenum) + "_CH" + num2str(i+1) //Add "_" in the front as you can't start a wave name using a number, unless you are doing liberal name. The next line of code will fix this for you anyways though.
//It is always important to verify your wave name is valid.
//Alternatively, you can use CreateDataObjectName for more options only if you are using Igor Pro 9.0 or later.
wvName = CleanupName(wvName, 0)
duplicate/o $loadedWvName, $wvName
killwaves $loadedWvName
endfor
Unrelated, yet related. I have created loaders for several different types of data files with meta data in it. The solution is similar to what hrodstein provided here. It involves a combination of FReadline and LoadWave (sometimes multiple of each).
Please correct me if I am wrong about this. One pet peeve of mine is that LoadWave/L seems to be limited by the number of columns in the firstLine of data which overrides the numColumns parameter if the later is larger. I am sure there are underlying reason why it does this, but part of me hope we can have an option to override that.
The situation is the worst when multiple chunks of data are stored in the same file with their individual meta data. Doing multiple loops of LoadWave/L can be very slow. FReadline appears to be faster (as the file is only opened once) but requires additional post loading wave handling. Some times I can try loading the entire file into one wave and separate each chunk out. It gets messy if the chunk sizes are dynamic.
July 30, 2024 at 08:37 am - Permalink
That's correct. It is a limitation that I have run into myself on occasion.
The underlying reason is that I didn't think of this situation when I wrote this code (about 35 years ago ;)
I think a flag that says "don't automically determine the number of columns in the file from the first line of data, follow /L instead" is doable.
August 1, 2024 at 10:18 am - Permalink
In reply to One pet peeve of mine is… by hrodstein
That will be cool. I am looking forward to that feature in future updates.
August 6, 2024 at 12:22 pm - Permalink