Load waves from my csv file

Share icon

Hi, I am trying to load the wave header to get the Start and Increment of my csv file.


    // Here is the beginning of an example file:
    // X,                 CH2,        CH3,                     Start,      Increment
    // Sequence,   VOLT,        VOLT,                   -6.00E    -6,1.00E-8
    // 0,                -8.00E      -2,8.00E-2
    // 1,                8.00E-2,    0.00E+0
    // 2,                -8.00E-2,    0.00E+0
    // 3,                 8.00E-2,     0.00E+0
    // 4,                -8.00E-2,      0.00E+0

I am trying to use 

LoadWave/D/J/O/L={0,1,0,3,0}/A=dummy /p=my_path fileName

if the number of channels change then my L ={} will change correspondingly

I think but there should be a  way for all possible # of channels. So is there a way to load the waves and get the number of columns first, lets call that ncolums, then L will be L={0,1,0,ncolums-2,0}, so that I could always load the wave header of last two columns of my data. Thanks 

 

 

I have cleaned up (I think) the data you provided. I think "-6,1.00E-8" should be "1.00E-8".

Also "-6.00E" should be "-6.00E+0" and there was an extraneous "2," in the first line of data.

This gives the following:

X,    CH2,    CH3,    Start,    Increment
Sequence,   VOLT,        VOLT,    -6.00E+0,    1.00E-8
0,    -8.00E,    8.00E-2
1,    8.00E-2,    0.00E+0
2,    -8.00E-2,    0.00E+0
3,    8.00E-2,    0.00E+0
4,    -8.00E-2,    0.00E+0

Given that data, the following code determines the number of data columns to be loaded and loads them:

Function FindNumberOfDataColumns(pathName, fileName)
    String pathName     // Name of an Igor symbolic path or ""
    String fileName     // Name of file or full path to file
    
    Variable refNum
 
    Open/R/P=$pathName refNum as fileName
    
    String buffer, text
    Variable line = 0
    
    do
        FReadLine refNum, buffer
        if (strlen(buffer) == 0)
            Close refNum
            return -1                       // Error: No more data
        endif
        text = buffer[0,7]
        if (CmpStr(text,"Sequence") == 0)
            // This line tells us the number of columns
            int numberOfColumns = ItemsInList(buffer,",")
            int numberOfDataColumns = numberOfColumns - 2   // We want to skip the last two columns
            Close refNum
            return numberOfDataColumns
        endif
        line += 1
    while(1)
    
    return -1       // We will never get here
End
 
Function FindNumberOfDataColumnsAndLoad(pathName, fileName)
    String pathName     // Name of an Igor symbolic path or "" for dialog
    String fileName     // Name of file or full path to file or "" for dialog
 
    // First get a valid reference to a file
    if ((strlen(pathName)==0) || (strlen(fileName)==0))
        // Display dialog looking for file.
        Variable refNum
        Open/D/R/P=$pathName refNum as fileName
        fileName = S_fileName           // S_fileName is set by Open/D
        if (strlen(fileName) == 0)      // User cancelled?
            return -2
        endif
    endif
 
    // Now find the number of data columns in the file
    Variable numberOfDataColumns = FindNumberOfDataColumns(pathName, fileName)
    if (numberOfDataColumns <= 0)
        Print "FindNumberOfDataColumnsAndLoad Error - Could not find number of data columns"
        return -1
    endif
    
    // Now load the data
    int firstDataLine = 2
    LoadWave/Q/J/D/P=$pathName/E=1/K=0/L={0,firstDataLine,0,0,numberOfDataColumns}/N=temp fileName
        
    return 0
End

There are some improvements that I would do:

1. Skip the first column which is of no use assuming that it is always a sequence starting from 0 and stepping by 1.

2. Provide better wave names using the /B flag.

3. Assuming that Start and Increment refer to the starting X value and delta X value, use these to set the X scaling of the loaded waves.

I can help you with those tasks but I'd prefer to work on an actual data file which you can upload as an attachment.

July 26, 2024 at 11:54 am - Permalink

Yes, what I am trying to do is to write a Macro that could load the waves. And the first thing to do is get the starting X value and delta X value, so I could use these to set the X scaling of the loaded waves. But I would like to write a macro that could work for all possible # of channels instead of only two. That is why I am asking how to get the header of last two columns. Thanks

July 26, 2024 at 04:35 pm - Permalink

The code above does load all columns of data regardless of how many there are.

Here is an updated version that:

1. Skips the first column which is of no use assuming that it is always a sequence starting from 0 and stepping by 1.

2. Names the output waves CH0, CH1,...

3. Sets the X scaling of each output wave based on the last two numbers in the "Sequence" line.

This code works assuming that the format of the text in the file matches the format of my cleaned-up text in the post above.

If you attach an actual data file, I will test to make sure this code works with your actual data file.

Function ExtractScaling(buffer, x0, dx)
    String buffer       // e.g., "Sequence, VOLT, VOLT, -6.00E, 1.00E-8"
    double& x0          // Output
    double& dx          // Output
 
    x0 = 123
    dx = 456
    
    String list = ReplaceString(" ", buffer, "")        // Remove spaces
    int numItems = ItemsInList(list, ",")
    String x0Str = StringFromList(numItems-2, list, ",")
    x0 = str2num(x0Str)
    String dxStr = StringFromList(numItems-1, list, ",")
    dx = str2num(dxStr)
    
    String expr = ""
    
    return 0    
End
 
Function SetWaveScaling(listOfWaveNames, x0, dx)
    String listOfWaveNames
    double x0, dx
    
    int numItems = ItemsInList(listOfWaveNames)
    int itemNumber
    for(itemNumber=0; itemNumber<numItems; itemNumber+=1)
        String name = StringFromList(itemNumber, listOfWaveNames)
        WAVE w = $name
        SetScale/P x, x0, dx, w
    endfor
End
 
Function FindNumberOfDataColumnsAndScaling(pathName, fileName, x0, dx)
    String pathName     // Name of an Igor symbolic path or ""
    String fileName     // Name of file or full path to file
    double& x0          // Output
    double& dx          // Output
    
    Variable refNum
 
    Open/R/P=$pathName refNum as fileName
    
    String buffer, text
    Variable line = 0
    
    do
        FReadLine refNum, buffer
        if (strlen(buffer) == 0)
            Close refNum
            return -1                       // Error: No more data
        endif
        text = buffer[0,7]
        if (CmpStr(text,"Sequence") == 0)
            // This line tells us the number of columns
            int numberOfColumns = ItemsInList(buffer,",")
            int numberOfDataColumns = numberOfColumns - 2   // We want to skip the last two columns
            Close refNum
            ExtractScaling(buffer, x0, dx)
            return numberOfDataColumns
        endif
        line += 1
    while(1)
    
    return -1       // We will never get here
End
 
Function FindNumberOfDataColumnsAndLoad(pathName, fileName)
    String pathName     // Name of an Igor symbolic path or "" for dialog
    String fileName     // Name of file or full path to file or "" for dialog
 
    // First get a valid reference to a file
    if ((strlen(pathName)==0) || (strlen(fileName)==0))
        // Display dialog looking for file.
        Variable refNum
        Open/D/R/P=$pathName refNum as fileName
        fileName = S_fileName           // S_fileName is set by Open/D
        if (strlen(fileName) == 0)      // User cancelled?
            return -2
        endif
    endif
 
    // Now find the number of data columns in the file
    double x0, dx
    Variable numberOfDataColumns = FindNumberOfDataColumnsAndScaling(pathName, fileName, x0, dx)
    if (numberOfDataColumns <= 0)
        Print "FindNumberOfDataColumnsAndLoad Error - Could not find number of data columns"
        return -1
    endif
    
    // Now load the data
    int firstDataLine = 2
    int firstColumnToLoad = 1       // Skip column 0 which is just 0,1,2,...
    LoadWave/Q/J/D/P=$pathName/K=0/O/A=CH/E=1/L={0,firstDataLine,0,firstColumnToLoad,numberOfDataColumns}/N=temp fileName
    
    SetWaveScaling(S_waveNames, x0, dx)
    
    return 0
End

 

July 26, 2024 at 08:11 pm - Permalink

Much appreciated!

Here is the Macro that I have so far that only works for two channels, could you try to modify that, so it could work for al possible number of channels? I am hesitating to give you my actual data, since it is important.

Macro loadtrace(tracenum,filename,pathname)
//7/22/2024 This macro loads a Rigol scope trace.
    variable/d tracenum // sets naming of waves
    String fileName // name of file to load or "" to get dialog
    String Pathname // name of path or "" get dialog
    variable/d num_waves_loaded
    String listItem
    variable/d start,increment
    String startwavename, incrementwavename
    // Silent 1; PauseUpdate
    String w0, w1, w2, w3, w4 //  Wavenames of loaded waves.
    // can have up to 4 traces, and time is first column
    String newname0,newname1,newname2,newname3,newname4 // New wave names
    // Here is the beginning of an example Rigol file:
    // X,CH2,CH3,Start,Increment
    // Sequence,VOLT,VOLT,-6.00E-6,1.00E-8
    // 0,-8.00E-2,8.00E-2
    // 1,8.00E-2,0.00E+0
    // 2,-8.00E-2,0.00E+0
    // 3,8.00E-2,0.00E+0
    // 4,-8.00E-2,0.00E+0
    // First we have to load the wave header to get the time scale since that is how Rigol formats their csv files:
    //LoadWave/D/J/O/L={0,1,0,3,0}/A=dummy /p=$Pathname fileName
    LoadWave/D/J/O/L={0,1,0,3,0}/A=dummy /p=my_path fileName
    // /L={nameLine, firstLine, numLines, firstColumn, numColumns}
    print("Loaded S_fileName=")
    print(S_fileName)
    print("Loaded S_fileName=")
    print(S_path)
    print(S_waveNames)
    startwavename=StringFromList(0, S_waveNames)
    incrementwavename=StringFromList(1, S_waveNames)
    Pathname=S_path // for next load command, if user selected using dialog
    fileName=S_fileName// for next load command
    // start = ....
    start=$startwavename[0]
    // increment = ...
    increment=$incrementwavename[0]
    print("start=")
    print(start)
    print("increment=")
    print(increment)
    killwaves $startwavename
    killwaves $incrementwavename
    // Next, load in waves.
    print ("loading filanemame=")
    print filename
    //LoadWave/A=dummy/Q/D/G/O/p=my_path filename
    LoadWave/D/J/O/L={0,2,0,0,3}/W/A/Q /p=my_path fileName
    if (V_flag==0) // Now waves loaded. Perhaps user cancelled.
        return
    endif       
    print("Loaded S_fileName=")
    print(S_fileName)
    print("Loaded S_fileName=")
    print(S_path)
    print("S_waveNames=")
    print(S_waveNames)  
    //Print ItemsInList(stringList, ",")
    print("num_waves_loaded=V_flag=")
    print(V_flag)
    print("done printing....")
    w0= StringFromList(0,S_waveNames) // this is the point number, we don't want it
    w1= StringFromList(1,S_waveNames)
    w2= StringFromList(2,S_waveNames)
    w3= StringFromList(3,S_waveNames)
    w4= StringFromList(4,S_waveNames)
    // put the name of waves into string variables
    if (strlen(w0) != 0) // w0 exists
        print("w0 ok!")
        w0= StringFromList(0,S_waveNames) 
        killwaves $w0 // this is the point number, we don't want it
    endif
    if (strlen(w1) != 0) // w1 exists
        print("w1 ok!")
        w1= StringFromList(1,S_waveNames)
        newname1=w1+"_" + num2str(tracenum)
        //newname1="data_1_" + num2str(tracenum)
        duplicate/o $w1, $newname1
        killwaves $w1
        // set scale to volts       
        SetScale/P x start,increment,"s", $newname1
        SetScale d 0,0,"V", $newname1
    endif
    if (strlen(w2) != 0) // w2 exists
        print("w2 ok!")
        w2= StringFromList(2,S_waveNames)
        newname2=w2+"_" + num2str(tracenum)
        //newname2="data_2_" + num2str(tracenum)
        duplicate/o $w2, $newname2
        killwaves $w2
        // set scale to volts       
        SetScale/P x start,increment,"s", $newname2
        SetScale d 0,0,"V", $newname2
    endif
    if (strlen(w3) != 0) // w3 exists
        print("w3 ok!")
        w3= StringFromList(3,S_waveNames)
        newname3=w3+"_" + num2str(tracenum)
        //newname3="data_3_" + num2str(tracenum)
        duplicate/o $w3, $newname3
        killwaves $w3
        // set scale to volts
        SetScale/P x start,increment,"s", $newname3
        SetScale d 0,0,"V", $newname3
    endif
    if (strlen(w4) != 0) // w4 exists
        print("w4 ok!")
        w4= StringFromList(4,S_waveNames)
        //newname4="data_4_" + num2str(tracenum)
        newname4=w4+"_" + num2str(tracenum)
        duplicate/o $w4, $newname4
        killwaves $w4
        SetScale/P x start,increment,"s", $newname4
        SetScale d 0,0,"V", $newname4
        // set scale to volts       
    endif
EndMacro

 

July 26, 2024 at 09:39 pm - Permalink

I wonder, did you look at what hrodstein has provided? Did it not work for you? It seems a very neat solution to me... Also, macros are very old tech and should be avoided in favor of functions.

Anyway, here is a code fragment which wraps your manual assignment into a simple loop. Things could be done much more neatly (see Howards solution), but if you just want to quickly modify what you already have...

... code until the main load command ...
 
    LoadWave/D/J/O/L={0,2,0,0,3}/W/A/Q /p=my_path fileName
    if (V_flag==0)
        return
    endif
    Variable i
    for (i=1; i<ItemsInList(S_waveNames);i+=1)
        string currW = StringFromList(i,S_waveNames)
        string Wname = currW + "_" + num2str(tracenum)
        Duplicate/o $currW, $Wname
        KillWaves $currW
        SetScale/P x start,increment,"s", $Wname
        SetScale d 0,0,"V", $Wname
    endfor
 
... whatever else you want to do ...

 

July 27, 2024 at 09:15 pm - Permalink

 

minghaoj wrote:

 

   LoadWave/D/J/O/L={0,2,0,0,3}/W/A/Q /p=my_path fileName


Agree to chozo's take. It really comes down to the /L={0,2,0,0,3}. The 3 dictates how many columns you are loading (in your case the X, CH2, and CH3). You can increase this number accordingly if you have additional channels of data. Since your meta data (Start and Increment) are in the first rows, you won't have issue loading your data.

I am guessing your question is how to have your function be versatile to handle things beyond (the repetitive chunks of code for) w0 to w4??

Personally, for data like this, I prefer to load them into a 2D wave matrix by adding /M to LoadWave and go from there.

Another option would be creating a function that does your post loading processing and then feed the waves generated to that function in a loop. I see you already found the list of names of waves are stored in S_waveNames. You can use ItemsInList to determine the number of channels/waves loaded. Then use a For loop to repeat the  process.

int i, numWaves
string wvName, newWvName
numWaves = itemsinlist(S_waveNames)
for(i =0; i< numWaves;i++)
    wvName= StringFromList(i,S_waveNames)
    newWvName = wvName + "_" + num2str(tracenum)
    duplicate/o $wvName, $newWvName
    killwaves $wvName
    // set scale to volts
    SetScale/P x start,increment,"s", $newWvName
    SetScale d 0,0,"V", $newWvName
endfor

Personally, I would change the order of operation as follow and use name the waves differently (you can also use the LoadWave/Name added in Igor Pro 9 or later to handle your wave name changing. Note to myself, start using this).

int i, numWaves
string loadedWvName, wvName
numWaves = itemsinlist(S_waveNames)
for(i =0; i< numWaves;i++)
    loadedWvName= StringFromList(i,S_waveNames)
    // set scale to volts
    SetScale/P x start,increment,"s", $loadedWvName
    SetScale d 0,0,"V", $loadedWvName
    //Chose one below
    //If you prefer the same channel of data from different files are grouped together in Data Browser use this line
    wvName = "CH" + num2str(i+1) + "_" + num2str(tracenum)
    //If you prefer data of different channels from the same data file are grouped together in Data Browser use this line.
    wvName = "_" + num2str(tracenum) + "_CH" + num2str(i+1) //Add "_" in the front as you can't start a wave name using a number, unless you are doing liberal name. The next line of code will fix this for you anyways though.
    //It is always important to verify your wave name is valid. 
    //Alternatively, you can use CreateDataObjectName for more options only if you are using Igor Pro 9.0 or later.
    wvName = CleanupName(wvName, 0) 
    duplicate/o $loadedWvName, $wvName
    killwaves $loadedWvName
endfor

Unrelated, yet related. I have created loaders for several different types of data files with meta data in it. The solution is similar to what hrodstein provided here. It involves a combination of FReadline and LoadWave (sometimes multiple of each).

Please correct me if I am wrong about this. One pet peeve of mine is that LoadWave/L seems to be limited by the number of columns in the firstLine of data which overrides the numColumns parameter if the later is larger. I am sure there are underlying reason why it does this, but part of me hope we can have an option to override that.

The situation is the worst when multiple chunks of data are stored in the same file with their individual meta data. Doing multiple loops of LoadWave/L can be very slow. FReadline appears to be faster (as the file is only opened once) but requires additional post loading wave handling. Some times I can try loading the entire file into one wave and separate each chunk out. It gets messy if the chunk sizes are dynamic.

July 30, 2024 at 08:37 am - Permalink

One pet peeve of mine is that LoadWave/L seems to be limited by the number of columns in the firstLine of data which overrides the numColumns parameter if the later is larger.

That's correct. It is a limitation that I have run into myself on occasion.

I am sure there are underlying reason why it does this

The underlying reason is that I didn't think of this situation when I wrote this code (about 35 years ago ;)

but part of me hope we can have an option to override that.

I think a flag that says "don't automically determine the number of columns in the file from the first line of data, follow /L instead" is doable.

 

August 1, 2024 at 10:18 am - Permalink

hrodstein wrote:

I think a flag that says "don't automically determine the number of columns in the file from the first line of data, follow /L instead" is doable.

 

That will be cool. I am looking forward to that feature in future updates.

August 6, 2024 at 12:22 pm - Permalink