Load and process waves

Share icon

Hello

I try to write a procedure to do:

1 - load all the .CSV file located in a folder (or .txt Space-Delimited Text File)

2 - There is three column In each file, so i have to put the name of the file for the first one (eg fileName), the second : fileName_PM1 and third : fileName_PM2

3- Make the sum (fileName_PM1 + fileName_PM2) and put the result in a new wave fileName_Sum.

For now I managed to achieve the first two points, but for the third one I could not find the correct way to do it !!!! problems with the names of wave / wave reference / string ... ( All the commented part in the end the procedure)

I know this is a little problem but...

is ther a simple solution/function to do the third point ?

Here is the procedure:

Function importAllFiles()
 
 
    Variable formatSelector // selector of the file format : 1 for .csv format and  2 for .txt format
    String fileFormat = ""
    Variable  firstLine = 2 // start to load from the line number 2 to skip header 
    Variable  numColumns = 3 //  the number of columns to load.
    
    String extension= "*.csv; *.txt;"
    
    String separator = ""
    
    Prompt formatSelector, "Sélectionnez un format de fichier", popup extension
    
    DoPrompt " File Format ", formatSelector
    if (V_Flag)
        return -1                               // User canceled
    endif
    
        
    if ( formatSelector == 1) // 1 for .csv format
    
        fileFormat = ".csv" 
        separator = ";"                 
        
    elseif (formatSelector == 2)  // 2 for .txt format
    
        fileFormat = ".txt" 
        separator = "   "   // Space-Delimited Text File            
    
    endif
    
    String messageSt = " Emplacement des fichiers " + fileFormat+ " ? "  //  put the file format in the prompt message.
          
    NewPath/M=messageSt/O PathName  // get the folder in which the files are
     
    if (V_flag == -1)   //User cancelled dialog
        return -1
    endif
     
    string ListOfFiles = IndexedFile(PathName, -1, fileFormat)  // Get list of *.csv files in folder
        
    Variable  nbrOfFiles = itemsinlist (ListOfFiles ) // number of files in the folder 
        
    String fileToLoad   // name of the file to load
    variable i
        
    String columnInfo = ""
            
    Edit/N=Traitement // Ceat a Table "Traitement" and append the loaded waves to this table
        
    for( i = 0; i<nbrOfFiles; i=i+1)    
            
            fileToLoad = StringFromList(i, ListOfFiles) 
                    
            if(strlen(fileToLoad)==0)   // no more file to import
                break
            endif                   // Condition;update loop variables
        
            String name = RemoveEnding(fileToLoad, fileFormat ) // remove the .csv from file name
                
            columnInfo +=  "N='" + name + "';" //  take the name of the file for the firts column
        
            columnInfo += "N=" + name + "_PM1;"    // add _PM1 for the second column
        
            columnInfo += "N=" + name + "_PM2;"    // add _PM2 for the second column
                
            LoadWave/a/E=2/d/j/b=columnInfo/L={0, firstLine, 0, 0, numColumns }/V={separator, "$", 0, 0 }/P=PathName fileToLoad //  
            
                if (V_flag==0)              // No waves loaded. Perhaps user canceled.
                    return -1
                endif
            
 
            
//          wave waveSum = $(name + "_Sum")
    
//          string waveNameSum  = name + "_Sum"
//          
//          string waveNamePM1 = name + "_PM1" 
//          
//          string waveNamePM2 = name + "_PM2" 
//          
//          Make/O $waveNameSum
//          
//          Make/O $waveNamePM1
//          
//          Make/O $waveNamePM2
//          
//          wave waveSum = $waveNameSum
//          
//          wave wavePM1 = $waveNamePM1
//          
//          wave wavePM2 = $waveNamePM2
//          
//          waveSum = wavePM1 + wavePM2
            
            columnInfo = "" 
                                
//          Wave/Z waveSum= $(name + "_Sum")
//          
//          Wave/Z wavePM1= $( name + "_PM1")
//          
//          Wave/Z wavePM2= $(name + "_PM2")
//          
//          waveSum = wavePM1 + wavePM2
 
                
    endfor                                              
    
 
 
 
    
    
end
 
 
 <pre><code class="language-igor">
After LoadWave, do this:

String listOfWavesLoaded = S_waveNames      // S_waveNames is created by LoadWave
String firstName = StringFromList(0, listOfWavesLoaded)
Wave firstWave = $firstName
String secondName = StringFromList(1, listOfWavesLoaded)
Wave secondWave = $secondName
String thirdName = StringFromList(2, listOfWavesLoaded)
Wave thirdWave = $thirdName


Now you can use the wave references in subsequent commands.

March 16, 2016 at 10:37 am - Permalink

Thanks a lot hrodstein,

After a few tries...it works.

That may not be the best way to do. The concept of wavename and wave reference is not easy to figure out....

One more question, is there any tips to speed up loading very large files?

many thanks

And here is the new procedure :

 
Function importAllFiles()
 
 
    Variable formatSelector // selector of the file format : 1 for .csv format and  2 for .txt format
    String fileFormat = ""
    Variable  firstLine = 2 // start to load from the line number 2 to skip header 
    Variable  numColumns = 3 //  the number of columns to load.
    
    String extension= "*.csv; *.txt;"
    
    String separator = ""
    
    Prompt formatSelector, "Sélectionnez un format de fichier", popup extension
    
    DoPrompt " File Format ", formatSelector
    if (V_Flag)
        return -1                               // User canceled
    endif
    
        
    if ( formatSelector == 1) // 1 for .csv format
    
        fileFormat = ".csv" 
        separator = ";"                 
        
    elseif (formatSelector == 2)  // 2 for .txt format
    
        fileFormat = ".txt" 
        separator = "   "   // Space-Delimited Text File            
    
    endif
    
    String messageSt = " Emplacement des fichiers " + fileFormat+ " ? "  //  put the file format in the prompt message.
          
    NewPath/M=messageSt/O PathName  // get the folder in which the files are
     
    if (V_flag == -1)   //User cancelled dialog
        return -1
    endif
     
    string ListOfFiles = IndexedFile(PathName, -1, fileFormat)  // Get list of *.csv files in folder
        
    Variable  nbrOfFiles = itemsinlist (ListOfFiles ) // number of files in the folder 
        
    String fileToLoad   // name of the file to load
    variable i
        
    String columnInfo = ""
            
    Edit/N=Traitement // Ceat a Table "Traitement" and append the loaded waves to this table
        
    for( i = 0; i<nbrOfFiles; i=i+1)    
            
            fileToLoad = StringFromList(i, ListOfFiles) 
                    
            if(strlen(fileToLoad)==0)   // no more file to import
                break
            endif                   // Condition;update loop variables
        
            String name = RemoveEnding(fileToLoad, fileFormat ) // remove the .csv from file name
                
            columnInfo +=  "N='" + name + "';" //  take the name of the file for the firts column
        
            columnInfo += "N=" + name + "_PM1;"    // add _PM1 for the second column
        
            columnInfo += "N=" + name + "_PM2;"    // add _PM2 for the second column
                
            LoadWave/a/E=2/d/j/b=columnInfo/L={0, firstLine, 0, 0, numColumns }/V={separator, "$", 0, 0 }/P=PathName fileToLoad //  
            
                if (V_flag==0)              // No waves loaded. Perhaps user canceled.
                    return -1
                endif
            
            String listOfWavesLoaded = S_waveNames      // S_waveNames is created by LoadWave
            
            String firstName = StringFromList(0, listOfWavesLoaded)
            
            duplicate/O $firstName waveSum
            
            String secondName = StringFromList(1, listOfWavesLoaded)
            Wave secondWave = $secondName
            
            String thirdName = StringFromList(2, listOfWavesLoaded)
            Wave thirdWave = $thirdName
            
            waveSum = secondWave + thirdWave
            
            string newName =  name + "_Sum" 
            
            Duplicate/O waveSum, $newName; KillWaves waveSum
            
            columnInfo = "" 
                
    endfor                                              
    
 
end

March 17, 2016 at 01:39 am - Permalink

You may be able speed up things a bit (depending on how big you data is) by removing the last duplicate, which is unnecessary. Duplicate eats memory and processing power. It is always better to use what is already there.
    for( i = 0; i<nbrOfFiles; i=i+1)    
            fileToLoad = StringFromList(i, ListOfFiles) 
            if(strlen(fileToLoad)==0)   // no more file to import
                break
            endif                   // Condition;update loop variables
 
            String name = RemoveEnding(fileToLoad, fileFormat ) // remove the .csv from file name
 
            columnInfo += "N='" + name + "';" //  take the name of the file for the firts column
            columnInfo += "N=" + name + "_PM1;"    // add _PM1 for the second column
            columnInfo += "N=" + name + "_PM2;"    // add _PM2 for the second column
 
            LoadWave/a/E=2/d/j/b=columnInfo/L={0, firstLine, 0, 0, numColumns }/V={separator, "$", 0, 0 }/P=PathName fileToLoad //  
            if (V_flag==0)              // No waves loaded. Perhaps user canceled.
                return -1
            endif
 
            String listOfWavesLoaded = S_waveNames      // S_waveNames is created by LoadWave
            String newName = name + "_Sum"
 
            duplicate/O $StringFromList(0, listOfWavesLoaded) $newName
            Wave waveSum = $newName
            Wave secondWave = $StringFromList(1, listOfWavesLoaded)
            Wave thirdWave = $StringFromList(2, listOfWavesLoaded)
 
            waveSum = secondWave + thirdWave
 
            columnInfo = "" 
    endfor 

March 17, 2016 at 03:59 am - Permalink

Thank you chozo for this optimization. My files can be up to some hundred of MB !

March 17, 2016 at 05:44 am - Permalink

Quote:
is there any tips to speed up loading very large files?


See the section "Loading Very Large Waves" in the help for the LoadWave operation.

Using a solid state hard drive will probably help a lot.

March 17, 2016 at 07:46 am - Permalink

The first duplicate is also not needed, or?
Pass the wavename to LoadWave with /N and tell it to overwrite with /O.

March 17, 2016 at 09:05 am - Permalink

Isn't "loading very large files" already taken care of by the parameter /L={0, firstLine, 0, 0, numColumns } in loadwave or am I missing something?
HJ

March 17, 2016 at 05:15 pm - Permalink

@ HJDrescher : that's what I was wondering and that's why I asked for another tip !!!

@ thomas_braun : I did not really understand what you mean.

March 18, 2016 at 12:43 am - Permalink

Destination wave names can be passed to LoadWave via the /B flag. Syntax is a little bulky: have a look at the /B flag in the help file (and the N parameter therein).
Here's an example for two single precision columns: /B=("C=1,F=0,N=CalData; C=1,F=0,N=QMSData;")

HJ
@TB You also got me at first with the /N flag!

March 18, 2016 at 03:08 am - Permalink

Okay, okay. I should have posted a full fledged example ;)

LoadWave/J/N=wave0/O "C:Users:thomas:Desktop:34561_01.csv"


gives

Delimited text load from "34561_01.csv" Data length: 40, waves: wave00, wave01, wave02, wave03, wave04, wave05, wave06, wave07 and: wave08, wave09, wave010, wave011, wave012, wave013, wave014, wave015, wave016 and: wave017, wave018, wave019, wave020, wave021, wave022, wave023, wave024, wave025 and: wave026, wave027, wave028, wave029, wave030, wave031, wave032, wave033, wave034 and: wave035, wave036, wave037, wave038, wave039, wave040, wave041, wave042, wave043 and: wave044, wave045, wave046, wave047, wave048, wave049, wave050, wave051, wave052 and: wave053, wave054, wave055, wave056, wave057, wave058, wave059, wave060, wave061 and: wave062, wave063, wave064, wave065, wave066, wave067, wave068, wave069, wave070 and: wave071, wave072, wave073, wave074, wave075, wave076, wave077, wave078, wave079 and: wave080, wave081, wave082, wave083, wave084, wave085, wave086, wave087, wave088 and: wave089, wave090, wave091, wave092, wave093, wave094, wave095, wave096, wave097 and: wave098, wave099, wave0100, wave0101, wave0102, wave0103, wave0104, wave0105 and: wave0106, wave0107, wave0108, wave0109, wave0110, wave0111, wave0112, wave0113 and: wave0114, wave0115, wave0116, wave0117, wave0118, wave0119, wave0120, wave0121 and: wave0122, wave0123, wave0124, wave0125, wave0126, wave0127, wave0128, wave0129 and: wave0130, wave0131, wave0132, wave0133, wave0134, wave0135, wave0136, wave0137 and: wave0138, wave0139, wave0140

This requires no /B fiddling.

March 18, 2016 at 04:16 am - Permalink

To test my procedure I am trying to load large file.
I have a file that contains a bunch of data about 1.5GB! when I try to load it, charging starts and after a moment : error LoadWave "out of memory" !
what's wrong ?

Here is the procedure :

Function importAllFiles()
 
 
    Variable formatSelector // selector of the file format : 1 for .csv format and  2 for .txt format
    String fileFormat = ""
    Variable  firstLine = 2 // start to load from the line number 2 to skip header 
    Variable  numColumns = 3 //  the number of columns to load.
    
    String extension= "*.csv; *.txt;"
    
    String separator = ""
    
    Prompt formatSelector, "Sélectionnez un format de fichier", popup extension
    
    DoPrompt " File Format ", formatSelector
    if (V_Flag)
        return -1                               // User canceled 
    endif
    
        
    if ( formatSelector == 1) // 1 for .csv format
    
        fileFormat = ".csv" 
        separator = ";"                 
        
    elseif (formatSelector == 2)  // 2 for .txt format
    
        fileFormat = ".txt" 
        separator = "   "   // Space-Delimited Text File            
    
    endif
    
    String messageSt = " Emplacement des fichiers " + fileFormat+ " ? "  //  put the file format in the prompt message.
          
    NewPath/M=messageSt/O PathName  // get the folder in which the files are
     
    if (V_flag == -1)   //User cancelled dialog
        return -1
    endif
     
    string ListOfFiles = IndexedFile(PathName, -1, fileFormat)  // Get list of *.csv files in folder
        
    Variable  nbrOfFiles = itemsinlist (ListOfFiles ) // number of files in the folder 
        
    String fileToLoad   // name of the file to load
    variable i
        
    String columnInfo = ""
            
    Edit/N=Traitement // Ceat a Table "Traitement" and append the loaded waves to this table
        
    for( i = 0; i<nbrOfFiles; i=i+1)    
            
            fileToLoad = StringFromList(i, ListOfFiles) 
                    
            if(strlen(fileToLoad)==0)   // no more file to import
                break
            endif                   // Condition;update loop variables
        
            String name = RemoveEnding(fileToLoad, fileFormat ) // remove the .csv from file name
                
            columnInfo +=  "N='" + name + "';" //  take the name of the file for the firts column
        
            columnInfo += "N=" + name + "_PM1;"    // add _PM1 for the second column
        
            columnInfo += "N=" + name + "_PM2;"    // add _PM2 for the second column
                
            LoadWave/a/E=2/d/j/b=columnInfo/L={0, firstLine, 0, 0, numColumns }/V={separator, "$", 0, 0 }/P=PathName fileToLoad //  
            
                if (V_flag==0)              // No waves loaded. Perhaps user canceled.
                    return -1
                endif
            
            String listOfWavesLoaded = S_waveNames      // S_waveNames is created by LoadWave
            
            String firstName = StringFromList(0, listOfWavesLoaded)
            
            duplicate/O $firstName waveSum
            
            String secondName = StringFromList(1, listOfWavesLoaded)
            Wave secondWave = $secondName
            
            String thirdName = StringFromList(2, listOfWavesLoaded)
            Wave thirdWave = $thirdName
            
            waveSum = secondWave + thirdWave
            
            string newName =  name + "_Sum" 
            
            Duplicate/O waveSum, $newName; KillWaves waveSum
            
            columnInfo = "" 
                
    endfor                                              
    
 
end

March 21, 2016 at 10:08 am - Permalink

Couple of things.

Avoid the last duplicate command. 
string newName =  name + "_Sum" 
duplicate/O $firstName $newName  // this replaces the first duplicate

should work.
Also have a look at the Rename command.
Is there a reason why you duplicate the fristWave and assign the sumWave as second + third?
Duplication of the second and adding the third might be faster
string newName =  name + "_Sum" // more string declarations
wave sumWave=$newName, thirdWave=$thridName
duplicate/O $secondName $newName  // again, this replaces the first duplicate
sumWave+=thirdwave


Is double precision (~15 digits) necessary or would single (~7 digits) or even integers (maybe coded ["*100"] -- not recommended per se) also work (~half the memory!)? If not, omit the /D flag.

Wave sizes are limited to 2GB in Igor6.x. Depending on your data, this limit might be reached here -- although I doubt it: Your text data would have to be more compact than the binary format in Igor -- like single text-formated bit values stored in double precision later.
Maybe the automatic calculation of the maximal number of rows in /L is causing trouble. Try a reasonable large but constant value. If the error is gone now AND all data is loaded it might be a bug.

@TB /N provides a base name. For this example, we need the /B fiddling. ;-)
"/N=baseName Same as /N except that Igor automatically assigns wave names of the form baseName0, baseName1."

HJ

March 22, 2016 at 09:29 am - Permalink

HJDrescher wrote:

@TB /N provides a base name. For this example, we need the /B fiddling. ;-)
"/N=baseName Same as /N except that Igor automatically assigns wave names of the form baseName0, baseName1."

Yes, you are right. Thanks.

March 22, 2016 at 12:41 pm - Permalink