Automating Analysis

Hey everyone,

I've got a couple questions. I'm trying to automate the analysis of multiple data sets within the same file. For each data set I need to average 200 traces, display the graph so I can manually check that they don't look wrong, and then bin the data points into 8 bins, find the variation about the mean within each bin, and display these 8 numbers in a final table. I've figured out how to make and analyze a graph for an individual data set, but can't seem to find a way to automate it. I could just copy and paste a repeat of the same commands with the number changed each time to designate a new data set, as I show below (the first set of commands makes waves tagged with the number 1 to correspond to data set 1, the second waves tagged with 2, etc...) but I figured there's probably a more eloquent way to do this. Especially since I have around 40 data sets per file, and a dozen files to analyze. I tried using a variable Series=1 and then making a loop to make Series increase by 1 with each loop, but it error'd on me. :/ If anyone has any suggestions I'd appreciate it.

As an appended question: my files contain data sets from multiple types of experiments. Over half of the data sets in a single file aren't going to be analyzed by this method. If there is a way to automate the data set chosen, can I also somehow tell it to only analyze certain data sets, like set 3, 6, 8, 9, 10, 15...?


fWaveAverage(wavelist("PMPulse_1_1*_Adc-1", ";",""), "",1, 1, "AverageWaves1", "Average_SD1") //make an averaged wave of all traces that start with "PMPulse_1_1" and end with "_Adc-1".
Display AverageWaves1 //display the new averaged wave, can check by eye that it doesn't look wonky
Duplicate/O /R=(0.02005, 0.06) AverageWaves1,Points1               //copies the "AverageWave#", and narrows region of interest to period of stimulation (in this case, 20.05ms-60ms). Saves as new wave called "Points#"
Redimension/N=(100,8) Points1     // Set the number of rows and columns in the wave "Points#", creating 8 columns with 100 points of data each (which is all of the data points for each doublet)
MatrixOP/O Variation1=VarCols(Points1) //creates a new wave called "Variation#" which takes the variation about the mean for each of the 8 bins in "Points#." VarCols only creates one dimensional waves, so no new lines can be added to it.

fWaveAverage(wavelist("PMPulse_1_2*_Adc-1", ";",""), "",1, 1, "AverageWaves2", "Average_SD2")
Display AverageWaves2
Duplicate/O /R=(0.02005, 0.06) AverageWaves2,Points2              
Redimension/N=(100,8) Points2    
MatrixOP/O Variation2=VarCols(Points2)



Okay, so my second question: I use VarCols in MatrixOP. The problem with VarCols is, it only creates a 1-dimensional wave. If possible, I'd like to just put all my variation numbers into a single table, the first row being from data set 1, the second from data set 2, etc. Should I just create a new wave at the end that consolidates all of these "Variation#" files? Or is there a better way to do it?

btw I'm starting to learn some of the language, but I have no experience programming. Be gentle with me.

Thanks!
ItalliaVP wrote:
automate it. I could just copy and paste a repeat of the same commands with the number changed each time to designate a new data set, as I show below (the first set of commands makes waves tagged with the number 1 to correspond to data set 1, the second waves tagged with 2, etc...) but I figured there's probably a more eloquent way to do this. Especially since I have around 40 data sets per file, and a dozen files to analyze. I tried using a variable Series=1 and then making a loop to make Series increase by 1 with each loop, but it error'd on me. :/ If anyone has any suggestions I'd appreciate it.


you can construct wavenames as strings within the loop and create wave references using $:

string s_newAverage, s_newSD, s_data
variable series=0
do
   series+=1
   sprintf s_newAverage "AverageWaves%d", series
   sprintf s_newSD "Average_SD%d", series
   sprintf s_data "PMPulse_1_%d*_Adc-1", series
   fWaveAverage(wavelist(s_data, ";",""), "",1, 1, s_newAverage, s_newSD)
   wave w_average=$s_newAverage
   wave w_SD=$s_newSD
   // and so on
   
while (not finished)

ItalliaVP wrote:

Okay, so my second question: I use VarCols in MatrixOP. The problem with VarCols is, it only creates a 1-dimensional wave. If possible, I'd like to just put all my variation numbers into a single table, the first row being from data set 1, the second from data set 2, etc. Should I just create a new wave at the end that consolidates all of these "Variation#" files? Or is there a better way to do it?


This is fine. If you store each of your variation waves using unique names (as advised above) you just need to combine them at the end.
So VarCols will give a 1 row 8 col 2D wave for your data. Use the Concatenate command with /NP=0 to make the matrix you want. Something like Concatenate/O/NP=0 WaveList("Variation*",";",""), myMatrix. Use the /KILL flag if you want to dispose of them all at the end but keep your matrix.
tony wrote:
ItalliaVP wrote:
automate it. I could just copy and paste a repeat of the same commands with the number changed each time to designate a new data set, as I show below (the first set of commands makes waves tagged with the number 1 to correspond to data set 1, the second waves tagged with 2, etc...) but I figured there's probably a more eloquent way to do this. Especially since I have around 40 data sets per file, and a dozen files to analyze. I tried using a variable Series=1 and then making a loop to make Series increase by 1 with each loop, but it error'd on me. :/ If anyone has any suggestions I'd appreciate it.


you can construct wavenames as strings within the loop and create wave references using $:

string s_newAverage, s_newSD, s_data
variable series=0
do
   series+=1
   sprintf s_newAverage "AverageWaves%d", series
   sprintf s_newSD "Average_SD%d", series
   sprintf s_data "PMPulse_1_%d*_Adc-1", series
   fWaveAverage(wavelist(s_data, ";",""), "",1, 1, s_newAverage, s_newSD)
   wave w_average=$s_newAverage
   wave w_SD=$s_newSD
   // and so on
   
while (not finished)


Thanks! So I think I understand what the code is doing right up until the "wave" commands--what is the purpose of those? I can't find "wave" in the Igor Help Browser, nor can I find "do". Maybe if I know the purpose of wave and do (and while), I can figure out what's supposed to come next in the code. At the moment I'm not seeing the way forward in completing my analysis.

EDIT: I just found the do-while and wave in the Programming section. I think "wave" is used because Igor wouldn't have recognized my data as waves, so we use the wave command to tell Igor that these are waves? That would make sense, but I still don't understand what I'm supposed to put after while. What is expression? If I just copy and paste the command lines above, it errors at the "do" step.
sjr51 wrote:
ItalliaVP wrote:

Okay, so my second question: I use VarCols in MatrixOP. The problem with VarCols is, it only creates a 1-dimensional wave. If possible, I'd like to just put all my variation numbers into a single table, the first row being from data set 1, the second from data set 2, etc. Should I just create a new wave at the end that consolidates all of these "Variation#" files? Or is there a better way to do it?


This is fine. If you store each of your variation waves using unique names (as advised above) you just need to combine them at the end.
So VarCols will give a 1 row 8 col 2D wave for your data. Use the Concatenate command with /NP=0 to make the matrix you want. Something like Concatenate/O/NP=0 WaveList("Variation*",";",""), myMatrix. Use the /KILL flag if you want to dispose of them all at the end but keep your matrix.


Thanks, that worked perfectly!
ItalliaVP wrote:


EDIT: I just found the do-while and wave in the Programming section. I think "wave" is used because Igor wouldn't have recognized my data as waves, so we use the wave command to tell Igor that these are waves? That would make sense, but I still don't understand what I'm supposed to put after while. What is expression? If I just copy and paste the command lines above, it errors at the "do" step.


do... while(boolean) will loop while the expression in parentheses is true (i.e., while boolean is not equal to 0). So

variable count=0
do
   count+=1
while (count<100)


will loop 100 times

You can also loop using for() ... endfor. See the help for more details.
Quote:
do... while(boolean) will loop while the expression in parentheses is true (i.e., while boolean is not equal to 0). So

variable count=0
do
   count+=1
while (count<100)


will loop 100 times

You can also loop using for() ... endfor. See the help for more details.


I tried this

string s_newAverage, s_newSD, s_data //designates each of these as strings
variable series=0 //makes a variable called "series" set equal to zero
do
   series+=1 //makes the variable "series" increase by 1 number for each loop
   sprintf s_newAverage "AverageWaves%d", series //sprintf acts like printf operation, except instead of printing to the history, it will print to the above string,
                        //with %d being replaced by the series, which is a list of numbers increasing in increments of 1
   sprintf s_newSD "Average_SD%d", series
   sprintf s_data "PMPulse_1_%d*_Adc-1", series //anytime you need any of the green quoted things, just type the first line after sprintf with a $ in front to do it to all.
   fWaveAverage(wavelist(s_data, ";",""), "",1, 1, s_newAverage, s_newSD)
   wave w_average=$s_newAverage
   wave w_SD=$s_newSD

while (series<100)


but it still errors on the "do" step.
ItalliaVP][quote wrote:
do... while(boolean) will loop while the expression in parentheses is true (i.e., while boolean is not equal to 0). So


I tried this

string s_newAverage, s_newSD, s_data //designates each of these as strings
variable series=0 //makes a variable called "series" set equal to zero
do
   series+=1 //makes the variable "series" increase by 1 number for each loop
   sprintf s_newAverage "AverageWaves%d", series //sprintf acts like printf operation, except instead of printing to the history, it will print to the above string,
                        //with %d being replaced by the series, which is a list of numbers increasing in increments of 1
   sprintf s_newSD "Average_SD%d", series
   sprintf s_data "PMPulse_1_%d*_Adc-1", series //anytime you need any of the green quoted things, just type the first line after sprintf with a $ in front to do it to all.
   fWaveAverage(wavelist(s_data, ";",""), "",1, 1, s_newAverage, s_newSD)
   wave w_average=$s_newAverage
   wave w_SD=$s_newSD

while (series<100)


but it still errors on the "do" step.


Other than the lack of a fWaveAverage function, I don't see a problem. I wrapped your code above in a function and commented out the fWaveAverage call and added a statement to print the strings in the loop:

Function fTest()
    string s_newAverage, s_newSD, s_data //designates each of these as strings
    variable series=0 //makes a variable called "series" set equal to zero
    do
       series+=1 //makes the variable "series" increase by 1 number for each loop
       sprintf s_newAverage "AverageWaves%d", series //sprintf acts like printf operation, except instead of printing to the history, it will print to the above string,
                            //with %d being replaced by the series, which is a list of numbers increasing in increments of 1
       sprintf s_newSD "Average_SD%d", series
       sprintf s_data "PMPulse_1_%d*_Adc-1", series //anytime you need any of the green quoted things, just type the first line after sprintf with a $ in front to do it to all.
    //   fWaveAverage(wavelist(s_data, ";",""), "",1, 1, s_newAverage, s_newSD)
       wave w_average=$s_newAverage
       wave w_SD=$s_newSD
    print s_newAverage, s_NewSD, s_data
     
    while (series<10)
End


prints:

AverageWaves1 Average_SD1 PMPulse_1_1*_Adc-1
AverageWaves2 Average_SD2 PMPulse_1_2*_Adc-1
AverageWaves3 Average_SD3 PMPulse_1_3*_Adc-1
AverageWaves4 Average_SD4 PMPulse_1_4*_Adc-1
AverageWaves5 Average_SD5 PMPulse_1_5*_Adc-1
AverageWaves6 Average_SD6 PMPulse_1_6*_Adc-1
AverageWaves7 Average_SD7 PMPulse_1_7*_Adc-1
AverageWaves8 Average_SD8 PMPulse_1_8*_Adc-1
AverageWaves9 Average_SD9 PMPulse_1_9*_Adc-1
AverageWaves10 Average_SD10 PMPulse_1_10*_Adc-1

Runs on both Igor 7.05 and Igor 6.38 on Win7.
jtigor wrote:


Other than the lack of a fWaveAverage function, I don't see a problem. I wrapped your code above in a function and commented out the fWaveAverage call and added a statement to print the strings in the loop:

Function fTest()
    string s_newAverage, s_newSD, s_data //designates each of these as strings
    variable series=0 //makes a variable called "series" set equal to zero
    do
       series+=1 //makes the variable "series" increase by 1 number for each loop
       sprintf s_newAverage "AverageWaves%d", series //sprintf acts like printf operation, except instead of printing to the history, it will print to the above string,
                            //with %d being replaced by the series, which is a list of numbers increasing in increments of 1
       sprintf s_newSD "Average_SD%d", series
       sprintf s_data "PMPulse_1_%d*_Adc-1", series //anytime you need any of the green quoted things, just type the first line after sprintf with a $ in front to do it to all.
    //   fWaveAverage(wavelist(s_data, ";",""), "",1, 1, s_newAverage, s_newSD)
       wave w_average=$s_newAverage
       wave w_SD=$s_newSD
    print s_newAverage, s_NewSD, s_data
     
    while (series<10)
End


prints:

AverageWaves1 Average_SD1 PMPulse_1_1*_Adc-1
AverageWaves2 Average_SD2 PMPulse_1_2*_Adc-1
AverageWaves3 Average_SD3 PMPulse_1_3*_Adc-1
AverageWaves4 Average_SD4 PMPulse_1_4*_Adc-1
AverageWaves5 Average_SD5 PMPulse_1_5*_Adc-1
AverageWaves6 Average_SD6 PMPulse_1_6*_Adc-1
AverageWaves7 Average_SD7 PMPulse_1_7*_Adc-1
AverageWaves8 Average_SD8 PMPulse_1_8*_Adc-1
AverageWaves9 Average_SD9 PMPulse_1_9*_Adc-1
AverageWaves10 Average_SD10 PMPulse_1_10*_Adc-1

Runs on both Igor 7.05 and Igor 6.38 on Win7.


Thanks, for some reason it lets me use that when I copy and paste yours, but not when I go edit my own to look just like it. I must have a slight difference that I can't find.

Okay so that all works, but how do I refer to each averaged wave when I'm trying to do the final commands, i.e. all the commands starting with Display AverageWaves and the ending with the MatrixOP. I've tried referring to these as "AverageWaves%d" or "w_average" or "$s_newAverage" but it won't accept any of these...so I still have to do the last few lines by entering each number manually, like Display AverageWaves1, Display AverageWaves2, etc.
ItalliaVP wrote:


Okay so that all works, but how do I refer to each averaged wave when I'm trying to do the final commands, i.e. all the commands starting with Display AverageWaves and the ending with the MatrixOP. I've tried referring to these as "AverageWaves%d" or "w_average" or "$s_newAverage" but it won't accept any of these...so I still have to do the last few lines by entering each number manually, like Display AverageWaves1, Display AverageWaves2, etc.


In each loop you make waves with names like AverageWaves7.
s_newAverage contains the name as a string.
the line
wave w_average=$s_newAverage
sets w_average to reference the wave that was created in the loop.

So, you need to use w_average in place of the wave name.
for example,
print nameofwave(w_average)
should print "AverageWaves7" (or whatever).

For more info:
displayHelpTopic "wave"
Finally got it to work--thanks everyone! MatrixOp doesn't allow operators like $ so I had to just make a new "Points" wave for each loop that overwrote the old one, but that's fine because I really want the Variance and those were saved for each data set and then concatenated at the end. Here's the final product:

Function Var()
  string s_newAverage, s_newSD, s_data, s_variance
  variable series=0
    do
       series+=1
       sprintf s_newAverage "AverageWaves%d", series
       sprintf s_newSD "Average_SD%d", series
       sprintf s_data "PMPulse_1_%d*_Adc-1", series
       sprintf s_variance "Variance%d", series          //Anytime you need any of the green quoted things,
                                    //type $s_whatever in place of the wave name.
       fWaveAverage(wavelist(s_data, ";",""), "",1, 1, s_newAverage, s_newSD)
       wave w_average=$s_newAverage
       wave w_SD=$s_newSD
       Display/k=1 $s_newAverage
       Duplicate/O /R=(0.02005, 0.06) $s_newAverage,Points
       Redimension/N=(100,8) Points
       MatrixOP/O $s_variance=VarCols(Points)
    while (series<32)                       //Change to the last series number you want to analyze.
    Concatenate/O/KILL/NP=0 WaveList("Variance*",";",""), FinalVariance
End