How to iteratively process and append data to a single table

Share icon

Hello,

I am trying to create a program that will pick peaks out of a few hundred spectrums at a time. I have adapted a code utilizing the program <Peak AutoFind> to iteratively go through the list of waves without having to do each wave one by one. My problem is that not only does it make a new table for each wave, the data in each table is the data from the most recently processed wave, resulting in many copies of the same table. I would like to have the processed peaks appended to the same table so I can simply scroll through and look at each peak without clicking through multiple tables, but I cannot seem to get it working. I have included the relevant part of the code below. 

#pragma TextEncoding = "UTF-8"
#pragma rtGlobals=3    
#include <Peak AutoFind>

Function DataProcess()
    string Waves = Wavelist("*wave*",";","DIMS:1.TEXT:0,CMPLX:0")
    String xdata="_calculated_"
    Variable minPeakPercent
    Variable maxPeaks = 100
    Prompt minPeakPercent, "Minimum Peak Amplitude (% max)"
    DoPrompt"Automatically Find Peaks", minPeakPercent
    variable i
    for (i=60;i<=65;i+=1)
        WAVE/Z w= $stringfromlist(i,'Waves')
        WAVE/Z wx=$"_calculated_"
        Variable pBegin=0, pEnd= numpnts(w)-1
        Variable/C estimates= EstPeakNoiseAndSmfact(w,pBegin, pEnd)
        Variable noiselevel=real(estimates)
        Variable smoothingFactor=imag(estimates)
        Variable peaksFound= AutoFindPeaks(w,pBegin,pEnd,noiseLevel,smoothingFactor,maxPeaks)
        Wave/Z W_AutoPeakInfo
                   // Remove too-small peaks
        peaksFound= TrimAmpAutoPeakInfo(W_AutoPeakInfo,minPeakPercent/100)
        if( peaksFound >0)
            // Make waves to display in a graph
            Make/O/N=(peaksFound) WA_PeakCentersY = w[W_AutoPeakInfo[p][0]]
            AdjustAutoPeakInfoForX(W_AutoPeakInfo,w,wx)
            Make/O/N=(peaksFound) WA_PeakCentersX = W_AutoPeakInfo[p][0]

            // Show W_AutoPeakInfo in a table, with dimension labels
            SetDimLabel 1, 0, center, W_AutoPeakInfo
            SetDimLabel 1, 1, width, W_AutoPeakInfo
            SetDimLabel 1, 2, height, W_AutoPeakInfo
            CheckDisplayed/A W_AutoPeakInfo
            edit W_AutoPeakInfo.ld
            endif
   EndFor
End

A few notes:

I have tried using appendToTable instead of edit, but get the same results. I have also tried creating a table with edit on the first iteration and appendToTable on all further iterations, and still get the same results of having multiple tables.

The included iteration is very small so I don't have to close hundreds of tables on each iteration, but in practice this loop would need to iterate several hundred times.

The list it is iterating on is just the wavelist, and each wave would be imported from an excel file outputted by our analytical instrument. 

 

Any thoughts or obvious mistakes I made? Any help is greatly appreciated, Thank you!

Hmm, calling Edit on the first iteration and then using AppendToTable for successive iterations should do the trick just fine. The simplest way to do that is to check if i equals your start value and call Edit or AppendToTable otherwise. Maybe your branching code had a mistake. Could you please post your example with above variant in place so that we can have a look?

June 28, 2021 at 01:12 pm - Permalink

Another problem is that every time through the loop he's creating/overwriting the same W_AutoPeakInfo wave; there aren't multiple waves to append to a table, it is just the same one.

If you intend to record the processed peaks so you can view each set concorrently, you need separate "output" waves.

June 28, 2021 at 02:34 pm - Permalink

JimProuty wrote:

Another problem is that every time through the loop he's creating/overwriting the same W_AutoPeakInfo wave; there aren't multiple waves to append to a table, it is just the same one.

If you intend to record the processed peaks so you can view each set concorrently, you need separate "output" waves.

That is what I was thinking as well. Do you know of a good way to iteratively create new waves? I was trying to include the iteration number in the name of the wave, but couldn't figure out how to do it. 

June 29, 2021 at 09:13 am - Permalink

chozo wrote:

Hmm, calling Edit on the first iteration and then using AppendToTable for successive iterations should do the trick just fine. The simplest way to do that is to check if i equals your start value and call Edit or AppendToTable otherwise. Maybe your branching code had a mistake. Could you please post your example with above variant in place so that we can have a look?

Sure, here is version with the different first iteration, this version creates a single table that again only contains the data of the last processed wave. 

Function DataProcess()
    string Waves = Wavelist("*wave*",";","DIMS:1.TEXT:0,CMPLX:0")
    String xdata="_calculated_"
    Variable minPeakPercent
    Variable maxPeaks = 100
    Prompt minPeakPercent, "Minimum Peak Amplitude (% max)"
    DoPrompt"Automatically Find Peaks", minPeakPercent
    variable i
    variable k=0
    for (i=60;i<=65;i+=1)
        WAVE/Z w= $stringfromlist(i,'Waves')
        WAVE/Z wx=$"_calculated_"
        Variable pBegin=0, pEnd= numpnts(w)-1
        Variable/C estimates= EstPeakNoiseAndSmfact(w,pBegin, pEnd)
        Variable noiselevel=real(estimates)
        Variable smoothingFactor=imag(estimates)
        Variable peaksFound= AutoFindPeaks(w,pBegin,pEnd,noiseLevel,smoothingFactor,maxPeaks)
        Wave/Z W_AutoPeakInfo
                   // Remove too-small peaks
        peaksFound= TrimAmpAutoPeakInfo(W_AutoPeakInfo,minPeakPercent/100)
        if( peaksFound >0)
            // Make waves to display in a graph
            Make/O/N=(peaksFound) WA_PeakCentersY = w[W_AutoPeakInfo[p][0]]
            AdjustAutoPeakInfoForX(W_AutoPeakInfo,w,wx)
            Make/O/N=(peaksFound) WA_PeakCentersX = W_AutoPeakInfo[p][0]

            // Show W_AutoPeakInfo in a table, with dimension labels
            SetDimLabel 1, 0, center, W_AutoPeakInfo
            SetDimLabel 1, 1, width, W_AutoPeakInfo
            SetDimLabel 1, 2, height, W_AutoPeakInfo
            CheckDisplayed/A W_AutoPeakInfo
            If (k==0)
                edit W_AutoPeakInfo.ld
                k=1
                else   
                appendToTable W_AutoPeakInfo.ld
                endif
            endif
   EndFor
End

 

June 29, 2021 at 09:15 am - Permalink

I don't have your data to try this, but this is one way to create numbered output waves.

#include <Peak AutoFind>
Function DataProcess()
    string Waves = Wavelist("*wave*",";","DIMS:1.TEXT:0,CMPLX:0")
    String xdata="_calculated_"
    Variable minPeakPercent
    Variable maxPeaks = 100
    Prompt minPeakPercent, "Minimum Peak Amplitude (% max)"
    DoPrompt"Automatically Find Peaks", minPeakPercent
    variable i
    variable k=0
    for (i=60;i<=65;i+=1)
        WAVE/Z w= $stringfromlist(i,Waves)
        WAVE/Z wx=$"_calculated_"
        Variable pBegin=0, pEnd= numpnts(w)-1
        Variable/C estimates= EstPeakNoiseAndSmfact(w,pBegin, pEnd)
        Variable noiselevel=real(estimates)
        Variable smoothingFactor=imag(estimates)
        Variable peaksFound= AutoFindPeaks(w,pBegin,pEnd,noiseLevel,smoothingFactor,maxPeaks)
        Wave/Z W_AutoPeakInfo
        // Remove too-small peaks
        peaksFound= TrimAmpAutoPeakInfo(W_AutoPeakInfo,minPeakPercent/100)
        if( peaksFound >0)
            // Make waves to display in a graph
            Variable n = i-59 // 1, 2, 3...
            String wyName="PeakCentersY_"+num2istr(n)
            String wxName="PeakCentersX_"+num2istr(n)
            Make/O/N=(peaksFound) $wyName = w[W_AutoPeakInfo[p][0]]
            AdjustAutoPeakInfoForX(W_AutoPeakInfo,w,wx)
            Make/O/N=(peaksFound) $wxName = W_AutoPeakInfo[p][0]
            WAVE pkY=$wyName
            WAVE pkX=$wxName
            If (k==0)
                Display pkY vs pkX
            else  
                appendToGraph pkY vs pkX
            endif

            // Show a numbered copy of W_AutoPeakInfo in a table, with dimension labels
            SetDimLabel 1, 0, center, W_AutoPeakInfo
            SetDimLabel 1, 1, width, W_AutoPeakInfo
            SetDimLabel 1, 2, height, W_AutoPeakInfo
            String wInfoName="AutoPeakInfo_"+num2istr(n)
            Duplicate/O W_AutoPeakInfo, $wInfoName
            WAVE wInfo = $wInfoName
            If (k==0)
                edit wInfo.ld
                k=1
            else  
                appendToTable wInfo.ld
            endif
        endif
    EndFor
End

 

June 29, 2021 at 10:49 am - Permalink