Fast calculation of linear correlations

Share icon

Hello, 

I have to calculate linear correlation coefficients for a 2D matrix with large dimensions (e.g. 30000 rows and 2000 or more columns). This can take a while to run and I have a couple such matrices . I am wondering if there is a faster way to run correlations in IGOR. Please advise. 

Best, 

Peeyush

I don't fully understand your problem; when you say "correlation coefficients" of a matrix do you mean covariance?

If so, 

DisplayHelpTopic "MatrixOP"

and look under Functions for covariance(w).

August 14, 2024 at 09:28 am - Permalink

Hi s.r.chinn, 

Yep, I mean Pearson correlation coefficients to check how well the individual columns linearly correlate. Thanks for the tips!

Best,

Peeyush

August 15, 2024 at 02:25 pm - Permalink

I have a form of Pearson correlation in my Image Profile tools to align rows or columns. I use Correlate/AUTO for that, which seems to be reasonably fast. I also implemented fitting to determine substep offsets. Here is my code from that tool. in1 and in2 at the two columns to compare (I loop over all columns to compare against the first):

static function SpectrumCorrelationShift(wave in1, wave in2, variable sub)
    Duplicate/Free in2, work
    Correlate/AUTO in1, work
    WaveStats/Q work
    variable error = 1, result = 0
    variable optLoc = sub ? V_maxLoc : (V_maxRowLoc - x2pnt(work, 0))   // if sub-shift is active, return in scaled wave units
    if (V_sum != 0)
        if (V_maxRowLoc >= 3 && sub)
            try
                CurveFit/M=0/Q gauss, work[V_maxRowLoc-3,V_maxRowLoc+3]; AbortOnRTE
                wave W_coef
                if (numtype(W_coef[2]) == 0 && W_coef[2] > optLoc-3 && W_Coef[2] < optLoc+3)
                    result = W_coef[2]
                    error = 0
                endif
            catch
                if (V_AbortCode == -4)
                    variable cfError = GetRTError(1)
                endif
            endtry
        endif
        if (error)
            result = optLoc
        endif
    endif
    return result
end

 

August 15, 2024 at 08:23 pm - Permalink