Lispix format

Is there any possibility to import raw-files in Lispix format into Igor:

https://www.nist.gov/services-resources/software/lispix
I would like to use Igor to handle "cube" data sets acquired from electron microscopy, which would be equivalent to 3D-waves, e.g. of the size "Make/B/U/N=(500, 500, 1024) cube". Those data are generated by spectral imaging, where each pixel of an image contains an energy dispersive spectrum.

Any ideas or suggestions how to import such data would be highly appreciated!

Cheers
Christian

The file format seems clearly documented. GBLoadWave is your friend.

You could first extract the parameters from the .rpl file using, e.g., Open, FReadLine and sscanf, and then use that information as input for the GBLoadWave operation. The nomenclature ("little-endian" etc.) is exactly the same in GBLoadWave and the documentation links you posted.

Wolfgang Harneit
... should go something like this (not tested! There's a little ambiguity in the doc about "little endian" being "low byte first"; also, scaling information is lacking in the official documentation)

#pragma rtGlobals=1     // Use modern global access method.

// adds a new menu to the right of the help menu
Menu "Lispix Support"
    "Import Lispix Files"
End

// to be called by user (e.g., through menu)
function/S ImportLispixFiles()
    variable refNum
    Open/D/R/T=".rpl"/M="Select one or more ripple (.rpl) files"/MULT=1 refNum
    string filePaths = S_fileName
    if( strlen(filePaths) == 0 )
        print "user cancelled."
    else
        variable i, numFilesSelected = ItemsInList(filePaths, "\r")
        for( i = 0; i < numFilesSelected; i += 1 )
            string path = StringFromList(i, filePaths, "\r")
            ReadLispixFilePair(path)
        endfor
    endif
    return filePaths
end

// to be called by ImportLispixFiles()
function ReadLispixFilePair(ripplePath)
string ripplePath       // known to be a valid path to a ripple (.rpl) file

// 1. make sure there is an associated raw file, get its name
    string rawPath = ripplePath[0,strlen(ripplePath)-3] + "aw"
    GetFileFolderInfo/Q/Z rawPath
    if( V_Flag != 0 )
        print "no raw file found for "+ParseFilePath(0, ripplePath, ":", 1, 0)
    else
   
// 2. read ripple file, convert to keyword-value string
        variable refNum
        Open/R refNum as ripplePath
        string info = ""
        do
            string line
            FReadLine refNum, line
            if( strlen(line) == 0 ) // end of file
                break   // exit do-while loop
            endif
            if( !cmpstr(line[0],";") )  // comment line
                continue    // skip
            endif
            line = ReplaceString(" ", LowerStr(line), "")   // remove spaces and convert to lower case
            string keyStr, valStr
            sscanf line, "%s %s", keyStr, valStr
            info += keyStr + ":" + valStr + ";" // convert to Igor-style keyword-value string
        while( 1 )  // exit is through break statement
        Close refNum

// 3. extract info from keyword-value string
        variable offset = NumberByKey("offset", info), byteOrder = 0
        if( !cmpstr(StringByKey("byte-order", info), "little-endian") )
            byteOrder = 1
        endif
        variable igorType = 0, length = NumberByKey("data-length", info)
        string dataType = StringByKey("data-type", info)
        strswitch(dataType)
            case "float":
                if( length == 4 || length == 8 )
                    igorType = length/2
                endif
                break
            case "unsigned":
                igorType = 64
                // fall through
            case "signed":
                if( length == 1 || length == 2 || length == 4 )
                    igorType += 8*length
                else
                    igorType = 0
                endif
                break
        endswitch
        if( igorType == 0 )
            abort "illegal data type/length specification: "+dataType+"/"+num2str(length)
        endif
           
// 4. load raw file
        GBLoadWave /A=lispix /B=(byteOrder) /Q /S=(offset) /T={igorType,igorType} rawPath
       
// 5. report to history and rename wave to filename
        if( V_flag == 0 )
            printf "no waves"
        else
            printf S_waveNames
        endif
        print " loaded from "+S_path+S_fileName+"."
        if( V_Flag == 1 )   // should be one
            string oldName = StringFromList(0, S_waveNames)
            string newName = ParseFilePath(3, ripplePath, ":", 0, 0)
            wave oldWave = $oldName
            Duplicate/O oldWave, $newName
            KillWaves oldWave
            print oldName+" renamed to "+newName

// 6. redimension data cube
            variable width = NumberByKey("width", info)
            variable height = NumberByKey("height", info)
            variable depth = NumberByKey("depth", info)
            Redimension/N=(width, height, depth) $newName
           
// 7. set data scaling
            SetScale/P X, 0, 1, "Å", $newName
            SetScale/P Y, 0, 1, "Å", $newName
            if( !cmpstr(StringByKey("record-by", info), "vector") )
                variable Zscale = NumberByKey("ev-per-chan", info)
                SetScale/P Z, 0, Zscale, "eV", $newName
            else
                SetScale/P Z, 0, 1, "Å", $newName
            endif
   
        endif
    endif
end
harneit wrote:
... should go something like this (not tested! There's a little ambiguity in the doc about "little endian" being "low byte first"; also, scaling information is lacking in the official documentation)


Hi Wolfgang,

thanks for pointing me towards GBLoadWave! I feel quite guilty now, seeing that you wrote this code apparently between 2 and 4 a.m.!
Thanks for your effort - using this I got some test data imported straight away, although it looks different to what I expected - I will look more carefully at the parameters!

Thanks again,

Cheers
Christian
Do you get strange numbers like 6.5082e-317? then it's the byte-order logic and / or data format that is screwed up.
Wolfgang
the data range looks ok - it's more that the EDS spectra stored in the 1024 layers (I've expected to find counts per 1024 channels of the EDS detector), don't look like EDS spectra at all. Might be a problem of the data set, or the data need further conversion.... something I need to sort out.....

Cheers
Christian
The statement:
Redimension/N=(width, height, depth) $newName

assumes that the data in the file is stored in "column major order" (see http://en.wikipedia.org/wiki/Row-major_order).

For a matrix, this means that, as you sequentially step through memory, all of the elements for column 0 appear consecutively, then all of the elements for column 1, and so on. Column major order is used by Igor and, according to the wikipedia article, by Fortran and Matlab.

C and many programs use row major order. For a matrix this means that, as you sequentially step through memory, all of the elements for row 0 appear consecutively, then all of the elements for row 1, and so on.

I'm a bit unclear on how this translates to 3D.

If your data is in row major order then you will need to shuffle it to put it in the right order. You would do this by duplicating it to create a temporary wave and then using wave assignment statements to pick the data out of the temporary wave and put it in the output wave in the right order.

it seems that this is likely the problem, because the imported 3D wave shows some regular patterns such that rows, columns or layers might have been mixed up.
I will start with a 1D wave and try to figure out the order.

Thanks
Christian
On re-reading the doc ("...stored row by row (each row spectrum by spectrum)..."), it should probably be
Redimension/N=(depth,width,height) $newname
for vector cubes. Then, you would have images in the YZ dimensions, and the X dimension represents the spectra. I guess you would want XY as image dimensions and Z as the spectral dimension. This could be achieved by leaving the Redimension line as it is and then picking out the data with
newWave[][][] = oldWave[(p + q*width)*depth + r]
or so. Of course, in any case you'd have to modify the surrounding code a bit (e.g. defer the KillWaves oldWave statement, declare Wave NewWave= $newName, etc.)

Wolfgang

Last week I completed a hack to load raw WiRE files (proprietary filetype used in Renishaw Raman spectrometers, may hold 1d or 2d arrays of spectra and spatial coordinates). Jamie Boyd's BinaryReader procedure was indispensable for figuring out the structure of the binary files. I've used it to hack several file types, including Bruker FTIR files. If you haven't already tried it, you should check it out!

Edit: Oops, I just realized that this is an old thread that was resurrected when links were fixed. Still, I recommend BinaryReader to anyone reading this!