Loading the 2D data as a matrix with its file name as the wave name

Share icon

Hello,
I have about 50 ASCII file in a folder which are 2D data with size 1040x1376, and I could load them (at present one by one) as a matrix using following command:
LoadWave/G/M/D/A=wave ""
In this code, /A flag automatically name the wave.
But, I would like to name the wave by its file_name.
For the same, I try to use /W flag. But, did not work. It appears that with /G/M, I may not able to use /W.
Is there any in-build code that would name the wave by its file_name and also load all ASCII files in a folder.
I thank you in advance for your help.
Regards,
Maniraj
Thank you very much for providing link to the source code.
I have changed the file extension and then could load the all files which are having same file extension, successfully.

However, I am facing following problem:
If file name starts with a number, for example,

01_Test
02_Test
03_Test
....
Then, the filename automatically gets "X" as prefix and moreover, the data file format (.dat, .acs, ..) in the suffix (as shown below) !

X01_Test_dat0
X02_Test_dat0
X03_Test_dat0
....
I have tried for both .dat and .acs and the issues are same for both the cases.

If file name starts with any Alphabets, then there is no prefix "X" but suffix

I could locate source for 0 in the suffix of filename which is 0 in the below line
matrixName = UniqueName(matrixName, 1, 0)

But, I exactly could not locate the reason why file format is added and why X is added in prefix.
Please let me know if you have any solution for it.
Thanks you in advance for you help.

July 4, 2014 at 04:09 am - Permalink

Have a look at the rules for the names of waves:
DisplayHelpTopic("Object Names")

They must start with an alphabetic character.

Hope this helps,
Kurt

July 4, 2014 at 04:44 am - Permalink

Well, I do not completely understand it.
When we rename a wave with number as starting character, it works!
When I try to give base name in /A=0123, then also it works!

I could command out (marked by ***) some of the lines in the original code and manage to get number as starting character, but I am still struggling to get rid of data file format at suffix of filename.

The modified code is

Function LoadAllMatrixFilesFromFolder(pathName, extension, makePlots)
String pathName // Name of an Igor symbolic path or "" to get a dialog
String extension // File name extension - e.g., "txt" or ".dat"
Variable makePlots // 1 to display each matrix in a plot

if (strlen(pathName) == 0)
NewPath/O/M="Choose a folder containing Matrix Files" LoadIndexedMatrixPath
if (V_flag != 0)
return -1 // User cancelled
endif
pathName = "LoadIndexedMatrixPath"
endif

String fileName
Variable index

index = 0
do
fileName = IndexedFile($pathName, index, extension)
if (strlen(fileName) == 0)
break // No more files.
endif

String matrixName =fileName// CleanupName(fileName, 0) ***
//Print matrixName ***
//if (exists(matrixName) != 1) ***
// matrixName conflicts with some other Igor name so make it unique ***
//matrixName = UniqueName(matrixName, 1, 5) ***
//endif ***

String columnInfoStr
sprintf columnInfoStr, "N='%s';", matrixName
LoadWave/G/D/P=$pathName/M/B=columnInfoStr fileName ///original line for .dat file format
Wave matrix = $matrixName // Create a wave reference

if (makePlots)
NewImage matrix
endif

index += 1
while (1)
End

July 4, 2014 at 05:28 am - Permalink

To get rid of the file format in the name use fileName = RemoveEnding(fileName, extension).

July 4, 2014 at 05:32 am - Permalink

Okey, Great.
When I used below line (as you have suggested), it worked nicely.
String matrixName= RemoveEnding(fileName, extension)

Thanks a lot.

July 4, 2014 at 06:28 am - Permalink

The "X" prefix is the result of the CleanUpName(fileName, 0) function. If the 0 is replaced with 1, then names are permitted to be "liberal" and the wave names can begin with a number. The X will not be prepended. This may have implications further down in the code. I don't really know, being somewhat conservative, I tend to avoid liberal names.

For more details look up CleanUpNames in the help file; this will also lead to more info on naming waves and other Igor objects.

July 4, 2014 at 07:20 am - Permalink

Finally, the following code works to load .asc data.
Function LoadAllMatrixFilesFromFolder(pathName, extension, makePlots) 
/// Original code can be found in Link, http://www.igorexchange.com/node/5843 
    String pathName             // Name of an Igor symbolic path or "" to get a dialog
    String extension                    // File name extension - e.g., "txt" or ".dat"
    Variable makePlots              // 1 to display each matrix in a plot
 
    if (strlen(pathName) == 0)
        NewPath/O/M="Choose a folder containing Matrix Files" LoadIndexedMatrixPath
        if (V_flag != 0)
            return -1               // User cancelled
        endif
        pathName = "LoadIndexedMatrixPath"
    endif
 
    String fileName
    Variable index
 
    index = 0
    do
        fileName = IndexedFile($pathName, index, extension)
        if (strlen(fileName) == 0)
            break           // No more files.
        endif
        String matrixName= RemoveEnding(fileName, extension)// CleanupName(fileName, 0)                 ***
        String columnInfoStr
        sprintf columnInfoStr, "N='%s';", matrixName
        LoadWave/G/D/P=$pathName/M/B=columnInfoStr fileName   ///original line for .dat file format
        
        Wave matrix = $matrixName           // Create a wave reference
        if (makePlots)
            NewImage matrix
        endif
 
        index += 1
    while (1)
End

Now, I am trying to add option to read the header info and auto scale the matrix.
Header info looks as follows:
# # ImageWidth: 1376 pixel # ImageHeight: 1040 pixel # KineticEnergy: 14 eV # PassEnergy: 160 eV # EnergyScalingOffset: 3.44767 eV # EnergyScalingFactor: 0.0153488 eV/pixel # AzimuthalScalingOffset: 6.99327 deg # AzimuthalScalingFactor: -0.0134615 deg/pixel # 00002400 00001800 00004000 .......
For scaling, I need EnergyScalingOffset, .... to....AzimuthalScalingFactor. But, I would also like to load all this info and store it in wave note.
In connection to this, I saw a forum post Here).
To use the code given in Link to above pasted program, I am not able to get proper starting point. Specifically, how to assign "refNum".
I am looking for info, how to assign refNum in the above pasted code, which line.

July 8, 2014 at 07:20 am - Permalink

I think that you need the Open operation. Given a path and filename, it will open the file for reading and/or writing and will return refNum, this is a handle on the open file and is used for other file manipulation operations. The following is a relevant fragment from the code that you linked to:

    Variable refNum
    Open/R/P=$pathName refNum as filePath
    if (refNum == 0)            // File not opened - bad pathName or fileName
        return -1           // Failure
    endif


Here the file will be opened for read only ("/R" flag). So in your functioning code, feed your path and file to Open and change the remainder of the code to find the header parameters that are needed.

Hope that I have correctly understood your question.

July 8, 2014 at 08:30 am - Permalink

Firstly, thank you very much for your interest.
I have tried to use this set of line to assign refNum, but it does not work.
Since, this code open file to load data, are there any way to directly assign refNum. So, that I can start further to load lines.
To be frank, it appears to me that it is higher level programming!

July 8, 2014 at 08:45 am - Permalink

Quote:

Specifically, how to assign "refNum".


refNum is set by the Open operation:
Function ReadHeaderInfo(pathName, filePath, width, height)
    ...
 
    Variable refNum
    Open/R/P=$pathName refNum as filePath   // *** This sets refNum ***
    if (refNum == 0)            // File not opened - bad pathName or fileName
        return -1           // Failure
    endif


Here is another example that shows the big picture:
http://www.igorexchange.com/node/908

Execute this and read the help to make sure you understand what an Igor symbolic path is and that you are passing a valid value for pathName:
DisplayHelpTopic "Symbolic Paths"

You can pass "" for pathName if you pass a full path for the filePath parameter. If filePath is a partial path or a simple file name then it is relative to the folder referenced by the symbolic path named by pathName. A full path starts with a volume name (e.g., "C:" on Windows or "hd" on Macintosh). A partial path starts with a separator - typically colon but could be backslash.

It is best to use colon as the separator in file paths because backslashes are escape characters in Igor. To read about this, execute this:
DisplayHelpTopic "Path Separators"

Quote:

I have tried to use this set of line to assign refNum, but it does not work.

I need to know specifically what you mean by "does not work".

It might be useful if you posted your code that attempts to read header information.

July 8, 2014 at 08:58 am - Permalink

After going through you suggestion and code, finally I could get things done to some extent.
The present code is 
Function LoadAllMatrixFilesFromFolder2(pathName, extension, makePlots)  
    String pathName             // Name of an Igor symbolic path or "" to get a dialog
    String extension                    // File name extension - e.g., "txt" or ".dat"
    Variable makePlots              // 1 to display each matrix in a plot
    String filePath, fileName,buffer,units,temp
    Variable index=0,i=0,number,refNum
    Variable Energyoffset,Energystep,Thetaoffset,Thetastep
 
    if (strlen(pathName) == 0)
        NewPath/O/M="Choose a folder containing Matrix Files" LoadIndexedMatrixPath
        if (V_flag != 0)
            return -1               // User cancelled
        endif
        pathName = "LoadIndexedMatrixPath"
    endif
    
    
    do
        fileName = IndexedFile($pathName, index, extension)
        if (strlen(fileName) == 0)
            break           // No more files.
        endif
        String matrixName= RemoveEnding(fileName, extension)// CleanupName(fileName, 0)                 ***
        String columnInfoStr
        sprintf columnInfoStr, "N='%s';", matrixName
        LoadWave/G/D/P=$pathName/M/B=columnInfoStr fileName   ///original line for .dat file format
        
        Wave matrix = $matrixName           // Create a wave reference
        if (makePlots)
            NewImage matrix
        endif
    
        if ((strlen(pathName)==0) || (strlen(fileName)==0))
        Open /D /R /P=$pathName /T=(extension) refNum as fileName
        fileName = S_fileName           // S_fileName is set by Open/D
            if (strlen(fileName) == 0)      // User cancelled?
                return -1
            endif
        // fileName is now a full path to the file.
        endif
 
        // Read the header lines
        Open /P=$pathName /R refNum as fileName
        // Read header here using FReadLine
        buffer=""
        do
            FReadLine refNum, buffer
            //Print buffer
            if (strlen(buffer) == 0)
            Print "File ended without finding Header!"
            return -1   // Failure
            endif
            
            temp = buffer[0,22]
            if (CmpStr(temp,"#  EnergyScalingOffset:") == 0)
                //print ":)", buffer
                sscanf buffer, "#  EnergyScalingOffset: %g %s", number, units
                if (V_flag != 2)
                    // Did not get two items
                    //Print "ReadHeaderInfo: sscanf failed for EnergyScalingOffset"         
                    return -1   // Failure
                else
                    Print "number=",number," and units=", units
                    Energyoffset=number
                endif
            endif
        
            temp = buffer[0,22]
            if (CmpStr(temp,"#  EnergyScalingFactor:") == 0)
                print ":)", buffer
                sscanf buffer, "#  EnergyScalingFactor: %g %s", number, units
                if (V_flag != 2)
                    // Did not get two items
                    Print "ReadHeaderInfo: sscanf failed for EnergyScalingFactor"           
                    return -1   // Failure
                else
                    //Print "number=",number," and units=", units
                    Energystep=number
                endif
            endif
        
            temp = buffer[0,25]
            if (CmpStr(temp,"#  AzimuthalScalingOffset:") == 0)
                print ":)", buffer
                sscanf buffer, "#  AzimuthalScalingOffset: %g %s", number, units
                if (V_flag != 2)
                    // Did not get two items
                    Print "ReadHeaderInfo: sscanf failed for AzimuthalScalingOffset"            
                    return -1   // Failure
                else
                    //Print "number=",number," and units=", units
                    Thetaoffset=number
                endif
            endif
            
            temp = buffer[0,25]
            if (CmpStr(temp,"#  AzimuthalScalingFactor:") == 0)
                print ":)", buffer
                sscanf buffer, "#  AzimuthalScalingFactor: %g %s", number, units
                if (V_flag != 2)
                    // Did not get two items
                    Print "ReadHeaderInfo: sscanf failed for AzimuthalScalingFactor"            
                    return -1   // Failure
                else
                    //Print "number=",number," and units=", units
                    Thetastep=number
                endif
            endif       
        i+=1
        while (i<10)
        Close refNum
        
        Print "Energy offset=",Energyoffset,"Energy step=",Energystep,"Theta offset=",Thetaoffset,"Theta step=",Thetastep
        SetScale/P x Thetaoffset,Thetastep,"", matrix;DelayUpdate
        SetScale/P y Energyoffset,Energystep,"", matrix
    index += 1
    while (1)
End

This code works good and loaded all .asc files in the folder. However, Header reading works only for first file and then it uses same values for 2nd data file onwards. I am confused, where is the problem. Can any one help me. For model data please find the attached file, you may change values and try. Note: I have attached data in .txt format because webpage allowed .txt and I have also deleted so many points to minimize the memory.
01_Ta03072014.txt (3.02 MB)

July 9, 2014 at 07:40 am - Permalink

The problem is that you set the variable i to 0 only once. You need to set it to zero before the start of the loop that reads the header.

I discovered this using the Igor debugger. It is invaluable. To read about it, execute:
DisplayHelpTopic "The Debugger"

I rejiggered your code to make it more modular and expressive.

I changed the variable name "i" to "lineNumber" which is more expressive. I removed the declaration of the variable from the beginning of the function to just before the loop which makes it more clear where it is used. I also changed the do loop to a for loop which guarantees that lineNumber is initialized each time the loop starts.

I removed this which is not necessary. The purpose is to display an Open File dialog to choose the file but it not useful when loading all of the files in a folder and also, if you were going to do it, you would have to do it before the first use of pathName and fileName which means before the LoadWave call.
        if ((strlen(pathName)==0) || (strlen(fileName)==0))
            Open /D /R /P=$pathName /T=(extension) refNum as fileName
            fileName = S_fileName           // S_fileName is set by Open/D
            if (strlen(fileName) == 0)      // User cancelled?
                return -1
            endif
            // fileName is now a full path to the file.
        endif


I also split the header reading out as a subroutine. This makes the code more modular, easier to understand and easier to debug, because you don't have to wade through a monolithic block of code to find what you are interested in.

Here is the resulting code:

static Function ReadHeader(pathName, fileName, Energyoffset, Energystep, Thetaoffset, Thetastep)
    String pathName     // Input - Name of an Igor symbolic path
    String fileName         // Input - Name of file from which header is to be read
    Variable& Energyoffset  // Output
    Variable& Energystep    // Output
    Variable& Thetaoffset   // Output
    Variable& Thetastep // Output
    
    Variable refNum
    Open /P=$pathName /R refNum as fileName
 
    Variable lineNumber
    for (lineNumber=0; lineNumber<10; lineNumber+=1)
        String buffer
        FReadLine refNum, buffer
        //Print buffer
        if (strlen(buffer) == 0)
            Print "File ended without finding Header!"
            return -1   // Failure
        endif
 
        String units
        Variable number
 
        String temp = buffer[0,22]
        if (CmpStr(temp,"#  EnergyScalingOffset:") == 0)
            //print ":)", buffer
            sscanf buffer, "#  EnergyScalingOffset: %g %s", number, units
            if (V_flag != 2)
                // Did not get two items
                //Print "ReadHeaderInfo: sscanf failed for EnergyScalingOffset"         
                return -1   // Failure
            else
                Print "number=",number," and units=", units
                Energyoffset=number
            endif
        endif
 
        temp = buffer[0,22]
        if (CmpStr(temp,"#  EnergyScalingFactor:") == 0)
            print ":)", buffer
            sscanf buffer, "#  EnergyScalingFactor: %g %s", number, units
            if (V_flag != 2)
                // Did not get two items
                Print "ReadHeaderInfo: sscanf failed for EnergyScalingFactor"           
                return -1   // Failure
            else
                //Print "number=",number," and units=", units
                Energystep=number
            endif
        endif
 
        temp = buffer[0,25]
        if (CmpStr(temp,"#  AzimuthalScalingOffset:") == 0)
            print ":)", buffer
            sscanf buffer, "#  AzimuthalScalingOffset: %g %s", number, units
            if (V_flag != 2)
                // Did not get two items
                Print "ReadHeaderInfo: sscanf failed for AzimuthalScalingOffset"            
                return -1   // Failure
            else
                //Print "number=",number," and units=", units
                Thetaoffset=number
            endif
        endif
 
        temp = buffer[0,25]
        if (CmpStr(temp,"#  AzimuthalScalingFactor:") == 0)
            print ":)", buffer
            sscanf buffer, "#  AzimuthalScalingFactor: %g %s", number, units
            if (V_flag != 2)
                // Did not get two items
                Print "ReadHeaderInfo: sscanf failed for AzimuthalScalingFactor"            
                return -1   // Failure
            else
                //Print "number=",number," and units=", units
                Thetastep=number
            endif
        endif       
    endfor
    Close refNum
    
    return 0            // Success
End
 
Function LoadAllMatrixFilesFromFolder2(pathName, extension, makePlots)  
    String pathName             // Name of an Igor symbolic path or "" to get a dialog
    String extension                    // File name extension - e.g., "txt" or ".dat"
    Variable makePlots              // 1 to display each matrix in a plot
 
    String filePath, fileName,buffer,units,temp
    Variable index=0,number,refNum
 
    if (strlen(pathName) == 0)
        NewPath/O/M="Choose a folder containing Matrix Files" LoadIndexedMatrixPath
        if (V_flag != 0)
            return -1               // User cancelled
        endif
        pathName = "LoadIndexedMatrixPath"
    endif
 
    do
        fileName = IndexedFile($pathName, index, extension)
        if (strlen(fileName) == 0)
            break           // No more files.
        endif
        String matrixName= RemoveEnding(fileName, extension)// CleanupName(fileName, 0)                 ***
        String columnInfoStr
        sprintf columnInfoStr, "N='%s';", matrixName
        LoadWave/G/D/P=$pathName/M/B=columnInfoStr fileName   ///original line for .dat file format
 
        Wave matrix = $matrixName           // Create a wave reference
        if (makePlots)
            NewImage matrix
        endif
        
        Variable Energyoffset,Energystep,Thetaoffset,Thetastep
        Variable result = ReadHeader(pathName, fileName, Energyoffset, Energystep, Thetaoffset, Thetastep)
        if (result != 0)
            return result                       // Failure
        endif
  
        Print "Energy offset=",Energyoffset,"Energy step=",Energystep,"Theta offset=",Thetaoffset,"Theta step=",Thetastep
        SetScale/P x Thetaoffset,Thetastep,"", matrix;DelayUpdate
        SetScale/P y Energyoffset,Energystep,"", matrix
    index += 1
    while (1)
    
    return 0                                // Success
End


The ReadHeader uses pass-by-reference parameters to return results to the calling routine. To read about pass-by-reference parameters, execute:
DisplayHelpTopic "How Parameters Work"

If you were reading a large number of header parameters you would want to use a structure to return the values from ReadHeader to the main function. But for your situation pass-by-reference parameters are fine.

July 9, 2014 at 09:16 am - Permalink

Thanks a lot hrodstein.
Special thanks to your interest and introducing me to step by step debugger tool.
I finally added Note option and completed the function.

July 10, 2014 at 03:57 am - Permalink

I am trying to modify discussed code to load only one specific matrix wave. Modified code is as follows:
Function LoadSingleMatrixFileFromFolder()  
    String pathName             // Name of an Igor symbolic path or "" to get a dialog
    String extension                    // File name extension - e.g., "txt" or ".dat"
    Variable makePlots=0                // 1 to display each matrix in a plot
 
    String filePath, fileName,buffer,units,temp
    Variable index=0,number,refNum  
    String message = "Select one or more files"
    String fileFilters = "Data Files (*.txt,*.dat,*.csv,*.asc):.txt,.dat,.csv,.asc;"
    fileFilters += "All Files:.*;"
    Open /D /R /MULT=1 /F=fileFilters /M=message RefNum
    //print "filename is", ParseFilePath(3, S_fileName, ":", 0, 0) 
 
 
//  if (strlen(pathName) == 0)
//      NewPath/O/M="Choose a folder containing Matrix Files" LoadIndexedMatrixPath
//      if (V_flag != 0)
//          return -1               // User cancelled
//      endif
//      pathName = "LoadIndexedMatrixPath"
//  endif   
 
 
 // do
        fileName = S_fileName//IndexedFile($pathName, index, extension)
        String matrixName= ""
        matrixName=ParseFilePath(3, S_fileName, ":", 0, 0)//RemoveEnding(fileName, extension)
        print "matrixName is ",matrixName
        
        if (exists(matrixName) != 1)  // It checks whether matrixName exist/already loaded from the same folder, if exist/loaded, that file(s) are skipped
            String columnInfoStr
            sprintf columnInfoStr, "N='%s';", matrixName
            LoadWave/G/D/P=$pathName/M/B=columnInfoStr fileName   ///original line for .dat file format
 
            Wave matrix = $matrixName       // Create a wave reference
            if (makePlots)
                NewImage matrix
            endif
 
            Variable Energyoffset,Energystep,Thetaoffset,Thetastep
            Variable result = ReadHeader(pathName, fileName, matrix, Energyoffset, Energystep, Thetaoffset, Thetastep)
            if (result != 0)
                return result                       // Failure
            endif
 
            Printf "Energy offset=%g, Energy step=%g, Theta offset= %g, Theta step=%g \r",Energyoffset,Energystep,Thetaoffset,Thetastep
            SetScale/P x Thetaoffset,Thetastep,"", matrix;DelayUpdate
            SetScale/P y Energyoffset,Energystep,"", matrix
        
            //integrated spectrum extraction part
            //Activation below lines will automatically create the integrated spectra
            
            //ImageTransform sumAllCols matrix
            //String intname=""
            //intname=matrixName+"_sum"
            //wave W_sumCols
            //SetScale/P x Energyoffset,Energystep,"",W_sumCols
            //rename W_sumCols $intname
            //integrated spectrum extraction part
        
        endif  // It checks whether matrixName exist/already loaded from the same folder, if exist/loaded, that file(s) are skipped
        
    //index += 1
    //while (1)
 
    return 0                                // Success
End


I am getting "Attempt to use a null string variable" error for below segment 
Wave matrix = $matrixName       // Create a wave reference
            if (makePlots)
                NewImage matrix
            endif

I could not get what is missing. Any help is greatly appreciated.

November 4, 2015 at 03:05 pm - Permalink

mslv86 wrote:

I could not get what is missing. Any help is greatly appreciated.

Have you run it with the debugger turned on?
This should help you track down the issue.
HTH,
Kurt

November 4, 2015 at 10:59 pm - Permalink

Yes, I used debugger which show where was problem. But, matrixName has filename stored yet I get null string. So got confused. My file name starts with number. Will this makes any problem?

November 4, 2015 at 11:42 pm - Permalink

Sometimes it is best to try to reproduce the problem by starting from the ground up.
As a starter, here is some code that tests a wavename starting with a digit...
Function RunTest()
    string sWaveName
    
    sWaveName = "MyWave" // simple wavename
    print "1: Wave", sWaveName, ", exists:",exists(sWaveName)
    wave wMyWave = $sWaveName
    
    Make/O $sWaveName
    print "2: Wave", sWaveName, ", exists:",exists(sWaveName)
    wave wMyWave = $sWaveName
    KillWaves/Z $sWaveName
    
    sWaveName = "9MyWave" // start with digit
    print "3: Wave", sWaveName, ", exists:",exists(sWaveName)
    wave wMyWave = $sWaveName
    
    Make/O $sWaveName
    print "4: Wave", sWaveName, ", exists:",exists(sWaveName)
    wave wMyWave = $sWaveName
    KillWaves/Z $sWaveName
    
    sWaveName = "9 MyWave" // start with digit and has space
    print "5: Wave", sWaveName, ", exists:",exists(sWaveName)
    wave wMyWave = $sWaveName
    
    Make/O $sWaveName
    print "6: Wave", sWaveName, ", exists:",exists(sWaveName)
    wave wMyWave = $sWaveName
    KillWaves/Z $sWaveName
End

This runs as I would expect...
•RunTest()
  1: Wave  MyWave  , exists:  0
  2: Wave  MyWave  , exists:  1
  3: Wave  9MyWave  , exists:  0
  4: Wave  9MyWave  , exists:  1
  5: Wave  9 MyWave  , exists:  0
  6: Wave  9 MyWave  , exists:  1


Can you add to this to reproduce your error message?

HTH,
Kurt

November 5, 2015 at 12:56 am - Permalink

Just a thought...

Are you sure the error is here:
mslv86 wrote:


I am getting "Attempt to use a null string variable" error for below segment 
Wave matrix = $matrixName       // Create a wave reference
            if (makePlots)
                NewImage matrix
            endif

...and not in the previous line?
LoadWave/G/D/P=$pathName/M/B=columnInfoStr fileName   ///original line for .dat file format

November 5, 2015 at 01:13 am - Permalink