XPS data from Phy Multipak
hegedus
Has anyone have experience with .spe file from a Phy Multipak? I have a data file (attached) that I am trying to extract the data from. There should be 6 spectra that are defined in the header. Simpler files with only 1 data range I can sort of get, though there is a bit of garbage at the beginning of the wave after export.
edit: The exported CSV file is also attached.
My goal is to extract the data directly, because the export software does not support batch processing.
The multiple ranges a bit of a stump for me. Anyone with a magic .ipf that they can share?
Andy
string key // keyword after which a spectrum begins
string base // base name for spectra
variable skip // line to skip after keyword
variable refNum
Open/D/R/F="*.csv"/M="Select SPE-CSV File" refNum
string filePath = S_fileName
Open/R refNum as filepath
Make/FREE/O/T/N=0 w
variable i =0
do
string line
FReadLine refNum, line
if( strlen(line) == 0 ) // end of file
break // exit do-while loop
endif
line = RemoveEnding(line, "\r")
InsertPoints i, 1, w
w[i] = line
i+=1
while( 1 )
Close refNum
i=0
variable filenum = 0
for(i=0; i<numpnts(w); i+=1)
if(cmpstr(w[i], key) == 0)
i+=skip
Make/O/N=(numpnts(w)) $base + num2str(filenum)
wave ww = $base + num2str(filenum)
ww = NaN
endif
string ThisLine = w[i]
variable strStart = strsearch(ThisLine, ",", 0)+1
ww[i] = str2num(ThisLine[strStart, inf])
if(strlen(w[i]) == 0)
filenum +=1
WaveTransform zapNaNs ww
endif
endfor
end
try:
The entire file is first loaded as text wave using FReadLine (maybe LoadWave would be easier), then I use a keyword (here "Area5") to figure out where a spectrum begins.
You need to add a bit of code to either extract x-data or to set the wave scaling.
April 15, 2014 at 01:42 am - Permalink
Thank you very much for your code. I used it as a starting point and produced this using some patterns in the file. I would really love to be able to do this off of the .spe file. It has a text header with a lot of information needed including the number of spectra and the ranges to set the scaling. The actual data is in binary and I have not been successful in parsing it into the individual spectra.
//get file
variable refNum
Open/D/R/F="*.csv"/M="Select SPE-CSV File" refNum
string filePath = S_fileName
Open/R refNum as filepath
Make/O/T/N=0 w
Make/O/N=0 SpectraStart
variable i,j =0
do
string line
FReadLine refNum, line
if( strlen(line) == 0 ) // end of file
break // exit do-while loop
endif
line = RemoveEnding(line, "\r")
InsertPoints i, 1, w
//check to see if it is the start of a new spectra and note the line number
if(stringmatch(line,"*Area*")==1)
InsertPoints j, 1, SpectraStart
SpectraStart[j] = i
j +=1
endif
w[i] = line
i+=1
while( 1 )
Close refNum
//make individual spectra
variable SpectraIndex,index,offset
For(SpectraIndex=0;SpectraIndex<numpnts(SpectraStart);SpectraIndex +=1)
//Read in data
Make/O/N=(0,2) DataIn
offset = spectrastart[spectraindex]+4
index=0
Do
if(strlen(w[(offset+index)])==0)
break
endif
insertpoints index,1, Datain
DataIn[index][0] = str2num(stringfromlist(0,w[(offset+index)],","))
DataIn[index][1] = str2num(stringfromlist(1,w[(offset+index)],","))
index +=1
// print index
while(1)
//Make Spectra with Name
Make/O/N=(dimsize(datain,0)) $w[spectraStart[spectraIndex]+2]
Wave DataOut = $w[spectraStart[spectraIndex]+2]
Dataout[]=Datain[p][1]
//Set wave Scaling
SetScale/I x, datain[0][0], datain[dimsize(datain,0)-1], dataout
EndFor
End
April 15, 2014 at 11:33 am - Permalink
In [1] I've found a file format specification for SPE files, maybe these are the same formats.
Thomas
[1]: ftp://ftp.princetoninstruments.com/Public/Manuals/Princeton%20Instrumen…
April 16, 2014 at 11:04 am - Permalink
Same extension different files. Thanks for the idea.
Andy
April 16, 2014 at 01:05 pm - Permalink
Using a hex file editor, it isn't too difficult to figure out how to load the spectra. See the code below. The purpose of some elements of the file aren't clear (to me); however, the procedure loads these values into waves. The values are interpreted in ways that seem to make sense (i.e. as integers or floats) but I may have guessed incorrectly. Hopefully, they will make some sense to you. Also it wasn't clear how to get the wavelength information, but, maybe you already know how to do this. Oh, yes, this procedure will likely need modification if you want to load files with spectra from more than one position.
The procedure is embedded here and as a ipf file. Also attached is an experiment file with the results for your example file.
Hope this helps.
Jeff
Function ImportXPS_bin()
//Read .spe files from Phy Multipak XPS
Variable vFileRef
Variable vIndex = 0
Variable vLoc
Variable vNumSpectra
Variable vPointsInSpectrum
Variable vReadVal
String sFilePath
String sLine
String sSpectrumWaveName
//Get name/path to file to open
Open/D/R/F="Multipak Files (*.spe):.spe;"/M="Select SPE-CSV File" vFileRef
sFilePath = S_fileName
if( StrLen( S_fileName ) == 0 )
return -1 //user cancelled file selection
endif
Open/R vFileRef as sFilePath
//Text Header
Make/O/T/N=0 wHeader
//Read the File Header one line at at time; read terminates on LF/CR combinations, per the help file
//In this case the terminator is LF
Do
FReadLine vFileRef, sLine
FStatus vFileRef
if( V_filePos > V_logEOF )
print "Attempting to read past end of file"
return -1 //attempting to read past end of file
endif
sLine = RemoveEnding( sLine, "\r" )
InsertPoints vIndex, 1, wHeader
wHeader[vIndex] = sLine
//get number of spectra in file from header containing
if( stringmatch( sLine,"NoSpectralReg:*" ) == 1 )
vLoc = strsearch( sLine, ":", 0 )
vNumSpectra = str2num( sLine[vLoc + 1, inf ] )
print "Number of Spectra in file: ", vNumSpectra
endif
//End of header is given by "EOFH" string; quit when this is found
if( stringmatch( sLine,"EOFH" ) == 1 )
print "End of header found"
FStatus vFileRef
print "Current file position: ", V_filePos
break
endif
vIndex += 1
While( 1 )
//Binary headers
//brief (16 bytes) header; seems to be 4 4 byte unsigned integer values, second valued may give number of spectra
//otherwise the purpose of this part is not clear
Make/O/N=4 wBinHeader_1
//each spectrum gets 24 unsigned 4 byte integers; fifth value is number of points in corresponding spectrum
Make/O/N=( 24 * vNumSpectra ) wSpectraHeader
//read the binary headers
FBinRead/B=0/F=3/U vFileRef, wBinHeader_1
FStatus vFileRef
print "Current file position after bin header read: ", V_filePos
FBinRead/B=0/F=3/U vFileRef, wSpectraHeader
FStatus vFileRef
print "Current file position after spectra header read: ", V_filePos
//Mystery value at end of each spectrum...
//4 byte floating value at end of spectrum; don't know what it is for; read it anyway
Make/O/N=( vNumSpectra ) wMysteryBytes
//load each spectrum into a separate wave
for( vIndex = 0; vIndex < vNumSpectra; vIndex += 1 )
//name for spectrum wave; name is "wSpectrum_" plus number corresponding to position in file
sprintf sSpectrumWaveName, "wSpectrum_%02d", vIndex + 1
print sSpectrumWaveName
//get number of points to load for the spectrum
vPointsInSpectrum = wSpectraHeader[ 5 + vIndex * 24 ]
//make the wave for the current spectrum
Make/O/N=( vPointsInSpectrum ) $sSpectrumWaveName
Wave wSpectrum = $sSpectrumWaveName
//read spectrum; format is 64 bit float
FBinRead/B=0/F=5 vFileRef, wSpectrum
//deal with 4 bytes at end of a spectrum; either read into wave just for this or skip
//code for skipping is commented out
//read 4 bytes at end of spectrum, interpret at 32 bit float
FBinRead/B=0/F=4 vFileRef, vReadVal
wMysteryBytes[ vIndex ] = vReadVal
// //next 4 bytes have unknown purpose, skip for now
// FStatus vFileRef
// FSetPos vFileRef, V_filePos + 4
endfor
Close vFileRef
End
April 18, 2014 at 06:18 pm - Permalink
I'm also working on a PHI machine and I'm also interested in directly loading the spe data.
April 20, 2014 at 01:53 am - Permalink
I used an application (a "hex editor", see the Wikipedia article.. http://en.wikipedia.org/wiki/Hex_editor) that permits viewing the contents of a file in binary form, bit by bit, if necessary. The text header in the file gave insights into interpreting the binary data and, certainly, having the text export version of the spe file was a big help. With those hints, you have to use the hex editor to examine the file for clues to its structure and make some guesses.
There was more information in the binary portion of the file that I did not understand. This was saved in a couple of other waves that, may make sense to someone with experience using the instrument. Hopefully, those mysterious portions were interpreted (as integer or floating point values) correctly.
Also, as I mentioned in my original post, it appears that spectra from different regions of the sample can be combined in the same data file. I don't think the code provided will handle this without some changes.
By the way, are you able to load your data with the Igor procedure that was posted?
April 20, 2014 at 04:19 pm - Permalink
Thank you very much. I have never used a hex editor and this is certainly a good place to start. The spectra do make sense and match the ASCII values so your guesses were correct.
This great!
Thanks again.
Andy
April 21, 2014 at 08:06 am - Permalink
Thanks for your positng.
Recently I strongly require data loader of the .SPE files.
The pxp or ipf file you uploaded gives an error "While sxecuting FBinRead, the following error occured: logical end-of file was reached unexpectedly uring read operation." when I execute 'importXPS_bin()".
I attach an example of SPE files.
Can you check it up? I am beginner of igor program.
JW
May 2, 2016 at 06:31 am - Permalink
I will look into this.
Jeff
May 6, 2016 at 05:17 am - Permalink
http://www.igorexchange.com/project/phispefileloader
Please contact me when you would like to join in the project.
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
September 3, 2016 at 07:16 am - Permalink
FWIW, the mistake in the existing code was to read "mystery bytes" at the end of each spectrum.
Please test and confirm. I'll post a non-beta release based on feedback.
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
September 5, 2016 at 07:30 am - Permalink
I tested your IGOR.6.30.x-1.0 on both Igor versions 6.37 and 7.02.
As you noted, the error message always pops up.
In addition, there are several issues as the following.
The spe file that I uploaded consists of four spectra. But your macro only loads one of them.
I need a single wave with proper parameters such as row number, start, step per each spectrum. Total 7 waves are not necessary at all.
As you compare the Igor display with real spectrum in the attached figure, spectral shape and scale change a lot. Even the background is removed. I do not understand what happens...
Longing for better version,
realMAX
January 31, 2017 at 06:56 pm - Permalink
Would you do me the favor and ...
* Log the error report to the Package (so that I can keep track of it there)
* Include a ZIP archive of your SPE file (so that I can troubleshoot on hand the data you have)
I suspect this is because the SPE file that I tested only had one "region", whereas yours has more than one.
I prefer not to create wave formats that are different than what are currently used. The loader can always be run with a function that subsequently cleans up behind it or does other formatting/calculating steps, for example ...
LoadPHISPE()
setscale/I ...
killwaves/Z ....
return 0
end
As long as the data are all loaded correctly, decisions about what to do with the values and waves can be left to each user.
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
February 1, 2017 at 07:59 am - Permalink
The error message is as follows
"While executing Make, the following error occurred: Expected name"
The zip file is attached again, the same file above BP_118. This is composed of 4 spectra.
Most serious problem is that the imported wave is different from real spectrum as I reported yesturday.
realMAX
February 1, 2017 at 09:27 pm - Permalink
SpectralRegDef: the XPS region definition
SpectralRegDef2: ??
SpectralRegDefBackgrond: ??
SpectralRegHero: ??
SpectralRegIR: ??
I will adjust the loader to skip past everything except the XPS regions.
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
December 23, 2017 at 07:00 am - Permalink