Transpose and rename large data set
Greetings everyone! I have a large dataset that is output from a vector network analyzer. The data comes out in the wrong orientation though, so I need to transpose it. The attached screenshot is an example of what a portion of my data looks like. Here: wave0 should be the name of all of the waves. The top row is the time delta of reflection measurements from the VNA (this should be my X axis values in a simple XY plot), and the rest of the data is the magnitudes measured by the VNA (this would be my Y axis values in a simple XY plot).
My goal is to transpose the waves, rename each of the new waves so the new names match up approximately with the entries in the original wave 0 column, then be able to graph the transposed data in traditional XY plots. I tried doing the transpose operation in Excel, but there's too much data here. Excel just crashes trying to do any sort of manipulations with it. So I dedicated some time to figuring out how to do it in igor.
After many attempts on my own, I couldn't get what I wanted. Now, admittedly I'm not very good at coding, so I thought I'd see if chatgpt could help generate something that worked so I could learn. I went through a few dozen iterations. The code kept getting longer and longer, but the operation seemed to always get stuck on renaming the waves. When I run it, I get an error that says "while executing rename, the following error occurred: expected name of wave, variable, or string". I appreciate any help you all may be able to offer me. Below is the last code it gave me that would compile successfully:
// Automatically detect the number of waves and points
variable i,j
Variable numColumns = 0
Variable numPoints
// Step 1: Duplicate wave0 into a new text wave to safely reference it
Wave wave0 = $ "wave0" // Reference wave0 directly
numPoints = numpnts(wave0) // Get the number of points in wave0
Duplicate/T wave0, wave0Copy // Create a copy of wave0 as wave0Copy
// Create waveNamesList as a text wave to hold sanitized names
Make/T/N=(numPoints) waveNamesList
// Loop through wave0Copy, sanitize each entry, and store it in waveNamesList
for (j = 0; j < numPoints; j += 1)
String rawName = wave0Copy[j] // Get the original name from wave0Copy
// Remove " AM" and " PM" from the rawName if they are present
rawName = ReplaceString(rawName, " AM", "")
rawName = ReplaceString(rawName, " PM", "")
// Initialize sanitizedName with prefix and add sanitized rawName
String sanitizedName = "VNA_" + rawName
sanitizedName = ReplaceString(sanitizedName, " ", "_")
sanitizedName = ReplaceString(sanitizedName, ":", "_")
// Check if sanitizedName is empty and provide a fallback
if (strlen(sanitizedName) == 0)
sanitizedName = "VNA_Unknown_" + num2str(j) // Fallback name if sanitizedName is empty
endif
// Store the sanitized name in waveNamesList
waveNamesList[j] = sanitizedName
endfor
// Create waveNames as a text wave and copy names from waveNamesList
Make/T/N=(numPoints) waveNames
for (j = 0; j < numPoints; j += 1)
waveNames[j] = waveNamesList[j] // Copy sanitized names to waveNames
endfor
// Step 2: Detect the number of numeric waves
do
String waveName = "wave" + num2str(numColumns + 1) // Start from wave1, skipping wave0
if (!exists(waveName)) // Check if the wave exists
break // Stop if wave does not exist
endif
numColumns += 1
while (1)
// Step 3: Create a 2D matrix wave for numeric data only
Make/N=(numPoints, numColumns) combinedWave // Numeric wave for all columns excluding text
// Step 4: Loop through each numeric wave and fill combinedWave
for (i = 1; i <= numColumns; i += 1)
String columnWaveName = "wave" + num2str(i) // Generate wave name as a string
Wave columnWaveRef = $columnWaveName // Reference the numeric wave
for (j = 0; j < numPoints; j += 1)
combinedWave[j][i - 1] = columnWaveRef[j] // Copy each value individually to ensure correct indexing
endfor
endfor
// Step 5: Transpose the combinedWave to align columns with points
MatrixTranspose combinedWave
// Step 6: Rename each wave using sanitized values from waveNames
for (j = 0; j < numPoints; j += 1)
String newWaveName = waveNames[j] // Get the sanitized name for each wave from waveNames
String originalWaveName = "combinedWave[" + num2str(j) + "]" // Construct the original wave name
// Validate the newWaveName before renaming
if (strlen(newWaveName) == 0)
newWaveName = "VNA_Fallback_" + num2str(j) // Final fallback name if empty
endif
// Print renaming information for each entry
print "Renaming combinedWave index ", j, " to ", newWaveName
// Rename the wave
Rename originalWaveName, newWaveName // Rename the wave
endfor
end
It might be more efficient to use the Concatenate operation to create the original 2D matrix. Once you have the matrix, you may not even need to transpose it in order to plot either rows or columns, e.g.,
// or
Display mat2d[someRow][] // display the data of someRow
If you have another reason to break the data into 1D waves in the orthogonal direction to the original data you can transpose the matrix and use SplitWave.
November 21, 2024 at 12:47 pm - Permalink
I would also move the information of wave 0 and the first row in each wave into the row/column labels of a concatenated 2D wave. This way you would have only your data inside the 'active part' of the wave, which would make plotting as proposed by Igor much easier. If you could provide a small subset of your data (such as you have posted in the screenshot), we could come up with a small code example easily.
BTW, I am surprised that ChatGPT is able to give you useful Igor code at all. I thought c++ yes, but Igors scripting would be way to niche for that.
November 21, 2024 at 08:29 pm - Permalink
If you want to transfer a large dataset from Excel (although I'm not sure if this is the case here) you can use the following snippet to copy & paste everything into a single matrix.
// row, col = 1, if data should have row or col headers as labels
LoadWave/U={row,0,col,0}/J/M "Clipboard"
end
E.g. if you want to assign the time stamp and first row as dimension labels copy the data in excel, then call
from the command line and the dataset will be available as wave0 (assuming a fresh experiment). Then you can use MatrixTranspose to change orientation.
PS:
if you use a 2D wave with labels and want to plot data, this might be useful:
https://www.wavemetrics.com/code-snippet/display-rowscols-labelled-2d-w…
November 22, 2024 at 12:41 am - Permalink
In reply to I would also move the… by chozo
Yeah, it wasn't very good at generating igor code. 90% of the attempts it made ended with compile errors. I was hoping I could essentially teach it to be better at it by continuing to iterate with it on this one, but I eventually gave up after a few dozen iterations.
Thank you so much for offering to help come up with the code for this task. I've attached a small section of the data in an excel file as requested.
November 22, 2024 at 06:59 am - Permalink
Here this should do ... run via e.g., postProcessWaves("loadedData")
if (strlen(outputName) == 0)
Abort "You need to specify a name for the output."
endif
string list = WaveList("wave*",";","TEXT:0;DIM:1")
Concatenate/O/NP=2 list, $outputName // merge all waves
wave outwave = $outputName
int i
for (i=0; i<DimSize(outwave,1); i++) // set column labels from first row
SetDimLabel 1, i, $num2str(outwave[0][i]), outwave
KillWaves/Z $("wave"+num2str(i+1))
endfor
DeletePoints 0,1, outwave // delete first row
wave/Z/T rowNames = wave0
if (WaveExists(rowNames) == 0) // there is no wave
return 0
endif
for (i=0; i<min(DimSize(rowNames,0),DimSize(outwave,0)); i++) // set row labels from wave0
SetDimLabel 0, i, $CleanupName(rowNames[i],1), outwave
endfor
Duplicate/O rowNames, $(outputName+"_time")
DeletePoints 0,1, $(outputName+"_time")
KillWaves/Z wave0
return 0
end
This code assumes that you have just loaded such a table with default names. You might want to combine the code with a loading procedure to automate the whole process. I could easily modify the code so that you just need to drop such a table into Igor to automatically generate a single matrix each time. But first you should check how long each step takes with your full data volumne. I left wave0 as a separate wave after all, since the row labels cannot save the time stamp in its original format (e.g., "3:02:26 PM", since colons are not allowed in labels). The matrix also has time-stamp labels, but in a slightly different format ("3_02_26 PM"). If you want, you could also Transpose the matrix now, but this depends on what you want to do with the data. Simply plotting this as traces also works from a single matrix wave.
November 22, 2024 at 08:02 am - Permalink
In reply to chozo wrote: I would… by DrPFF
I agree with your assessment of ChatGPT. We have a version for internal use at my employer. I think chozo is correct that (at the moment, at least) Igor is too niche for reliable results from ChatGPT. It seemed to have some surface knowledge of Igor, but tended to invent operations/functions that seemed to be extrapolated from other languages.
Better to stick with ChatI(gor)E(xchange).
November 22, 2024 at 10:23 am - Permalink