Transpose and rename large data set
Greetings everyone! I have a large dataset that is output from a vector network analyzer. The data comes out in the wrong orientation though, so I need to transpose it. The attached screenshot is an example of what a portion of my data looks like. Here: wave0 should be the name of all of the waves. The top row is the time delta of reflection measurements from the VNA (this should be my X axis values in a simple XY plot), and the rest of the data is the magnitudes measured by the VNA (this would be my Y axis values in a simple XY plot).
My goal is to transpose the waves, rename each of the new waves so the new names match up approximately with the entries in the original wave 0 column, then be able to graph the transposed data in traditional XY plots. I tried doing the transpose operation in Excel, but there's too much data here. Excel just crashes trying to do any sort of manipulations with it. So I dedicated some time to figuring out how to do it in igor.
After many attempts on my own, I couldn't get what I wanted. Now, admittedly I'm not very good at coding, so I thought I'd see if chatgpt could help generate something that worked so I could learn. I went through a few dozen iterations. The code kept getting longer and longer, but the operation seemed to always get stuck on renaming the waves. When I run it, I get an error that says "while executing rename, the following error occurred: expected name of wave, variable, or string". I appreciate any help you all may be able to offer me. Below is the last code it gave me that would compile successfully:
function transposewaves() // Automatically detect the number of waves and points variable i,j Variable numColumns = 0 Variable numPoints // Step 1: Duplicate wave0 into a new text wave to safely reference it Wave wave0 = $ "wave0" // Reference wave0 directly numPoints = numpnts(wave0) // Get the number of points in wave0 Duplicate/T wave0, wave0Copy // Create a copy of wave0 as wave0Copy // Create waveNamesList as a text wave to hold sanitized names Make/T/N=(numPoints) waveNamesList // Loop through wave0Copy, sanitize each entry, and store it in waveNamesList for (j = 0; j < numPoints; j += 1) String rawName = wave0Copy[j] // Get the original name from wave0Copy // Remove " AM" and " PM" from the rawName if they are present rawName = ReplaceString(rawName, " AM", "") rawName = ReplaceString(rawName, " PM", "") // Initialize sanitizedName with prefix and add sanitized rawName String sanitizedName = "VNA_" + rawName sanitizedName = ReplaceString(sanitizedName, " ", "_") sanitizedName = ReplaceString(sanitizedName, ":", "_") // Check if sanitizedName is empty and provide a fallback if (strlen(sanitizedName) == 0) sanitizedName = "VNA_Unknown_" + num2str(j) // Fallback name if sanitizedName is empty endif // Store the sanitized name in waveNamesList waveNamesList[j] = sanitizedName endfor // Create waveNames as a text wave and copy names from waveNamesList Make/T/N=(numPoints) waveNames for (j = 0; j < numPoints; j += 1) waveNames[j] = waveNamesList[j] // Copy sanitized names to waveNames endfor // Step 2: Detect the number of numeric waves do String waveName = "wave" + num2str(numColumns + 1) // Start from wave1, skipping wave0 if (!exists(waveName)) // Check if the wave exists break // Stop if wave does not exist endif numColumns += 1 while (1) // Step 3: Create a 2D matrix wave for numeric data only Make/N=(numPoints, numColumns) combinedWave // Numeric wave for all columns excluding text // Step 4: Loop through each numeric wave and fill combinedWave for (i = 1; i <= numColumns; i += 1) String columnWaveName = "wave" + num2str(i) // Generate wave name as a string Wave columnWaveRef = $columnWaveName // Reference the numeric wave for (j = 0; j < numPoints; j += 1) combinedWave[j][i - 1] = columnWaveRef[j] // Copy each value individually to ensure correct indexing endfor endfor // Step 5: Transpose the combinedWave to align columns with points MatrixTranspose combinedWave // Step 6: Rename each wave using sanitized values from waveNames for (j = 0; j < numPoints; j += 1) String newWaveName = waveNames[j] // Get the sanitized name for each wave from waveNames String originalWaveName = "combinedWave[" + num2str(j) + "]" // Construct the original wave name // Validate the newWaveName before renaming if (strlen(newWaveName) == 0) newWaveName = "VNA_Fallback_" + num2str(j) // Final fallback name if empty endif // Print renaming information for each entry print "Renaming combinedWave index ", j, " to ", newWaveName // Rename the wave Rename originalWaveName, newWaveName // Rename the wave endfor end
It might be more efficient to use the Concatenate operation to create the original 2D matrix. Once you have the matrix, you may not even need to transpose it in order to plot either rows or columns, e.g.,
If you have another reason to break the data into 1D waves in the orthogonal direction to the original data you can transpose the matrix and use SplitWave.
November 21, 2024 at 12:47 pm - Permalink
I would also move the information of wave 0 and the first row in each wave into the row/column labels of a concatenated 2D wave. This way you would have only your data inside the 'active part' of the wave, which would make plotting as proposed by Igor much easier. If you could provide a small subset of your data (such as you have posted in the screenshot), we could come up with a small code example easily.
BTW, I am surprised that ChatGPT is able to give you useful Igor code at all. I thought c++ yes, but Igors scripting would be way to niche for that.
November 21, 2024 at 08:29 pm - Permalink
If you want to transfer a large dataset from Excel (although I'm not sure if this is the case here) you can use the following snippet to copy & paste everything into a single matrix.
E.g. if you want to assign the time stamp and first row as dimension labels copy the data in excel, then call
from the command line and the dataset will be available as wave0 (assuming a fresh experiment). Then you can use MatrixTranspose to change orientation.
PS:
if you use a 2D wave with labels and want to plot data, this might be useful:
https://www.wavemetrics.com/code-snippet/display-rowscols-labelled-2d-w…
November 22, 2024 at 12:41 am - Permalink
In reply to I would also move the… by chozo
Yeah, it wasn't very good at generating igor code. 90% of the attempts it made ended with compile errors. I was hoping I could essentially teach it to be better at it by continuing to iterate with it on this one, but I eventually gave up after a few dozen iterations.
Thank you so much for offering to help come up with the code for this task. I've attached a small section of the data in an excel file as requested.
November 22, 2024 at 06:59 am - Permalink
Here this should do ... run via e.g., postProcessWaves("loadedData")
This code assumes that you have just loaded such a table with default names. You might want to combine the code with a loading procedure to automate the whole process. I could easily modify the code so that you just need to drop such a table into Igor to automatically generate a single matrix each time. But first you should check how long each step takes with your full data volumne. I left wave0 as a separate wave after all, since the row labels cannot save the time stamp in its original format (e.g., "3:02:26 PM", since colons are not allowed in labels). The matrix also has time-stamp labels, but in a slightly different format ("3_02_26 PM"). If you want, you could also Transpose the matrix now, but this depends on what you want to do with the data. Simply plotting this as traces also works from a single matrix wave.
November 22, 2024 at 08:02 am - Permalink
In reply to chozo wrote: I would… by DrPFF
I agree with your assessment of ChatGPT. We have a version for internal use at my employer. I think chozo is correct that (at the moment, at least) Igor is too niche for reliable results from ChatGPT. It seemed to have some surface knowledge of Igor, but tended to invent operations/functions that seemed to be extrapolated from other languages.
Better to stick with ChatI(gor)E(xchange).
November 22, 2024 at 10:23 am - Permalink