How to use JointHistogram with /W=weightWave more efficient/fast?

Hi Johan, (happy to see you in the forum. Hope you are doing well)

Looks like you are trying to write some code for TOF data. I have also the experience that things can be quite slow with 100.000s of hits to process.

The first question would be if you really, really need dynamic linking, or whether an update function will do which is periodically called. In the latter case your slow wave assignments could be reformulated with much faster functions such as MatrixOP. Here is how you can generate your weight wave:

This should be MUCH faster, since it is only one assignment and also MatrixOP is way more optimized. You can also choose to use multiple CPU threads here using the /NTHR flag. You do not seem to use All_lim later, but maybe this part has been omitted here.

As for this part:

I am not sure what you are trying to achieve here. This looks to me very much like you are just copying the data (make with some linear shift). Why not use test_2D_Selected_Points for Display in a graph directly? Could you please tell a bit more about what you want to achieve here?

I am not sure JointHistogram can be sped up further, but maybe someone else has an idea.

Hi Stephan!

Long time no see, I hope you are doing well! As far as I know you are now living quite far away from where we met... 

I've been seeing you here every now and then and also have one of your macros installed in Igor here... You are way more advanced in Igor than I am... I might even use parts of your marco in mine, the part where you "straighten out" curved lines on a 2D image... Let's see when I get there... since my data is in x,y format and not an image perhaps I will need to write my own - I really want to keep it in x-y to limit the size of the Igor experiment as the 2D image quickly grows in size when you have too many datasets..

The data is actually RIXS spectra but from a data-process point of view perhaps somewhat similar to ToF data...

Thanks A LOT for the MatrixOP, this is (one of) the things I was looking for - that will probably speed up the procedure significantly... I will test this out tonight when I have time...

As for the All_lim I use it in the JointHistogram /W=All_Lim - this is my way of omitting "bad" datapoints - this was the fastest way I could find. I am really only using it as a mask for what data is considered good (within the limits I set) and bad (outside the limits).

As for the projection I am projecting the 2D matrix along one axis. If we have a matrix as follows:

1, 2, 3

4, 5, 6

I am simply making the a wave containing three elements and assigning them to 1+4, 2+5, 3+6 (summing vertically). I am sure there is a MatrixOP for this as well that will speed it up...

My current plan would be to create a function (let's call this JS_UpdateAll() for simplicity) with your MatrixOP as well as a function to do the equivalent of Projection := test_2D_Selected_Points[p][x] and then I will call this function with a userHook whenever any of the limits are changed - I think that is doable...

So basically if I change any of the variables (or waves) that contain my limits (in the example above this would be eg. x_min and x_max) I will simply re-calculate the MatrixOP thus creating my "mask" (All_lim), then I will perform the JointHistogram and lastly the "projection" (or whatever it is called mathematically - I really need to brush up on math!). This will probably be somewhat slow but hopefully "fast enough"... Otherwise I will need to have an "update button" that calls such a function...

While I have you here - I've been looking into hooks yesterday (I never used them) and I wonder if you have readily an example that looks for changes in certain waves/variables or even a datafolder (all settings are saved in a folder). So whenever I update a value in that folder/wave/variable then the function "JS_UpdateAll()" will be run. That way I can "dynamically" update the waves whenever needed... I have also seen that hooks can be turned on/off so I can enable this hook when I need dynamic updating and disable it when I do not need it...

Thanks a lot for your time and help, it is greatly apprechiated!!!

Hi Johan,

Great to hear from you again. Yes, I am a bit far away now, but still also somewhat close (virtually). Let me know if you need specific help with some of your other endeavors with Igor, either here or via email. So this is RIXS data then. I was just thinking of TOF data because of the x,y,t (time?) variables. As you write, data analysis is not so different across spectroscopies.

Anyway, it seems I didn't read your initial text properly, since I overlooked the use of All_lim with JointHistogram. You are using All_lim as a 'hard' (binary) weighting wave, but JointHistogram still has to work on the full data in the end (and decide on the fly if each point is useful or not). Maybe it is possible to first reduce the data volume (temporarily) to work with a somewhat reduced data size in JointHistogram. I will think about it.

Regarding the Projection wave, I think your code currently doesn't do what you want.

is the same as:

if the Projection wave is not scaled (I assume it is not). This simply gives you the diagonal elements of the input (1,5,...) in your example. If you want to sum all elements in vertical direction (or in other words sum the whole column), then you can use this simple command:

Appending ^t makes sure that the result is in the correct orientation.

As for auto-updating the calculation, there is no hook function which can directly watch folders / waves etc. (as far as I know). You could cheat by (again) creating a dependency of some wave or variable, but this is a rather dangerous and confusing approach. I assume you (the user) changes one or multiple parameters 'by hand' anyway. Then it is easy to trigger the update function from this user interaction.

You can assign hooks to graph updates, cursor movement, panel interaction, user controls etc. No need to have Igor watch the folder for changes. This depends a bit on how you would like to implement the change in variables. It is very easy when you have something like SetVariable controls: simply call JS_UpdateAll() each time a variable has been touched. But other methods are imaginable as well, as long as you have some graph or panel to interact with. You then can also have a checkbox to toggle 'auto updates'. Let me / us know how you would like to proceed from here, then we can give a bit more detailed tips.

In case anyone would end up reading this I have re-written this and will eventually publish how I did this - in the end it was quite quick... As long as I can figure out how to access MatrixOP within a function as 

I will be set, right now I end up doing

which does not work since my string can easily be too long...

P.S. I am happy I figured out how to write Igor code in here...

Hello Johan,

It would help if you could explain why you need to express a MatrixOP command as a string.

You got good advice from Chozo with one minor clarification: the /NTHR flag does not really play a role unless you are processing multiple layers.

A.G.

Hi A.G.

You are correct, I got brilliant help from Chozo - and I know him from earlier so we are in contact via e-mail regarding some of these issues.

The reason I want to use MatrixOP as a string is the following (somewhat simplified for this particular example):

I have a large dataset where one parameter is time, and the time is a periodic function. I want to discard the data that falls outside the "time-windows" of interest. In order to do this I folded (thanks https://www.wavemetrics.com/code-snippet/wave-folding) the periodic data and obtained an equation where the time-window is. The equation in question is simple. Below is snippets of my code that attempts to do this. Since it is not working it is not tested, but if the string is shorter (I do this elsewhere in my code) I can get it to work, so in principle it should work

Here is an example of the periodic time. I have other waves of the same length and I want to discard all data that is not on the same rows as the "peaks" below - the relevant data is only in the peaks.

Here is how the above wave looks if I fold it onto itself with the correct boundaries

We see that the low limit is 700 and the high limit is 1000, and we see (or know) that this repeats with a periodicity of 1458. So I want to extract the rows in the range 

Below is a code-snipped (not tested since I fail in figuring out how to use the string when it becomes too long - but it should work I think) that attempts at doing this... There are other steps (divide by the output from  MatrixOP and zipNaNs) that I later need to do in order to get to the final wave - but those are details...

I hope that this was possible to understand. I tried to write as little as possible (!) and still make it possible to understand... Let's hope for the best...

Thanks!

Johan

Just for the record, I am also in email contact and have received some test data as well, so I might be in the unique position to have a complete picture of this particular problem. As Johan mentions, the goal is to select a limited 'region of interest' from a repeating data series. The region of interest is in the time data of an x,y,time data set (i.e., three or more separate waves) which record 'random' events in time on any x,y plane. For this reason, a periodic ROI section has to be tested against the time wave to create a 'mask' for the x,y or any other data which was associated with this time series. The code fraction above is for the purpose of creating this mask. I will work on it, but if somebody has some idea how to solve this in a neat way then please let me know.

Hi Johan,

Thanks for the detailed explanation.  I think I can see what you are trying to do with MatrixOP.

I'd never dream of using MatrixOP like that.  The approach is appropriate when you have one or two ranges but not many.  Instead, in your case, I prefer to create a 1D mask wave which should efficiently select the relevant segments.

A.G.

Dear all.

I have now solved this, much thanks to all who helped out. Since the wave I want to exclude is periodic I simply created a new wave that was set to 

This is now a "complete" project but strictly geared towards data from a particular spectrometer.

I will see how I go about to make this project available for the general public - if it is even interesting beyond a small group of people... Until then anyone who has data in the format of 

x,y,t,....,N-1, N... columns and want to reduce this to:

1. An image of x,y pairs

2. Filtering on time (or other colums)

3. (Optional) Create a 1D spectrum of the 2D image

might benefit from my code - feel free to write here and I will happily share my code. At present it is ugly but working, in the long run it might become beautiful and working...Maybe...

Best,

Johan Söderström