How fast can high-speed, multichannel data acquisition run?
TSzuts
I have a high-bandwidth data acquisition task that I initially doubted could be done in Igor. To my surprise, it can be done, but not quite at the speed I need it to run.
The task is to acquire analog input from 64 channels at 10 kHz, save it to disk, analyze each chunk of data as it comes in, and then output to a digital port. My hardware is an NI M-series card (USB-6255) and a PC with Windows 7. The rub is that I’d like the read-write-analyze loop to run every 10 ms to minimize the delay between input and output.
Currently, Igor can reach a loop speed of 40 ms when setting up the acquisition with
DAQmx_Scan and a background procedure that calls FIFO2wave, etc. When I decrease the loop speed to 30 ms, Igor simply crashes, even though the background procedure execution time is about 5 ms. The loop period can go down to 30 ms when tying the FIFO directly to a file, but then I can’t access the input signals AND it’s still too slow.So my questions are:
Would a faster machine help me get to a 10 ms loop period?
Are XOPs my best and only option from here?
Many thanks,
Tobi
You realize that you are pushing the envelope in a program that wasn't originally written for real-time applications, right?
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
December 1, 2011 at 05:27 pm - Permalink
2) What is the analysis you have to do?
One possible design pattern would be to have 1 thread reading the data and posting it to a queue, 1 thread to read the queue and analyze. Depending on the chores in that thread you could do the digital out from there.
Unfortunately the FIFO ops don't look like they are threadsafe, perhaps that 1st thread could be a background task.
Could you post some code (if it's not too long?)
December 1, 2011 at 07:12 pm - Permalink
if you want to do it fast, the standard way is typically to have:
1: hardware which does acquisition alone, just getting a trigger from outside and have a cache to save the data
2: get the data asynchronous from the machine
3. if you need to do real fast calculation do it directly on the hardware (DSP etc.)
4. don't use very slow interfaces like USB ...
Stefan
December 2, 2011 at 02:48 am - Permalink
You're getting some good advice here.
I think the solutions proposed by andyfaff and stefanm would require that you write your own custom XOP. That might not be much harder than coercing Igor and NIDAQ Tools MX to do it for you.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
December 2, 2011 at 09:34 am - Permalink
Too true; the main reason that I’m doing this is so that the analysis code can be written in Igor, which everybody in my laboratory uses regularly! That 20 ms loop speed means that ANY analysis done in Igor will have a minimal latency of 20 ms – that’s the key parameter here.
We save to disk so we can also do off-line analysis. Since the signals come from a mildly complicated biological preparation, they’re worth keeping around.
The analysis is a little ill-defined right now, and will depend a lot on the end user (not me: I’m just the developer!). Minimal case is finding thresholds and a little bit of matrix math (almost trivial); next highest is multi-dimensional waveform discrimination based on the shape of short segments (aka, spike sorting). The ultimate goal is to compute the response properties of neurons or neural populations. Ultimately, there’s much more flexibility here than in the acquire step.
The multi-thread queue idea is a good way to speed up the analysis loop: if I write the acquire task as an XOP, that’s where I’ll start.
StefanM:
Fortunately, these task parameters are pretty stable and unlikely to get more demanding! The hard part is to make a system that’s flexible and easy enough for others to take advantage of...hence why I’m trying to do it in Igor in the first place.
I was surprised that USB is fast enough for this data stream, too!
Summing up this advice, it seems like I could achieve an output response latency of >20 ms with Igor – less than half of the current speed (40+ ms). A latency of 1-10 ms comes from the custom XOP loop (say) and 20 ms from Igor - might be good enough.
Thank you all for your help!
Tobi
December 2, 2011 at 02:14 pm - Permalink
In this regard, could you gain anything on processing speed by selecting the absolute minimal processing requirement that is demanded by the feedback, leaving all other processing for later? If all you need for example is the average intensity of a signal over a time span as a feedback to adjust an input gain, then a simple sum of all incoming intensity in that time span should serve (followed by a hardware gain adjustment control). All other processing can be dumped off to a "non-real-time" mode.
You may already be this far in your developments ... just thought I'd ask.
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville
December 2, 2011 at 03:25 pm - Permalink
That's exactly it. I wish the analysis step were the rate limiting one--it's much easier to speed that up. The problem is that nothing can go faster than Igor's 20 ms loop time. So you don't gain anything by reducing your analysis speed from 10 to 1 ms!!
Tobi
December 2, 2011 at 04:38 pm - Permalink
Not so. One solution is to make use of asynchronous acquisition and/or background threads. Basically you need to make some or all of your tasks independent of the timing of the main loop, which means the following:
1) As stefanm suggests, run the acquisition on the device itself, and read out the data asynchronously. Unfortunately, Igor's Nidaq tools cannot not be run in a preemptive thread, so you would do readout using a background task. That is not a problem since the device will do its own buffering. Also, the added advantage is that this will likely guarantee you the highest timing accuracy (depending on the device). So I would seriously recommend looking into this.
2) If you cannot run the acquisition on the device itself, your best option is to run it in a preemptive thread, and to transfer the data from the preemptive thread to a background task. Since Igor's Nidaq is not thread-safe, you would have to write your own XOP to communicate with the device. If you have any familiarity with XOP programming then that is actually not as hard as it may seem.
3) If any of your downstream processing is slow (and writing to disk certainly is), you will also want to decouple this from the main loop, and run it in an Igor thread. Basically this thread receives data folders with new data to write (think
ThreadGroupGetDFin the thread andThreadGroupPutDFin the background task) and takes care of it. Even better is that this is fire-and-forget, so the background task doesn't have to care about any of it once the data has been submitted.4) The same is true for the analysis. If it is slow, then run it in its own thread. If not then just do it in the main thread.
See also the "Slow Data Acq" example experiment included with Igor.
Threading can be complex. However, if you have a clear mental picture of how everything fits together it can actually be surprisingly straightforward. Where it becomes ugly is usually in situations involving error reporting, progress updates, or aborts. And if you can get the device to function asynchronously (strongly recommended), then you may not need to do any threading at all.
By the way, if you were using something other than Igor, you would likely end up with very similar solutions.
December 3, 2011 at 03:41 am - Permalink
December 3, 2011 at 07:04 pm - Permalink
True - I was being overly general when I said that "nothing can go faster than Igor's 20 ms loop time." What I meant to say was that the process chain (acquire, analyze, output), will be limited to that loop time if any one part of it is written as an Igor function - slowest step wins. Johnweeks says that the time limit applies to background tasks too, unfortunately. If I write everything in XOPs, that takes away the advantage of letting others write analysis code in Igor quickly and easily.
Tobi
December 5, 2011 at 11:09 am - Permalink
I don't really see the limitation. First of all, I don't think there is typically a need to perform GUI updates or data saving at a rate exceeding 50 Hz. If faster GUI updates are important to you, Igor is probably not the appropriate tool for the job.
I think the only aspect where the loop period is excessively slow is in the data acquisition. If you perform direct sampling of the I/O ports on the DAQ device from the background task, you will be limited to sampling every 20 ms. However, I think that you will want to avoid doing that for a number of reasons. Simply use the Nidaq tools to set up a continuous sampling, and read that out in the background task (disclaimer - I'm not all that familiar with nidaq tools – but I would find it surprising if that functionality didn't exist).
Then set up threads for the data saving and analysis. You can have the logging function take multiple samples at once, which means that you can simply fire off a new batch every 20 ms. Have the background task call ThreadGroupPutDF with the new samples, and have the thread block on ThreadGroupGetDF until data is available. Because these are queues, Igor will make sure that your data is safe while it's waiting to be written to disk. Here's an overview:
myData = GetMyData() // your acquisition function. Calls into nidaq tools at some point.
DFREF freeFolder = NewFreeDataFolder()
Duplicate myData, freeFolder:myData
ThreadGroupPutDF mySavingThreadID, freeFolder
// do other stuff or end this iteration of the task
// in the thread that saves the data
for (;;)
DFREF newData = ThreadGroupGetDF(0, inf) // wait forever -- possible deadlock if you're doing things wrong
// check for some magic 'stop' value in newData to stop the thread
// you need to send this magic data when the acquisition stops
wave myNewData = newData:myData
savemynewdata(myNewData) // your function. Uses Fprintf or whatever
endfor
The same is true for the analysis. Have the background task call ThreadGroupPutDF, and the analysis thread likewise blocks on ThreadGroupGetDF. Then, to get the results back out, you do the reverse, but make sure that the background thread calls ThreadGroupGetDF with a small timeout value so that the background task isn't blocked while the newest data unit is still executing.
You can have Igor code in the analysis and use it without blocking the background task by using threading. The only limitation is that the analysis needs to be threadsafe. Consider the following:
// but you're free to modify this template to whatever you want.
ThreadSafe Function /WAVE MyAnalysisPrototype(data)
wave data // contains the newly acquired analysis data
End
ThreadSafe Function DoAnalysisWorker(myAnalysisFunction)
FUNCREF MyAnalysisPrototype myAnalysisFunction
for (;;)
DFREF newData = ThreadGroupGetDF(0, inf) // wait forever -- possible deadlock if you're doing things wrong
wave myNewData = newData:myData
wave result = myAnalysisFunction(myNewData) // this is cool – the analysis can be completely different
// depending on what function the user provides in ThreadGroupCreate, as long as it is
// based on MyAnalysisPrototype. But you don't have to change any part of the threading!
// make a new free data folder and return it to the main thread with ThreadGroupPutDF
endfor
End
And then simply pass the appropriate function when you call
ThreadGroupCreate. The users of your software can make whatever function they want, in Igor code, provided that it is threadsafe and conforms to your prototype, and it will play nice with your code. Best of all is that, even if the function is slow, your background task is still firing away at 20 ms because it doesn't have to care about the preemptive thread! (Assuming that you're not sucking up all the CPU power in the background threads - unlikely on modern PCs with more than one CPU and if your analysis uses just a single thread.)December 5, 2011 at 01:20 pm - Permalink
Actually, I wasn't quite clear about the 20 mS limit. What I meant was that the recurrence time for a background task is limited by the main event loop, which runs at 20 mS intervals. A tight loop in a user-defined function can run much faster than 20 mS. BUT, a user-defined function *prevents* the main loop from running, and NIDAQ Tools depends on the main loop to get messages for transferring data. You can work around that problem by calling DoXOPIdle in your user-defined function. Calling DoXOPIdle is just like running the main event loop, except that you decide when it should happen.
An alternative might be to use fDAQmx_ScanGetAvailable() to get data in a tight loop inside a user function. That might be better than DoXOPIdle in that it will only run the NIDAQ Tools XOP, and not every other XOP in your Igor installation.
It may not be quite applicable to your situation, but my intention for this sort of feedback loop was that you would call DAQmx_AI_SetupReader and fDAQmx_AI_GetReader to run your servo loop. Those will work in a tight loop in a user function, and should be much faster than what you're trying to do. Your loop would have to save results to the FIFO itself, rather than relying on DAQmx_Scan to do it, though.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
December 5, 2011 at 04:14 pm - Permalink
It seems to me, the only "analysis" parts you need to write in XOP are those that are needed to calculate the feedback signal from the input signal in order to apply it to the output signal. Everything else could be Igor code.
Unless of course you are expecting that your users will want to operate at a refresh rate above 50 Hz too :-)
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville
December 5, 2011 at 04:14 pm - Permalink
Tobi
December 6, 2011 at 08:25 am - Permalink
Even that wouldn't work- background tasks are checked and run once each time around the main event loop. If you have 5 background tasks whose times have all expired, they will all get run one after another. They will all then not run again until the main event loop goes around again.
That's true. Keep in mind that even a tight loop in a user-defined function is much slower than C/C++ code in an XOP.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
December 7, 2011 at 09:54 am - Permalink