speed of visaread vs visareadbinary

Share icon

Hello everyone,

I am using Igor Pro for data acquisition with various instruments. I am running v6.3.6.4 on Win7 Pro. The instrument is controlled using standard GPIB commands (SCPI) using a LAN connection.
I need to transfer relatively large amount of data from an instrument (for instance a Vector Network Analyzer). Just to give an idea I might need to transfer several thousands of data points.
I have noticed a dramatic difference in the time it take to do so depending on whether I do it using the "visaread" (sending data in ascii) or "visareadbinary" (sending data in binary) functions.
Transfering the exact same data with both function it took the following:
-85.2s for visaread (34296 bytes were transfered)
-35ms (yes, 0.035s) for visareadbinary (8007 bytes were transfered)

Here is a typical piece of code I used to do so:
Function timingtest()
    NVAR ZVL13  //reference to my instrument
    Variable timerRefNum
    Variable microSeconds
    Variable n
    string junkstring2=""
    n=100
       //visawrite zvl13, "form ascii" //for ascii transfer
    visawrite zvl13, "form real" //for IEEE754
        timerRefNum = startMSTimer
    if (timerRefNum == -1)
        Abort "All timers are in use"
    endif
    do
        visawrite zvl13,":CALC:DATA? SDAT"  //I do not think this one is fully universal but it is slightly instrument dependent
        //visaread /n=100000 /T="\n" zvl13,junkstring2 //ASCII
        visareadbinary /s=100000 /T="\n" zvl13,junkstring2 //BINARY
        n -= 1
    while (n > 0)
    microSeconds = stopMSTimer(timerRefNum)
    Print microSeconds/100, "microseconds per iteration"
End

I understand the number of bytes to be transfered is generally smaller when working with binary compared to ascii (about a factor 4 in my example) but clearly the time does not scale with the number of bytes.

I can also add that the 35ms I got using "visareadbinary" seems to be to some extent limited by the time it takes to actually send the GPIB command and the effect of the number of transfered bytes is seen only when it starts to be really large. On the other hand, using "visaread" the time somehow scales with the data size.

I came up with the conclusion that it is due to the way Igor handles ascii for the following reasons:
- I noticed the same thing with different instruments
- colleagues working with Python do not see such differences
- if I use NIMAX to transfer the ascii data, it goes way faster than with Igor

I wonder if it comes from the fact that Igor internally deals with binary or whether it is due to the OS I use (I have seen in the past large differences in fitting time running on Windows or Mac but I did not have the chance to test the current issue on Mac).

To conclude, as long as the instrument can send binary data, it should not be a problem but if anyone has some insight into this issue I would be happy to know more.

Regards,

Julien
Quote:
Transfering the exact same data with both function it took the following:
-85.2s for visaread (34296 bytes were transfered)
-35ms (yes, 0.035s) for visareadbinary (8007 bytes were transferred)


I find that very puzzling as the code for VISARead is nearly identical to the code for VISAReadBinary.

In both cases, because you have specified a terminator character, the VISA XOP has to read one byte at a time, so it can stop when the terminator is read.

If the instrument asserts the GPIB END signal with the last byte then you can use /T="". In this case, VISAReadBinary would ask for 100000 bytes and stop reading when END is received. VISARead, however, would still read one byte at a time. Given that VISAReadBinary is already fast, I would stick with the way you are doing it to avoid dependence on the instrument's implementation of the END signal.

Side note:
NVAR ZVL13  //reference to my instrument


Your function will fail if you change the current data folder. You should do this instead:
NVAR ZVL13 = root:ZVL13  //reference to my instrument

June 10, 2015 at 09:49 am - Permalink

hrodstein wrote:



If the instrument asserts the GPIB END signal with the last byte then you can use /T="". In this case, VISAReadBinary would ask for 100000 bytes and stop reading when END is received. VISARead, however, would still read one byte at a time. Given that VISAReadBinary is already fast, I would stick with the way you are doing it to avoid dependence on the instrument's implementation of the END signal.




Thanks a lot hrodstein for your comment.
Just to be complete, in the real function I wrote, I am using the VISAReadBinaryWave and VISAReadWave versions. Since I know exactly the amount of data I am expecting I can dimension the wave and thus I do not need to worry about termination and the length (i.e the /s flag in the previous example).
It would look like:
make /C /o /n=(tempsize) junkwave=nan  // with tempsize being the pre-determined number of data points. And I am using /C because I know I have 2 numbers per data point.
visareadbinarywave  /type=2  zvl13,junkwave // for binary, 32 bits
//visareadwave /T="\n" zvl13,junkwave //for ascii. I still give the termination even though I probably do not need it.


Julien

June 11, 2015 at 02:01 am - Permalink

hrodstein wrote:

In both cases, because you have specified a terminator character, the VISA XOP has to read one byte at a time, so it can stop when the terminator is read.

If the instrument asserts the GPIB END signal with the last byte then you can use /T="". In this case, VISAReadBinary would ask for 100000 bytes and stop reading when END is received. VISARead, however, would still read one byte at a time. Given that VISAReadBinary is already fast, I would stick with the way you are doing it to avoid dependence on the instrument's implementation of the END signal.



I realize I did not understand correctly what you meant when I first read!
It looks indeed like since I give /T="\n", VISA XOP reads one byte at a time. If I do not give it, VISAReadBinary runs even much faster. But it makes no difference for VISARead, as you wrote. Why is that? Is there no way for VISARead to read everything with a single read? It seems that it is what NIMAX is doing for instance. Still it does not quite explain the huge speed difference that I see between the two functions (for instance when I compare them giving "\T=\n" to both).
Any idea?

Julien

June 12, 2015 at 01:15 am - Permalink

Quote:
It looks indeed like since I give /T="\n", VISA XOP reads one byte at a time. If I do not give it, VISAReadBinary runs even much faster. But it makes no difference for VISARead, as you wrote. Why is that? Is there no way for VISARead to read everything with a single read?


VISARead has to read one byte at a time so that it can stop reading when it gets a terminator. This is necessary so that the next call to VISARead will start reading from the right place. In the past I used buffering to avoid this but it added complexity/fragility to the code so I abandoned it for a straightforward approach. Since the driver presumably has its own buffering, buffering in the XOP is not a big win.

Quote:
It seems that it is what NIMAX is doing for instance.


NI-MAX does not need to worry about assigning the right characters to the right variables - it just displays everything that is sent.

Quote:
Still it does not quite explain the huge speed difference that I see between the two functions (for instance when I compare them giving "\T=\n" to both).


I agree and I don't know why it is so slow in your case. I did not see this when developing VISA XOP, about a decade ago. And you are the first to mention this so I don't think it affects everyone.

June 12, 2015 at 10:09 am - Permalink

hrodstein wrote:

I agree and I don't know why it is so slow in your case. I did not see this when developing VISA XOP, about a decade ago. And you are the first to mention this so I don't think it affects everyone.


Thanks again hrodstein for your answer.
Do you see anything in the way I am initializing the connection that might be wrong:
function initlan()  //initialize lan
   
    VISAControl /Q testMode=0, dumpTestBuffer, clearTestBuffer, killIO
    variable session,instr
    string resourceName
    viOpenDefaultRM(session)

    variable /g root:zvl13
    NVAR zvl13=root:zvl13
    resourceName = "TCPIP0::ZVL13-100886::inst0::INSTR"
    viOpen(session, resourceName, 0, 0, instr)
    zvl13=instr
   
end


If anyone reads this and do/do not experience the same speed problem for visaread, I would be happy to know. I have experienced this on two totally different setups (OS/Igor version/instruments), the only thing that did not change was myself (and I realize that might very well be the weak link there!).
Regards,

Julien

June 15, 2015 at 04:22 am - Permalink

Quote:
Do you see anything in the way I am initializing the connection that might be wrong


No.

June 15, 2015 at 09:04 am - Permalink

I think I understood what is going on... I was not really using ethernet but an USB-->ethernet adapter. It appears that this is slowing down things dramatically (and this is consistent with the fact that I already saw that using a full USB connection). I would never have thought that USB drivers could cause anything in the ms timescale.
The final tests gave the following:
VISAWrite (or VISARead for only 1 Byte) takes about 0.7ms using ethernet but 2.4ms on USB.
Then if I read large amount of data using VISARead, it scales with the number of bytes (so that it can really takes tens of seconds for thousands of bytes). On the other hand, using binary, I am limited by the time it takes to actually execute one single VISARead (or VISAReadBinary). The data size only starts to matter in that case when it exceeds a thousand bytes. See the attached graph.
I guess the last thing I still do not quite understand is what is the intrinsic difference between VISARead and VISAReadBinary that makes one able to read everything in a single read and not the other one.
Thanks again hrodstein for your help.
Julien
Timing_test.pdf (3.49 KB)

June 16, 2015 at 04:29 am - Permalink

Quote:
I guess the last thing I still do not quite understand is what is the intrinsic difference between VISARead and VISAReadBinary that makes one able to read everything in a single read and not the other one.


Imagine that the instrument sends this data:
A,B<LF>

Now you execute this:
String str
VISAReadBinary/S=3 str


VISAReadBinary reads three bytes and is done.

Now imagine you execute this instead:
String aStr, bStr
VISARead/T=",\n"/N=3 aStr
VISARead/T=",\n"/N=3 bStr


If the first VISARead call read 3 bytes then there would be nothing left to read for the second VISARead call. The first call must stop after a terminator character is received so that the second VISARead call will read what it is supposed to read.

If you use VISAReadBinary with /T="any string", then VISAReadBinary behaves the same as VISARead - it reads one byte at a time for the same reason.

I have a vague recollection that, long ago, I tested the speed of the one-byte-at-a-time method and compared it to read-everything-all-at-once. I found that the one-byte-at-a-time method was slower but not dramatically. However I don't remember if I was using GPIB or serial port. It was definitely not Ethernet.

June 16, 2015 at 09:25 am - Permalink

hrodstein][quote wrote:

If you use VISAReadBinary with /T="any string", then VISAReadBinary behaves the same as VISARead - it reads one byte at a time for the same reason.


I guess what I got confused with is that the two functions have different default /T (meaning they look for different terminations). But I wonder why when I compare:
visareadbinary /s=100000 /T="" zvl13,junkstring2

and
visaread /n=100000 /T="" zvl13,junkstring2


VISARead needs to run several viread instances while VISAReadBinary do not. Is it just because VISAReadBinary is looking for an END character anyway while VISARead does not know what to look for?
Would there be a way to tweak VISARead to make it work the way VISAReadBinary does at least in some cases? But I guess if I prefer reading ascii, I could just use VISAReadBinary with ascii...

Julien

June 17, 2015 at 09:11 am - Permalink

Quote:
Is it just because VISAReadBinary is looking for an END character anyway while VISARead does not know what to look for?


The difference is that I made a special case for VISAReadBinary if /T="", which is the default. In this special case, VISAReadBinary asks for N bytes. The read automatically stops when N bytes are read or when the END signal is asserted by the sender.

I did not make a special case for VISARead when /T="" because I was thinking of its purpose as reading small amounts of data into one or more Igor variables.

Quote:
Would there be a way to tweak VISARead to make it work the way VISAReadBinary does at least in some cases?


I could make a special case for VISARead/T="". I'll give it some thought.

OK ... I made the change. See the attachment.

This change will ship with Igor7 (E.T.A. unknown), not with Igor6 because I don't want to risk introducing a bug in Igor6.
VISA XOP 1.10.zip (112.86 KB)

June 17, 2015 at 11:46 am - Permalink

hrodstein wrote:


I could make a special case for VISARead/T="". I'll give it some thought.

OK ... I made the change. See the attachment.


That looks great!
Just two comments:
-after installation of the XOP, Igor complained about a missing "msvcr120.dll" when I started it. But it looked like I had one with that name on my system anyway... I solved that with the installation of some Visual C++ packages from Microsoft. I am not sure what this problem was about. If that matters, I am running on Win7 Pro 64 bits (and running Igor 32 bits).
-in the help file with the revisions details, you mention
Quote:
VISARead attempts to read the number of bytes specified by the /S flag in one call rather than reading one byte at a time

I imagine it should be changed to the "/N flag".
Thanks for the work!
Julien

June 18, 2015 at 12:17 pm - Permalink

Quote:
after installation of the XOP, Igor complained about a missing "msvcr120.dll" when I started it.


I forgot that the new version of the XOP is targeted for Igor7. Igor7 ships with the required DLL but Igor6 does not.

Quote:
I imagine it should be changed to the "/N flag".


Thanks for catching that. I will fix it.

June 18, 2015 at 02:00 pm - Permalink