Fbinread equivalent for textfiles

Share icon

Is there an equivalent of
Fbinread file, tmpwave
for reading a fixed number of lines containing some numbers into a wave for a text file?
Loadwave does somthing like that but it does not read from the current position of an opened file. I would have to open the file separately and counted the lines until the current position.
FReadLine can read lines into a string.
You can then use sscanf to get out numeric data.
Compared to LoadWave this will be really slow of course.

July 4, 2014 at 02:12 am - Permalink

thomas_braun wrote:
FReadLine can read lines into a string.
You can then use sscanf to get out numeric data.
Compared to LoadWave this will be really slow of course.


Right now I am using Freadline in a loop. And yes thats very slow for big arrays and I want to speed that up with a function similar to Fbinread.

July 4, 2014 at 04:40 am - Permalink

You have not described your situation and motivation with enough detail for me to give you definitive advice but ...

Quote:

Loadwave does somthing like that but it does not read from the current position of an opened file.

If you know the line number (zero-based) where you want to start loading the data you can use LoadWave with the /L flag. This is how I would do it.

One way to determine the line number of interest is to read the entire file, or the first so-many bytes, into a string variable and then search through the string variable.

Here is code that shows how to read the file into a string variable:
http://www.igorexchange.com/node/5846

Another approach is to load the whole file into a text wave and then analyze that wave.

This all depends on factors such as how big the file is, what are you trying to get out of it and the general organization of the file.

July 4, 2014 at 08:30 am - Permalink

Ok, a quick explanation:
I have huge text files 30++MB with different data blocks (each 20k+ lines) separated by text headers (and subheaders ...) which I also have to evaluate.
And loading one of these files takes several minutes when I have to load each data line in a loop. Thats why I was looking for something similar to the fbinread function.

But I guess the only option is really to use the loadwave function with /L flag and increase a linecounter each time I read a line to get the offsets to the datablocks.

Loading the file up to the current position into a string and then do some fast line counting will also not be effective for these big files.

July 4, 2014 at 12:33 pm - Permalink

The only fast way to load it would be to load the whole thing in one LoadWave call and then extract sections of it into the desired final waves. The speed of this approach is in the fact that the text has to be traversed only once.

Whether this is practical or possible depends on the details of the file format.

July 4, 2014 at 02:42 pm - Permalink

hrodstein wrote:
The only fast way to load it would be to load the whole thing in one LoadWave call and then extract sections of it into the desired final waves.


Ok, is there a way to load the complete file in a single text wave?

thomas_braun wrote:
@Callisto Is that a standardized format?


It uses some kind of XML structure.

July 5, 2014 at 11:58 am - Permalink

Quote:
Ok, is there a way to load the complete file in a single text wave?

This command should load the entire file into a text wave with one line of the file stored in the corresponding point of the text wave:
LoadWave/J/V={"", "", 0, 0}/K=2


This command specifies no delimiter character (first "" in /V) and that all waves are to be created as text waves (/K=2).

With the text in the text wave you can then access the elements of the text wave quickly. If you change elements in the text wave it will be very slow but just reading the elements should be quick.

How you proceed from there depends on the details of the file and what you want to do with it.

Quote:
It uses some kind of XML structure.

There are some projects for loading XML:
http://www.igorexchange.com/project/XMLutils
http://www.igorexchange.com/project/udStFiLrXML

I don't have any experience with them and don't know if they would be fast with very large files.

July 5, 2014 at 03:55 pm - Permalink

Callisto wrote:
...

It uses some kind of XML structure.


Perhaps with XML and an XSL converter, you could convert to a data structure that might load faster in to Igor.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville

July 7, 2014 at 05:52 am - Permalink