
read in data (txt file) with special time format

zek
I try to read in a file like this
Datum Zeit Tmp_ak WGms WR DatumZeit
17.09.2016 09.50.00.02 15.8 1.4 248 17.09.2016 09.50.00.02
17.09.2016 09.50.00.12 15.7 1.3 253.6 17.09.2016 09.50.00.12
17.09.2016 09.50.00.22 15.7 1.3 254.8 17.09.2016 09.50.00.22
17.09.2016 09.50.00.32 15.7 1.4 255.1 17.09.2016 09.50.00.32
17.09.2016 09.50.00.42 15.7 1.3 254.9 17.09.2016 09.50.00.42
17.09.2016 09.50.00.52 15.7 1.3 253.2 17.09.2016 09.50.00.52
17.09.2016 09.50.00.62 15.7 1.3 253.5 17.09.2016 09.50.00.62
17.09.2016 09.50.00.72 15.7 1.3 250.9 17.09.2016 09.50.00.72
17.09.2016 09.50.00.82 15.7 1.3 250.4 17.09.2016 09.50.00.82
17.09.2016 09.50.00.92 15.7 1.2 250.7 17.09.2016 09.50.00.92
First line is the column name; first and second column gives date and time, which I want to merge in one column. The column time has a very special time format. The other columns are temperature, wind speed and wind direction and DateTime.
I tried to load this file with the Data-Loader from Igor Pro, but I has problems with this special time format. Does anyone has a solution or has somebody already some code, which is useful for the read-in of such type of files?
Best regards
Kerstin
I recommend saving this as an Igor Procedure file so that it will be automatically loaded when you launch Igor. Execute this for details:
DisplayHelpTopic "Special Folders"
If this does not work with your file then it is probably due to formatting subtleties. In that case, attach a zip archive of your file so I can see exactly what the file contains.
July 10, 2017 at 10:02 am - Permalink
The procedure works, :-).
Not very fast, probably of the big data file.
I attached one of my data file.
Best regards
Kerstin
July 14, 2017 at 02:32 am - Permalink
Why do you have duplicate columns in the data file? (Datum, Zeit) in the beginning, (Datum, Zeit) at the end? Each line begins and ends with this tuple. For the file that you posted if you remove the last two columns, the reduction is substantial: 70173015 bytes > 42524260 bytes (~67MB > ~41MB).
Under a linux or cygwin environment you'd do something like:
cat wind_test_file.txt | awk '{print $1,$2,$3,$4,$5}' > wind_test_file_out
Another recommendation is to change the delimiter to a comma. Then, use the
Data > Load Waves > Load delimited text...
menu, to load the data keeping the time as a text. Parse it after loading.Under linux or cygwin in vim you'd issue the following to substitute space for comma:
:1,$s/ /,/g
Another recommendation is to combine the date and time into one variable ala julian date, unix time, etc. This will provide further savings on the file size, but will increase processing upon file loading.
edit: As a matter of fact, you don't need the date. It's a waste of bytes. Tell the person generating these files to name the filenames with the corresponding date, or put it in the beginning of each file as a header or something, and dump data only for this particular day in this particular file. I would also use fractional time. Even though a delta t would be enough (which btw is not precisely 100ms, just a tiny bit less, ~99.99(93)ms)
best,
_sk
July 14, 2017 at 06:24 am - Permalink