If Igor could retrieve a web page, what would you do with it?

Share icon

 

I've been playing with an idea about adding to Igor the ability to retrieve the HTML of a web page into a string (probably) or a text wave (maybe). And probably a way to use HTTP to download a file, too, similar to FTPDownload.

So, two commands:

 

String text= GetHTTPText("http://www.whatever.com/file.html") // returns the HTML

HTTPDownload "http://www.whatever.com/file.dmg", "/Users/Me/Documents" // downloads the file to the directory


(Note: BrowseURL does something different than HTTPDownload: BrowseURL tells your default web browser to open a URL. GetHTTPText and HTTPDownload wouldn't invoke a web browser.)

It seems like a good starting point for adding more "web-savvy" to Igor.

If done, what would *you* do with a feature like this?
 
Hmm,
some aspects might be interesting. On out neutron scattering instruments we have Data Acquisition servers that control all our detector banks. The public face of these is a webpage which we can interact with. One of the buttons on the webpage tells the computer to save a local copy of the data. This is saved as an HDF file that can then opened into IGOR.

To do the pressing of the buttons without using the webpage I use CURL (with a script), from the DOS command line. This is turn is controlled from IGOR, via ExecuteScriptText.

My wishlist during this period has been that some aspects of curl could be integrated into IGOR, so that I could press buttons, etc, and get the results.

However, this practice is somewhat of a stopgap... Some of our datafiles are 180 Mb in size, which takes a while to transfer, even over 100Mbit/s networks. I would also like IGOR to be able to perform TCP/IP socket communication (faster than file transfer). Some people out there have written some XOP code already, but this has to be adapted everytime.

September 12, 2007 at 03:19 pm - Permalink

JimProuty wrote:

If done, what would *you* do with a feature like this?


One possibility would be to do what I mention at http://www.igorexchange.com/node/175#comment-43

Basically, it's possible (though not currently enabled) to query the IgorExchange server to find out the most recent release of a project and various other information about it. The output is XML which could potentially be processed with Jeff Weimer's XML processing procedures. As is, I think the only potential way to do this automatically would be for the server to periodically write these XML files and then use FTPDownload to download the file. But being able to just grab the XML via GetHTTPText would be a lot easier and more flexible.

Of course other types of XML data feeds would be fetchable in a similar way, which I'm sure could be useful to many.

It would also be nice to be able to do extended HTTP connections and send data via GET and POST requests. SSL would be cool too but would probably involve a lot more work on your part. I'm thinking of the functionality of the Unix tool wget here. It might be possible for one to use ExecuteScriptText to call wget from within Igor and pipe the output of that to a file that is then opened within Igor.

Adam

September 12, 2007 at 07:17 pm - Permalink

JimProuty wrote:
I've been playing with an idea about adding to Igor the ability to retrieve the HTML of a web page into a string (probably) or a text wave (maybe). And probably a way to use HTTP to download a file, too, similar to FTPDownload.


I wonder what could be done with web services. I don't know enough to answer the question but I do think Igor should have more web-interaction capability.

September 13, 2007 at 12:26 pm - Permalink

I would like to be able to do file transfer through an ssh tunnel (scp) from within Igor. Presumably on the mac scp is part of OSX, but on the pc one has to install a command line ssh client (eg putty) and invoke that through ExecuteScriptText. The built in FTPUpload and FTPDownload are of little use now that everyone requires secure server connections. When I use Igor for data acquisition I use ExecuteScriptText to run a batch file to sequentially zip and scp data to a web server using a private key, which satisfies the security requirements for the systems administrator. When I distribute the code I need to include open source code for command line compression, transfer and mail (7-zip, putty and blat for the pc), and making those things work cross platform is a pain.

I've been toying with the idea of controlling igor remotely by putting something on a server that Igor can check periodically. GetHTTPText could simplify that sort of thing. If I ever have the nerve to do this, it'll be done via the web server because I don't want to open up the computer that runs Igor.

September 14, 2007 at 10:23 am - Permalink

 

JimProuty wrote:
I've been playing with an idea about adding to Igor the ability to retrieve the HTML of a web page into a string (probably) or a text wave (maybe). And probably a way to use HTTP to download a file, too, similar to FTPDownload.

So, two commands:

 

// Get the HTML
String text= GetHTTPText("http://www.whatever.com/file.html")

// Download the file to the directory
HTTPDownload "http://www.whatever.com/file.dmg", "/Users/Me/Documents"


(Note: BrowseURL does something different than HTTPDownload: BrowseURL tells your default web browser to open a URL. GetHTTPText and HTTPDownload wouldn't invoke a web browser.)

It seems like a good starting point for adding more "web-savvy" to Igor.

If done, what would *you* do with a feature like this?



Could be quite a lot, if you use some standard tools...we are all supposed to be using XHTML these days so I would gear it towards using XML as the basic format, perhaps running any HTML through a validator and/or a utility like htmlTidy or xmllint to make it well formed. Then presumably you want to extract data. For that I would use an XSLT template and add an XSL engine in Igor to output the data to waves. Clearly the user has to know something about the format to be able to do this. (With formats which are not too complex it should be possible to parse directly, or generate the XSL template on-the-fly with a little user input).
And the use? Well (at CERN) we already have web pages and web services serving things like temperature and environmental conditions for our detectors, this is gradually being extended so our databases have web interfaces which return xml.
Some databases (Oracle) can generate XML directly from SQL and have a standard format, so it would be possible to write a standard XSLT which would be pretty flexible.

January 19, 2008 at 11:09 am - Permalink

Shaun_Roe wrote:

... so it would be possible to write a standard XSLT which would be pretty flexible.


Out of curiousity, why build an XSLT engine into Igor Pro when both MacOS and WinXX have free, well-developed XSLT engines already available?

Or are you just saying to use a standard XSL transform????

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH

January 21, 2008 at 07:23 pm - Permalink

jjweimer wrote:
Shaun_Roe wrote:

... so it would be possible to write a standard XSLT which would be pretty flexible.


Out of curiousity, why build an XSLT engine into Igor Pro when both MacOS and WinXX have free, well-developed XSLT engines already available?

Or are you just saying to use a standard XSL transform????

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


I mean write an XSL transform, and hopefully use the in-built xslt processor

January 25, 2008 at 09:16 am - Permalink

At the moment, I can't think of anything scientific that I would do with it, but I would use it to scrape data from databases on the web. I do this now with PHP and curl, and then parse out the good stuff, and then put that info into Igor for visualization. A direct GetHTTP operation would save me a step.

In general, more network capability would be desirable. My friend is working on a networking XOP as we speak, although it's not his top priority right now.

Rick

February 22, 2008 at 12:22 pm - Permalink

A networking XOP would be manna to me. At the moment we have an instrument control server that runs an experiment. But then I have to transfer data manually to IGOR to analyse, before returning to type in the results by hand. If I had an XOP to take care of the connection things would be so much simpler.

Andy

March 4, 2008 at 07:49 pm - Permalink

 

An XOP is now available for dealing with URLS:
http://www.igorexchange.com/project/easyHttp

It is based on the LIBCURL libraries. You can get webpages, perform ftp downloads, get files from other filesystems, etc.

e.g.

 

 
//get a webpage and put it into the S_gethttp string
easyHttp "www.wavemetrics.com"
print S_gethttp
//save into a file
easyHttp/File="C:/file.dmg" "http://www.whatever.com/file.dmg"
//get a remote file over your network and put it into S_getHttp
easyhttp "file:///C:/Program%20Files/WaveMetrics/Igor%20Pro%20Folder/Igor%20Extensions/About%20Igor%20Extensions.txt"

 


Future plans include authentication + FORM/POST.

 

March 9, 2008 at 05:19 pm - Permalink

Igor 6 and later have the FetchURL command:

FetchURL(urlStr )

The FetchURL function returns a string containing the server's response to a request to get the contents of the URL specified by urlStr.

If urlStr contains a URL that uses the file:// scheme, the contents of the local file is returned. 

July 27, 2018 at 11:43 am - Permalink

And Igor 7 and beyond have URLRequest, which is much more capable and flexible, though a bit more complicated than FetchURL.

July 27, 2018 at 01:55 pm - Permalink