easyHttp

Share icon

An XOP is now available for dealing with URLS:
http://www.igorexchange.com/project/easyHttp

It is based on the LIBCURL libraries. You can get webpages, perform ftp downloads, get files from other filesystems, etc.

e.g.
 
//get a webpage and put it into the S_gethttp string
easyHttp "www.wavemetrics.com"
print S_gethttp
//save into a file
easyHttp/File="C:/file.dmg" "http://www.whatever.com/file.dmg"
//get a remote file over your network and put it into S_getHttp
easyhttp "file:///C:/Program%20Files/WaveMetrics/Igor%20Pro%20Folder/Igor%20Extensions/About%20Igor%20Extensions.txt"


Future plans include authentication + FORM/POST. This should be fairly easy with libcurl
Finally put POST into the XOP. Using the /POST=poststr flag you can do posts to an HTTP server, e.g.

/POST="name=andrew&surname=nelson"

April 21, 2011 at 01:07 am - Permalink

FetchURL can only do a HTTP GET request to get information. The addition of POST to easyHttp means that it can be used to send data to a HTTP server. For example, it might be possible to upload a file or simulate filling out a web form. FetchURL couldn't do either of those things (see caveat below). I do plan to eventually introduce HTTP POST support into Igor itself, but that hasn't happened yet.

caveat: Very simple web forms, such as a search keywords form, often use GET instead of POST to communicate the information to the server. More complicated forms almost always use POST.

April 21, 2011 at 09:30 am - Permalink

johnweeks wrote:

Since I know very little about this sort of stuff-- how is this different from what we provide via FetchURL?
support@wavemetrics.com


As Adam said. The two most supported requests over HTTP are GET and POST. I always think of POST being were you wouldn't want the request being sent twice, e.g. buying something, or deleting something - in other words where you are changing the state of the server. GET is a simple request for information.

However, easyHttp can also do things that FetchURL can't at the moment. i.e. HTTPS, proxies, authentication, FTP, SFTP, LDAP, HTTP POST, Of those POST, authentication and proxies are the most missed. Most people in corporate environments have to use proxies. If they can't get out through the proxy, then the operation is unusable.

April 21, 2011 at 07:44 pm - Permalink

For some reason I receive an error from the /P flag, which Igor says it doesn't recognize. Placing the whole path in the fileNameStr works, so there is that workaround.

My real issue is seem to have SSL problems using easyHttp to pull down data from https sites. http works fine, but I want to pull data from https, namely dropbox. But even "https://encrypted.google.com" throws the same error.

easyhttp /TIME = 10 "https://encrypted.google.com"
gives S_Value of
" SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"

any thoughts?

February 21, 2013 at 01:52 pm - Permalink

tjpeshek wrote:
For some reason I receive an error from the /P flag, which Igor says it doesn't recognize. Placing the whole path in the fileNameStr works, so there is that workaround.

My real issue is seem to have SSL problems using easyHttp to pull down data from https sites. http works fine, but I want to pull data from https, namely dropbox. But even "https://encrypted.google.com" throws the same error.

easyhttp /TIME = 10 "https://encrypted.google.com"
gives S_Value of
" SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"

any thoughts?


Does the path specified by /P exist?
That operation works for me:
•easyHttp "https://encrypted.google.com"
print S_getHttp

results in:
<!doctype html><html itemscope="itemscope" itemtype="http://schema.org/WebPage&quot;><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta itemprop="image" content="/images/google_favicon_128."<br />


What OS/OS version are you using? What is your Igor version? Are you behind a proxy/firewall? Are you using the latest XOP? If you are on OS X what is the result of typing the following from Terminal.app?
<br /> curl "https://encrypted.google.com"<br />

February 21, 2013 at 03:06 pm - Permalink

Thanks for the quick reply. More info: OS: Windows 7; Igor: v6.22; latest XOP of easyHttp? yes, downloaded it yesterday.

I have cygwin installed and I did perform

curl "https://encrypted.google.com"
and received a proper response.

I even performed the curl operation on the site I'm trying to access through easyHttp and it gave me the data. However, for experimentation I tried a
curl -2 MyURL
and received an unknown SSL protocol error, but
curl -3 MyURL
worked also.

/P path exists, but Igor chokes on the /P flag itself. Only at runtime, at compiletime the ipf compiles.

I noticed in this thread: http://www.igorexchange.com/node/3180
that CURLOPT_SSL_VERIFYHOST is set to 2, I may try setting it to zero to get it working in the near term.

My goal is remote data logging whereby the data are saved to a dropbox folder. Since dropbox has an unsecure http public option none of my current issues are blocking, but a better solution is worth putting effort into.

February 22, 2013 at 07:29 am - Permalink

Apparently I removed the /P option from the XOP, but forgot to document it in the help file. So that won't work. I'm going to look into the other issues.

A.

February 22, 2013 at 12:34 pm - Permalink

I think the reason why the https connection doesn't work is because there is no location for certificates specified in the XOP. This will require a bit more investigation. In the meantime it's possible to build the XOP to tell it not to check the certificate at all. This is the solution in the other forum post. This is insecure, but will allow you to do what you want to do.
Unfortunately this will mean a recompile of the XOP.

On OS X I think it works because the inbuilt openSSL library knows where all the certificates live on a system.

I suspect it will be possible to get the XOP to do this on windows, but it will need a bit of persuading.

A.

February 22, 2013 at 01:55 pm - Permalink

Dear tjpeshek,
I have spent the afternoon looking into this. Instead of using openssl in libcurl I am now using schannel. The difference is that openssl requires certificates from the XOP, whereas schannel is the windows inbuilt version of ssl and has access to the certificate store of the machine. Thus, I believe that the new version will do what you need it to. Can you download it and try it out?

A.

February 26, 2013 at 03:05 pm - Permalink

Hi,

I was curious what the state of the easyHTTP XOP was nowadays. I was attempting to use it to access a google docs spreadsheet and I ran into what seemed like the same Open SSL issues tjpeshek had mentioned above. Currently the error I am getting is the following:

S_value = "SSL certificate problem: unable to get local issuer certificate"

for reference is the info obtained from setting the URL string to "licence":
S_getHttp = "easyHttp uses: libcurl, libssh2, libproxy, libz, openssl. Please see the COPYING.txt file from: http://www.igorexchange.com/project/easyHttp"

and for "curl_version_info":
S_getHttp = "version:7.32.0;host:i386-pc-win32;features:669;ssl_version:OpenSSL/1.0.0d;libz_version:1.2.6;PROTOCOLS:dict,file,ftp,ftps,gopher,http,https,imap,imaps,ldap,pop3,pop3s,rtsp,scp,sftp,smtp,smtps,telnet,tftp,;libssh2_version:libssh2/1.4.0;"

the call I used in the command line was easyHTTP url string for an https type URL, I did try setting the PASS flag to my user authentication (with and without percent encoding) and got the same result.

Any info would be greatly appreciated!

-Albert

andyfaff wrote:
Dear tjpeshek,
I have spent the afternoon looking into this. Instead of using openssl in libcurl I am now using schannel. The difference is that openssl requires certificates from the XOP, whereas schannel is the windows inbuilt version of ssl and has access to the certificate store of the machine. Thus, I believe that the new version will do what you need it to. Can you download it and try it out?

A.

August 4, 2014 at 01:26 pm - Permalink

The XOP is still being maintained but I'm not currently adding extra features as it's already fairly feature rich. The XOP doesn't work with certificate stores at the moment (https does work for a lot of things though). Sorry, I'm not able to help you further myself. I would recommend trying and experimenting with curl/libcurl. If you get anywhere let me know and I may add features.

A.

August 4, 2014 at 05:31 pm - Permalink

Thanks for the response! I did find some interesting observations for anyone's curiosity:

1) easyHttp works fine on OS X (I am using mavericks) when trying to access https type URL's, specifically google docs
2) private google docs will only download the html text of the web page from the url (which I guess is to be expected, since google doc is built around their scripting API?)
3) you CAN import a text delimited google doc spreadsheet but you have to do the following:
a) publish the google doc
b) obtain a link to formatted data (e.g. csv, etc.)
4) easyHttp has issues accessing certain https on Windows (I am using Win 7), specifically google docs

I didn't troll around online too much but from cURL themselves there is this link http://curl.haxx.se/docs/sslcerts.html discussing SSL but didn't seem to shed anymore light than what you had already discussed.

I suppose the hope was to access a text delimited google spreadsheet (easier to work with, doesn't require a full blown database) and import that into Igor for some metadata tricks we use for organizing analysis. But the requirement to publish the spreadsheet to get CSV formatted text is not ideal. Anyone, fun diversion but looks like I'll be tinkering around with other methods.

Cheers,

Albert

August 5, 2014 at 12:09 pm - Permalink

aaumentado... wrote:

I suppose the hope was to access a text delimited google spreadsheet (easier to work with, doesn't require a full blown database) and import that into Igor for some metadata tricks we use for organizing analysis. But the requirement to publish the spreadsheet to get CSV formatted text is not ideal. Anyone, fun diversion but looks like I'll be tinkering around with other methods.


Could you follow up on a separate thread with your progress on this. Especially in particular, when by "other methods" you mean to find other ways to pull the contents of a cloud-stored GoogleDocs spreadsheet in to Igor Pro instead of via easyHTTP.

Thanks!

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville

August 5, 2014 at 12:55 pm - Permalink

aaumentado wrote:

I suppose the hope was to access a text delimited google spreadsheet (easier to work with, doesn't require a full blown database) and import that into Igor for some metadata tricks we use for organizing analysis. But the requirement to publish the spreadsheet to get CSV formatted text is not ideal. Anyone, fun diversion but looks like I'll be tinkering around with other methods.


Google has a Google Drive application (https://tools.google.com/dlpage/drive) that allows you to automatically two-way sync a disk directory with a Google Drive account. Perhaps you could set that up and then just access the local copy of the document.

Adam

August 5, 2014 at 03:25 pm - Permalink

Unfortunately I don't have the time to look into SSLcerts, etc. This kind of work requires a lot of dedicated time and effort, as well as close attention to the security aspects of what one is trying to do. Unfortunately I don't have the time to dedicate to this.

August 5, 2014 at 04:56 pm - Permalink

I guess the problem with https URLs is that libcurl is still using openssl for checking the certificates.
-easyHttp "curl_version_info" Value = version:7.32.0;host:i386-pc-win32;features:669;ssl_version:OpenSSL/1.0.0d;libz_version:1.2.6;PROTOCOLS:dict,file,ftp,ftps,gopher,http,https,imap,imaps,ldap,pop3,pop3s,rtsp,scp,sftp,smtp,smtps,telnet,tftp,;libssh2_version:libssh2/1.4.0;

I've also tried to compile curl on windows (mingw) with schannel instead of openssl but encountered similiar problems.

I guess for now the easiest thing would be to turn off SSL certificate verification e.g.
--- easyHttp.cpp Sun Dec 22 15:58:32 2013 +++ easyHttp_orig.cpp Wed Aug 6 13:20:18 2014 @@ -348,6 +348,10 @@ if(p->TIMEFlagNumber > 0) curl_easy_setopt(curl, CURLOPT_TIMEOUT, (long)p->TIMEFlagNumber); + /* TODO only do that if some flag is given */ + curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, (long) 0); + curl_easy_setopt(curl, CURLOPT_SSL_VERIFPEER, (long) 0); + /* do the operation */ try{ res = curl_easy_perform(curl);

There is no compilation tooling in the easyHttp package included so I could not try it out.

August 6, 2014 at 06:26 am - Permalink