Embedded Python

Share icon

Python is a scripting language with a large number of numerical libraries for data analysis. Unfortunately, graphical data exploration using even the most recent Python libraries is terribly primitive and awkward compared to Igor. One way to co-opt a very large number of the numerical libraries for use in Igor would be to embed the Python scripting language into Igor. If I understand correctly, this could be done in an XOP using the strategies outlined here:

http://docs.python.org/extending/embedding.html

Python scripts and commands could be run from an Igor operation, and the resulting Python data structures could be converted by the XOP to C data structures and then to Igor data structures. This would help connect Igor more widely to a growing code base.

Does anyone have any interest or insight into such a project?

Rick


I've thought about this and think it would be relatively trivial to write a quick XOP to do specific jobs in the python interpreter. However, for it to be really useful one has to deal with the output that comes back from the python script/command/method. This output is obviously varied:
1) Tuple
2) List
3) Dictionary
4) Iterators
5) Variable
6) Object
7) Deque

semi adinfinitum. Probably one would want to write an API to deal with the most common IGOR objects, such as waves (numerical/text), variables, function references, etc. It's not clear to me how one would go about it. Having said all this I know Matlab has a Matlab/Java bridge. There are also projects to allow Matlab to be called from Python/Java. Both these options would be useful.

One other thought that I had was an XOP that would pipe commands to other programs and read their stdout, putting it into a string. You could implement a wide variety of things with a technique like that.

Andy

March 24, 2010 at 06:52 pm - Permalink

andyfaff wrote:

One other thought that I had was an XOP that would pipe commands to other programs and read their stdout, putting it into a string. You could implement a wide variety of things with a technique like that.


That was my first thought, although I hadn't imagined it being so general. Of course, if you are dealing with large arrays, you have to wait for them to fill up stdout, or write to disk and grab them there. And then you have to parse them.

It looks like doing the two of the three kinds of embedding described in http://docs.python.org/extending/embedding.html is fairly easy (sections 5.1 and 5.3). 5.2 would be the hard part where you track all of the data types. But since the Python API already offers a straightforward way of converting the Python datatypes into C datatypes, and the Igor XOP Toolkit API offers a straightforward way of converting several C datatypes into Igor datatypes, I imagine this is doable, too. It would just take a lot more time. I will work on the 5.1 and 5.3 methods and see how things work. The only pitfall here is getting the right libraries to link correctly, but they even have a section on how to get that right.

March 26, 2010 at 11:34 am - Permalink

As a Python fan, I applaud this idea. It seems totally feasible to me, judging from the Py/C API docs. Every Python type just becomes a PyObject*, which you can then manipulate via the API. But I'm not an XOP'er, so I can't speak to the final step of doing the C-->Igor typecasting. If someone can demonstrate this, it might be the straw that breaks my back and makes me buy/dig into the XOP toolkit. Python is that rad.

Here's a little more "out there" idea: What if WM someday had a "PyXOP toolkit", with no need for users to code C? I suspect this could be built on top of the existing toolkit by doing the reverse of what you're talking about, Rick -- calling C from Python (http://docs.python.org/extending/extending.html). To guys like me (OK, maybe we're not all that common) this would be a huge selling point for Igor. Again, Python is that rad. It's just so approachable and beautifully OO.

Nick

March 26, 2010 at 11:42 am - Permalink

RGerkin wrote:


That was my first thought, although I hadn't imagined it being so general. Of course, if you are dealing with large arrays, you have to wait for them to fill up stdout, or write to disk and grab them there. And then you have to parse them.



I've already written operations to take binary strings and convert them to numerical arrays. It was one of the easiest XOP's to write, as all you have to do is check that the string is the right length for the wavetype you want. All you do then is make a wave and do a wholesale memcpy. I do like the idea of using popen, I tried it but it didn't work and I don't know why. I think it would be easier to pipe stuff to another program, not so easy to listen to stdin if Igor is already running. THere are things that I don't know. For example if one wrote an XOP to listen to stdin (threaded of course), would that intercept keyboard strokes destined for the main IGOR thread, etc?

March 29, 2010 at 04:39 pm - Permalink

OK, I just wrote a little test XOP for embedded Python, but all it will do it execute Python functions with zero arguments, and print the numeric return value in the Igor history window. There is little to no error checking, and I have compiled it for Windows only.

The syntax in Igor is:
PythonFunc "smith","testIgor"


To make sure it's executing correctly, I also have the testIgor function in smith.py create a dummy text file on the desktop.

I always start with XOP1 from the toolkit, and I haven't bothered to change the name. To run the test, download the attached .zip file, install the XOP, and put smith.py and python26_d.dll in your Igor Pro Folder. If you want to compile it yourself, I've included the source and environment settings, but you will also need to unzip the entire Python 2.6.5 source into the XOP1 folder in your Igor XOP Toolkit. My environment settings are set to include the requisite static libraries from the Python source.

Starting from something like this, someone's (Andy's?) expertise in type conversion could get us most of the rest of the way. I really don't think this is a very hard project when you consider what a tremendous volume of existing code this would give us access to. If there is at least one other potential contributor, let me know and I'll put what I have up on IgorExchange as a project. I'm also happy to start from scratch.

Rick
pyIgor.zip (1.24 MB)

April 6, 2010 at 07:12 am - Permalink

RGerkin wrote:
OK, I just wrote a little test XOP for embedded Python, ... install the XOP, and put smith.py and python26_d.dll in your Igor Pro Folder. ...


Presumably, due to the .dll, this install will NOT work with MacOS X. However, Python installs do exist for MacOS X, so having the functionality there could also be of interest, for example to link to sage (free symbolic math only on MacOS X and Linux).

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville

April 5, 2010 at 08:45 am - Permalink

I think it would be fairly simple to compile for OS X. On OS X you simply link against the system libpython.dylib which is in /usr/lib:
libpython.dylib -> libpython2.dylib
libpython2.5.dylib -> libpython2.dylib
libpython2.dylib -> ../../System/Library/Frameworks/Python.framework/Versions/2.5/Python

Please note that I redirect the last symlink to
libpython2.dylib -> /System/Library/Frameworks/Python.framework/Versions/Current/Python

On other machines it may be
/Library/Frameworks/Python.framework/Versions/Current/Python

The difficulty is that each persons python path is likely to be different. One trouble with the linking is that the compiled XOP would have to use a compatible python dylib when a user uses it. Which means a) choosing the right symlink to link against. b) Ensuring that the user has that version of python installed (and that it's in the PATH).

AFAIK there is no libpython.a on OS X. Alternatively one can just copy the python framework into the XOP, but that will increase the size of the XOP by ~140Mb.

April 5, 2010 at 06:41 pm - Permalink

Quote:
The difficulty is that each persons python path is likely to be different.


If libpython is under /usr/lib, would the dynamic linker not find it without the need to use a specific path?

April 6, 2010 at 01:27 pm - Permalink

hrodstein wrote:


If libpython is under /usr/lib, would the dynamic linker not find it without the need to use a specific path?


It would find it. What I was worried about was someone linking to python3, but the system python is python2.5 or 2.6, or the other way around. I'm not sure if there are problems between 2.5 and 2.6 as well.

April 6, 2010 at 03:28 pm - Permalink

I compiled Ricks XOP using system libpython2.dylib. (resolving to /System/Library/Frameworks/Python.framework/Versions/2.5/Python on my machine).
I had to put the smith.py file in /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages. This is where my PYTHONPATH locates to.

I also had to change the python script to:

def testIgor():
import os
f = open('test.txt','w')
f.write('Test\n')
f.close()
return 0


As the original code supplied by Rick doesn't work on OS X. When I executed the code the file test.txt turns up at "/test.txt", i.e. at the root level. THis of course could be changed. It seems that the pilot code written by Rick needs it's error handling improved. (lots of places need to return specific error messages).
XOP1.xop.dmg (36.5 KB)

April 6, 2010 at 06:58 pm - Permalink