#### Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Options

# Python, NumPy, SciPy

I'm trying to get myself set up at home so I can play around with coding stuff that might be relevant for Azimuth. I like Sage, but am running Windows, and am not crazy about running Sage via VMware.

So I decided to try to set up Python, NumPy, SciPy, etc directly under Windows.

When I try these kinds of things, I tend to make every possible mistake you could make. If we make any progress, it would be great if we could share code, so I thought I would summarize my experience so that maybe others can avoid making the same mistakes.

First, if you download Python 2.7, make sure you also download NumPy for Python 2.7 and SciPy for Python 2.7. It sounds obvious, but I tend to grab the first .exe I see. You'd think it would be backward compatible so downloading NumPy for Python 2.6 should work with Python 2.7, but apparently not. The versions have to all align.

• Options
1.
edited March 2011

I'm not a great Python expert, but I gather there's two kinds of python "module":

1. Modules that's written purely in Python. These tend to be backward compatible.

2. Modules that provide Python interfaces but under the hood are basically C libraries. Because these tend to use implementation details that change over time they tend to be more likely to be version dependent. I gather NumPy is one of these kind of things.

Comment Source:I'm not a great Python expert, but I gather there's two kinds of python "module": 1. Modules that's written purely in Python. These tend to be backward compatible. 2. Modules that provide Python interfaces but under the hood are basically C libraries. Because these tend to use implementation details that change over time they tend to be more likely to be version dependent. I gather NumPy is one of these kind of things.
• Options
2.

I'm playing around with Python, NumPy, SciPy, and matplotlib.

These three together, so far seem to be enough for any foreseeable numerical work I might want to do.

Once I get some substantial code, I tend to like to throw a GUI around it, so I might at some point be interested in GUI development in Python, but for now I'm pretty happy with what I see so far. Data visualization with matplotlib seems very good.

Comment Source:I'm playing around with Python, NumPy, SciPy, and matplotlib. These three together, so far seem to be enough for any foreseeable numerical work I might want to do. Once I get some substantial code, I tend to like to throw a GUI around it, so I might at some point be interested in GUI development in Python, but for now I'm pretty happy with what I see so far. Data visualization with matplotlib seems very good.
• Options
3.

they are good and there is one installation that contains them all from a company called Enthought.I'd recommend installing Ipython (interactive shell which is simply too good to be true). Also easy_install is recommended to manage your Python installation and adding packages (http://pypi.python.org/pypi/setuptools). In Sage uses you can all three of these, and lots of the 2d plots is based in matplotlib, but for 3d they use Jmol and the Tachyon raytracer and I like both for different purposes

I'd recommend the books from Langtangen on Springer. His "Primer on scientific programmming for people without programming experience. see Computational science. He has another from 2008 3e called "Python Scripting for Computational Science" which is for seasoned programmers. But these are Springer books and thus not cheap. The Computational Science page here on Azimuth also has some free Python books.

I am no fan of the VMWare either, so I am going to install one of the smaller .exe I mentioned on another thread

Comment Source:they are good and there is one installation that contains them all from a company [called Enthought](http://www.enthought.com).I'd recommend installing Ipython (interactive shell which is simply too good to be true). Also easy_install is recommended to manage your Python installation and adding packages (http://pypi.python.org/pypi/setuptools). In Sage uses you can all three of these, and lots of the 2d plots is based in matplotlib, but for 3d they use Jmol and the Tachyon raytracer and I like both for different purposes I'd recommend the books from Langtangen on Springer. His "Primer on scientific programmming for people without programming experience. see [[Computational science]]. He has another from 2008 3e called "Python Scripting for Computational Science" which is for seasoned programmers. But these are Springer books and thus not cheap. The Computational Science page here on Azimuth also has some free Python books. I am no fan of the VMWare either, so I am going to install one of the smaller .exe I mentioned on another thread
• Options
4.

Created a page Python. The installation of NumPy failed because I have the version 2.7 and 3.x of Python installed, but not 2.6. Why does it depend on 2.6?

Comment Source:Created a page [[Python]]. The installation of NumPy failed because I have the version 2.7 and 3.x of Python installed, but not 2.6. Why does it depend on 2.6?
• Options
5.

Here's one download link of some version of NumPy for Win32 Python 2.7 (I don't know if Win32 is what you want). Tried to add some text to the page to explain the architecture of Python and libraries.

Comment Source:Here's [one download link](http://sourceforge.net/projects/numpy/files/NumPy/1.5.1/numpy-1.5.1-win32-superpack-python2.7.exe/download) of some version of NumPy for Win32 Python 2.7 (I don't know if Win32 is what you want). Tried to add some text to the page to explain the architecture of Python and libraries.
• Options
6.

Ok, I'll try that, I was curious about what numpy dependend upon and what could have changed between 2.6, 2.7 and 3.x. Java has been backward compatible which means that there is a lot of obsolete baggage in the latest JDK, but Phyton hasn't, I gather?

Comment Source:Ok, I'll try that, I was curious about what numpy dependend upon and what could have changed between 2.6, 2.7 and 3.x. Java has been backward compatible which means that there is a lot of obsolete baggage in the latest JDK, but Phyton hasn't, I gather?
• Options
7.

It's all in the difference between the Python language and the CPython C internals, a bit like the difference between Java and the JVM (AIUI). The Python language has remained backwards compatible apart from a deliberate allowance of changes that break compatibility between the 2.x and the 3.x series. So a random piece of pure Python code downloaded off the internet should run on any version. However, the CPython C internals is allowed to change how it implements the language between releases. Unfortunately, a lot of computationally intensive Python libraries are not just pure Python but include low-level C code which is dynamically loaded by the Python process (even if it's just wrapping up an external library so it's objects behave like Python objects).

Comment Source:It's all in the difference between the Python language and the CPython C internals, a bit like the difference between Java and the JVM (AIUI). The Python _language_ has remained backwards compatible apart from a deliberate allowance of changes that break compatibility between the 2.x and the 3.x series. So a random piece of pure Python code downloaded off the internet should run on any version. However, the _CPython C internals_ is allowed to change how it implements the language between releases. Unfortunately, a lot of computationally intensive Python libraries are not just pure Python but include low-level C code which is dynamically loaded by the Python process (even if it's just wrapping up an external library so it's objects behave like Python objects).
• Options
8.
edited May 2011

Good that you started this Tim! I will add some ref's (both books and libs) on monday. I will move the 3-4 python specific books on Sage and add as a freely accessible book "how to think like a Comp Scientist" which is short free and to the point

For advanced programmers ill keep the "Dive into Python" and add Beazleys "Python:Essential Reference. Or if you have any better suggestions just add :-)

Comment Source:Good that you started this Tim! I will add some ref's (both books and libs) on monday. I will move the 3-4 python specific books on [[Sage]] and add as a freely accessible book "how to think like a Comp Scientist" which is [short free and to the point](http://greenteapress.com/thinkpython/) For advanced programmers ill keep the "Dive into Python" and add Beazleys "Python:Essential Reference. Or if you have any better suggestions just add :-)
• Options
9.

NumPy has some interesting functions, but I'm also missing a lot of functions that are implemented in the gsl (gnu scientific library). Since it is supposed to be easy to wrap a C library in Python, has no one done this for the gsl?

For the Burgers equation I'll need Bessel functions and the evaluation of sums, for example, and I haven't found that in the NumPy documentation.

Staffan wrote:

For advanced programmers ill keep the "Dive into Python" and add Beazleys "Python:Essential Reference...

Java has Joshua Bloch's effective Java, and I know similar books for C and C++: Good programming style for experienced developers (which you cannot appreciate before you have written several thousands lines of code). I haven't seen anything like this for Python...

Comment Source:NumPy has some interesting functions, but I'm also missing a lot of functions that are implemented in the gsl (gnu scientific library). Since it is supposed to be easy to wrap a C library in Python, has no one done this for the gsl? For the Burgers equation I'll need Bessel functions and the evaluation of sums, for example, and I haven't found that in the NumPy documentation. Staffan wrote: <blockquote> <p> For advanced programmers ill keep the "Dive into Python" and add Beazleys "Python:Essential Reference... </p> </blockquote> Java has Joshua Bloch's effective Java, and I know similar books for C and C++: Good programming style for experienced developers (which you cannot appreciate before you have written several thousands lines of code). I haven't seen anything like this for Python...
• Options
10.

We should have a page on Numpy Scipy. There are some orders of magnitude that u can gain or not by using the former right. See this on arxiv. In essence one should use in place operations (*=,-=)

here is one thread of getting cython to work with gsl

Comment Source:We should have a page on Numpy Scipy. There are some orders of magnitude that u can gain or not by using the former right. [See this on arxiv](http://arxiv.org/abs/1102.1523). In essence one should use in place operations (*=,-=) here is one thread of [getting cython to work with gsl](http://comments.gmane.org/gmane.comp.python.cython.user/2192)
• Options
11.
edited May 2011

Thanks for the links, but Hmpf!

There are some orders of magnitude that u can gain or not by using the former right.

From an engineering perspective there should be only one way to use it, the right one. In a project that runs 5 years with a varying teams of dozens of developers you can't expect that every single one of them, including the freshmen from college, get everything right all the time all by themselves :-)

But I'm very curious about the details of the implementation of the array class and if it can be emulated in Scala :-)

Edit: Ok, it can't because it relies on direct manipulation of memory.

Comment Source:Thanks for the links, but Hmpf! <blockquote> <p> There are some orders of magnitude that u can gain or not by using the former right. </p> </blockquote> From an engineering perspective there should be only one way to use it, the right one. In a project that runs 5 years with a varying teams of dozens of developers you can't expect that every single one of them, including the freshmen from college, get everything right all the time all by themselves :-) But I'm very curious about the details of the implementation of the array class and if it can be emulated in Scala :-) Edit: Ok, it can't because it relies on direct manipulation of memory.
• Options
12.
edited May 2011

Tim wrote

From an engineering perspective there should be only one way to use it, the right one. In a project that runs 5 years with a varying teams of dozens of developers you can't expect that every single one of them, including the freshmen from college, get everything right all the time all by themselves :-)

Again I feel compelled to point out that you're applying values and trade-offs that make sense in your situation as if they applied to all programming contexts. There's a multi-dimensional space of program debugability, program simplicity, program development time/cost, program performance, program portability, program maintainability and probably a few others. Not all points in the space are attainable, and the choices that are made are heavily dependent on the context in which things are being done. The one absolute statement I'm comfortable making is "There's no absolute statement about programming trade-offs that's applicable in all contexts" :-)

In this context: an order of magnitude performance difference may justify having mulitple ways of expressing "equivalent" programs, particularly if you're promoting a language for "reasonably good for both quick numerical scripting and medium performance numerical codes". Equally if you're programming a huge application, particularly for an enterprise context where predictability of development/refactoring/maintenance planning may be more important than exposing the full spectrum of performance options, settling on a minimal interface may be the right choice. You can't analyse programming trade-offs separately from the greater context in which the programming task is being done.

Comment Source:Tim wrote > From an engineering perspective there should be only one way to use it, the right one. In a project that runs 5 years with a varying teams of dozens of developers you can't expect that every single one of them, including the freshmen from college, get everything right all the time all by themselves :-) Again I feel compelled to point out that you're applying values and trade-offs that make sense _in your situation_ as if they applied to all programming contexts. There's a multi-dimensional space of program debugability, program simplicity, program development time/cost, program performance, program portability, program maintainability and probably a few others. Not all points in the space are attainable, and the choices that are made are heavily dependent on the context in which things are being done. The one absolute statement I'm comfortable making is _"There's no absolute statement about programming trade-offs that's applicable in all contexts"_ :-) In this context: an order of magnitude performance difference may justify having mulitple ways of expressing "equivalent" programs, particularly if you're promoting a language for "reasonably good for both quick numerical scripting and medium performance numerical codes". Equally if you're programming a huge application, particularly for an enterprise context where predictability of development/refactoring/maintenance planning may be more important than exposing the full spectrum of performance options, settling on a minimal interface may be the right choice. You can't analyse programming trade-offs separately from the greater context in which the programming task is being done.
• Options
13.
edited May 2011

Sure.

However I wasn't thinking about my situation as in "developing (business) applications for a big international company", because the whole setup is not applicable to this situation (Python, C, no no!). I was thinking about "my situation" as in "what would I think is important if I had to set up a project for the development of a climate model from scratch".

This would entail a developer team of several dozens of people who would need to have some qualifications in software development, geosciences, physics and mathematical modelling. I'd rather not have them concentrate on questions as how an array should be manipulated on the level of abstraction as it is explained in the NumPy array paper. Of course that paper and the whole topic is not directed primarily to people working on climate models, but sometimes I get the impression that climate modellers are thinking at an inappropriate level of abstraction. Like people who go with a cleaning mop to a breach in a dike.

Maybe I should add an explanation like this as a "disclaimer" to my statements :-)

Comment Source:Sure. However I wasn't thinking about my situation as in "developing (business) applications for a big international company", because the whole setup is not applicable to this situation (Python, C, no no!). I was thinking about "my situation" as in "what would I think is important if I had to set up a project for the development of a climate model from scratch". This would entail a developer team of several dozens of people who would need to have some qualifications in software development, geosciences, physics and mathematical modelling. I'd rather not have them concentrate on questions as how an array should be manipulated on the level of abstraction as it is explained in the NumPy array paper. Of course that paper and the whole topic is not directed primarily to people working on climate models, but sometimes I get the impression that climate modellers are thinking at an inappropriate level of abstraction. Like people who go with a cleaning mop to a breach in a dike. Maybe I should add an explanation like this as a "disclaimer" to my statements :-)
• Options
14.
edited May 2011

Tim wrote

Like people who go with a cleaning mop to a breach in a dike.

My personal experience, which was in academic computer vision, was that it's find the balance point of doing the kind of software engineering practices that helps achieve the things you want to do with the codebase and not doing the software engineering practices that get in the way of what you want to do with the code base. As an example, when I needed to integrate with an existing codebase my heart would always sink when I found there was a really strict interface and class hierarchy with "one way to do everything", because I'd almost always want to experiment trying something that hadn't been an envisaged use-case (maybe even implemented as a quick-n-dirty test before cleaning up the approach if it seemed to work), which would often be impossible with something that had been carefully top-down designed with expectations of exactly how things would be used.

Instead of the mop/dike analogy, I kind of think more like the difference between the army corps of engineers and civilian "peacetime bridge" engineers in building bridges: the civilian engineers build bridges where it's very clear a bridge is needed and will do a lot of analysis and testing on their bridges so they know what loads they can support, are optimally laid-out for efficient vehicle usage, etc. The army engineering corps typical bridge building at the river just coming up and is much more ad-hoc/flexible (depending on your viewpoint) typically with lots of multi-use components, there's a lot of mental guesstimates rather than precise estimates of loading, etc. If a bridge proves to be an important crossing point the army corps can "upgrade" it to a heavy duty bridge (like the civilian engineers would have built in the first place), but the army often finds out that either it wasn't actually that important to cross that river (failed experiment), or that the real "heavy traffic" bridge needs to be built in a slightly different place (changing requirements after getting preliminary results).

Obviously that begs the question: is the modelling/science behind a global climate model at the point where that task is most appropriately an "army corps" task or a "peacetime bridge" task? Maybe the knowledge of the dominant physics and the numerical simulation algorithms are well-understood enough to make this a "peacetime bridge". I haven't remotely studied the problem enough to have an opinion.

I'm not against languages that provide high-level semantics designed for both programmer simplicity and efficiency (that's why I hope something like SaC takes off), but until then I think a good knowledge of flexible components at various levels of efficiency/abstraction is an important "working skill" for a scientific programmer.

Comment Source:Tim wrote > Like people who go with a cleaning mop to a breach in a dike. My personal experience, which was in academic computer vision, was that it's find the balance point of doing the kind of software engineering practices that helps achieve the things you want to do with the codebase and not doing the software engineering practices that get in the way of what you want to do with the code base. As an example, when I needed to integrate with an existing codebase my heart would always sink when I found there was a really strict interface and class hierarchy with "one way to do everything", because I'd almost always want to experiment trying something that hadn't been an envisaged use-case (maybe even implemented as a quick-n-dirty test before cleaning up the approach if it seemed to work), which would often be impossible with something that had been carefully top-down designed with expectations of exactly how things would be used. Instead of the mop/dike analogy, I kind of think more like the difference between the army corps of engineers and civilian "peacetime bridge" engineers in building bridges: the civilian engineers build bridges where it's very clear a bridge is needed and will do a lot of analysis and testing on their bridges so they know what loads they can support, are optimally laid-out for efficient vehicle usage, etc. The army engineering corps typical bridge building at the river just coming up and is much more ad-hoc/flexible (depending on your viewpoint) typically with lots of multi-use components, there's a lot of mental guesstimates rather than precise estimates of loading, etc. If a bridge proves to be an important crossing point the army corps can "upgrade" it to a heavy duty bridge (like the civilian engineers would have built in the first place), but the army often finds out that either it wasn't actually that important to cross that river (failed experiment), or that the real "heavy traffic" bridge needs to be built in a slightly different place (changing requirements after getting preliminary results). Obviously that begs the question: is the modelling/science behind a global climate model at the point where that task is most appropriately an "army corps" task or a "peacetime bridge" task? Maybe the knowledge of the dominant physics and the numerical simulation algorithms _are_ well-understood enough to make this a "peacetime bridge". I haven't remotely studied the problem enough to have an opinion. I'm not against languages that provide high-level semantics designed for both programmer simplicity and efficiency (that's why I hope something like SaC takes off), but until then I think a good knowledge of flexible components at various levels of efficiency/abstraction is an important "working skill" for a scientific programmer.
• Options
15.

I'll have to find a better metaphor, to express the concentration on "small scale concrete programming problems" instead of large scale engineering problems, the mop-dike-metaphor does not catch that.

I think I understand what you mean. Steve Easterbrook has written that the development process is more akin to agile programming than to the classic V-model, but it is not quite the same as agile programming as it is understood in the software engineering, it's something different.

With regard to the software structure the stage where you have a small code base that you can play around with is passed when you have more than 1 million lines of code that comprises the model, I don't count all the libraries that are included. (For several models I know that 1 million lines of code is actually the code that directly computes solutions to equations, infrastructure code is not included).

In order to do what you say you'd like to do, to have some flexibility to play around, you'd first need to have a good concept of what a component should be in such a system. That's the kind of level of abstraction that people should think about. When everybody understands what a component should be, you can start to structure your software into components, and finally, eventually, regain the ability to play around with a small part of the model. So, large scale architecture is actually about regaining flexibility to play around in large systems. Today, this does not seem to be possible.

I'd like to enable people who specialize in a certain scientific aspect of the model like atmosphere-ocean heat coupling to be able to play around with their module coupled to the whole model (a frozen version thereof) on their laptop :-) That's my vision.

Anyway, yesterday evening I was playing around with Swig to compile a little toy example in C to a Python module, but it did not work, the linking of the object files resulted in a lot of error messages. So, I did not even succeed in getting the Swig-Python-toy example to work. Is there some documentation that tells me what versions of what software (gcc, Swig, Python) at least should work and which are known not to?

Comment Source:I'll have to find a better metaphor, to express the concentration on "small scale concrete programming problems" instead of large scale engineering problems, the mop-dike-metaphor does not catch that. I think I understand what you mean. Steve Easterbrook has written that the development <i>process</i> is more akin to agile programming than to the classic V-model, but it is not quite the same as agile programming as it is understood in the software engineering, it's something different. With regard to the software <i>structure</i> the stage where you have a small code base that you can play around with is passed when you have more than 1 million lines of code that comprises the model, I don't count all the libraries that are included. (For several models I know that 1 million lines of code is actually the code that directly computes solutions to equations, infrastructure code is not included). In order to do what you say you'd like to do, to have some flexibility to play around, you'd first need to have a good concept of what a component should be in such a system. That's the kind of level of abstraction that people should think about. When everybody understands what a component should be, you can start to structure your software into components, and finally, eventually, regain the ability to play around with a small part of the model. So, large scale architecture is actually about regaining flexibility to play around in large systems. Today, this does not seem to be possible. I'd like to enable people who specialize in a certain scientific aspect of the model like atmosphere-ocean heat coupling to be able to play around with their module <i>coupled</i> to the whole model (a frozen version thereof) on their laptop :-) That's my vision. Anyway, yesterday evening I was playing around with Swig to compile a little toy example in C to a Python module, but it did not work, the linking of the object files resulted in a lot of error messages. So, I did not even succeed in getting the Swig-Python-toy example to work. Is there some documentation that tells me what versions of what software (gcc, Swig, Python) at least should work and which are known not to?
• Options
16.

Anyway, yesterday evening I was playing around with Swig to compile a little toy example in C to a Python module, but it did not work, the linking of the object files resulted in a lot of error messages. So, I did not even succeed in getting the Swig-Python-toy example to work. Is there some documentation that tells me what versions of what software (gcc, Swig, Python) at least should work and which are known not to?

I think you're on Windows? If so I have no experience -- Staffan might know.

Comment Source:> Anyway, yesterday evening I was playing around with Swig to compile a little toy example in C to a Python module, but it did not work, the linking of the object files resulted in a lot of error messages. So, I did not even succeed in getting the Swig-Python-toy example to work. Is there some documentation that tells me what versions of what software (gcc, Swig, Python) at least should work and which are known not to? I think you're on Windows? If so I have no experience -- Staffan might know.
• Options
17.

Yes, I use the notebook I got from my employer, and we're all forced to use Windows, in this case Windows 7. Thank gods there is Cygwin.

Comment Source:Yes, I use the notebook I got from my employer, and we're all forced to use Windows, in this case Windows 7. Thank gods there is Cygwin.
• Options
18.

UUUUUUfffff....I just succeeded in building a C++-extension for Python on my windows laptop. Geez, that wasn't easy.

But it means that I'll be able - I hope I hope I hope - to take C++ solutions to common numerical problems and supplement my Python installation with it, for example the Bessel functions, which are needed for one of the problems on the Burgers equation, for the "solution in closed form" (which is actually quite involved to evaluate numerically).

I think I'll document the way I do this on the wiki somewhere, maybe on the Python page.

Comment Source:UUUUUUfffff....I just succeeded in building a C++-extension for Python on my windows laptop. Geez, that wasn't easy. But it means that I'll be able - I hope I hope I hope - to take C++ solutions to common numerical problems and supplement my Python installation with it, for example the Bessel functions, which are needed for one of the problems on the [[Burgers equation]], for the "solution in closed form" (which is actually quite involved to evaluate numerically). I think I'll document the way I do this on the wiki somewhere, maybe on the [[Python]] page.
• Options
19.

But it means that I'll be able - I hope I hope I hope - to take C++ solutions to common numerical problems and supplement my Python installation with it...

Cool!

Comment Source:> But it means that I'll be able - I hope I hope I hope - to take C++ solutions to common numerical problems and supplement my Python installation with it... Cool!
• Options
20.

Still running into mundane problems - this time I had to realize that I have forgotten how C and C++ handle their source file structure. The gsl has a main folder and a lot of subfolders dedicated to different problems, like "specfunc" for special functions. Source files in the folder "specfunc" reference header files in different subfolders by including them, but not with the relative path which would be

../otherfolder/other.h

but by

otherfolder/other.h

i.e. not the relative path from their location but the relative path from the main folder. Which works for make, but not for distutils which is also supposed to compile the C++ files first.

How does build management for bigger C++ projects work, anyway?

Comment Source:Still running into mundane problems - this time I had to realize that I have forgotten how C and C++ handle their source file structure. The gsl has a main folder and a lot of subfolders dedicated to different problems, like "specfunc" for special functions. Source files in the folder "specfunc" reference header files in different subfolders by including them, but not with the relative path which would be ../otherfolder/other.h but by otherfolder/other.h i.e. not the relative path from their location but the relative path from the main folder. Which works for make, but not for distutils which is also supposed to compile the C++ files first. How does build management for bigger C++ projects work, anyway?
• Options
21.
edited June 2011

DISCLAIMER: Linux guy, windows may require stuff like sacrificing a goat every third Tuesday as well for all I know.

Relative paths do work, but (AIUI) you've got to be aware what the current directory for the compiler instance is (eg, if I'm in "/home/dt/blah" and type "gcc dir1/dir2/file.c" then gcc's current directory is /home/dt/blah not where the source file is), as paths are relative from there rather than the directory where the source file is. So if the distutils build system "cwd"s into a source directory while the structure is expecting it to run in the toplevel directory (where the Makefile is) things won't work.

Alternatively what appears to be relevant to your "issue" may be another feature: if you look at the command line you may well see some "-Igsl/thing/dir" switches to the compiler and linker. This adds gsl/thing/dir (relative to the complier current directory I think) to the search path for any files to look up. Don't know distutils but you can probably persuade it to pass such a switch to the command line.

I gather MSVC++ does a lot of magic to resolve paths for you, on Linux the most popular idea seems to be to (1) split things into separate library projects wherever possible and (2) take care with makefiles/other build alternatives. (One useful hack just to quickly get something to build on Linux is the ability to do symbolic links: if the project file expects a file somewhere other than where it is you can create a symlink. Fixing the problems is recommeded long-term.)

Comment Source:DISCLAIMER: Linux guy, windows may require stuff like sacrificing a goat every third Tuesday as well for all I know. Relative paths do work, but (AIUI) you've got to be aware what the current directory for the _compiler_ instance is (eg, if I'm in "/home/dt/blah" and type "gcc dir1/dir2/file.c" then gcc's current directory is /home/dt/blah not where the source file is), as paths are relative from _there_ rather than the directory where the source file is. So if the distutils build system "cwd"s into a source directory while the structure is expecting it to run in the toplevel directory (where the Makefile is) things won't work. Alternatively what appears to be relevant to your "issue" may be another feature: if you look at the command line you may well see some "-Igsl/thing/dir" switches to the compiler and linker. This adds gsl/thing/dir (relative to the complier current directory I think) to the search path for any files to look up. Don't know distutils but you can probably persuade it to pass such a switch to the command line. I gather MSVC++ does a lot of magic to resolve paths for you, on Linux the most popular idea seems to be to (1) split things into separate library projects wherever possible and (2) take care with makefiles/other build alternatives. (One useful hack just to quickly get something to build on Linux is the ability to do symbolic links: if the project file expects a file somewhere other than where it is you can create a symlink. Fixing the problems is recommeded long-term.)
• Options
22.

I use Eclipse with MinGW as gcc emulation, which should do a lot of less magic behind the doors than MSVC++.

...if you look at the command line you may well see some "-Igsl/thing/dir" switches to the compiler and linker. This adds gsl/thing/dir (relative to the complier current directory I think) to the search path for any files to look up. Don't know distutils but you can probably persuade it to pass such a switch to the command line.

disutils works via a python method that is executed from the base folder, so that it should be able to resolve all paths, as it has the same starting point as make :-)

but maybe one has to pass the folders in some way you describe it, I'll have to try it (resp. find it in the documentation).

Comment Source:I use Eclipse with MinGW as gcc emulation, which should do a lot of less magic behind the doors than MSVC++. <blockquote> <p> ...if you look at the command line you may well see some "-Igsl/thing/dir" switches to the compiler and linker. This adds gsl/thing/dir (relative to the complier current directory I think) to the search path for any files to look up. Don't know distutils but you can probably persuade it to pass such a switch to the command line. </p> </blockquote> disutils works via a python method that is executed from the base folder, so that it should be able to resolve all paths, as it has the same starting point as make :-) but maybe one has to pass the folders in some way you describe it, I'll have to try it (resp. find it in the documentation).
• Options
23.

I think you should check out Cython again. there has been a tremendous dev there and now u can generate C and use C and C++ libraries.

Langtangen says that "....SWIG cannot handle automatically handle the mapping between Numpy arrays and plain C or C++ arrays..." so i'd recommend using Cython,

Btw I think the current content of Python is far from the idea stated in the beginning :-( I can take a stab at adding more content if you want?

Comment Source:I think you should check out [Cython](http://cython.org/) again. there has been a tremendous dev there and now u can generate C and use C and [C++ libraries](http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html). Langtangen says that "....SWIG cannot handle automatically handle the mapping between Numpy arrays and plain C or C++ arrays..." so i'd recommend using Cython, Btw I think the current content of [[Python]] is far from the idea stated in the beginning :-( I can take a stab at adding more content if you want?
• Options
24.
edited July 2011

I think python(x,y) is a platform that includes most of the desired features. It might ease some of the install troubles. Sorry if I missed a previous link to it above.

Have you thought about trying a purely functional language like Haskell or OCaml? Robert Harper has some information about these, and why he thinks they're superior to any imperative language, on his blog Existential Type. I see there are some comments including Haskell and Scala in the Azimuth code project thread, but I might start a new thread specifically on functional programming if anyone is interested. Let me know if I missed one that already exists in my search!

Comment Source:I think [python(x,y)](http://www.pythonxy.com/) is a platform that includes most of the desired features. It might ease some of the install troubles. Sorry if I missed a previous link to it above. Have you thought about trying a purely functional language like [Haskell](http://en.wikipedia.org/wiki/Haskell_(programming_language)) or [OCaml](http://en.wikipedia.org/wiki/Objective_Caml)? [Robert Harper](http://www.cs.cmu.edu/~rwh/) has some information about these, and why he thinks they're superior to any imperative language, on his blog [Existential Type](http://existentialtype.wordpress.com/). I see there are some comments including Haskell and Scala in the [Azimuth code project](http://www.math.ntnu.no/~stacey/Mathforge/Azimuth/comments.php?DiscussionID=526) thread, but I might start a new thread specifically on functional programming if anyone is interested. Let me know if I missed one that already exists in my search!
• Options
25.

Cameron wrote:

I think python(x,y) is a platform that includes most of the desired features. It might ease some of the install troubles. Sorry if I missed a previous link to it above.

I don't think this has been mentioned before, thanks for the tip!

However, I'm looking for a way of continuous integration of external libraries written in C or Fortran into a modern programming framework like Python as "gluing code". I would like to make the lots and lots of numerical routines written in these languages available to others.

Have you thought about trying a purely functional language like Haskell or OCaml? Robert Harper has some information about these, and why he thinks they're superior to any imperative language, on his blog Existential Type.

My main objective is to bridge the gap between software engineering and scientific computing. In the 1960ties, if you were at the top of scientific computing you were also at the top of software engineering. This is long over. Today both communities have almost no overlap. Since pure functional languages have had almost no impact outside of academia, I haven't considered to use them. Scala is a candidate that could, some day, have a significant impact outside of academia, because it takes one of the best known and widespread language and adds to it. But Sala is of course not a pure functional language, it is a procedural-object oriented - functional hybrid. And frankly, I'm quite disappointed at the whole toolchain of Scala (starting with the IDE plugins).

Comment Source:Cameron wrote: <blockquote> <p> I think python(x,y) is a platform that includes most of the desired features. It might ease some of the install troubles. Sorry if I missed a previous link to it above. </p> </blockquote> I don't think this has been mentioned before, thanks for the tip! However, I'm looking for a way of continuous integration of external libraries written in C or Fortran into a modern programming framework like Python as "gluing code". I would like to make the lots and lots of numerical routines written in these languages available to others. <blockquote> <p> Have you thought about trying a purely functional language like Haskell or OCaml? Robert Harper has some information about these, and why he thinks they're superior to any imperative language, on his blog Existential Type. </p> </blockquote> My main objective is to bridge the gap between software engineering and scientific computing. In the 1960ties, if you were at the top of scientific computing you were also at the top of software engineering. This is long over. Today both communities have almost no overlap. Since pure functional languages have had almost no impact outside of academia, I haven't considered to use them. Scala is a candidate that could, some day, have a significant impact outside of academia, because it takes one of the best known and widespread language and adds to it. But Sala is of course not a pure functional language, it is a procedural-object oriented - functional hybrid. And frankly, I'm quite disappointed at the whole toolchain of Scala (starting with the IDE plugins).
• Options
26.

Have you thought about trying a purely functional language like Haskell or OCaml? Robert Harper has some information about these, and why he thinks they're superior to any imperative language, on his blog Existential Type.

I really like Haskell, and use it (even at the moment) for the things that I think it works well with (eg, compilers, structure generation/symbolic processing, etc). I hear that OCaml is really impressive speedwise, but after Haskell I just find the syntax of ML variants to be excessive and unnecessary. (It's a shame Eager Haskell never took off.)

The problem with using them for most of the "climate type" problems is that most of these are computationally intensive problems and for all the notational elegance, it's still currently the case that most of the time the optimisation routines for things like Haskell or OCaml still can't infer some things about computations that you need for really high-performance numerical computation (partly because it seem, IMO, that a lot of type theory research has focussed on "safeness" properties with little on "optimisational" properties. For example, it'd hopefully be useful to the optimiser if I could write something like

sum :: Num a => ([a]:::Full) -> a

where the ::: is a made-up notation expressing that the entire data-structure [a] is used in the computation. Of course sum is sufficiently simple that it probably does get analysed, but it'd be nice for more complicated instances. And I'm sure there's room for a lot more by someone who has thought more about type theory.)

Comment Source:Cameron asked > Have you thought about trying a purely functional language like Haskell or OCaml? Robert Harper has some information about these, and why he thinks they're superior to any imperative language, on his blog Existential Type. I really like Haskell, and use it (even at the moment) for the things that I think it works well with (eg, compilers, structure generation/symbolic processing, etc). I hear that OCaml is really impressive speedwise, but after Haskell I just find the syntax of ML variants to be excessive and unnecessary. (It's a shame Eager Haskell never took off.) The problem with using them for most of the "climate type" problems is that most of these are computationally intensive problems and for all the notational elegance, it's still currently the case that most of the time the optimisation routines for things like Haskell or OCaml still can't infer some things about computations that you need for really high-performance numerical computation (partly because it seem, IMO, that a lot of type theory research has focussed on "safeness" properties with little on "optimisational" properties. For example, it'd hopefully be useful to the optimiser if I could write something like sum :: Num a => ([a]:::Full) -> a where the ::: is a made-up notation expressing that the entire data-structure [a] is used in the computation. Of course sum is sufficiently simple that it probably does get analysed, but it'd be nice for more complicated instances. And I'm sure there's room for a lot more by someone who has thought more about type theory.)
• Options
27.

Have you thought about trying a purely functional language

I'll take a look at the blog article you link, but I suspect the answer to the following question is no:

In your favourite functional language, can I make a one-byte change to a gigabyte dataset without making a copy?

Assuming the answer is no, I leave it as an exercise to the reader why functional languages are unknown in scientific computing.

Comment Source:Cameron asks: > Have you thought about trying a purely functional language I'll take a look at the blog article you link, but I suspect the answer to the following question is no: In your favourite functional language, can I make a one-byte change to a gigabyte dataset without making a copy? Assuming the answer is no, I leave it as an exercise to the reader why functional languages are unknown in scientific computing.
• Options
28.

In your favourite functional language, can I make a one-byte change to a gigabyte dataset without making a copy?

It's a bit more subtle than that. Precisely because there are no non-obvious side-effects, functional languages are very good at doing copy-on-write and maintaining modifications to datasets in space proportional only to the modifications. However:

1. It's generally tricky to, say, add 1 to each element of a 1G dataset without allocating 1G and marking 1G for "garbage collection".

2. Maintaining modifications rather than update-in-place generally means cache locality suffers a lot.

(There are also effects resulting from the evaluation order of functional languages, particularly lazy ones, which mean that although it's nice for some things it's more difficult to get the "straight line" code you want in inner loops.)

All of these are soluble with explicit annotations (about evaluation strategy, allowable update-in-place, etc) and structuring, but if you've got to do all that you've lost most of the advantages of writing a language with abstractions: if you're going to specify details like evaluation order you might as well do it in Fortran/C/whatever.

Incidentally, there have been a lot of variants on existing functional languages that attempt to tackle high-performance numerical computing: one I like is Single-Assignment C. However, none of them have got a big enough developer base that I'm comfortable commiting to developing a long-term code-base in them.

Comment Source:Victor asked > In your favourite functional language, can I make a one-byte change to a gigabyte dataset without making a copy? It's a bit more subtle than that. Precisely because there are no non-obvious side-effects, functional languages are very good at doing copy-on-write and maintaining modifications to datasets in space proportional only to the modifications. However: 1. It's generally tricky to, say, add 1 to each element of a 1G dataset without allocating 1G and marking 1G for "garbage collection". 2. Maintaining modifications rather than update-in-place generally means cache locality suffers a lot. (There are also effects resulting from the evaluation order of functional languages, particularly lazy ones, which mean that although it's nice for some things it's more difficult to get the "straight line" code you want in inner loops.) All of these are soluble with explicit annotations (about evaluation strategy, allowable update-in-place, etc) and structuring, but if you've got to do all that you've lost most of the advantages of writing a language with abstractions: if you're going to specify details like evaluation order you might as well do it in Fortran/C/whatever. Incidentally, there have been a lot of variants on existing functional languages that attempt to tackle high-performance numerical computing: one I like is [Single-Assignment C](www.sac-home.org). However, none of them have got a big enough developer base that I'm comfortable commiting to developing a long-term code-base in them.
• Options
29.

I think python(x,y) is a platform that includes most of the desired features. It might ease some of the install troubles. Sorry if I missed a previous link to it above.

Have you thought about trying a purely functional language like Haskell or OCaml? Robert Harper has some information about these, and why he thinks they're superior to any imperative language,...

Funny I downloaded python(x,y) during the summer so now I have an excuse to try it more (I was running P.2.7 I think it was not available for that yet. but it looks promising

the other forum which has been around forever is LTu lambda the ultimate which is still alive and kicking. very good and slight bias towards FP

Tim wrote:

However, I'm looking for a way of continuous integration of external libraries written in C or Fortran into a modern programming framework

Did I mention this already? I just came back from vacation, so I might have. In the advanced book by Lantangen "Python Scripting for Computational Science" he recommends not to use SWIG

Comment Source:> I think python(x,y) is a platform that includes most of the desired features. It might ease some of the install troubles. Sorry if I missed a previous link to it above. > Have you thought about trying a purely functional language like Haskell or OCaml? Robert Harper has some information about these, and why he thinks they're superior to any imperative language,... Funny I downloaded python(x,y) during the summer so now I have an excuse to try it more (I was running P.2.7 I think it was not available for that yet. but it looks promising the other forum which has been around forever is LTu [lambda the ultimate](http://lambda-the-ultimate.org/) which is still alive and kicking. very good and slight bias towards FP Tim wrote: > However, I'm looking for a way of continuous integration of external libraries written in C or Fortran into a modern programming framework Did I mention this already? I just came back from vacation, so I might have. In the advanced book by Lantangen "Python Scripting for Computational Science" he recommends not to use SWIG