A comparison of programming languages in economics

by on July 2, 2014 at 1:16 pm in Data Source, Economics, Science, Web/Tech | Permalink

There is a new NBER working paper with that title, by S. Borağan Aruoba and Jesus Fernandez-Villaverde. Here is the abstract:

We solve the stochastic neoclassical growth model, the workhorse of modern macroeconomics, using C++11, Fortran 2008, Java, Julia, Python, Matlab, Mathematica, and R. We implement the same algorithm, value function iteration with grid search, in each of the languages. We report the execution times of the codes in a Mac and in a Windows computer and comment on the strength and weakness of each language.

Here are their results:

1. C++ and Fortran are still considerably faster than any other alternative, although one needs to be careful with the choice of compiler.

2. C++ compilers have advanced enough that, contrary to the situation in the 1990s and some folk wisdom, C++ code runs slightly faster (5-7 percent) than Fortran code.

3. Julia, with its just-in-time compiler, delivers outstanding per formance. Execution speed is only between 2.64 and 2.70 times the execution speed of the best C++ compiler.

4. Baseline Python was slow. Using the Pypy implementation, it runs around 44 times slower than in C++. Using the default CPython interpreter, the code runs between 155 and 269 times slower than in C++.

5. However, a relatively small rewriting of the code and the use of Numba (a just-in-time compiler for Python that uses decorators) dramatically improves Python ’s performance: the decorated code runs only between 1.57 and 1.62 times slower than the best C++ executable.

6.Matlab is between 9 to 11 times slower than the best C++ executable. When combined with Mex files, though, the difference is only 1.24 to 1.64 times.

7. R runs between 500 to 700 times slower than C++ . If the code is compiled, the code is between 240 to 340 times slower.

8. Mathematica can deliver excellent speed, about four times slower than C++, but only after a considerable rewriting of the code to take advantage of the peculiarities of the language. The baseline version our algorithm in Mathematica is much slower, even after taking advantage of Mathematica compilation.

There are ungated copies and some discussion here.

Oliver Sherouse July 2, 2014 at 1:24 pm

Interesting, but not at all surprising based on how these languages work. But more importantly, this seems like the wrong metric. In an age of cheap hardware and fast processors, who cares about machine time? It’s programmer time (or in this case, economist time) that’s truly scarce. C++ runs fast, for example, but you have to understand pointers and the terrible standard library. Python is slower to execute, but is beautifully simple and makes it easy to understand what’s going on. It strikes me, especially if we’re trying to be scientific about it, that simplicity and comprehensibility are the more valuable traits.

Reply

Beliavsky July 2, 2014 at 1:33 pm

The cheap hardware also makes it possible to store ever-larger amounts of data, and for some algorithms, CPU time can vary with the square of the number of observations, so there are still lots of problems where speed matters.

Reply

albert magnus July 2, 2014 at 1:40 pm

I’m a computational physicist and there are things you need to go fast and things where it is not so important.

I agree Python is a very useful language, but unless you take significant optimization steps it is very slow. However, it is easy to translate into C++ if it is not too long.

Reply

R July 2, 2014 at 1:43 pm

I think the standard workflow when using Python is to take advantage of it’s strengths to quickly put out a maintainable working version, then profile and optimize the hotspots if necessary. Never pre-maturely optimize.

And there are a lot of very effective ways to optimize Python code. For these applications you should probably be using numpy/scipy from the start, but you can also leverage stuff like numexpr and Numba for increased speeds or use Cython to automatically translate code into C.

Reply

albert magnus July 2, 2014 at 1:52 pm

One of things that python does badly, which is also something I do a lot, is read in very large complicated text files and pull data out of them. The type conversions are incredibly slow and the file reading functions are somewhat worse than Matlab’s. The optimization suggestions don’t really help with that situation.

Reply

Oliver Sherouse July 2, 2014 at 1:59 pm

I find that surprising; many people who use Python–myself included–cite data mangling as one of the key strengths of Python over other languages. The string functions are powerful, and intelligent loops are relatively simple to write. What kind of problems do you run into?

albert magnus July 2, 2014 at 2:09 pm

There are packages that read in very specific file formats quickly (pandas is sort of set up to do some things), but if the file is not in that format and is a mix of text and floating points you have to parse each line and then convert the floats from string format. The code is straight-forward to read, but very slow.

PerlMonk July 3, 2014 at 12:45 am

Vive la perl!

Rahul July 3, 2014 at 10:40 am

@albert magnus

Don’t you guys use netcdf or some such format?

If you care about performance then reading in very large complicated text files seems like a bottleneck in itself.

albert magnus July 3, 2014 at 12:26 pm

The text files I’m trying to process are output from a very large, complex decades-in-development computer programs which I would prefer not to modify. It’s much easier to write a C++ code to re-format or interpret the output.

I’m just pointing out that Python has some deficiencies where I wouldn’t expect them. It would be nice if it there were one code that I could do everything in, but it doesn’t quite exist yet.

Adrian Ratnapala July 2, 2014 at 3:27 pm

Yes. My favourite strategy in numerics is to do it in Python (using Numpy, Scipy etc) and then rewrite what needs to be rewritten in C. What have I needed to rewrite in C? Nothing, ever. That’s partly because Numpy & co are already written in C, but it’s mostly because its generally easier to
just wait for the computer than to optimise it. That’s for research work, if I were building a tools for others, speed would matter more.

Reply

Silas Barta July 2, 2014 at 5:27 pm

Two words: numpy.

Reply

Adrian Ratnapala July 2, 2014 at 5:45 pm

I doubt numpy is enough for Albert M. It’s all very nice have big matrices and vectorise, but that’s only fast compared to interpreted Python. The True Believers in interpreted numerics are usually people who don’t compare real world scripts to C / Fortran equivalents. Even I do that rarely (since I rarely care about speed), but when I do, I am shocked by how slow even nicely vectorised code.

I expect it is all about caching. Doing 300 floating point operations on a huge matrix one after the other uses a lot of memory bandwidth. Whereas doing 300 flops at one location before moving to the next leaves plenty of time for the cache to fill.

Reply

corporate serf July 2, 2014 at 6:27 pm

You guys should really consider using numba (it understands numpy and scipy). This was also mentioned in the paper.

Ryan July 2, 2014 at 1:41 pm

This is probably a fair point in industry, but among academics computing time is still a very serious constraint. The model I built for my dissertation takes sometimes 24 hours to solve on 180 processors, and it’s nothing special (heterogeneous agent model, modified version of Buera, Kaboski, Shin 2011). Try solving it enough times to calibrate (or estimate) and you could make a PhD take a very long time without a fast language. I use Fortran; the most time-intensive parts of the solution algorithm took 10x longer in Matlab. Maybe if I had a ton of money I could run it on better machines and the language would matter less–but even then, parallellization has diminishing returns. Basically when I push much above 180 processors I get no speedup.

This stuff matters, because the most interesting and productive part of macro (in my view) is the rise of micro data-based macro analysis, and the het agent DSGE models associated with it.

Reply

Eric Rasmusen July 2, 2014 at 4:26 pm

Actually, is 24 hours so bad? How long do you spend on programming vs. on running the programs? If you could cut your programming time in half, would that be enough to justify using a clunkier program?

Reply

Finch July 2, 2014 at 4:36 pm

From experience, how do you develop and debug programs that take more than a couple of days to run?

The set of people who run multi-week or multi-month programs is very, very small. And they typically have very large budgets and no real alternatives. For some of my old customers, the alternative to a multi-month simulation involved a treaty violation.

Reply

zbicyclist July 2, 2014 at 11:01 pm

“From experience, how do you develop and debug programs that take more than a couple of days to run?”

Seldom does it make sense to develop and debug programs on a full set of data. Not never, but seldom. You can almost always find almost all of your bugs using small sets of data.

Finch July 3, 2014 at 9:39 am

That depends on what you’re doing. I used to do a lot of simulation of large computer systems, and often you couldn’t even approach the problem you were trying to investigate unless you had the whole system instantiated, at least at the level of having several dozen ASICs in there. How are you supposed to find problems that only come out with a congested network if you can’t build the big configuration?

And from my knowledge of customer experience, a lot of problems are like that. The stuff you really need to work on doesn’t even manifest until the scale gets large.

Ryan July 2, 2014 at 4:42 pm

24 hours is pretty bad. It’s not like you solve the model once then write a paper about it. Calibrating or estimating a model means solving it dozens or hundreds of times. In the early stages of working with a model, you are still trying to get a feel for how it works, and waiting a long time to see how various modifications affect it makes things pretty unproductive. Just general debugging is really hard this way. Cutting programming time in half in exchange for doubling or tripling computation time would be a bad tradeoff. But this is not the case for, eg, representative agent models, which can often be solved in seconds so computation time is a nonissue.

Also, Fortran is really easy. It’s not much harder than Matlab.

Reply

derek July 2, 2014 at 5:35 pm

Most of these projects are iterative, meaning that you run it multiple times as you are developing it. A 24 hour test cycle early on would be a pain in the neck.

Reply

Alise July 2, 2014 at 2:14 pm

I totally agree with Oliver — nothing that interesting here if you know how the languages work. With C++ you get more low-level control but is pretty hard for novice programmers to use. On the other hand, Python as a high-level language has an extremely low learning curve. Python lags behind R/MATLAB for data analysis, although libraries like numpy are great.

Reply

NPW July 2, 2014 at 3:17 pm

I find all this Python love odd. I can very get vector math working in matlab very quickly, and I still haven’t gotten Python to do what I want. I did get it to add two vectors once, but then I shut down my computer. I ran the exact same file after rebooting, and it wouldn’t work. Seriously, who does that?

Granted I worked with C and Matlab first so I’m more familiar, but I would have thought that with all the people claiming Python’s ease of use that I’d learn it faster than I did Matlab.

I was led to believe that there was an awesome community with tons of free code. There are a dozen different ways posted on how to vector math….zero that worked. I’ve never had a problem with Matlab. I can cut and paste just about anyone’s code and it works the first time.

I’m going to get Python to work not because it is easy to use, but because I can’t beat the price. Matlab’s student edition doesn’t have the toolboxes. I was hoping that numpy would bridge the gap.

I just want simple code for converting TLE to state vectors. Ok, that was what I was starting at, but I’ll need to do some other things like second order DE. Using excel was mind-numbing.

I don’t really care about speed, but easy to use language would be nice.

Reply

Zach July 2, 2014 at 3:27 pm

For anything that uses arrays or matrix operations, numpy or scipy are so standard that for all intents and purposes, they’re part of the core python language.

In numpy, adding two vectors with the same shape is as easy as A+B.

Reply

Adrian Ratnapala July 2, 2014 at 4:02 pm

I also worked with C and Matlab first. And I find Numpy far superior to Matlab. They attack similar sorts of problems, so while other languages might have important advantages, I can confidently say there is no situation in which I would prefer to use Matlab over Python.

Matlab is better in many ways. The toolkits are more complete, the language has a nice dumbness that lets you not worry about “real software” issues, and although Numpy integrates pretty well with the core language, there are unnavoidable rough edges.

The problem Matlab is *so dumb* that it any script above 400 lines punishes you and makes you want a tool for real software. And “price” is more than price. It means you have to have passed through bureaucratic hoops to get the license in the first place, then you have to hope the license server is up, then you have to worry about whether the strange proprietry files you saved last year work with the current version. The free stuff is always there, ready to use.

Reply

Rahul July 3, 2014 at 11:58 am

In my experience, if I need to do some quick exploratory analysis, or play with a new method etc. MATLAB wins.

OTOH, if I wanted to write a large piece of code that I’d expect others to be using, & portable, and with error checking on I/O, interfacing with other APIs and Libraries etc. I’d go for Python.

Jesus Fernandez-Villaverde July 2, 2014 at 2:30 pm

Dear Oliver

Thanks for your comment. There are many standard applications in economics where even the best hardware faces limitations. Think about a standard Krusell-Smith model with many state variables. Even with aggressive parallelization (I use, for example GPUs all the time), if you have many state variables, it may take weeks to compute.

Reply

Oliver Sherouse July 2, 2014 at 3:57 pm

A fair point, and I’ll also mention that you did point out the need for other avenues of comparison in your conclusion, including readability. In projecting trends forward, however, I do suspect that Moore’s law will make hardware cheaper and faster more quickly than the economics profession will learn proficiency at low-level programming languages. In any case, many thanks for drawing attention to this important frontier in economic practice.

Reply

rpl July 9, 2014 at 12:07 pm

Moore’s Law only helps you if your code is written in a way that allows it to take advantage of the wider vector units and additional cores that CPU manufacturers are adding to their new chips. If your code is single-threaded and non-vectorized, it will be just as slow on the processors five years from now as it is on today’s processors.

Reply

Viral Shah July 2, 2014 at 3:46 pm

The choice of programming language does not matter if the problem you are solving runs in a reasonable time in the language of your choice. The appetite for running ever larger problems is only increasing though, and rewriting programs written in Matlab/Python/R into C/C++/Fortran is excruciating. Many non-computer scientists are not even accustomed to these lower-level languages. In fact, I had dabbled a little with coding macroeconomic models in both, Matlab and C, and the experience was frustrating. In C, the computer was productive, and in Matlab, I was productive. It is exactly this kind of frustration that led us to start our work on the Julia programming language, which achieves C levels of performance with Matlab/R/Python style productivity. Disclaimer: I am one of the co-creators of the Julia programming language (http://www.julialang.org).

Reply

David Wright July 3, 2014 at 2:10 am

First, let me congratulate you on getting a whole new language with some interesting aspects and a great mission off the ground and running.

Second, let me express a bit of skepticism, although I would be happy to have that skepticism rebutted.

Presumably a great part of Julia’s good performance on basic CS algorithms is because of JIT compilation. But Cython can compile Python, and Mathematica can output a compiled form of any function you write, so why shouldn’t comparisons be made against the compiled form of programs in those languages, and why shouldn’t users familiar with those languages stay with them and take advantage of their compilation facilities instead of switching to a new language?

Presumably a good part of Julia’s good performance on basic linear algebra algorithms is because matrices are 1st class objects and basic matrix operations are routed to compiled LINPACK code. That’s great for something like a simple matrix multiply, but if I implement my own matrix algorithms in Julia (e.g. my own SVD), presumably I’m not going to see performance comparable to LINPACK because I won’t be routed to the LINPACK implementation of that algorithm.

Reply

John Shahbazian July 17, 2014 at 1:32 pm

Rewriting from Matlab to Fortran is actually exceptionally easy. And Fortran is ridiculously fast.

Reply

dbg July 2, 2014 at 7:18 pm

Python with Numba has outstanding performance and is also easier to code and much more beautiful (subjectively, anyway, just my opinion) than C++ code. Its a great choice for this type of problem.

Reply

Frank July 2, 2014 at 7:24 pm

Personally, I’ve gotten a lot of mileage out of the Armadillo C++ linear algebra library combined with the R package RcppArmadillo:

http://dirk.eddelbuettel.com/code/rcpp.armadillo.html

This lets me do the bulk of my work in R and call in fast C++ code when needed. Armadillo’s syntax is really nice: quite similar to Matlab but 10 to 100 times faster for many of the things I’ve tried.

Reply

John Hall July 6, 2014 at 11:14 pm

I have also used Armadillo, more so in C++ than in R, but I have used both. Given the choice, I would probably rather write in Python than C++, but Python is just so slow for matrix math. Armanpy (armadillo on python) is a thing, but I haven’t given it a try.

Reply

ismail July 3, 2014 at 4:05 am

*Time* is the most scarce resource yes, but not always programmer time. For example some computations are highly time sensitive. i.e Realtime rating of prepaid calls.

Reply

Axa July 3, 2014 at 6:07 am

As a developer before sending a huge task to a cluster and having a result…….you have to write it, test it on small scale, correct and then go big. You write things and if it doesn’t finish after 5 minutes in your PC, you will distract, enter MR or laugh with cat pictures. Fast code keeps you focused while making corrections on the code.

As a model end user, it’s fine. You can spend 2 years to create the input, run it in a week on a cluster and anyway you get your Ph D.

Reply

JB July 3, 2014 at 1:59 pm

It depends on what you are doing. Users care about machine time. Nearly every application is too slow to start up and too slow to respond in many cases. Not all of these problems are solved by choosing a faster programming language but some of them are. If there is enough leverage in the user : programmer ratio then the faster language will often be worth it unless there is some other constraint on the time of feature development.

If you look at the amount of effort that goes into reducing friction for users by improving application performance or latency (especially in the consumer space) then people choosing C++ or something faster and lower level will make sense in those contexts. Again, this isn’t always programming language related. As an example, take a look at this presentation from instagram* or read about user frustration with scrolling in android vs iOS. These kinds of performance features are often a huge difference in attracting or retaining users.

*
https://speakerdeck.com/mikeyk/secrets-to-lightning-fast-mobile-design

Reply

genauer July 2, 2014 at 1:28 pm

what is the reason NBER papers are gated and 5$ demanded.

Those folks are on the public payroll, or?

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 2:31 pm

Dear Genauer

The paper is available at my webpage http://www.econ.upenn.edu/~jesusfv. There is also a github page for the code. The NBER is not a government institution.

Thanks

Reply

genauer July 2, 2014 at 6:34 pm

Jesus,

thanks for the reply.

I did of course get the free version, before your post. I can read the words “ungated” in Tylers post, and I didnt need the invitation,and I do get to any NBER I want, without disclosing my information or credit card.

My question was in general about NBER

Reply

ZC July 4, 2014 at 2:23 pm

Speculation, but my guess is to tamp down traffic. This comes from the fact that older papers are completely ungated. My guess is NBER doesn’t have a lot of capacity for a ton of users to download long papers, so it cuts off the non-academic stream.

Reply

Curt F. July 2, 2014 at 1:31 pm

Awesomely, they put all their code on github. https://github.com/jesusfv/Comparison-Programming-Languages-Economics

Their R code looks pretty bad. I’m not 100% sure but I suspect that the R code could be improved by orders of magnitude by using apply() instead of for loops. I’d expect that compiled R or properly written R to be about 10-20x slower than C++, not 700x.

Reply

Malthusian Delight July 2, 2014 at 1:44 pm

That R code is pretty ugly. For loops in R are bad as is. Triple nested for loops inside a while loop? You’re just asking to wait days for that to run.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 2:34 pm

Dear Curt and Malthusian

As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper.

Yes, we know how to code better and prettier R code with the plyr package and related tools. Again, that was not what we wanted to do in this exercise. But we will be thrilled to see alternative versions on github. That is why we posted all our code there.

My personal guess is that, given how large our capital grid is, plyr will go less far than some people would have forecast, but I’d be happy to be proven wrong.

Reply

Nathan W July 2, 2014 at 3:07 pm

Probably some folks will find ways to tweak things to save a few percent or more in computing time, after picking through the code.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 3:44 pm

Yes. As I have mentioned in other forums, we are more interested in “orders of magnitude” than in concrete numbers (which I think will provide a false sense of precision). Programmers in the Julia user forum were able to reduce computing time by around 15-20% after quite a few of tweaks. But if I can do the same (or equivalent tweaks) in Matlab, then the relative speed (which is what we really care about) would be around the same.

Doug July 2, 2014 at 4:38 pm

Nathan, it’s not just an issue of a few percentage points. It’s orders of magnitude. R’s base functions are implemented in FORTRAN, which is benchmarked as faster than C. Proper R implementation on most numeric applications should be on par or even better than C, not 700x slower. That’s just silly.

Nathan W July 4, 2014 at 2:41 pm

Thanks. I sometimes forget these are orders of magnitude questions, even when I’m presently thinking of it :)

A few percent difference doesn’t go far towards 70,000%, unless there are 2,000 such tweaks.

Curt F. July 2, 2014 at 3:09 pm

Mr. Fernandez-Villaverde, I understand where you are coming from and appreciate your response. I worry however that your choice of design parameter effectively locks in the coding style to one that is best matched to C++ or Fortran.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 3:45 pm

C++ actually suffers a bit in our codes from the way we implement our loops (we are not very efficient with cache misses). Fortran and Matlab are most likely the “winners” of our approach. R, as mentioned in another reply probably suffers a bit. Python and Julia I think should more or less break even.

Chris T. July 2, 2014 at 4:55 pm

Agreed. Why not use some objective measure of what idioms are most prevalent for a certain task in a certain language? (Based on mining github code, maybe? Harder to implement, but more useful.)

Data.table and plyr provide multiple-order-of-magnitude improvements over base R, and are becoming pretty widely adopted. I’m a relative beginning in R because of the JHU Data Science courses on Coursera, and I know how to use both those packages. The paper (otherwise interesting) gives the mistaken impression that R is a terrible choice for doing this work.

Alan C July 2, 2014 at 4:04 pm

Think of it this way, Jesus: imagine comparing the performance of cars with different automatic vs. manual transmissions for an average driver –by using the stick shift in the exact same way every time.

For some programming languages, there is a bare minimum threshold of literacy taught to beginners from the outset, due to its outsize effect on performance (e.g. in Stata, using ‘forvalues’ to explicitly set values when ‘gen’ would do). For a more apples-to-apples comparison of “the objective virtues” of programming languages, your code needs to acknowledge that. There are many examples of such comparisons in applied technical fields: I invite you to check them out to get an idea.

Reply

Doug July 2, 2014 at 4:37 pm

“As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper.”

Sorry to be blunt but I think this decision is just plain dumb. When a user selects a language he doesn’t just grab one off the shelf for the day. If someone is using an R environment they assuredly know how to use the apply complex. If you sit down someone without any knowledge R is slow, but for people who have experience using it, it’s actually blazingly fast on numeric applications.

Reply

Leon du Toit July 2, 2014 at 5:08 pm

For the record, plyr has been superseded by dplyr – the next iteration of plyr. It is orders of magnitude faster.

Reply

Leon du Toit July 2, 2014 at 5:27 pm

Hi

I think an important detail is to use the latest version of R. I se in your paper that you made use of R 3.0.3 and R 3.0.1. The latest version is 3.1.0 and was released in April 2014. This is important because some fundamental changes were introduced regarding memory management which, in my guess, would have an impact on the timings.

Very interesting paper btw.

Reply

Viral Shah July 2, 2014 at 3:56 pm

It is not that the for loop per say, is bad in any language. It is just that in a dynamic such as R or Python, when running in the interpreter, you do not have enough type information. With no type information, the overhead of interpretation is really high (can be 100-1000 times), and this is what makes the programs slow. In theory, you could have type inference and a JIT compiler, to get a fast code. This is what we do in Julia. The design of the language makes it easier for the compiler to generate good code inside tight inner loops. While vectorization is a useful construct, I feel that vectorization for purely performance reasons often distorts the code and is only a half-measure compared to having a real compiler.

Reply

Rahul July 3, 2014 at 10:17 am

I was scanning through the Julia performance micro benchmarks on the official site and was surprised to see Julia reported as 10% to 15% *faster than C* for the Fibonacci & Mandelbrot functions.

Is that really true? I’m curious how do you manage that? It sure sounds impressive but the skeptic in me needs more convincing.

Reply

Kabal July 2, 2014 at 11:02 pm

apply() is just R loops in disguise, though…

Reply

Watchmaker July 3, 2014 at 10:55 am

In disguise and running orders of magnitude faster.

IIRC, apply() is significantly slower than other apply functions such as sapply() and vapply().

Reply

Ryan July 2, 2014 at 1:43 pm

The new speed advantage of C++ over Fortran is interesting (and surprised me a bit). I still think Fortran is the best choice for computationally intensive modeling–it’s much easier than C.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 2:36 pm

Dear Ryan

It is indeed the case that Fortran used to be faster, specially since C++ pays an abstraction penalty. However, there has been huge amount of effort at developing C++ compilers while Fortran compilers have stagnated a bit. In fact, some of the current Fortran compilers use a C back-end.

Also, please note that our C++ is not particularly efficient in the way we implement the loops (rows vs. columns).

Given the current state of compilers, the power of C++ (metaprogramming, lambda functions, full OO) is really attractive.

Reply

Beliavsky July 2, 2014 at 3:13 pm

I posted a comment about your paper in comp.lang.fortran a few weeks ago, leading to a lively discussion https://groups.google.com/forum/#!searchin/comp.lang.fortran/economics%7Csort:date/comp.lang.fortran/zUn52g_61Kk/oP40TyhZeRYJ . The first reply was as follows, edited for language.

“This comparison is [nonsense].

I knew it was [nonsense] when I saw that they have g++ as considerably faster than clang++ and icpc on Mac. They show gfortran as faster than ifort, which is also [nonsense]. They show g++ and gfortran as being equal on Windows; I compiled and ran both and found gfortran to be around 3% faster.

First of all, if you take a look at the codes, you’ll see the absolutely stellar coding skills (e.g. including cmath right after including math.h in the C++ version, transposing an array that never changes inside a do loop in the Fortran version).

Second of all, and probably the biggest problem, is the method of timing. The C++ version uses clock() which is CPU (user+sys) time on POSIX and is wall-clock time on Windows. The Fortran version uses ETIME which is a compiler extension and seems to be implementation defined as to whether or not it gives wall-clock time or CPU time. From Intel’s ETIME page: “On single processor systems, returns the elapsed CPU time, in seconds, of the process that calls it. On multi-core or multi-processor systems, returns the elapsed wall-clock time, in seconds.”

You can also see on his GitHub that they have a separate Intel version of the Fortran program which simply has the ETIME call commented out because they didn’t know to use ifport. This raises the question, how did they time the Fortran program on the Intel compiler?

I’ve done quite a few compiler and programming language comparisons and I’ve found Fortran to usually be about 10% faster than C/C++, and as far as compilers go, Intel is the fastest followed by Clang, then GCC. I don’t test Microsoft stuff. “

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 4:12 pm

Dear Beliavsky

Thanks for your careful comments.

1) Timing. That was a mistake when we loaded the original codes into github. The ones in github should work fine now (for example, there is no separate intel fortran version any longer). We included some timing routines when we were developing and debugging the code and somehow we did not load the most recent versions into github the first time. The timing reported in the paper, though, is the right one and yes, we were very careful in measuring CPU time.

2) Yes, we are fully aware that we did things that were not optimal. As we explain in the paper, we were trying to implement the algorithm in the way that we found, as macroeconomists, the easiest to follow, even if it was not the most efficient one. It was a 100% conscious decision. Since those inefficiencies affect all languages and our interest is in relative times, we find your objection relatively unimportant.

3) Finally, as I mentioned above, we are interested in orders of magnitude. Is your machine generating fortran executable that is 3% faster than C++? We report a 5% difference in favor of C++. Does that change anything of substance? No. The message is “look, Fortran used to be quite faster than C++. Is that the case as of today?” Not really.

Reply

reviewer July 3, 2014 at 3:22 pm

Submitted to: Journal for Carpentry Research

Title: On the relative speeds of tools for cutting wooden boards

Abstract: We compare the speeds for cutting a standard 8ft long 2×4 board into two pieces 4ft pieces using (1) a standard handsaw and (2) an electric circular saw. Our findings suggest that the hand saw is roughly 100x times faster than the circular saw.

Methods: We obtained two 8ft long 2×4 pine boards (non-pressure treated). The first we cut in half using the hand saw by drawing the blade back and forth across the center of the board with one hand, while bracing the 2×4 with the other hand. To keep the comparison as fair as possible, we used as close to the same algorithm for the circular saw. The circular saw was grasped in two hands and the blade we pulled back and forth across the center of the 2×4.

Discussion: It took us roughly 100x as long to cut the board in half using the circular saw. We feel that carpenters may find this information useful when deciding which tool to use for their next project.

———————————————————————————-

Reviewer comment: Surely this isn’t a fair or useful comparison, since you aren’t using the circular saw as it was intended to be used?

Author response: This was a conscious decision. We did not want to “adapt” the algorithm to each tool to improve speed, because it would open the door to too many “uncontrolled variables” (how good one is at using that tool and so on). Since those inefficiencies affect all tools and our interest is in relative times, we find your objection relatively unimportant.

Reviewer response: WTF?

——————————————————————————-

Note: neither tool above is meant to represent any particular language. It is the methodology that I am commenting on. The only thing more embarrassing than the fact that time and effort was actually expended on this “research” is that Tyler wasted so many people’s time by bringing so much attention to it.

Jonathan July 2, 2014 at 2:01 pm

I think the message here is that you can write fast R code, or you can write slow R code. You can write fast python or slow python. All this really tells us is which languages the researchers were proficient in (hint: not R).

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 2:38 pm

Dear Jonathan

I answered that question above when we explained the parameters of our comparison exercise. I’d love to see more efficient R code. But the speed of R is a main concern in the R community. The fist webpage I check every morning when I wake up is r-bloggers.com and I use it extensively for my time-series work, so I am not 100% unfamiliar with the language.

Reply

Nathan W July 2, 2014 at 2:55 pm

I think with R, often people are designing a specific fix for a specific problem. While they are aware that computational efficiency is important, probably they do not think so much about repeating the iteration 1,000,000,000,000 or more times.

In time, I think open source alternative like R will become more efficient. But first the libraries will expand, be poorly linked, inefficient, etc. As stopgap fixes to technical issues become increasingly rare, I think you will see more people dedicating grey matter to more efficient foundational code underlying the whole thing. Probably it’s obvious that I’m not a programmer and only kind of get this stuff, but I look forward to the day where there is an easy-to-use, computationally efficient open source option on the market. This will not happen overnight when the program relies on the volunteerism of open source communities, but it will happen in time.

Reply

Kabal July 2, 2014 at 11:07 pm

R loops are notoriously slow, so any comparison that leans on loops will tend to result in R getting crushed.

R’s strengths are in flexible, easy subscripting and widely available statistical libraries.

I’m sure the paper’s authors know that, though.

Reply

Jon July 2, 2014 at 2:02 pm

I once replaced a set of Fortran cash flow libraries with a purchased product that took 10X as much CPU to value a deal.

The reason was it took sometimes well over a day for programmers to create the module that valued the deal using the Fortran cash flow libraries; but with the purchased product we were able to start the valuation when the deal was offered. Thus the purchased libraries ended up being a far faster path to the deal valuation. And we were able to redeploy the people who reversed engineered the deals.

Reply

Justin Irving July 2, 2014 at 2:05 pm

Shame of them for misusing the plural “codes”. Only economists do this.

“The German codes were broken by British mathematicians”

“The code we wrote complied in no time at all”

Code != script

Reply

Kevin Denny July 2, 2014 at 3:33 pm

Ever greater shame for saying complied when you mean compiled. FFS.

Reply

Justin Irving July 2, 2014 at 3:37 pm

No you are wrong.

Reply

rpl July 9, 2014 at 12:19 pm

Computational scientists working in physics and related fields invariably refer to their programs as “codes”. The usage dates back to before universities even had computer science departments.

Reply

Mark July 2, 2014 at 2:08 pm

Is this just a sly ad for Julia? I’d never heard of it before.

Reply

Ricardo July 2, 2014 at 2:38 pm
Jesus Fernandez-Villaverde July 2, 2014 at 2:39 pm

Dear Mark

No, we are not part of the Julia core team or even Julia developers. It just worked really well for us. More and more people are using it.

Reply

Finch July 2, 2014 at 2:49 pm

The Julia team did something similar to this; they stipulated use of the same algorithm but adopted reasonable idioms from each language. Their results very roughly corroborate yours. See http://julialang.org/ ; the figure is a little down the page.

Reply

Dan Weber July 2, 2014 at 2:22 pm

Tyler may not be aware of language wars, but he’s about to get one erupting on his comment section. Welcome to Internet.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 2:40 pm

I know. I have already received quite a bit of correspondence about some of our choices.

Reply

theCoach July 2, 2014 at 2:28 pm

Haskell?

The problem with C++ is that it is invariably wrong, especially if you are using 180 cores.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 2:43 pm

We have a companion paper (not yet on the internet), where we look at Haskell, OCaml and some other functional programming languages.

Functional programming is sufficiently different that we decided to split the results. The Haskell code really, really looks very different. But yes, we love functional programming.

The only thing we really miss is Scala.

C++ with 180 cores: yes. I develop a lot using GPUs and it can get very frustrating.

Reply

theCoach July 2, 2014 at 4:13 pm

Thanks for the reply. I would be interested in the companion.
I would also be interested in a metric that measured correctness — i.e. how often did the program compute what it was supposed to. I have no idea how easy it is to make mistakes in this domain, but I would assume OCaml and Haskell would be more likely to be correct.
Another, obvious point, is that comparative analysis of programming languages is hard — you maybe interested in the following paper: http://www.leafpetersen.com/leaf/publications/ifl2013/haskell-gap.pdf

Reply

corporate serf July 2, 2014 at 6:37 pm

lush?

I have found that to be fast for numerical work in other contexts.

Reply

Santi July 3, 2014 at 9:14 am

Dear Jesus,

I’m just curious about how things work in Economics. In the paper you use a highly iterative problem as a generalisation of the field, in which there seems to be serious difficulties trying to vectorise algorithms in Matlab and R. But in this comment you say that you develop using GPUs, which only is convenient for highly parallelizable problems. So I’m confused.

Reply

Jyotirmoy Bhattacharya July 4, 2014 at 4:39 pm

I gave Haskell a try. The code I came up with runs about three times slower than C++.
https://github.com/jmoy/avb-econ-hs

Reply

Jyotirmoy Bhattacharya July 9, 2014 at 6:29 am

UPDATE: The Haskell program at the above link is now much more optimized and runs in about 1.7 times the time taken by the C++ version. This makes it comparable in speed to Python+Numba and Matlab+Mex.

Reply

Justin Rising July 2, 2014 at 2:35 pm

I think that the authors are possibly making a mistake in not using the proper idioms for each language to do the comparison. It would be much better to compare the code written by experienced programmers in every language, and to explicitly mention the learning curve of each language in the comparison. That’s impractical for the authors to do on their own, so this might make a good project for the community.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 2:45 pm

Dear Justin

We share your concerns. As I explained above, our choice is perhaps wrong but it was 100% deliberate. We posted all the codes in github precisely for that purpose.

I have already learned a lot from the feedback I got (for instance, Julia can be made much, much faster).

At the same time, we were interested in seeing what an “average economist” could do with Julia vs. Python, for example, without having to dive too deeply in the details of each language.

Reply

Rahul July 2, 2014 at 3:34 pm

I think a contest might’ve worked better. Frame a well defined problem & invite code submissions for various languages.

Reply

Santi July 3, 2014 at 7:47 am

That’s exactly what someone in Hacker News about this article proposed: a contest in TopCoder or another site like that. I second that.

Reply

Art July 3, 2014 at 12:58 pm

So you 100% deliberately decided to leave out an important element of this comparison? Just out of curiosity, how do you interpret the results from this testing. What was your hypothesis going in and do you feel that this was an adequate test of this hypothesis? At this point, the only thing I can see that this paper shows is that when you write C++-Style code in non C++-Style languages you get poor performance compared to C(++). Also, that if you compare code written by someone with a good understanding of C++ compared to code written by an R or Python novice you get better results with the C++ code. Is anyone surprised by these results?

Reply

Nathan W July 2, 2014 at 2:45 pm

So should economists learn C++ or work more closely with computer programmers?

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 2:47 pm

Dear Nathan

Yes and no. For many economists C++ may be an overkill. Also, many programmers do not use C++. But our results suggest that C++ maybe a powerful tool for those economists interested in particularly heavy computations.

We see our paper more as a guide than as an endorsement of one language vs. another.

Reply

Rahul July 2, 2014 at 3:28 pm

How long, in absolute wallclock time does a typical macroeconomic model run?

Reply

Finch July 2, 2014 at 3:39 pm

Across a couple of very different fields, I’ve found typical models run in wallclock time between 1 hour and three days regardless of the subject matter. People expand what they do to fit the computation available to them. But they need to get answers in a business-acceptable amount of time and they need to iterate to debug. So although they’d like to do computations that would take a month, they realistically can’t. And if they can do something in five minutes but get better answers with an hour run, that’s something they’ll do.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 4:03 pm

Huge variance. Some run in a second, some can take weeks.

Reply

KevinH July 2, 2014 at 2:53 pm

as a python fan, I’ll have to point out that Numpy ( http://www.numpy.org/ ) is probably the current standard way to do numerical manipulation in python, but wasn’t tested in this particular paper. And, as the whole thing is basically python hooks for some core c/c++/fortran functions, the speed difference between Python+Numpy and c++ is basically 0 from what I’ve read (eg http://stackoverflow.com/a/7614252/1701722 )

Though, I’m certainly biased. I was a long-time Matlab user, but then as I no longer had my Matlab licensed paid by someone else and wanted to expand from pure computation to interacting with data not on my machine (eg with AWS via the excellent library boto), the appeal of Python+numpy for almost best-in-class speed and ease of use is a real winner for me.

Reply

NPW July 2, 2014 at 3:21 pm

I haven’t found Python+numpy nearly as easy to use as Matlab. It appears that I’m a minority, but how was your transition? Did you find it easy?

Reply

Rahul July 2, 2014 at 3:30 pm

+1 for that. Matlab lets me think about the problem more. With Python I get bogged down by the language.

When I left the Uni & lost access to free Matlab I shifted to Octave.

Reply

Finch July 2, 2014 at 3:34 pm

If you put any dollar value on your time at all, your Matlab license will very quickly pay for itself when shifting from Octave.

Reply

Rahul July 2, 2014 at 3:45 pm

Maybe.

I just have a visceral aversion to paying for software. :)

Doug July 2, 2014 at 4:40 pm

Well, this is a problem until you spin some large problem up on a 1000 cores on AWS. That’s going to cost you a couple million bucks in matlab licenses.

Finch July 2, 2014 at 5:06 pm

If you really need to run on a 1000 core system today, you should probably be employing actual software engineers and not just economists.

That said, how much are the parallel tools in Matlab? It would astound me if this cost millions. It’s not like you have to buy 1000 copies of Matlab.

Further, the difference between Matlab and Octave was something like a factor of a thousand in recent memory. I gather that’s down somewhere between 2 and 100 these days, depending on what you’re doing. And the only way to really know for your application is to benchmark it yourself.

Rahul July 3, 2014 at 12:42 am

@Finch

I’ve worked on 100,000 core clusters (DoE / DoD) & I doubt even half the codes running had ever been touched by the hands of a software engineer. :)

No, a single license doesn’t cost millions. What @Doug’s talking about is per-node licensing. On a cluster that quickly adds up.

Finch July 3, 2014 at 9:48 am

> I’ve worked on 100,000 core clusters (DoE / DoD)

Are you talking about the stuff they do for PR or the real stuff? :)

When I worked in the business of designing those machines, that kind of customer would give us very specific feature requests that indicated they had very sophisticated software and performance people in their employ.

It would not surprise me if there had been no real attempt to optimize, say, the global climate code they run for some university. From the way they publicize those machines, you’d think they were all about the rainbows and butterflys.

But you are a real software person, aren’t you? I had that impression anyway…

Rahul July 3, 2014 at 10:31 am

@Finch: The core “mega” projects do get serious software engineering optimization attempts but the rest do not. Yeah, you could call these allocations DoE / DoDs PR effort, but that’s amounting to almost half their CPU allocations these days.

I guess the concept of a custom built Supercomputer (Cray?) that was optimized to solve one very specific problem is now quite dead. The vendors still try to upsell their tricks on interconnects and storage but most supercomputers are commodity hardware really. And a bulk of the stuff that gets run on them comes straight from the hands of a 1st year PhD student, bugs, warts, inefficiencies and all that.

And no, I’m not a software guy really, so I could be wrong.

Finch July 3, 2014 at 11:42 am

I worked on that sort of machine for two vendors, both of whom bought the parts of Cray. The microprocessors in those systems could be called commodity, in the sense that they implemented the same instruction set as the commodity processors, and everything on one side of the memory controller was common with what you would find on a desktop or a low-end server. But the memory and IO side of things is really quite different, and a lot of the money goes into that. I think that’s more or less still the case. Except for fairly specific problems it’s the interconnect that kills you on large machines. That said, most of the technology that went into the really big machines was actually designed around machines that fit in a dozen or so cabinets and probably run Oracle software, but it isn’t really fair to call that commodity.

Long term, you are probably going to see directory based cache coherence in your 1000-core desktop, but we aren’t there yet. I think the mainstream processor offerings still use some kid of snoopy protocol on die because the core counts are low enough for that to work and the GPUs have not yet adopted coherence, although with Bill Dally at the helm of Nvidia that’s probably coming.

Evan Harper July 2, 2014 at 2:56 pm

These gentlemen are economists, not programmers. It takes an expert programmer with a great deal of training and experience to write even moderately well-optimized code, and even then it is a struggle, and the programmer may always be surprised to find a significant optimization they left “on the table.”

Respectfully, their results are almost meaningless. Perhaps if 20 other economists from a variety of backgrounds all implemented the same algorithm, we would at least have a study about “what kind of mistakes do economists make when trying to be computer programmers.” But with n=1 this is just not useful at all.

Reply

will July 2, 2014 at 3:12 pm

Looking at what these guys have done, they could quit their jobs tomorrow and get a developer’s job very quickly. Not that this is important – they set out with a specific goal – which was not to code like expert programmers. Perhaps you should re-examine the facts and what you’ve written.

They’re a lot more competent than you give them credit for, and are not pretending to be anything other than economists who are pretty good at computation.

Reply

Nathan W July 2, 2014 at 3:13 pm

The people who will program stuff in economics are more likely to be economists than programmers, per se, so it seems like a better basis of comparison than taking the best team of programmers in the world.

I would assume that they used the most obviously applicable tool which gave the correct results. Most economists probably would do the same.

It would be interesting to compare to code of programmers, or economist-programmer teams, to see whether computational efficiency could be significantly increased.

Reply

Ryan July 2, 2014 at 3:53 pm

It’s amazing how many people are totally missing the point of the paper. It’s not meant to be the definitive guide to the speeds of different languages when each is properly optimized. It’s meant to be informative to economists, and it succeeds in that goal. Economists are not computer scientists. Our training is in economics. We learn enough computational tools to solve our models. We face a tradeoff between accumulating economics skills and accumulating computing skills. It’s nice to know how different languages perform using standard algorithms. Do Indy Car drivers freak out because Consumer Reports describes car models in a way that is not useful to Indy Car drivers?

There are grad students in economics trying to decide what language they should be learning. This paper will help them make that choice. A paper about what these languages look like for a bunch of crack programmers would not.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 6:31 pm

Dear Ryan

Your comment about “Do Indy Car drivers freak out because Consumer Reports describes car models in a way that is not useful to Indy Car drivers?” is beautiful. I hope you do not mind if I use it in the future :)

Reply

Will May July 3, 2014 at 12:35 am

The problem is that, despite the good intentions of the authors, this paper is very misleading.

Speed is a very poor indicator of the “benefit” of learning a language, which is what the authors imply. Sometimes speed is important, and sometimes it’s not. For all the cases where speed doesn’t matter (or matters little), this paper is misleading. In all of those cases, the slow languages are the ones that provide the most benefit!

Second, this speed comparison just isn’t appropriate, especially for R. In real life, a person wouldn’t write R code this way. They’d vectorize it. (Give me a bit of time, and I’ll fix their example so that it runs faster.) And the claiming that “economists” would write the code that way is a non sequitur. The paper portrays the results as the benefit of learning the language. To be consistent then, the authors should assume that the economist actually learned the language. No economist who learns R is going to write code like that.

I understand the authors have their justifications for the choices they made, but the end result is misleading on two different levels. The speed results are misleading and speed is a poor indicator of “benefit” to begin with.

Reply

Will May July 3, 2014 at 3:18 am

Actually, I take a lot of that back. I’m not getting anywhere trying to vectorize this. I would just call a C function from R. (As an aside, I thought it was odd that this was done for Matlab, but not for R.) If this is what macroeconomists are usually doing, I wouldn’t recommend using R, and this paper is pretty accurate.

Although.
If there are people interested in doing this in R, someone’s going to write a package for it in C or Fortran, and then R is going to be right back up there with the fastest languages.

Nathan W July 4, 2014 at 2:45 pm

I would choose R anyways, because it’s open source and free, unless I were working in very data heavy areas.

All that time spent learning STATA, SPSS and other standard softwares are much less useful if you do not have a budget to bill software costs to the department, company, or what have you. If I could redo it, I would spend much less time learning STATA (only enough to complete class assignments) and have tried to get a more practical level of skills with R. But at the time I thought maybe I’d get a job where I would actually use any of these skills. R would still be good for number crunching for various side projects. Much stronger ability to manipulate the database than with Excel.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 4:05 pm

Dear Evan

Thanks for your careful comment. We are just trying to measure what an average economist will find. Yes, we are not professional developers but other macros are not developer either. Our results seems to squarely fit with the “folk wisdom” of many of our colleagues as discussed over many years and many conferences. I do not find it totally uninformative to move the “folk wisdom” even if just by a very, very small amount into some formal numbers.

Reply

Rahul July 3, 2014 at 6:17 am

What about the risk of the “folk wisdom” turning into a self fulfilling prophecy? Is there a risk that you tune performance to the extent that it matches your prior expectations?

For any of the languages you considered did you have two members of your team code a solution independently to estimate the within language variation in performance?

What’s the chance that the programmer-to-programmer variation swamps any language-to-language variation in run-time?

Reply

roger July 2, 2014 at 3:10 pm

implementation in the go language?

Reply

Finch July 2, 2014 at 3:31 pm

The Julia team did that, somewhat more rigorously. Go was about a factor of four worse than C.

Their results very, very roughly corroborate the results here.

Reply

Bob July 2, 2014 at 3:12 pm

How much programming experience does the author have?
There are generic tests of execution time of programming languages out there where you will not get major biases caused by developer experience or an unequal amount of testing.

The paper talks about how Java is not worth it over C++ because the syntax is very similar, and Java is twice as slow. While I’d not use Java myself (Hint: If you want to use Java, Scala will almost always do it better), comparing a language that runs with automated memory management vs another that leaves said management up to you is not a fair comparison. While at first glance the languages look syntactically similar, the differences in the real world are large enough that you rarely find them both in the same problem space, without having to care about skill availability.

So when the paper says things like this, it’s hard to take seriously. You might as well say that keynesianism and market monetarism are the same thing, because they both state that a national economy can receive stimulus.

Reply

brad July 2, 2014 at 3:45 pm

Scala is a great programming language for people that love C++ but think it’s not quite complex enough, and doesn’t have quite enough features.

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 6:44 pm

That is a really nice comment, I never thought about it in that way. I actually like Scala a lot, I am embarrassed to say. But I also love Wagner, so I guess a have a perverse taste for complexity.

Reply

Ryan July 2, 2014 at 4:00 pm

Comparing languages using “generic tests of execution time” would not be as useful to economists as this paper, which compares languages based on how quickly they perform a pretty standard algorithm used for solving macro models. The intended audience of the paper is economists, not computer scientists. This is a very useful paper for economists, even if computer scientists don’t think so.

If you’re certain that the authors are wrong about Java, go get their code from github and translate their algorithm into Java. If it goes faster than their C++ tests, you will have a point.

Reply

Brad July 2, 2014 at 4:59 pm

The parent post didn’t claim Java was faster than C++, the paper’s conclusion that you pay a small speed penalty for it is completely believable. The part that is completely ridiculous is the claim that “java syntax is similar to c++ so there’s no reason to choose it over c++”.

Yes both are c decedents and so look on a superficial level similarly. But java is a considerably simpler language (if for no other reason than there isn’t a second turning complete language template language built in) and a considerably safer language given that you can’t access raw pointers.

Reply

Mark Thorson July 2, 2014 at 6:19 pm

The handling of floating-point arithmetic should be of great concern to economists. Java is deficient in this area. Perhaps this should be the first consideration in choosing a language/library environment, before performance. Trading away numerical stability, etc. for speed or ease-of-programming isn’t always a good decision, especially for programs that are sensitive to the difference as is the case in many problems run by economists.

http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 6:42 pm

Thanks Mark. That is a very nice point. As I mention in another comment, the amount of scientific computation done in Java seems to have stagnated about a promising start around a decade or more ago.

Mark Thorson July 2, 2014 at 7:05 pm

Somebody showed me a paper around 1978 or so (I can’t cite it, nor do I know when or where it was originally published) which showed the results of running the same multiple regression on several hardware/software systems (IBM, Control Data, Univac, etc.). It had a table comparing the results which underlined the correct digits in the results. The best had something like four or five correct digits, some had only one or two, and I think one system produced the wrong sign. Improper treatment of rounding was mostly responsible, though improper treatment of overflow may also have contributed.

This stuff matters! Wrong answers faster (or easier) is seldom a good choice.

Kronrod July 5, 2014 at 4:52 am

This is debatable. Unlike C++, Java guarantees that the same calculation always yields the same result, regardless of what machine it is executed on. With C++, this is not the case.

Jesus Fernandez-Villaverde July 2, 2014 at 6:38 pm

Dear Brad

Thanks for your comment. The intended audience for the paper is not a computer scientist, it is an economist. Issues such as raw pointers (or many of the other differences between C++ and Java) are, in my own personal experience, unlikely to show up in practical applications like the ones I am interested in exploring.
Think about it in this way: you are a second year grad student in economics and you have three-four weeks to pick up the basics of C++ or Java (you have either never coded or you have a cursory knowledge of Matlab and R). Will your life be much easier if you pick Java? Will you find many more numerical libraries in Java than in C++? Will you find many other programmers in economics using Java instead of C++?
Our subjective assessment is that your life will be, roughly, as difficult in Java as in C++, that you will find a more active community of people doing scientific computation in C++ (just search “scientific computation c++” and “scientific computation java” in Amazon) and that C++ will be somewhat faster.

Reply

brad July 2, 2014 at 7:22 pm

I don’t see how someone without any programming experience, even if we posit very bright grad students, is going to be able to write correct programs in either java or c++ in just four weeks. There are just too many sharp edges and gotchas. Heck the tooling alone is going to take a while to figure out. If those are the parameters Matlab, R, Mathematica, or *maybe* python seem the most realistic.

That said, as between java and c++, java has an easier learning curve, fewer ways of shooting yourself in the foot than java (though Mark Thorson makes a good point re: java floating point types), and generally more beginner tools (at least outside of windows where VS is very slick).

Mark Twain once wrote:
“My philological studies have satisfied me that a gifted person ought to learn English (barring spelling and pronouncing) in thirty hours, French in thirty days, and German in thirty years. ”

You could substitute python (or maybe VB) for English, java for french and c++ for German.

Jesus Fernandez-Villaverde July 2, 2014 at 7:38 pm

Dear Bob

Thanks for the comment. I have answered that comment in another place. Our wording may be poor. Our point was just about the relative difficulty for a grad students in economics of picking one or another.

Reply

Rahul July 3, 2014 at 12:51 am

Well, if you are “grad student in economics and you have three-four weeks to pick up the basics” should the *speed* of a language even be a factor in your choice?

Shouldn’t you be looking at stuff like programmer efficiency, steepness of learning curve, availability of libraries etc?

I think the whole notion of speed being important to an economist programmer is questionable. I suspect far more economists are stymied by bugs, maintainability, correctness, numerical stability and other issues than raw speed.

Reply

mkt July 3, 2014 at 3:01 am

Often true, but not always. If the problem to be solved is computation intensive, then raw speed becomes relatively more important than the steepness of the learning curve, ease of use, etc.

So there are tradeoffs in these language choices. But any economist knows that already. The question is how important is raw speed for the given research question, so that the economist can estimate how to balance the tradeoffs.

Reply

Rahul July 3, 2014 at 6:08 am

So what percent of economists (take a guess) work on problems where raw speed is the bottleneck?

My domain was Chemistry / Chemical Engineering and less than 10% of problems were such that raw power was the limiting resource.

ZC July 4, 2014 at 2:39 pm

@Rahul – A decent number–probably nearly anyone doing IO or macro, and most people doing structural work on other applied micro fields. Models where you’re specifying a parametric economic relationship can take a long time to estimate–something I recently worked on had a model that took 6-12 hours to estimate once.

Luis Pedro Coelho July 2, 2014 at 3:24 pm

Great Stagnation?

Fortran is older than Tyler and barely between by a 30 year upstart (C++)

Reply

Ryan July 2, 2014 at 4:02 pm

Fortran 2008 doesn’t look much like Fortran 77!

Reply

Demosthenes July 2, 2014 at 3:29 pm

Yeah, but how did Stata do?

Reply

greg July 2, 2014 at 7:12 pm

Haha. That’s what I was thinking. Of the econ profs I’ve known, almost all of them seemed to use SAS or Stata. A few used Matlab. How many economists do any actual programming, writing stuff from scratch in C++?

Reply

Jesus Fernandez-Villaverde July 2, 2014 at 7:40 pm

Stata or SAS are used in econometrics. Our paper is about macro. I do not think many people will use stata or sas to compute a macro model.

A fair number of people in macro use C++ or Fortran.

Reply

Bill July 2, 2014 at 5:18 pm

R Gnu C++ing?

Reply

mulp July 2, 2014 at 5:23 pm

“We solve the stochastic neoclassical growth model, …”

Aren’t all models, especially when put into a computer, just the basis for hoaxes?

Hoaxes perpetrated to make the authors of the computer models rich.

Reply

ivansml July 2, 2014 at 7:27 pm

I think it’s a cool paper. However, it seems to me that there are lot of additional, “softer” characteristics that would matter for an economist when choosing a language. What existing libraries for numerical computing are available, generally and for economic applications in particular? Do some languages have features that are particularly useful or expressive for economic problems? What about ease of installation and compiling/running programs, learning resources, community support, interoperability with other languages…?

I realize that many of these are subjective and time-varying, but still it would be interesting to hear author’s opinions, if not in this paper, perhaps somewhere else.

Reply

Andy Brice July 2, 2014 at 7:48 pm

>Execution speed is only between 2.64 and 2.70 times the execution speed of the best C++ compiler.

That is a rather a confusing turn of phrase. I assume you mean:

“Execution speed is only between 2.64 and 2.70 times **slower than** the execution speed of the best C++ compiler.”

Reply

agm July 3, 2014 at 3:56 am

It’s a nice idea for a gedanken experiment from the perspective of those without a lot of programming experience. There was an article on Ars Technica a few weeks ago and everyone who does HPC work expressed the same experience that Fortran and C++ were orders of magnitude faster than anything else for numeric work, but that should be evaluated with the consideration that your typical Ars programming thread reader will have a lot more experience than most economists.

More importantly, for roughly four weeks of prep time… if that’s the concern dominating your choice of language, your code is going to be riddled with bugs that will require a lot of time and testing to eliminate. Your model is most likely not calculating what you think it is. The computer is only smart enough to do what you tell it to, and there’s little guarantee you didn’t introduce logical bugs that change the algorithm you thought you were implementing to something else. If you only have four weeks, better pay someone who knows what they’re really doing to double check your work.

Reply

Rahul July 3, 2014 at 6:25 am

Having worked in HPC a fairly typical development cycle is like this: Someone starts working on a new problem and he’ll often use an “easy” language to start: e.g MATLAB, Mathematica, Python etc. At this point the project is small and the focus is on getting results, ironing out bugs, validating the approach on toy problems etc. Many such projects may die out at this stage because people lose motivation, the inherent approach sucks etc. Most often they are running serial mode at this stage.

As the method gets more and more established the time comes when run-time starts becoming a major constraint (e.g. people throw bigger “real” problems at the method) Now’s the right time to either (a) rewrite the whole code stack in a more efficient langauge or (b) at least profile the code and debottleneck the slow parts, inner loops etc. If the problem warrents moving to a cluster you are looking at a new set of constraints and headaches anyways. FWIW 80% of the codes I see on HPC clusters themselves are C / C++ & Fortran with more of the latter.

In any case, premature optimization is the root of all evil. If someone starts off working on a complicated brand new problem in C just because someone told him it is fast, that approach mostly always leads to disappointment.

Reply

Kevin Sheppard July 3, 2014 at 4:07 am

Rather than think about this as a computer science-type paper, lets consider it as an economics paper. First, what preferences does the agents have?

She seems to have a utility function that (a) depends only on computational runtime and is (b) weakly decreasing in its argument. These are extreme preferences, and probably are driven by a McNamara fallacy (http://en.wikipedia.org/wiki/McNamara_fallacy) – that is, measuring what can be easily measured, not what actually needs to be measured.

A more reasonable set of preferences would include (a) runtime (b) programming difficulty for someone who has some competence (c) tooling and infrastructure, e.g. usefulness of IDEs and profilers, (d) cost, (e) suitability as a general purpose do-all language and (f) quality and availability of the ecosystem (e.g. Dynare). Academic users are mostly invariant to (d), although it can matter for those who are using clusters to solve models. Academics who produce novel models are very sensitive to both (b) and (c), which is one of the reasons why languages like MATLAB and Mathematica are popular for them. Academics who do data analysis are sensitive to (f) which is explains R and Stata.

Another very important point for economists is how likely is it that the written code is actually a correct implementation of the underlying mathematical model. Considering some of the high profile mistakes, as well as the lack of transparency surrounding most economic research (much of which is driven simply by high barriers to entry, either in skills or data acquisition), this seems like an important concern, (e.g. http://www.economist.com/node/5246700 ).

This difficulty with a more realistic setup is that there are too many degrees of freedom, and that most of these are difficult to measures.

The paper is mostly a benchmark of how well a programming language’s runtime can compile loops. This is why C++ and Fortran do well, as do Numba/Python, Julia, and MATLAB (which is just running Java under the hood for simple loops), while languages that cannot compile (JIT) suffer (Python and R).

The current version suffers by its unwillingness to measure the sensitivity to language-specific changes that most users of the language would actually use (e.g. apply in R or some reasonable attempts at vectorization in languages that have good support of vectors). Without considering language-specific idiomatically correct implementations – that ones that end users who are competent in the language would produce – it is difficult to interpret this paper as providing guidance for someone interested in implementing a dynamic macroeconomic model.

Reply

Marian Kechlibar July 3, 2014 at 4:34 am

As far as efficiency is considered, it is quite suboptimal to code software like this in one programming language only. This is close to the “one man produces the whole pin” model.

The answer would be to pick a programming language in which the boring-but-necessary routines (file I/O, user interface) are easy (and safe!) to code. Code those routines and a preliminary implementation of the heavy stuff. Then ask an algorithm nerd to optimize the computational bottleneck in whatever language, and invoke the precompiled library from your program to run the optimized code.

To put some analogy from the real world: languages are like vehicles. If you drive a car and you need to cross a river, you will take ferry to get your car across. Designing an amphibious car which can get over rivers on its own is probably very inefficient, given that rivers constitute less than 1% of your journey.

Reply

Rahul July 3, 2014 at 6:12 am

That reminds me of the Unix command line model of pipes and several tools each doing one job well.

But is it true for large projects? Take any big software package (e.g. AutoCAD, Ansys, Fluent, SolidEdge etc.) ; was it coded in a mixture of languages?

Reply

Brad July 3, 2014 at 8:06 am

There’s almost always at least a DSL involved (if you believe Greenspun, an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.)

Reply

Guillaume July 3, 2014 at 5:34 am

It all depends on the code. Typically, I understand they wrote loops in R which is usually useless and slow. There are many ways to avoid this and to call more directly the underlying C / Fortran source code.

Reply

Jesus Fernandez-Villaverde July 4, 2014 at 7:45 am

Dear Guilleume

Thank for your comment. Yes, we wrote loops. If you look at the code, you will note that it is not trivial at all to vectorize the code (we mention this in the paper) because we take advantage of the monotonicity of the value function. Futhermore, the matrices and vectors to handle are huge, which goes against vectorization (they require large amounts of memory). While I cannot claim I coded the best possible vectorized version of the code, the one we implemented was actually slower. I’d be happy to see a faster R version without loops.

Reply

Jon July 3, 2014 at 8:21 am

Time for evaluating function f1=function(x) { log(x)*pnorm(x)} in R for an array of 200,000: for R 2.3.1
For loop: y[i]=f1(x[i] 1.79sec

apply function y1=apply(x,1,f1): 2.39 sec
through array operation: y2=log(x)*pnorm(x): 0.07 sec

I think it is most useful to compare the languages as they would be effectively used.
Code below:
ngrid=200000
f1<-function(x)
{
log(x)*pnorm(x)
}
x=array(0,c(ngrid))
x=as.array(exp(rnorm(ngrid)))
y=array(0,c(ngrid))
y1=y
pnt=proc.time()
for (i in 1:ngrid)
{
y[i]=f1(x[i])
}
mean(y)
proc.time()-pnt
pnt=proc.time()

y1=apply(x,1,f1)
mean(y1)
proc.time()-pnt
pnt=proc.time()
y2=log(x)*pnorm(x)
mean(y2)
proc.time()-pnt

Reply

Jon July 3, 2014 at 8:29 am

Sorry: The formatting did not work. Also I checked with the newer version of R(3.02 32 bit): The for loop speed is the same, the apply function becomes 20% faster than the for loop and the vectorized version slightly faster than the earlier version of R.

Reply

Semi Essessi July 3, 2014 at 8:22 am

where is the code?

this is an interesting set of data but without seeing the implementations i am curious… how far optimised are the C++ and Fortran versions?

the fact that in the paper the intel compiler is outperformed makes me extremely skeptical… this clashes with my experiences so far.

Reply

Jesus Fernandez-Villaverde July 4, 2014 at 7:45 am

In github. The link is in the paper.

Reply

Asher July 4, 2014 at 8:51 am

I am currently on the R/Matlab margin (finding Python too opaque, despite having mastered many other languages e.g. Fortran Basic Pascal C Lisp Vax assembler). I find that Matlab runs much faster but my obstacle is I have found no way to run it on the cloud. Cloud has many advantages including me here and my RA there, also being able to get massive resources in Amazon when you need them and not pay for them when you don’t. This tipped the scales in favor or R.

Also I want to write code in the same language as everyone else and I have been guessing that open source languages will dominate. I am writing with GNU license so that others can use and improve my code and I feel Matlab is an obstacle.

Has anyone found a way to run Matlab on Amazon or equivalent? Could tip the scales back.

(Yes this is a bleg.)

Reply

Nathan W July 4, 2014 at 2:48 pm

If programmers who cost $200k a year to employ can be paired with economists to reduce the use of supercomputers which cost thousands of dollars an hour to operate, it is easy to imagine that lots of money can be saved.

But that doesn’t matter much to me.

In the meantime, I can make coffee while R crunches much smaller datasets. Worst case scenario, I can ask it to select small subsamples, recreate certain distributive aspects of the dataset, then proceed as though 500 were the new 500 million.

Reply

https://play.google.com/store/apps/details?id=com.bestquotesever.babyquotes July 21, 2014 at 3:05 pm

I’m no longer positive the place you are getting your
information, but good topic. I must spend a while studying much more or understanding more.
Thank you for fantastic info I used to be searching for this info for my mission.

Reply

Marcin Gomulka July 22, 2014 at 9:29 am

Run my own tests. Here are the results:

| Language | N | Time |h
|R64|1e6| 198.18|
|R32|1e6| 125.61|
|Julia|1e6| 97.22|
|R64|1e5| 16.40|
|R32|1e5| 9.20|
|Julia|1e5| 8.30|
|R64|1e4| 1.57|
|R32|1e4| 0.84|
|Julia|1e4| 0.88|

Language N Time
R64 1e6 198.18
R32 1e6 125.61
Julia 1e6 97.22
R64 1e5 16.40
R32 1e5 9.20
Julia 1e5 8.30
R64 1e4 1.57
R32 1e4 0.84
Julia 1e4 0.88

Reply

Marcin Gomulka July 22, 2014 at 9:30 am

Language N Time

R64 1e6 198.18

R32 1e6 125.61

Julia 1e6 97.22

R64 1e5 16.40

R32 1e5 9.20

Julia 1e5 8.30

R64 1e4 1.57

R32 1e4 0.84

Julia 1e4 0.88

Reply

Marcin Gomulka July 22, 2014 at 10:04 am

Language N Time

R64 1e6 198.18

R32 1e6 125.61

Julia64 1e6 97.22

Julia32 1e6 127.05

R64 1e5 16.40

R32 1e5 9.20

Julia64 1e5 8.30

Julia32 1e5 10.01

R64 1e4 1.57

R32 1e4 0.84

Julia64 1e4 0.88

Julia32 1e4 1.70

R scripting front-end version 3.0.1 (2013-05-16)
julia version 0.3.0-rc1+113

Reply

Leave a Comment

Previous post:

Next post: