Why are some programmers paid more than others?

The law of one price does not seem to hold, so Brandon Berg poses a question to me:

I’m curious about the reasons for the wide regional variation in wages for software engineers. Computer software would seem to be the ultimate tradeable good, as it can be sent instantly around the world at zero cost. I’m a computer programmer and have recently been looking into employment opportunities in East Asia, and was surprised to find that typical wages for programmers varied by as much as a factor of 5, with the US and Japan at the upper extreme and mainland China at the lower extreme. Wages in Singapore are less than half of US wages, despite a similar per-capita GDP. Wages in Shenzhen are less than half of what they are in Hong Kong, just an hour’s train ride away.

What’s going on here? Why do firms continue to hire overpriced American and Japanese software engineers when they can get them for half price in Hong Kong, even less in Singapore and Taiwan, and at a 75-80% discount in China? I’ve had some people tell me that American and Japanese programmers are just better, but I’m skeptical of this, especially considering the difficulty level of the interviews I had in Shenzhen and Hong Kong.


It is perfectly possible for American and Japanese programmers to be 'better' despite the difficult Chinese interview questions. Perhaps the questions were not really difficult - where better than China to study to the test? Or perhaps the author's experience is unrepresentative. (No!) Perhaps the questions were difficult and the author's experience representative - of a few companies which don't substantially affect the overall averages. Perhaps the Chinese companies get good programmers, and waste them so there is little point in asking why they don't pay the proper premium, in which case we would then ask why aren't foreign companies poaching the good Chinese, and someone can reply that all the good Chinese and Japanese go to foreign companies either by school emigration, H1B visas, or foreign companies located in China like the huge Microsoft Research facilities in China. (I favor the waste explanation - Japanese companies seem to be really terrible at IT, and I compile a bunch of relevant citations at http://www.gwern.net/The%20Melancholy%20of%20Subculture%20Society#fn1 ) Those MR readers who are also Hacker News readers will likely be familiar with what Patrick Kalzumeus ( http://www.kalzumeus.com/greatest-hits/ ) has to say about Japanese IT. And so on.

I often have to work with Software Engineers in South Asia for outsourced work. Though a third of the price of onshore workers they often overrun by much more than three times when factoring in essential onshore quality assurance, extra management, and terrible lateness with whatever excuse comes to hand. False economy.

They have to compensate employees for the cost of living, at least.

Computer code doesn't have a zero cost of transmission. Security is one barrier. An employer has to make sure that its proprietary code doesn't get stolen, and has to vet employees, set them up with secure communication, etc. All of that is easier when their ties are local, or close to it (not international).

"They have to compensate employees for the cost of living, at least."

That was my first thought, but I don't think this is right. Employers don't have to hire American programmers at all. Yes, if they choose to hire American programmers, they have to pay them the prevailing wages or better, but the question is why the marginal American programmer is hired at all when the marginal Chinese programmer is available for a fraction of the price.

And to be clear, I'm not talking about contracting out specific work items to be done overseas. I'm talking about moving (or forming) entire firms or product units overseas. I get that there are problems with splitting development across 8-12 time zones.

The real kicker for me is the Hong Kong/Shenzhen differential. They're twenty miles apart.

As a software engineer who is recently surrounded by H1Bs, one important point is the cost of translation. I as an American can talk to the business producer and tech lead in their native tongues and societal norms. The cost to keep me on board is roughly equivalent to my wages + equipment. The H1Bs make less than I do because their cost is wages + equipment + additional time for communications. I've only been here for 2 months and already I've been fought over by PMs who want me over the Indians because they can communicate efficiently with me. Perhaps the real cost is on the PM's inability to communicate effectively across the language/culture barriers, but as I have yet to meet a PM with that ability, I'm not ready to place the cost there.

Now it seems you're more interested in why there is less out-sourcing. And although infrastructure for outsourcing is non-trivial, it's an initial cost with trivial rent. However, it goes straight back to the language/culture barrier. Even if HR is comfortable (and often they are not), communication over the internet is less effective than in person. Now multiple that problem with the language/culture problem and tack on the drastically lesser communications because of timezone issues and even a back-of-the-envelope calculation will show that paying me $40/hour here in Oregon is a better cost than paying someone around the world $10/hour. And if anyone is out-sourcing to Hong Kong they are doing it wrong. I would suspect programmers in Hong Kong work for Hong Kong companies.

What blows my mind is why there is less out-sourcing from Silicon Valley to my home state of Montana. The cost of living is drastically less and I could be paid somewhere from 2/3 to 3/4 of my rate if I could have stayed at home. The language and timezone problems are nearly non-existent and the culture issue is something easily solved by a well established consulting shop. I surmise the problem is HR and inertia, but I had to relocate for my current position, which meant a doubling to tripling of my rate back home. I won't complain and am finding my talents are needed in my new venture. It's a win-win!

At the risk of being horribly stereotypical, I suspect that a lot of hard-core programmers like being surrounded by geek-culture, which means they tend to migrate to expensive areas like big cities and, of course, Silicon Valley.

There may be something to that; there doesn't yet exist a good entry path into programming for those who don't socially fit with existing programmers, and the easiest path in is to have access to computers and at least one programmer growing up.

I would actually look to high-tech founders and the intensely incestuous network of VC funding to explain why start ups locate in a few expensive areas. Geek-culture isn't, for example, a major feature at Brogramming-rich Facebook, but it's in the Silicon Valley because A) that's where you go to get buzz and B) that's where you go if you want to hire people leaving Google, Apple or graduating from Berkley and Stanford. Why are wages high in Seattle? Because that happened to be where Bill Gates grew up. Why the Silicon Valley? Because a Navy base was there. Why Boston? MIT. Why NYC? Because that's where money lives.

I think you have a strong point on the language/culture issue (not to mention time zone difference). My own anecdote: I fielded a question about a ticker symbol on a report. Neither myself nor the questioner had seen it before. I happened to be on the same floor as the Indian guys who produced the report so I approached them directly. I wanted to know which asset the ticker represented and what source they'd gotten it from.

I showed one guy (the sender of the report) the symbol and asked where he pulled it from. He explained to me in broken English that it was a ticker symbol. I replied that I understood it was a ticker, but what did it represent? Another Indian guy approached, and they explained to me that a ticker is one of those things that scrolls across the screen with prices. I repeated that I understood the concept of a ticker and asked them again where they got it from. Then they proceeded to explain to me that a ticker is like a nickname relative to a full name. After a few minutes of futility, I went back to my desk. The original questioner had been able to find the ticker symbol on his own. My conclusion from this experience: H1Bs are fine for predetermined functions, but there is no way they can be used for troubleshooting.

You've pretty much described my daily experience.....

I am a Canadian H1B living in the Bay Area and working for a Bay Area company. I think your experience working with Canadian, Brazillian, German, or Australian H1Bs would be quite different.

While doubting neither your story nor your conclusion, I wonder if there could have been some way for you to make yourself understood. It can be very hard to communicate well with people with low education and/or English skills, but with patience and creativity sometimes you can meet them in the middle. If more people were good at this, these people would have lower barriers to being productive, and learn a little more about communication.

But sure, it is indeed frustrating to get your point across when you can't, say, ask a hypothetical "If X, then what about Y", without being understood as also saying "X is true."

"What blows my mind is why there is less out-sourcing from Silicon Valley to my home state of Montana."

It's easy to understand. There's probably not a large pool of potential hires there, so hires would have to be found elsewhere and relocated. Many desirable hires may not be interested in moving to a rural state, so you may have to pay them more. Others may be concerned about the *next* job, and would prefer to stay in a job-rich area. Other highly desirable prospects, the more startup/entrepreneurial types won't want to be out of the loop of Silicon Valley/New York/Cambridge/Seattle scenes.

A company offering Montana-based outsourced development would probably want to hire older workers with kids, for whom a rural setting could be a big plus, but that means their costs might be higher. And a company staffed with 'olds' might not be very attractive for clients looking for 20-somethings who can sleep 2 hours a night and be productive.


But you can move easily between Hong Kong and Shenzhen - especially if you're Chinese so that twenty miles might as well be 200.

Sorry CAN'T move

Programming computers seems deceptively like a technical discipline. It requires mathematical and logical understanding and comprehension of technical details. Programmers produce precise seeming digital output and pretend to follow engineering protocols.

Of course, programming is not a technical discipline. It is a creative and social act. The essence is not moving bits around but understanding the users and their quirks and mistakes. Machines have to adapt to men, not the other way around. Programmers need to be in close contact with their customers to do a good job. The business process, the cultural context, the hopes and dreams of the users add up to a living system that software has to grow into, not an engineering specification.

Also, wages for programmers are less than half their Boston and Silicon Valley levels in middle America. Whatever explains Singapore has to address Kansas City also.

+1 more so for business computing than technical stuff. You just have to look at the documentation you get with your hardware drivers to see that, since that's already done in cheaper places because it isn't user-focused.

Depends what sort of programs. There's a difference skillset needed for designing a pretty website and writing a network device driver.

True, but AFAIK most device driver code is written by people on full-time at the hardware maker - which means (as Oz noted) that it's typically some guy in Taiwan or China already.

Most of the programming work out there seems to be either back-end/LOB stuff or websites. There just isn't that much call for low-level device-and-kernel programming in comparison.

(FD: I'm a professional programmer, doing a user-facing commercial product.)

That then leads into .. which is more profitable to create? A website or a network device driver?

Yep. As a software developer, I used to worry about overseas competition, but I don't any more.

Firms played with overseas development, and some have done it with success, but for the most part they've returned back home.

Software is not a simple "dump requirements on programmer, get working product back" process. It's very iterative, and the customer usually does not know what they want when the process starts.

That is so, so true.

(We have some overseas guys, but we work with them iteratively and integrate them into the team as much as possible - they're Real Team Members [on a discount], not Guys We Have Do The Scutwork And Hope It Works Out.

If you do it right it's fairly efficient; they work while we sleep, we get their changes in the morning, work and then they get our changes when they get in for the day...)

I did a deal with a large computer company (you'd recognize their "two initial" name) about ten years ago. Our project manager described it as "They just dumped a pile of garbage on my table and asked us to fix it." He was so frustrated, he was nearly in tears. He got it done, though, it took a huge amount of time (billable) to sort out "what do you actually WANT? Specifically?"

We also did a job for another client, in which the client insisted on splitting duties between us and an Indian company. It was a real pain working with the Indians; even leaving aside the time zone differences, our programmers were vastly more proactive. Note: we were a linguistics (localization) company and 80% of our staff (including the president and vp) were foreign born, so this is not prejudice. If "cheaper" had been "more cost effective", we'd have contracted the Indian company in a heartbeat. But it wasn't.

"Programmers need to be in close contact with their customers to do a good job. "

In addition for much programming the programmer has to understand the business process. Often better than the customer can explain it, let alone explain it to somebody who doesn't speak English very well. Sometimes you have to understand it better than the customer actually does. ;) I think non-programmers assume all you need is a computer and you can code anything.

Sadly, most programmers feel the same. That's one of the reasons I don't sell myself as a programmer. Software engineer, thank you much!

Software Engineer sounds, if anything, more technical and less connected to customers.

I title myself Software Developer when I have to market my skills. I want to move up the food chain - that I understand the business and technical. Sadly, I find most employers moving the opposite direction - "agile" methods to break it down to disassociated tasks. I guess it works for a certain type of programmer, one that I'd likely outsource/offshore the task.

> Sometimes you have to understand it (the business) better than the customer actually does.

+1 on that, especially if you are talking about process.

However, in fairness, I have to say that when foreign programmers do come back with points about why what the customer requires is *not* what the customer needs, it's often received very poorly.

It's hard enough for management to handle the programmer mind-set when you're both face-to-face in a room, it's probably almost impossible when the comments/questions are coming from someone who you've never seen, isn't a native speaker of your language, isn't in the same time-zone, and isn't even a part of your company - you get a lot of "just do what the specs says", when the spec is ambiguous or ill-conceived.

Completely agree. They say that if you really want to understand something, teach it. Programmers teach computers how to do things.

"It’s hard enough for management to handle the programmer mind-set when you’re both face-to-face in a room, it’s probably almost impossible when the comments/questions are coming from someone who you’ve never seen, isn’t a native speaker of your language, isn’t in the same time-zone, and isn’t even a part of your company"

I think this is a big part of the answer. I also like CBBB's comment about the difficulty of travelling between Hong Kong and Shehnzen -- maybe that has something to do with it too.

I think gwern must also be onto something when he says that programmers in Japan and America might actually be 'better' than Chinese or Singaporean programmers, but then that raises the question of why that might be.

Maybe there was a first-mover advantage in the places where wages are higher -- are the regions with higher wages for programmers also the regions with the oldest software industries? (I'm seriously asking -- I have no idea)

Maybe there's also a network externality going on here -- that is, the more programmers are in one city or region, the more productive each programmer is in that place. Places like Silicon Valley or Hong Kong, for example, where programmers are relatively abundant, in close proximity, speak the same language, interact in person, etc., might be more valuable (each) than the programmers in China, where programmers just don't have as big a pool of proximal programmers to learn from, work with, and maybe even compete with.

For exactly the same reason -- the network externality -- in a country with many telephones per capita, each telephone is more useful than a country with just a few telephones. Maybe programmers are like telephones... which is an insensitive thought, but is there something to it..?

Good points, but is the difference really 2x in America? I could believe 50%, but the avg for an Oracle SQL developer here in IL is $107K according to (iirc) glassdoor, which would imply the average is >$214K on the coasts. I would have have difficulty believing that, though I would like to see some comparative salary surveys.

"Good points, but is the difference really 2x in America?"
Often way more, Dave.

On an individual basis, yes, but of course that's true within regions too. But are the region-to-region differences that large?

Not for my field. Im not a programmer, but for Websphere Aplication server admins, the rates vary by about %20 or so, maybe more if you really negotiate. I doubt the wage difference is x2 for anything but extreme cases.

As mofo points out, there is a massive difference between IT that requires creative ability (software development) and IT that requires a specific skillset (anything administration). I personally do not consider software development as part of "IT", but cultural norms don't permit me that nuance.

I think that for the same job the difference is much smaller. When I worked at major tech companies our cost-of-living adjustment was less than 10 percent top to bottom, ceteris paribus.

But the ceteris is not paribus. The difference is that top employers are a major presence in Silicon Valley, but the programmers in Iowa work for the local power company. Within each firm the tougher jobs are all done at headquarters, so the Mountain View office isn't doing the same things as the Charlotte office. The jobs aren't the same across geographies.

chicago != springfield

Yeah, I suspect the split may be more urban/rural within the U.S.

For kicks and giggles, I just put "Oracle Database Administrator" into Salary.com.
Median salary:
With zipcode 95101 (San Jose, CA): $118,772
With zipcode 02112 (Boston, MA): $110,557
With zipcode 59718 (Bozeman, MT): $69,284

Thats roughly the same wages when cost of living is taken into account.

This suggests to me that, programmers are a very inelastic good. Also that in software, the cost of taxes is borne by the investors of the company or possibly the customers, not the workers.

"The essence is not moving bits around but understanding the users and their quirks and mistakes. Machines have to adapt to men, not the other way around. Programmers need to be in close contact with their customers to do a good job."

Twenty years in software engineering tells me this is the correct explanation.


Computer programmers are communicators: translators, interviewers, readers, writers. It is not entirely fair to say that ours is not a technical discipline, because there is a bit of technical work, but it is far more important to be a good communicator than a good technician.

Systems Analysts, such as yours truly, are often former programmers who have found they can do their jobs even better if they don't actually write code: if they focus exclusively on the human-to-human communication parts of the job.


Other things to consider, in no particular order:

- Soft skills are extremely important to being a good programmer.

- Being in the same office as the rest of the team working on software almost always works far, far better than scheduled meetings with groups in different time zones.

- The difference in output of production-quality code between the best and the worst programmers is remarkably large.

- Contract programmers, especially anonymous ones working for code mills overseas, generally have a far smaller stake in project success.

I'm tempted to blame it on complementary factors of production, but that's pretty weak. Could it really be the case that East Asian co-workers work less effectively in teams or that technology isn't utilized as effectively in these regions?

Maybe it's some sort of intangible, but like creativity. There's solving a software problem, and then there's solving it in a simple, easy-to-maintain, elegant, standards-compliant and future-compatible way. Those are different goods. But, then again, it's hard to imagine (for me) reasons why several populations of workers would have incentives to be more or less creative.

One big complementary factor is management. The rise of Agile software development as a system for productivity enhancement has been difficult to implement in cultural environments that place great importance on hierarchy and rigid roles for managers and their subordinates. Agile requires individual programmers to be much more involved in planning and decision making, leaving the manager to a role more akin to that of a coordinator.
American culture is well suited to this approach and it has paid off well, but I often see it fall apart in other cultural contexts.

More generally, here is a test: tell your manager her/his idea is wrong in front of a group, and walk through step by step why it is wrong in 10 different ways. Then show why your idea is better. If you can't do that, the product will suffer.

Also, tough technical quizzes do not signal high productivity among their workers. In the end building a product relies less on knowing the ins and outs of multiple inheritance and more on using limited skills (and we are all limited), intelligence, creativity and communication to get the job done.

Also an excellent point - I have a boss. He is quite smart and usually right about things, but also occasionally completely wrong.

I can tell him "you're smoking crack" (I, in fact, often literally use those words) and, with a good argument, get him to change his mind, and nobody gets upset.

I shudder to think what kind of anti-productivity we'd have if nobody could do that...

Agile is awesome. It has made my life a lot easier and made me more productive.

I hadn't considered the team hierarchy component. That's a good catch, Ross. Agile fits the "decentralization" or "flattening" pattern of management that seems palatable to most American business students. There is probably something cultural there, though "East" versus "West" is probably too broad a stroke.

It depends a bit on what sort of software you're talking about, but context can be a big deal. If you're a company writing software that you plan to sell to companies in the US (who tend to have more money to spend on software than companies in lots of other parts of the world), you really are much better off with programmers whose native language is English and who reside in the US. Those programmers will be far more effective than programmers in, say, India, not because they are better programmers, but because they have better context - cultural context, geographical context (fighting against disparate time zones sucks), linguistic context, etc. The fact that software itself can be transmitted around the world at effectively zero cost does not mean that the requirements for said software, or the context necessary to properly implement those requirements, can be.

Berg is ruling out non-cash valuations of different properties of software engineers.

For example, an American Company, such as Activision-Blizzard, on top of not having to deal with time differences when collaborating, might value hiring American, if not local, programmers. A lack of outsourced labor can also contribute to a desired workplace culture/environment, where in-person collaboration on complicated software can be valued over long-distance (and cross-timezone) collaboration.

Furthermore, at least for Japanese companies, national pride and unity can account for a large portion of wage differentials. One could hardly expect a Japanese company, who does a majority of their work in Japan, outsourcing labor would be a severe cultural taboo.

Some economic decisions just don't easily translate into dollars and cents.

MA Economics

Japanese companies are perfectly capable of outsourcing even famously Japanese products almost in their entirety.

For example, anime. Most people know that American animation is often outsourced to South Korea (such as _The Simpsons_) but how many are aware that most anime is actually animated in SK? There are exceptions (Studio Ghibli does a lot in-house), but it's pretty common.

A few months ago I was transcribing an old _Animerica_ interview with Gainax co-founder Toshio Okada (http://www.gwern.net/docs/1996-animerica-conscience-otaking) where he mentions that the production of _Nadia_, the anime that brought Gainax to prominence Japan-wide and not just among otaku and was ripped off by Disney for its _Atlantis_ movie, was so chaotic that multiple episodes were were not just animated in South Korea but *directed* by the South Koreans as well!

Although programmer output is virtually costless to transmit, its quality is phenomenally costly to monitor - quite possibly more than the cost of initial development itself. So reliance on other practices to monitor the quality of output must be relied upon, like reputation or signals of conscientiousness already inbuilt to a culture.

Especially thanks to programmers' well known propensity to spend their time having cardboard sword fights while standing on their office chairs:


You use cardboard swords?


A few limbs get lost, and HR gets all bent out of shape. It's ridiculous.

+1 to this as a former software guy.

I think there is a network effect among good programmers. A group of good programmers can self-monitor more effectively. Combine this with the well documented Pareto distribution of programmer productivity and I think "innovative" software development is actually cheaper when done by a small group of of top 10% programmers.

There are costs to hire and manage a programming project overseas. People who are remote are more difficult to work with effectively, due to timezone differences, foreign language communication difficulties and telecommuting/travel costs. Engineers in China and India may be just as good, but they will be less effective working on a project for a US based company than local talent.

I have gotten very good programming work from some Chinese programmers. But there are additional costs to remote work that go beyond the hourly wage. And there are (sometimes) communication issues and communication between project management and developers is critical. And while the work product can certainly be sent around the world frictionlessly, there are times (more than most would think) when having everyone in the same room makes a big difference. I have worked with one of my current developers for a decade, and I can pretty easily work with him remotely. But for new developers that's not as easy or sometimes even possible. I have a developer I just brought into my team, and he's really top-notch, but working with him remotely takes a lot of effort and back-and-forth. The point of all this is that there might be excellent programmers somewhere else, but working with them from that somewhere else may be significantly less efficient than working with someone local.

What's going on is this: viewing programming as a fungible commodity grossly over-simplifies reality on the ground. The best programming isn't created with more effort or more time, but by more innovation and more creativity. These are cultural, not technical, differences, and take much longer to change.

The Economist had an article on how the internet would cause programmer off-shoring (anyone have a link?) I remember it being deeply wrong.

As a writer and a programmer I can tell you it's almost certainly because programming is essentially creative-writing. You could say the same thing about books. In this age of electronic books why can some books cost $.50 and some $500? Exactly the same reasons. There are Steven King's in programming. There are Faulkner's. And there are tons of self-published authors with little command of spelling or grammar.

There is truth to that, but it seems you are correlating all American programmers in Silicon Valley and the other tech areas with Steven King. King's work are specifically outstanding and deserve the praise he garners. Not all Silicon Valley developers are worth their weight in gold. Even barring the odd ZMP, most developers are no more capable in their chosen skill-sets than others, no matter the locale.

You say that. And yet throughout the entire world we still don't have enough programmers to meet demand. Even when they're handing out six figure salaries. Wall Street hands out these sort of salaries and every kid in America wants an economics or business degree. Silicon Valley hands them out and the industry has a labor shortage.
That says to me that programmers trivialize what they do and should probably be demanding double or quadruple the salary.

By and large the best programmers don't remain in China and India; they make a beeline for the west (quality of life aspects, mainly). So no wonder that American salaries are higher.

My personal experiences disagree. The H1Bs beelined here for the opportunities and have no desire to remain beyond their work life, which isn't too long considering many can move back home after 15-20 years and retire. This is mainly Indian co-workers, but even my Chinese brethren have said much the same.

"The H1Bs beelined here for the opportunities and have no desire to remain beyond their work life, which isn’t too long considering many can move back home after 15-20 years and retire."

Yes, but they are going back to retire! Thus the original contention that by and large the best programmers don't remain in China and India is still true. If they go back to their native country relatively rich and don't program anymore, then they are no longer among the 'best programmers'.

Very, very few people move back to India. It's nice here. No hassles, and things just *work*.

Many more say they will move back than actually do. The divergence is mostly a signalling issue.

You can't trust what a 25 year old says about what his life will be like at 35, much less 65. This is signaling, nothing more.

You can believe it when they say American programmers are just better. There are a lot of ways to get to a program that 'works'. A few clean, maintainable, flexible ways and a lot of terrible spegetti code messy ways. Indian expertise in spegetti code is unrivaled. Communication and infrastructure are also important inputs. How you structure the go between your users / stake holders and the people actually writing and editing the code is really important. Full time senior developers that interact with our outsource teams, project managers wiling to work off hours, etc. For developers in India especially additional infrastructure costs are huge. Battery backups for individual machines, diesel generators for buildings and server rooms that need to be able to run on a daily basis. If you actually start a company there you'll pretty much need to start your own bus service too. Salary is the least of the problems, overall costs I'd bet aren't that different over time.

Wages in San Fransisco as compared to almost anywhere else in the US are what really suprise me.

I can't speak to the quality of the software engineers, so let's assume they're the same. Good programming scales well so it may take the same amount of time to do the same project for firms of different size. If a programmer adds more value to the American or Japanese company, then it would make sense for them to be able to demand more money from the firm since it would take roughly the same amount of time to save them more money than a less profitable firm might get.

"I can’t speak to the quality of the software engineers, so let’s assume they’re the same. "

LOL. Let us assume that all people are the same, while we're at it. Of course, the conclusion will be useless and unreal, but so what...

Actually, the differences among software engineers are enormous. The worst ones actually have negative value, creating minefields of intractable bugs which their peers must later correct. The best ones may have 5 times more productivity than the average ones. Etc.

Lets assume all spending is the same...and all capital, labor and investment is homogeneuos as well.

Boy I sure am confused about why wages are different in different places and I really thought that housing bubble in 2006 was just due to productivity growth.

I'd say somewhere around 10-20% of working programmers I've met actively generate negative value. About 50% are probably about break even relative to their cost (wage + tax + rent + power etc.). The very very best are better by at least one order of magnitude, possibly two.

Comparing the productivity of good to average software engineers is like comparing the productivity of world-class architects to roofing workers.

This is exactly it!
Programmers are an incredibly heterogeneous good. The good ones are rock stars. The bad ones are arsonists.

You would think the high profiled companies would hire the best 'talent' (at least that's the impression generated). A look at the reviews of the iPhone Facebook app should make this company think twice about an IPO at the moment.

Tyler has very strong Keynesian methodological influences so its what you would expect to see here.

1. It's an imperfect tradable good. At least at small companies, one of the functions of the software team is to work very closely with the business guys to figure out the sweetspot of what you can do given the mixture of time available/what the customers want/what we can use off the shelf vs build it ourselves. That's a surprisingly interactive process, and offshoring it doesn't work very well.

2. The skills of software people vary widely. People usually talk about the good people being 10x better than the mediocre ones, and the bad ones have negative productivity. It's hard enough to figure out whether or not someone's good when you're sitting with them and don't have to deal with things like language barriers. Hiring software people is _HARD_; hiring software people remotely is even harder. An 80% discount isn't interesting if you can't figure out where the person is on the "smart and gets things done" scale. (http://www.joelonsoftware.com/items/2007/06/05.html)

3. Where is all the great Chinese software? If they're really good, they'll be contributing. Open source matters, a lot, for figuring out how good people are. As Chinese developers contribute more (and I see no reason why that won't happen) they'll get more credibility. My personal, baseless guess though is that China actually has quite a bit of internal demand for software, and lots of good people are working on stuff like Baido that's invisible to a non-Chinese audience. That soaks up enough good people that the guys who are available at 80% off aren't the cream of the crop.

4. The model that seems to work well is to set up local operations. Microsoft Beijing is probably getting good people. Hiring a outsourcing firm, though, seems to be a total trainwreck for anything that isn't really simple.

Excellent answer.

The best programmers are easily 10x the value of the average, and the worst do more harm than good. It's really really difficult to tell why, but it's a whole complex mix of logical, interpersonal and work-management stuff that's very hard to pin down, just like any craft I suspect.

The key here seems to be a combination of:

1. programming is NOT a commodity
2. cost of living


I have to disagree here as well. Programming IS a commodity. However, programming in a black hole is useless. There is so much more to being a developer than knowing how to put "Hello World" on the terminal.

But then again I find it particularly difficult to include the entire software lifecycle in the term programming.

Services are not commodities by definition.

rice not the rice farmer is a commodity.

Not sure, but it could be that the value of a (equally awesome) programmer is less in China than in Japan and the US. The reason why could be that other cost factors are more important in China (e.g., regulations, bribes, risk of arrest), or that the potential cash to be extracted from a given piece of software is less (no VCs or mature second market, no protected monopoly due to poor intellectual property protection, small advertising or premium software market, etc).

Nothing in Brandon's observations suggests that the law of one price does not hold. Companies that need physical bodies in desks have a subset of available programmers in a region. The programmers that fit above this pool simply work directly for telecommuting friendly foreign companies for a more competitive wage. Therefore, if you look for a job in one of these countries, one will find a lower wage compared to if they looked for a job they could perform while living in one of the mentioned countries.

I'm actually more puzzled why Wall Street hasn't been outsourced.

Wall St. runs the US they're not going to shoot themselves in the foot

Dubai is on that, and it's not working out too well.

The only people working in Dubai are ex-pats.

Are you kidding? Have you tried to understand what traders talk about? If you think accountants are difficult to parse, you should try to understand investment bankers - when they have 12 seconds of their precious time to spare.

The big-money jobs on Wall Street require either a lot of customer contact, schmoozing and salesmanship (IPOs, M&A, selling structured instruments), or they have the prerequisite of years and years of on-the-job learning in a highly-networked finance location (NYC, London, HK) -- after which nobody wants to take the salary cut of moving to a 3rd world country.

The small-money jobs in the back office administration or in office IT support or whatever are increasingly outsourced to India and China already/

The question is premised on the [nearly] free shipping of bits around the world, which is wholly correct, but developing software is not about moving bits so the remainder of the question is based on a failed connection to the premise: developing software is NOT about moving bits. (Frankly, I would expect someone who develops software to fully understand that said activity is not about moving bits.)

I was talking about doing software development locally in China and then shipping the finished product overseas. I think most boxed software actually gets physically manufactured in China anyway, doesn't it?

Why not write all books in china and have the manuscript shipped overseas?

We understood what you were talking about, Brandon. We just all think it was as goofy as if you said, "Jetliners mean you can ship produce anywhere within 24hrs. Land is cheapest in Nunavut. So why don't all the farmers grow their vegetables there?"

"Computer software would seem to be the ultimate tradeable good, as it can be sent instantly around the world at zero cost."

This is a major fallacy in your line of reasoning. Finished self-contained computer software is a highly tradable good. Once Microsoft writes Windows it indeed can ship it anywhere in the world at zero cost.

However 99.9% of programmers are working in larger teams, and not solely producing standalone products. Components of a software system have extremely high "transaction costs." Interface and implementation details of one component of the system has huge impact on design decisions of other components. Finding an optimal balance between the needs of each programmer on the team is an extraordinarily complex equilibrium. Especially on large systems (say >100k lines of code).

Programmers who are working on the same project but separated by geography, language, culture, company and/or timezone are at an extreme disadvantage to programmers where communication is frequent, easy and casual.

This is exactly right. Do not underestimate the value of water-cooler conversion and hallway debates when coordinating large software projects. I have a LOT of experience in this area. I've been working with mixed teams for more than a decade. It's always better to be co-located. And it's always better for a team to be culturally close to the customer, because you never have enough customer information and therefore you have to be able to put yourself in the customer's head to a certain degree if you want to do the job right. If the customer doesn't even live in your culture, that's nearly impossible.

I suspect that you'll only see high-paid Indian and Chinese software developers when they start making great products aimed at Chinese and Indian customers - and those customers have the money to pay for them at world rates.

Programming is not just a simple act of once and done production. Any programming must be designed, documented and tested and altered. It doesn't matter how technically proficient (you think) you are-if you can't read the specifications and write user peripherals or comminucate with quality assurance personnel-you are worthless.

comminucate or er ahh.. communicate

"I’m actually more puzzled why Wall Street hasn’t been outsourced."

American financial dominance is largely predicated on worldwide perception of the trustworthiness of US institutions along with the fairness of the American legal system. Same can be said for the UK.

Asian nations for all their industriousness, focus and intelligence (which gives them huge edge in manufacturing) are not perceived by Westerners (or even most Asians for that matter) as being trustworthy. Whether this perception is right or wrong (and as someone who traded Japanese equities for several years I can say there's more than a grain of truth to it) it seriously hurts would-be Asian financial competitors. Most people are simply not that comfortable entering into a bilateral swap with an Asian bank under an Asian legal jurisdiction, or investing in an Asian hedge fund with a local custodial bank, or shelling out several million dollars to buy a membership on an Asian exchange.


As many other people in these comments have noted, programmers are not a commodity good. The difference between a good programmer and a bad programmer is huge, and more importantly, is often inherent in the programmer's mental models or otherwise deeply-rooted, and very difficult to overcome even with good training and management, let alone bad/remote/outsourced management. Of course, not all American programmers are good, but the good ones are definitely paid more and bring up the average price for American programmers.

I wouldn't want to be a computer programmer for all the money in the world.

Differences in productivity.

I own a software company -- with twelve software engineers on staff. 8 local, 4 overseas.

Specifically to outsourcing to China -- we would never outsource to a Chinese development team because of the lack of respect for intellectual property. Way too many horror stories from friends of mine who've seen multi-million dollar investments in software developed in China, only to see the same code (line-for-line) a month later being sold by a new State-owned company.

Specifically to outsourcing to India -- Speaking from multiple experiences, working with remote Indian (and Pakistan) teams takes a LOT of hand holding to ensure quality. It takes a lot of time and costs a lot to manage outsourcing. This is often the case when there is a cultural divide.

Specifically to hiring locally in the USA -- nothing beats having your development teams sitting in the same offices. The marginal cost savings of outsourcing often do not scale well as teams grow in size. Plus, the best talent often wants to live here and are often willing to immigrate.

My two favorite countries for outsourcing - Russia and the Philippines. But nothing beats a good locally based developer who you can sit down with face-to-face every day.

A harvard grad is hardly 5x smarter than the average UCBerkeley grad. But people still hire the harvard guy at that price.

This comment and the one from Software Company Owner above speak to the mobility of the ZMP.

Actually he is-you can get into Berkeley with 'high' grades from a shoddy school.

As an American programmer who works with foreign imports, I'll chime in:

1) The ESL barrier is often a bigger problem that you'd think, moving from functional design to technical design to code without losing meaning can be a challenge. Americans who are able to do all three are very valuable.
2) Cultural differences also matter. Some of these can work in favor of Asians, but not all -- American software programmers are better at providing creative solutions in my experience. Shame cultures tend to struggle in some situations where intellectual honesty is of paramount importance.
3) Credentials have an effect too. A Master's degree or a business cert like CPA seem to make a large difference in earning power.

I also second Mike's point above, and James' point 1 is particularly important.

Finally, regarding the interviews -- per 2 above, Chinese programmers particularly tend to be very good at memorizing technical answers, but less good at innovation. This matters more in some contexts than others, of course.

The original post says American AND Japanese programmers are paid more, not just American programmers. Why do people keep eliding this?

Japan is a "shame culture", too, no?

That's a good point, but then I've never met any Japanese programmers. I get the impression they discourage immigration though, so I'd guess that's their biggest factor domestically.

I had a job a couple years ago where for cost reasons the company decided our US-based team could only hire new programmers in Singapore. It was a nightmare. Besides the timezone and communications issues, we had a great deal of trouble finding quality programmers who didn't require us to constantly hold their hands and recheck their work. We had programmers who would write exactly the code they were instructed to write and then send it to us without even bothering to verify that it ran.

Management suggested that we hand off the really easy pieces of the project to the Asian programmers but these pieces were all things we could write ourselves very quickly. We would spend more time explaining it to the Singapore guys and checking their work then we would spend writing it ourselves. It was the hard parts that we really needed to be able to hand off, but the Singapore team simply didn't have the skillsets to handle that stuff.

The number one issue with the programmers in Singapore was simply that they did not do creative problem solving. The most competent of them could write a program that delivered, in the most literal fashion possible, what you had asked. But the ability to step back from the problem and ask questions like "Is it efficient? Is it maintainable? Will it meet the clients' needs in a broad sense?" was virtually nonexistent.

Hiring was also a big, big problem. We would ask a general question over the phone, there would be a brief pause, and then we would get a word-for-word quote of some piece of Microsoft documentation. I have no idea if they had memorized all this stuff beforehand or were just really fast Googlers. Eventually we began requiring candidates to actually be present in the Singapore office when we conducted the phone interview with them so that we could verify that they were not reading their answers out of a manual.

Cost-saving corporate managers have attempted to reduce code writing to a rote technical skill which can be performed anywhere by anyone, but programming is simply not amenable to this approach, at least not yet. Until the tools get a lot, lot smarter, writing programs will continue to require non-fungible abilities such as creative thinking, adaptability, big-picture thinking and wide-ranging generalized knowledge of technical subjects. Even the idea that good programmers can be produced by a four-year degree program is illusory. Good programmers must be trained practically from birth. Of course very few parents are sitting their five-year-old down "C in a Nutshell," but Western parenting would seem to currently have the advantage when it comes to imparting the critical mindsets necessary for high level abstract problem solving.

assign +1.

Why should they be sitting their kid down to read "C in a Nutshell" - get that kid to Princeton and then to Wall St. and they'll be pissing wine worth more then a Google Engineer's salary.

If American programmers are on average just better than Asian programmers, then why don't employers offer American expats American wages? When a recruiter contacted me about a job in Shenzhen with a top-tier investment bank and asked me what sort of wage I was looking for, he told me that $30,000 per year would be overly optimistic.

Really? Shenzen isn't even THAT cheap

This doesn't match my understanding and admittedly anecdotal experience that Americans are sought-after and highly compensated in ex-pat positions. Do you have any data to back it up?

But not in programming, because no one respects that. I can bet you that the investment bankers there were making a hell of a lot more then 30K.

I can't counter with data, but from people I know, yeah, the programmers too. Admittedly, they were good. And I've certainly seen it in managerial positions.

Similar situation, different outcome.

Most of my team was in Kuala Lumpur. Once we figured out an appropriate strategy, my boss and I were pretty successful at getting results. What we would do is give the remote team some high level direction about what we were looking for and then ask them to give us a suitable design: No programming -- psuedocode and Visio. We used that as a carrot to get them to continue to do the tedious work that we had done but didn't have time for.

Of course, we still had issues with communication (the ethnic makeup of the team was such that the common language, before we even got on the phone, was English), timezones, and high turnover.

I had a toy that taught be binary before I knew how to count. Everyone I know who is a really good programmer had a parent or close relative who was a programmer, had a lot of free time as a kid to play around, later picked up some artistic hobby/interest and then returned to programming.

America has the advantage that personal computers were ubiquitous in schools early on, letting slightly more kids have the chance for that early creativity-driven exposure.

Well they DON'T hire American programmers, not really any way.

I'm still curious what languages you are proficient in and certifications you possess.

I will absolutely concede that it is much harder to break in if you are not hired directly from college, and that in turn is much harder to accomplish than it was 10 years ago. Time was they'd take anyone who could type their own name, and if half of them washed out in a year that was a good crop.

Depends what you mean by proficient since I can program but have very little experience in professional software development - the languages I know would be Java, C++, C#, and to a lesser extent Perl. No certifications though and I'm not looking to get into software development. But I still say it's a crappy field basically for the reason you hinted at - until more serious professions there's no real path to a career.

If you don't mind sharing with us, what do you (CBBB) do for a living? Presumably something computer-related, or you wouldn't know multiple languages? Also presumably something with a more well-defined career path than IT. As an experienced consultant/developer myself, I've seen how quickly a software development career can plateau. Careers advice would be much appreciated :)

Well I did a STEM degree - like Tyler Cowen recommends

So I'm unemployed

My unsolicited advice -- if you find you've plateaued within a technology or company, expand outside them. I can't even tell you how many people I know who migrated to Sharepoint and doubled their income. It's ridiculous. Know the market, learn the skills that are in demand.

Interesting. Are you trying to get into finance, then?

Software can be hard to get into, but once you're in for a few years, sky's the limit. It's one of the easiest skills to turn into a business, either consulting or developing a product. I would say if you want to be a millionaire someday, it's one of the best fields to throw your talents at.

I haven't pushed all that hard, and I think I'm making about 190K for 2011 all told -- and of course there are people doing a lot better than me, mostly those who were more aggressive (I got a bit lazy after I started my first company). My wife is pretty laid-back, and makes considerably less as a Java programmer but still a good living and she could be making 40%+ more in her position if she was willing to job-hop a bit.

And let me tell you, some of the people she works with... sheesh. You seem like a bright guy, so I can just about guarantee you would meet the minimum competence to hold a job, if you can get in the door.

Talldave- does sharepoint really make that much money? What do you do in it anyway?

It's apparently been the hot thing for a little while. At our company we had a few people migrate in that direction, and they left for six-figure jobs (one guy maybe 2 years out of college). Of course, they went to consulting companies too, which makes a difference.

My understanding is that Sharepoint is mostly configuration, some coding (C# generally), but I haven't looked at it that hard myself. A few years ago, almost no one knew it so it was a great opportunity. A lot of companies have Sharepoint sites and seem happy to have existing employees learn the skillset.

> I’m still curious what languages you are proficient in and certifications you possess

My experience is extensive but restricted to the Silicon Valley culture. But what I see is that possessing certifications is strongly correlated with weaker candidates for a programming job: weak in experience, and probably even weak in ability after controlling for experience. For sure, it's quite uncommon for very strong candidates to present any on their resume.

Breaking into programming is much tougher these days. Here's one nearly sure-fire recipe though (which is not to say it is easy or quick). Write a high-quality, interesting, and substantial open-source project; or become a significant contributor to a major existing project. Put this prominently on your resume (remember, people need to be able to link to the documentation and your source). Be somewhat creative in how you get your resume into people's hands (not much will be required, but submitting only to the HR department at big-company is a fail). Employment ensues.

Interesting. My experience looking at Oracle DBA positions was that everyone wanted OCP cert; many firms would not even talk to you without it. Of course that's a job that that emphasizes knowledge over creativity, and most would say it isn't "programming." My programming jobs have typically been in an obscure 4GL embedded database language supporting a specific application, so no certs applied, they just wanted to know what I'd done and if I knew the standard includes, etc. Gradually easing my way into PL/SQL and DBA work now.

I think your open-source advice is outstanding. Nothing sells your coding skills like code!

I know certifications are important in other niches. DBA I didn't know, but I recognize some of the Cisco networking certs seem very very highly regarded. My experience is just in programming though. Not to be too snobby about it but ... I've fought like hell to get someone to join our C++ shop who has never written a line of C++ or even C code in her life (we lost that one, to Google, sadly, new graduate; we should have gone up to $300k to get her but I got too timid). That someone needs to have experience in a particular language is strange to me; that someone thinks this is a valuable skill enough to get certification in their "X" is definitely a warning sign.

Yeah, very true, I was put on a project to do SQL Server code through Cranberry, with everyone's full knowledge that I'd never done anything in either. They just expected I would learn.

My first interview out of college was similar, they gave us programming tasks in this 4GL we'd never heard of, and we had to write code to perform the tasks. Some of the tasks were deliberately made unclear or even impossible as written, to see if we would ask the right questions.

$300K is very impressive for a new grad btw, I'm guessing this was a top prospect in a Master's program from somewhere like MIT or Berkeley? What kind of work were you looking to put them into?

Lately I've been toying with moving to something that combines programming and math or physics, I've learned a bit studying Polywell fusion, and the combination seems a bit more rarefied. OTOH I'm very comfortable where I'm at, but it's always fun to think about.

iPhone or Android apps can work too. The real trick is learning how to interview (write code on the board, talk about design patterns, know binary trees backwards and forwards, know grep and sed, be able to talk through your thought process while solving the puzzles, etc). Either network heavily or find a recruiter who thinks they can pitch your skillset and then profit. For example, if CBBB included a link to a technical blog and expressed interest in a specific geographic area a couple interviews might come of this ;-) Relatively few technical hires come out of slush piles, especially because technical interviews have such poor predictive value.

People are hiring coders like mad right now. It can be a little difficult to get a foot in the door because standing out from the flood of unqualified candidates is difficult, but no qualifications are required. Certifications often are a sign of people who came to the profession later and don't have the intuitive grasp the best programmers do; people who are looking to hire competent-but-not-outstanding programmers or people with a specific skill set may look for them, but the $200k a year jobs would consider them a black mark.

How about: ''All-inclusive software programmer''? Sort of like efficiency wages, good software programmers will do more than just the bare minimum, and will do the extra little tasks related to the main task they are paid to do. For the employer, this is a bit like the idea of paying for an all-inclusive resort. You pay more, but (1) you get a lot for your money, and (2) you avoid the hassle of finding someone to do every little task that crops up (search costs, transaction costs).

The employer, or client's, time and energy are valuable: if you can delegate to someone reliable, that has a lot of value.

Yes, in the multiplatform corporate environments I've been in, there is a high premium placed on being able to do several things in addition to your main job. Sometimes they have applications for which they don't want to pay a full-time developer but which need tweaks here and there.

1) Real programming is fundamentally an exercise in applied mathematics. (Given initial state s0, and inputs [i0, ... in], the program transitions to state s1 with outputs [o0...on].) The bulk of advancement in computer science has been to try to get around this problem, since there are not enough mathematicians in the world interested in working as programmers to fulfill the demand.

2) It is American culture and (for all its manifest faults) education system that produces people that can reach the margin that they can become productive programmers. I have seen multiple instances, with multiple permutations, of attempts to offshore software development to India. Negate ROI is the consistent result. On the other hand, three of the best programmers I know are American-educated Indians.

3) The time zone/people in the same room problem is real, but does not cost more than 25% when working with Israelis, despite the non-overlap in hours.

4) SF wages follow SF housing prices. Oddly enough, this does not hold for NYC.

5) The big advantage for me to stay in Austin (TX) is that I have a lot of potential buyers for my services right in town. From the standpoint of a working programmer, networking effects of locality are huge. I could take a 20% pay cut, move home, and live like a king--until I need to find my next job.

So, when I hear someone talking about offshoring their development, I just laugh & move on.

I am a mathematician turned a programmer, and although what you say about "exercise in applied mathematics" is true, it is true on such a level of abstraction that it is basically useless.

Getting the "feel" for producing good, maintainable code is a decidedly un-mathematical skill, and it takes time and practice to learn . Some people can never really do that.

+1 from a programmer who is fairly bad at math.

The only portions of mathematics my job regularly intersects are logic and set theory. There are of course subsets of the field where more math is required but if you're programming databases or windows apps the likelihood that you need to remember anything about your university calculus classes approaches zero.

Some people would argue that a familiarity in some mathematical fields (discrete math, logic, category theory) at the intersection of pure math and theoretical CS helps in producing reliable and maintainable software. There are plenty of anecdotal reports that becoming familiar with Haskell or similar technologies (which are heavy on mathematical formalism) noticeably improves code quality, even when using other languages in day-to-day work. Still, the comments here are spot on in practice; even some math graduates will have no more than a cursory familiarity with such areas (since they are are off the calculus/analysis/'continuous math' branch which is most commonly thought of as "applied math") so they are liable to fall back on informal reasoning.

1) Man that's bullshit - being a good programmer has very little to do with being good at math. Being good at math doesn't hurt of course but it hardly helps all that much. I know someone from university who works at Google - the university offered 2 types of CS degrees and he did the "watered down" one so he could avoid taking Vector Calc and a few other heavier math courses because he was crappy at math courses. But he was a huge nerd who basically lived on his computer and knew just about every major programming language and tool. THAT'S what counts if you want to be a programmer.

Knowing programming languages is not the same thing as being able to program.

Yeah that's true - but if you hand in a resume that says you know like 10 languages you're much more likely to get an interview then if your resume says 1 or 2 languages and a good background in math.
I'm sure this guy knew how to program quite well (Google interviews are not easy to bullshit through) - but what I'm saying is the whole "programming is applied math" line is a crock.

While I agree with much of what you're saying, your first point is completely off the mark. The bulk of today's programmers are not doing math - they're using high-level languages and frameworks and building the software is mostly an exercise in wiring together existing objects or creating new objects that lean heavily on a very rich layer of smaller objects.

That's not to say it's easy - the existence of those tools has simply enabled us to ramp up the complexity elsewhere, but that complexity is of a different kind. There's little need for people who can write a highly optimized sort algorithm, because we already have plenty of those. What we need instead are people who can be productive in an environment riddled with network effects, poor and incomplete information, hidden requirements, and who can find the right balance between being a good team player yet knowing when to push back when things aren't right.

I work for a large software firm, and we don't even require college degrees - a 2-year diploma is good enough if you have other desirable traits. We have a few people with no degree at all, and more working as programmers who have degrees in completely unrelated fields. And we place very little premium on masters degrees.

These days, we look more for 'soft skills'. Assuming you have the minimum technical requirements, what do you in your spare time and how quick on your feet you are in an interview tells me a lot more about how good a programmer you're likely to be than a bulleted list of languages you know or a list of math courses you took in college. Languages come and go, but people with high intelligence, good communications skills, creativity, and the right level of confidence/humiltiy are very hard to find. And unfortunately, degrees aren't a particularly good signal of those aspects, other than perhaps intelligence. And today, not even that in most cases. Being able to pick such people out in another culture can be nearly impossible.

I find it amusing that my statement is being attacked for what it does not say. As a mathematician who has learned to be a programmer, I fully understand the difference between the disciplines. Are any of you going to seriously challenge my first sentence given the second? How on earth do you demonstrate that your programs do what you want? More pointedly, if you don't understand that the second sentence is PRECISELY what testing is all about, you're going to make your customers miserable.

But now we come to my final sentence of my first point. The fact that there is any objection to my first sentence is either a demonstration of the complete ignorance of the working programmers on this list, or that we have succeeded quite well.

The issue that I was trying to stress with my first point is that without some understanding of what a proof looks like (recall HS Geometry), potential programmers are going to be significantly handicapped in their work. Furthermore, without the kind of directed creativity that is involved with solving those problems, a potential programmer is going to be hugely handicapped.

It is this directly creativity which is deeply ingrained in American culture, and that is why we continue to dominate.

all of this is true within the US as well. unemployment rates are high but it is still extremely difficult, slow, and expensive to hire a good programmer. and bad programmers impose enormous and ongoing costs in management, maintenance, and slowing down the work of good ones. and there is no degree or certification that helps much -- plenty of technical STEM PhD's think that brilliance + laziness + no training makes them qualified programmers.

Oh but those STEM PhDs don't need to worry about becoming programmers - Tyler Cowen says there's massive demand for STEM PhDs in fact our current economic stagnation is caused by a lack of people going into STEM.

Two big reasons for the wide range in salaries.

First, there is a huge difference in the productivity of average and very good programmers. The classical view in the industry is that the difference is 10 to 1 between average and a very good coder. A couple years ago Alan Eustace, a Google VP of engineering, claimed the difference was 300 to 1. My guess after 20 years of software development is somewhere in between. Most programmers are quite mediocre. You can throw huge numbers of them into a project and not see the kind of progress that occurs with the addition of one really good coder.

Second, there are big differences in understanding of the product domain. A really good US based programmer is likely to have a far better feel for the business rules that he's trying to implement. For instance, suppose you're putting together a project that involves finance. You might find a programmer in India who has solid technical skills, but is clueless concerning accounting. A developer with a basic understanding of the product domain is going to be a lot more productive than one who has no understanding of the underlying product.

Having been in the industry before it was an industry part of understanding the programmer productivity problem has a lot to do with how programming projects are managed. It is the number one determination of overall quality regardless of whether salaries are priced properly.

I will +1 the fact that culture & communications are a big fat hairy deal. One of the big secrets of the industry is that there is almost no technical domain in which master of a programming language or discipline does not come within 2-3 years. What smart people pay for is experience with those skills under a wide variety of circumstances *close to an industry specialty*. If you want a database programmer who has some facility with SQL, they are a dime a dozen. But apply that with specific experience for telecoms billing systems (the huge databases that track your every phone call (now a bit less prevalent than when long distance was billed separately)) and you quickly narrow down the availability. Then narrow that down again to someone specifically with Teradata database skills and you really up the price. Experience counts. Specific experience pays.

Another large factor is the acceptability of the contractor. Which is rather like Chicago politics. It ain't what you know, it's who sent you. Indian programming body shops can and will produce a SQL Server programmer who on a visa, pay that guy 30/hr and bill his services for $55. That's still, relatively speaking, commodity pricing. The same skill level programmer provided by Deloitte can be billed at $100. I have been on contracts through prestigious consulting firms that bill $250/hr for work doing macros in Excel. You'll see the same dynamics in dark work where clearances are necessary. In short, a lot of programming is guild work.

There is also system inflexibility. Which is to say, you have a lot of smart programmers who would, as the snarky t-shirt says, replace your staff with a very small shell script. But companies, having outsourced and cost-cut have painted themselves into arcane corners, with goofy toolsets and architectures.

You should also always make distinctions between programmers and software engineers. A software engineer is somebody hacks concepts and architects a solution in code - generally as a system that never existed before. A programmer is somebody who hacks code and makes all of that work for the customer. Engineer's time is very expensive but rarely funded. Programmer's time is not so expensive but requires, as has been mentioned, lots of customer contact.

I have found that the best metaphors for programmers of all sorts are in the law and in motorsports.
In the law, programmers are like Congressmen, basically any lawyer can serve, but the chance to write code/legislation that will last in its original form and have a lasting impact is like the difference between Greta Van Susteren and Oliver Wendell Holmes. There is more code out there than law. But nobody is going to rewrite much of 'constitutional' Linux or BSD kernels. In more everyday situations, you know better than too fool around with legal 'operators', and that's what the best programmers are like.

Every city has thousands of certified mechanics, but only a fraction of them could build a car from scratch, and a smaller fraction still could build competitive race cars in any class. The best programmers are like NASCAR or Formula1 pit crews. They can build cars from scratch, fix them when they explode, jury rig them to operate under any condition and put them together *fast*.

Most people don't really know what it's like to have world class systems rather in the same way most people don't have advocates like Melvin Belli or cars like a Maclaren F1. My guess is that Shenzen doesn't have the equivalent of a Formula One race track in systems, but Hong Kong, banking center that it is, does.

Cord Blomquist rocks it.

There's a big difference between computer scientists and programmers, and big differences between cloud computing and programming plug-ins for Excel. Do the jobs in question require the same skills, or might the higher paid jobs be for computer scientists? More information is needed before any conclusion can be arrived at.

If you are a computer programmer and your job can be sent around the world at zero cost, then you are not a very good computer programmer.

Programmers do not lay code like hens lay eggs. They get asked by managers how much time would something take, they have to be able to understand a bit of the business logic so that they know which part of the software are critical and should be handled with more care, they have to be able to reproduce on their systems bugs found by the QA team, they have to be able to communicate with the managers if they find that something is unclear in the specifications and of course they have to be able to read them without missing the point....all of this is very difficult to assess in job interviews home, and is even more difficult when you're outsourcing. The guy above that said that some programmers have negative productivity is absolutely right.
Some companies might be able to get (small) economies from outsourcing, but there can be a risky move and why make these when you're already having a margin in the tens? You know what you're losing (good and productive programmers home) but you don't know what you'll get.
I would like to know what the Chinese job interview questions were. When I got interviewed for my first job in my country (France) I got really easy technical questions, but that doesn't mean we are worse than the Chinese, it could also be a cultural (either company- or country-related) difference that some interviewers prefer to assess technical or non-technical skills.

From my experience, the outsourcing of some work we had no time for in Germany was a disaster. The German programmer just decided not to implement something that was written in bold and red (!!) in the functional specifications. Do you have any legal recourse if the job is not correctly done? What if the Chinese programmer is the only one able to understand his code? What if the economies are out weighted by the QA, legal, communication, HR and maintenance extra work?

Oh, and by the way, are wages in Hong Kong that low? Given the rents, I'd hardly say so.

"Oh, and by the way, are wages in Hong Kong that low? Given the rents, I’d hardly say so."

A recruiter I spoke to said that $40,000 a month (about $60,000 USD per year) was exceptional. The salary surveys I found were a bit less bleak, but still suggested that this was on the high end. Unfortunately, I haven't gotten any actual job offers yet, but I'm told that things will pick up in the new year when budgets get renewed.

I gather that having a roommate is much more common there. Combine that with low taxes and lowish prices for everything other than real estate, and you can get by.

Are you talking about entry-level jobs? It's possible that there's little premium for someone with an American education, but who hasn't worked in American firms.

Maybe the salary is really like 80K and the recruiter is pocketing a shitload.

This is another reason why being a programmer sucks - most firms insist on hiring through recruiters and most recruiters are morons with business degrees who earn tonnes of money by just sending out emails.

Solution? be a recruiter?

Could it be that there's simply a glut of programmers available in Korea, China and Japan vis a vis the United States?

I'm surprised no one brought up the wide variation of tax policies. Also, being physically present on-site is worth much more than the questioner seems to be assuming.

A few guesses:
First my understanding of the business. Excellence in computer science seems to be largely driven by being good at understanding what the client wants (I'm using client to mean both one's boss or outsourcing client) and being bright/creative enough to solve the problem effectively and quickly. Finally similar to construction few clients are good at specing anything beyond a very general, nebulous concept.

Also very slight differences in ability can mean huge differences in productivity. My boss can look at many problems that take others minutes to solve, impliment, and test and pop a different solution that takes seconds (and will be easier to test and verify as well).

So even if the foreign coder is equally talented at development, unless they are also very good at reading people and understanding speakers of a foreign language and culture, they may implement a solution that isn't correct because they were unable to understand the design requirements (and weren't as able to understand the nuance of the nebulus specs).

Let me add a few thoughts.

(1) Wages are unimaginably sticky in the software field. I've seen multi-million dollar projects blow up because management's refusal to give an extremely talented individual a pay raise to match an outside offer--a pay raise which would have cost the company an extremely small fraction of the cost it paid when the project failed. Generally salaries are determined by comparing salary patterns across other companies rather than understanding the value provided by a worker, and worse: some companies (famously Apple and Google) have illegally collaborated by agreeing not to hire each other's workers, which make it impossible for developers to bounce back and forth in an effort to uncover their actual value.

I think this is related to:

(2) The software field has one of the widest extremes in worker productivity of nearly any industry I've ever heard of. An excellent software developer can be 10x more productive than a good developer, who can be 10x more productive than an entry-level developer. And there are developers who actually are net negative in their productivity. Yet in the Los Angeles area salary ranges from perhaps $70k/year for someone out of college to $200k/year for someone at a very high level; a spread of a factor of 3 rather than the factor of 100 in productivity.

(3) I believe on very serious problem in the software field is a lack of ability for management to successfully measure and manage productivity. It seems like every week a new management fad (Agile being the latest) comes down the pike to help clueless managers manage their software developers.

But the reality is that, from the point of view of most management, they may as well be managing a cadre of sorcerers and witches: programming is black magic to most management folks who are in charge of determining schedules, costs and compensation schedules: lines of code may as well be eyes of newt. Worse, there is a lack of understanding on the part of those managers as to how much a software package should cost (in terms of time or effort) to deliver--and because software developers themselves have very little way to determine the market equilibrium of their services (thanks to imperfect information on the part of management, active collusion, a willingness to pay more for a failed project than to allow a market equilibrium to establish itself), software developers sometimes will undercut each other not knowing the true value of their services.

So calling this a situation of "imperfect market information" really doesn't even begin to describe just how broken the software market is.

(4) And that's in addition to everything else mentioned above--such as China's poor track record on IP, Indian companies histories of cost overruns, and the need for management to attempt to get some sort of handle on the development process by at least having their cadre of magical sorcerers and witches in the same room.

Sadly I think this is only going to get worse, as the demand for software developers explode, while the supply stagnates: it will draw in a lot of people who have no clue what they're doing, and like during the dot-com bubble of the 1990's it will become increasingly harder for management to find the one or two people who honestly can perform the job from the pool of thousands who have no clue but who can write a good resume.

I largely agree, but the truth is even more dramatic. Software developer productivity varies over at least 3 orders of magnitude (more than 1000:1), measured in terms of finished lines of code per unit time (few agree on an appropriate measure - viz, Dijsktra's objections to the lines/day metric), from days per finished line of code in the worst case to several thousand finished lines per day in some cases.

This isn't speculation; I've personally studied it, and observed it even in my own career. Yes, Virginia, there are academic studies of programmers along several dimensions, including productivity. But I don't do that anymore; I'm just a professional developer (albeit with a Ph.D. in CS).

The worst case is relatively easy to observe, over a range of application classes, but so is the best case. I personally have produced more than 2000 lines of finished code on single days, and averaged 300 per day over several months, on system level projects (drivers and standalone OSes). Anecdotally, I was once asked by a manager, who was amazed at my productivity over a period of time, to keep track of my output on the next working day, and on that day, I wrote 900 lines of code. And I know other developers (I won't mention names) who I have seen produce 1000 lines or more of finished code per day on an extended basis. Linus Torvalds is one who comes to mind; if one counts all the admin work he does maintaining the Linux kernel, I'm sure it works out to a pretty high number.

The best developers can write high-quality code as readily as a writer of prose can produce high-quality prose. In fact, I strongly believe the two skills are related. Sadly, though, most who are paid to write software for a living, seem to struggle mightily to be able to do it at all, even at a relatively low level of quality. Writers of prose would not keep their jobs in such cases; but software development managers are largely clueless by comparison. A good prose editor knows a good writer from a bad one; not so with software managers, it seems.

Speaking of Linux, though, I've also studied the impacts on developer productivity. My own findings suggested that in many organizations, less than 5% of a developers' day may be spent writing code. The implication is that a heavily bureacratic organization can be the biggest impediment to developer productivity, and that there is good reason that open source projects - which have little bureacracy to speak of - can be so highly productive compared to similar proprietary efforts.

Writing software is largely intellectual activity; it's not like playing a sport, when variations in performance are limited by, e.g., the range of variation in the physiology of the human body. The human mind is far more varied in its capacities than is the body. But economic perspectives of software development seem intent on viewing it in terms of "body counts," assuming without any evidence whatsoever - as did at least one commenter here - that all developers produce roughly the same. How foolish, but par for the course for economics, which generally waves its hands at phenomenon it doesn't know how to observe, let alone analyze, let alone measure effectively.

In terms of value to my employer, my most productive day as a developer was sent deleting 3 lines of code.

No disagreement from me; I've also seen 3 orders of magnitude in productivity between someone who was an excellent programmer verses someone who is simply "okay". And it gets worse in many ways: when managers impose metrics, for example (as happened at one job where I worked), poor programmers will happily game the system, producing extra "do-nothing" code or engage in "refactoring" (which at that job became a really bad joke for piling on a new layer of garbage on top of existing garbage), in order to boast their statistics.

So it can be more than just three orders of magnitude in terms of any sort of reasonable absolute metric (which, let me note in all your numbers about hundreds or thousands of lines of code in a day, this is in an industry that averages 15 lines of working code per day), since many developers have negative productivity.

Imagine in the housing industry, when building a house, having half your builders wander around with sledgehammers randomly smashing and destroying bits of the house as it's being built, and you get some idea what's actually going on out there.

"The Mythical Man-Month" (http://en.wikipedia.org/wiki/The_Mythical_Man-Month) is a great book on programming productivity, arguing the best programmer is 10 times more productive than the average. Certainly his ideas don't fit well with offshoring.

I wonder whether maintenance programming or such tedious changes as the Y2K modification aren't more exportable than creating new applications. Maybe India can keep up all the legacy COBOL programs?

As a software designer I have worked with programmers in India, Alabama, and the DC metro area. We use the ones in Alabama for most work now. They cost more than the Indians but the communication that takes place with the designer is critical.

We do have Indian programmers in DC and that can work too. They speak English better than their counterparts based in Mumbai and being able to meet face-to-face helps a great deal as well.

Again, to repeat, the basic misunderstanding is that most programming is a primarily a complex technical exercise. In business programming, (which most programming jobs are dedicated to,) quality is almost totally predicated on gathering and understanding business requirements.

Generally, the technical implementation aspects of business programming are extremely unimportant -- any working implementation often suffices, regardless of its aesthetic ugliness or inefficiency.

To create software: #1 answer what software you need to create, #2 come up with the logical structure to support it, #3 is type it out... #1 is really the critical factor. Simply concentrating on the effectiveness and efficiency of step 2, is a similar error to overly focusing on step 3 -- asking why 160 wpm typists would not have the highest programming salaries.

Only years of experience teach you the importance of #1. A couple mistakes fundamentally understanding requirements can ultimately invalidate and waste years of work on a project. Hundreds of step #2 type technical errors can be corrected nearly as quickly as they can be detected.

Anyway, due to a combination of language, locality and culture, US based programmers excel in requirements. Technical superiority is arguable...

I posted because I feel this overemphasis on technicalities is a pervasive and harmful misconception of programming. I think it scares off the vast majority of people who could use or be used with programming exposure. There is a certain bar of mathematical/logical skills needed to be a useful programmer, but it is lower than one would guess. Creativity and social skills carry the day.

The technical perception is perpetuated by most programmers, because technical people are drawn to the profession, and they take joy in the technical competencies constantly learned and refined over a career. Nearly every modern office worker could benefit by being comfortable with some scripting and querying knowledge, but an image persists of programming as some dark, arcane art, painful and inscrutable.

I forgot to say I'm very skeptical about the quality of the Chinese or Indian engineers in general (well, the ones that don't go to the West). When was the last time you were impressed by a Chinese phone, car, plane, video game? The answer for me is never. If you think about Germany, Mercedes-Benz or Porsche comes to mind, if you think about Japan, the Shinkansen, but if you think about China (apart from inventions that date from thousands of years like gunpowder?). Ripping off the Shinkansen idea is good and it requires talent to reverse engineer stuff, but having invented the Shinkansen requires much more talent and creativity.

I had some Chinese people in my class at the University and they were *awful*, speaking no French, no English, and of course no Java :). I couldn't believe it (are Chinese students in the US the same?). Paying for diplomas is also very common there, which adds to the risk.

P.S. : just for those of you who don't know and will get angry for no good reason, LG is Korean and Sony is Japanese, not Chinese.
P.S. 2 : generalizing is bad of course, especially when it comes to people, but necessary. Prejudice against English cooking have saved a lot of lives.

This is a bad argument because people said the same thing about the Japanese and then the Koreans. ESPECIALLY the Koreans - LG Started off making detergent and they built up their auto industry by insisting that any imported car must be assembled in Korea.
Good English cooking line though.

apart from being a prejudicial comment, it's slightly dated as well. Some of the very best programmers that I've ever come across are from asia -- Japan and Korea in particular.

Yeah I think he was bashing China (and comparing Korea and Japan favorably to China) specifically not Asia in general. Still not a good argument because people were saying the same things about Japan in the 1950s and Korea in the 1980s.

It's absolutely a good argument because the post that started this is about programming salaries *today*. It could well be that Chinese carmakers of 2050 will lead the world, but that wasn't his point.

Nah...his argument seemed to be that the Chinese are fundamentally less creative and he compared them explicitly to the Japanese.

Well, while we are being rude let me speak...
It's not the individuals of course, but the culture. And culture is why Americans have succeeded in programming. So let me throw out some stereotypes that actually work as broad guidelines. When I speak of culture I am speaking of the ways of doing business and education in native countries. So when you work at a Japanese company, the corporate culture is very different than an American company. When you know how Germans are educated you know it makes a difference in their work ethic.

In Japan, collaboration and steadfastness are cultural hallmarks. Basically, when you work in Japanese terms. you get everybody together on an idea, you commit to the idea and get everyone's agreement. Failure in Japanese terms means losing that consensus over time, or deviating from the promised goal. This makes for horrible software, because you always learn something new during development. But you must stick to the plan. On the upside, Japanese can be very imaginative and take great risks on long-term projects because of that consensus and dedication, just pick the right project..

In America, you pick a goal, then find a team to work on that goal find a funder to pay for it. If the goal changes, you change the team. That means there are always negotiations going on in the middle of a project. That's good for software because when you learn something new, or need something new, you can reconfigure people, budgets, goals. On the downside it means a lot of stuff is never finished and sold too soon (to pay for all the change management). Creativity and flexibility are key. Americans jump around too much - which is why there's only one Amazon.com and Bezos' approach to his business is exceptional. (Somebody mentioned that jumping from Apple to Google is prohibited, but it begs the question of why do people have such short attention spans that they feel the need to jump). I think the American expectation that there is always derivative income in secondary markets (and in second place, marketshare-wise) has deleterious effects on quality control. But since you can make money selling Chryslers or spinning rims for Chryslers.. The best Americans all know that which is why they have so little patience with second-best, thus Microsoft hate and haters in general.

In Germany, you find a technical boss and everybody follows the boss. That can be excellent, if your boss is excellent. That's good for software because it's not easy to be a German boss, and it requires standards and collaboration. You get high quality and consistency throughout. What you don't necessarily get is something that is intuitive to use or solves a problem people didn't know they had. The German software industry is growing - they're starting to do incubators and all that VC stuff. Interesting times are ahead.

My experience with Russians are that they are the best hackers. They trust themselves, they are self-contained and they work with meager resources and really sweat the details. I've worked with a few and I would love for all of my code to be eyeballed by a Russian programmer who'd make it work in less memory. I like the fatalism built into Russian engineering, makes stuff robust.

I also agree with what has been said about the Chinese. IP is meaningless. Chinese find monied interests and figure out what they're going to do. As long as the venture works out to the satisfaction of the owners, the rest are details for the workers to figure out. In China, you are your skill and there is no prestige to be had for anything outside of competence, so find friends and do as much business as you can. Response to markets? What's that? You work based upon the principles of your principals, period. It's an extremely competitive approach, but what is creativity?

Indians are something like Russians in that they take pride in delivering at a minimum cost, they are unafraid to take scut work. The best Indian programmers tend to be insufferable and that is because they are actually very good and are often expected to work with other Indians who are simply competent, or worse. And you know this if you've worked with large numbers of Indian immigrants here in the states. Class systems are quickly established amongst them, and failure is berated. So in that regard they are like Germans, but much more friendly in general. But as has been said, neither German nor Russians nor Indians put a high value on creativity and flexibility. The communications and class barriers for Indians can be formidable, but a lot of difficulties I've observed have to do with the social dynamic of how Indians are employed. It's complicated.

And to finish insulting everyone, I have never met a great French, Italian or Brit programmer. Great French sales guys, god you do not want to have to negotiate with them...

Though not in the business software realm, a common theme I've heard about the Japanese gaming industry is that they have a major tendency to reinvent the wheel on a regular basis (making new engines from scratch for every game) and have only within the last few years started becoming comfortable with using the assorted middleware and asset libraries that western developers have long taken as a given.

If I could hire any software technologists, I would avoid hiring from the South Asia region... (namely India)

As an Indian working in the US, I want to +1 the Indian parts in this.

As someone fighting against US' indiscriminate issuance of visas (particularly to Indians) I want to -1 the Indian parts in this.

Thanks for the plug!

I believe that situational and business knowledge accounts for much of the pricing differences.

The extreme example is quantitative finance.

A less extreme example would be programming a margin system. The programmer needs to understand how a margin system works.

I'm guessing the software used by Walmart to do their inventory management is much the same.

I think programming is vocational. I believe it is vocational because that is how programmers learn the business rules they need to program. I think often programmers are the institutional knowledge.

BTW I also think this is how programmers extract rents from their employers.

Co-owner of a small software house here.

Programming is a very social issue. The people working on a common project need to communicate, and they usually need to communicate often. By far the best way of communication is personal in this case.

If you put several timezones between members of the same development team, they will not develop in the same hours, and coordination gets worse quickly, even if they belong to the same culture, speak the same language and can hold teleconferences every day.

Add the barriers of culture, language and literally the other side of the world, and you'll get a piece of software which is completely unusable, far from your original intent, but hey, for half the price. Throwing the money out of a window outright would probably be more effective.

Also, if you develop software for a particular customer, the programming team will need to be updated about the customer comments, bug reports and demands - very often. No one out there really knows what he wants from the software, but will swiftly complain if a release does something unexpected or unwelcome or clumsy.

The ability of some senior programmers to translate customer-speak into technically sound descriptions for their colleagues is an important asset in the corporation. For that, the seniors must know the customer well and be in frequent contact with them. These are the best-paid programmers, and you can't outsource them.

Software as a finished product is extremely tradable. Software development is tradable only with extreme difficulty. They are actually the inverse of each other.

Good point. What is tradeable turns out to be a non-obvious empirical question. Answering customer support lines, for example, appears to be more tradeable than programing.

Also, a lot of the really outsourceable jobs that can be done by telecommunications (e.g. search engine evaluation, i.e. rating web pages) has a cultural component that you lose if you take jobs off shore. Increasingly, jobs are done by humans because only humans can do them because they take personal interaction and cultural capital, not just technical skill.

Think organic plus positive externalities of human capital. Take Silicon Valley. it's a whole eco-system that raises the productivity of everyone who works there. And I don't think it's a coincidence that it's on the west coast.

One of the odd things about the Indian economy is that young people often believe that there is only one route to success. In the past that was becoming a doctor, engineer or civil servant. In the last 10-15 years its become working in outsourcing. A lot of people end up being computer programmers without having much aptitude or interest in the job. I second the comment that the most intelligent get themselves to Europe/ America and (would add) then leave programming for say consulting (how many middle grade management consultants are ex-Indian software engineers?) The ones remaining in programming are probably not naturally very good. Its telling that the Steve Jobs type tinkering culture just does not exist in India as far as I can see.

Well this is the story of the whole world. There really is only a few very narrow routes to success anywhere.

I wouldn't say so. In my proximity, I have seen only a few people studying something that they hated in name of "success". Yet most of my peers are solid middle class at ~35 years of age, regardless of being engineers, economists, scientists etc.

But I rarely befriended slackers, so I can't speak for them and their success stories.

Very good and thoughtful responses.

The only thing I would add is resource complementarity: that if you think of software development as part of a vertical chain in the creation, manufacture and distribution of a product, you would expect it to be closer (in time, in space, in collaboration) with both the other components of the development and manufacturing process and the sales, marketing and after installation component of the business. If it has to be close to those elements of the business, it will colocate in the center of the business.

You might be interested to know that there are some folks who analyze intra-firm communication within a social network analysis model--with nodes, branches, etc.--and I would bet if you took a corporate structure diagram (or even e-mail network communication data) using social network analysis you would find the high end design archetecture and software modification programmers closer to the core of the business--that is, closer to management--and communicating with them more intently.

Finally, it is my observation also that senior programmers move into management, and they maintain, and are protective of, those that they grew up with in their network and maintain close communication with them, making it more difficult to outsource to the unknown.

Programming does require colocation for maximum productivity. Anyone who tells you otherwise is selling something. Therefore, it's really a question of why software businesses are located places where engineers are expensive.

The most valuable programmers (as opposed to some objectively-measured "best") are those with strong shared cultural and domain assumptions with the people they are creating software for. It is simply true that Americans create software that is more valuable to Americans and Japanese programmers create software that is more valuable to Japanese folks. The hard part of programming is not the math or the theory or the critical thinking skills: it is the social aspects of creating something with a group of people that fulfills the requirements of another group of people. Shared cultural assumptions do a lot to grease that process, and every time I have seen variations from this norm I have also seen significant loss in productivity and value created.

That said, wages do normalize relatively quickly. In India, for example, systems administrators are now being paid equivalent to their American counterparts, despite starting out an order of magnitude cheaper. Poland is right now cheap, but is rapidly rising as companies notice it is cheap and open offices there. The places where wages don't normalize are those where companies don't choose to move colocated centers to. China, for example, is a terrible place to do business, especially for tech companies; the Great Firewall has effectively isolated it from the rest of the world. Countries with insufficient infrastructure or government corruption also tend to be ignored. Additionally, coders like living places with high standards of living: NYC and the Silicon Valley all have high wages and lots of companies despite relatively high taxes and high costs of living because companies know that if they need more engineers tomorrow they can probably get them (for a price). To some extent software engineers have moved to where they want to live and companies that want to hire them follow.

Finally, the differences in pay sometimes obscure just how expensive some tech areas are. A coder in NYC will make 3x what a coder in South Carolina makes, but it's possible to buy a house in South Caroline for 8 months of NYC rent. You can hire coders in the US for what you can hire coders in Hong Kong, you just have to find those who are willing to live in places like South Carolina and they are likely to be less skilled than those in highly-networked hubs.

System administrator is a very different position from a programmer. That's like comparing a car mechanic to a car designer.

Also, system administrator is, in the ideal conditions, almost invisible to the user. If the servers work and do what they should, no one needs communication with the admin, save for rare occassions when a specific user right must be granted etc.

That is because the system administrator works on a collection of rather robust, conservatively chosen system utilities and programs. No one in their sane mind would put a fresh beta release of anything on their production server, unless they had no choice (for example, a critical bug fixed).

On the other hand, the programmer works on the "raw meat" of the future product, which means that communication regarding bugs permeats his workday.

"I wonder whether maintenance programming or such tedious changes as the Y2K modification aren’t more exportable than creating new applications. Maybe India can keep up all the legacy COBOL programs?"

You mean the critical COBOL programs that power the back office of most US banks and markets?

Perhaps its less to do with the programmers and more to do with the salary structures within western companies? Organisational hierarchy, salary level compressability and the local HR dept can have far more influence than generally realized. If you are classified at a certain grade, then that grade reflects your comp. If your grade has to fit in with many other JD's, then the pay isn't going to change much irrespective of your actual experience/skills. Note that as a programmer at grade x, you would probably be on par with Sales/Ops/HR manager grade x. Which in larger companies means that you probably come about level eight down from the President, and if he/she takes a pay cut then everyone suffers. The big companies end up leading the market as they can usually attract the best (either for pay, or conditions. Not necessarily so for contractors, however that usually comes out of the project budgets, not the operational ones. Not sure how that all plays out in the halls of academia?

I had some experience with mainland China trained programmers, and essentially, I have to use 0.5 of a supervisor for each mainland trained programmer while I use only 0.1 supervisor for each EU, US or Canadian trained ones. And even with this close supervision, I still don't have the productivities (rewrites are expensive), the promptness (rewrite takes time), and quality (you can't catch every trouble causing shortcuts these guys like so much). And it takes three times the time to convince them to follow good programming practices, and their documentation sucks.

The same logic can be applied to sales persons, doctors, lawyers etc. It's not a programmer thing.

Doctors and Lawyers benefit from huge legal and regulatory protections that programmers don't

That's not fair! Programmers should get the same protection!

What can I tell you? How about you take it up with consumer affairs

If dealing with things like medical records or legal documents, programmers can benefit from being bondable and/or American citizens.

There's already a significant body of programming jobs that need security clearances only accessible to citizens.

fungible labour, simple.

A lot of commentators have mentioned issues of timezone and culture. Having tried to work with programmers in India, I can understand that; but why are there no low-cost software houses in Latin America? Both the timezones and the culture are a lot closer.

I've heard stories of devs who already work remote in the US moving to Costa Rica and keeping their salaries intact. But never of large shops in LA.

I know someone who tried to work remote from Costa Rica and had to give up after a year or two because networking and personal client contact was too important to the success of the venture.

One data point. I work for a Hong Kong company, that has recently opened an office in NYC. The owner (living in HK) figured he would rather pay $$$ in US, not just in salary costs but also benefits, taxes, reg compliance, etc, but get much higher quality quants and developers. Those were his words.

I worked on a project for one of the largest Japanese electronics companies that was being done here in one of their Silicon Valley operations. It was a small office with just a few programmers working on CAD tools for chip design. This project was an equivalence checker -- a very advanced tool for determining whether a behavioral description of a chip is equivalent to its structural description. I asked the guy in charge why this work wasn't being done back in Japan, and he said it was people. It's easier to hire people who can do this kind of work here than in Japan. By the way, of the two hot-shot programmers that created the tool, I think one was an immigrant from India. (I didn't ask where he came from.)

I'm aware of a large Boston software firm that tried outsourcing and found that the cost savings did not match the negative impact it had on team functionality in most cases. The IP theft concerns with China (as well as a general climate of non-free internet communications) make that a particularly undesirable location. Also, I don't know of any firm that opens a foreign location without an expectation of at least a once of year visit from the home office, which is not easy to arrange in China, and is harder to arrange in others (I have a relative who has made some of those visits from the home office due to language abilities).

Another complication is that some of the juicest and most lucrative custom software programming work has national security elements that require security clearances and procedures that are much harder to comply with for the whole company if you have an offshore office. For example, a specialty in IT security is almost useless in the U.S. if you are a foreign national who didn't somehow manage to already get a security clearance.

A third problem is scale. A lot of programming shops are very small. Offshoring is a major one time set up cost that is about the same regardless of scale, and is also a not insignificant continuing cost (e.g. tax and money laundering and foreign corrupt practice act compliance) from a legal and compliance perspective. It is often cheaper to higher a lawyer to get a visa for the guy you want (which the prospective employee may prefer as well) and to pay a bit more, than it is to open offshore operations, unless you are a very big operation.

I have been living in Shenzhen for the last year and a half and can explain some key first hand differences.

Prices in Shenzhen for everything that is not western goods are approximately 5 times less than they are in the US. This includes rent, foot, utilities, variable payments, and taxes (assuming your employer knows how to pay you under the table).

If you made 20k usd per year in Shenzhen you would live as well as you would in the US making at least 80k. Further, Chinese workers have a much easier work day than most Americans, there is a nap time from 12-2pm and Chinese firms rarly work after 6.

The working condition's are very very different, and the price levels are a lot lower.

I remember some company screwing up a big project in Sweden by bringing in Indians.

There was a culture clash because the Indians didn't do anything without being instructed. I think productivity is quite low in under-developed countries because incitements disfavored people with initiative.

Not the most politically correct thing to say, I know.

There is a high coordination cost with programming. There needs to be good communication within the team members who are programming. Slightly less but still extremely good communication with the people defining the products requirements(in the sense of completely understanding) . Simple misunderstandings dramatically increase rework. This increases the cost and adds delays where time to market is often very important. To understand the requirements often takes a deep understanding of the market and as such needs individuals mentally in sync with their customers. A maximally efficient team works in close proximity. This leads us to wanting the product managers near the customers and also wanting the programmers near them. Timezone changes between team members further add coordination costs. Cultural differences between team members also lead to misunderstandings increasing costs. This last effect can be partially mitigated by physical indicators (facial expressions) that do not translate as well to remote sites.
For context I am a programmer who has worked with team members all in the same building and also worked on multi-site projects with team members in 3 countries.

One of the things that many people don't know about is the level of corruption in the outsourcing process. My evidence is anecdotal, obviously, but someone needs to do some investigative reporting on the subject. My understanding is that the contractors who supplies the H1Bs to the large corporations give some sort of kickbacks to the corporate decision makers who hire them. In one very egregious case, a Project Manager, himself an immigrant, who hired the H1Bs has the contractor deposit $X per hour per hire in a bank account in the PM's home country.

In order to get the offshore programmers to be successful you have to specify the requirements and design so tightly that the remaining code development is trivial. Also, there are some great people in India but too often you just get someone from the pool with a bunch of meaningless certificates (hiring a programmer based on certificates is like hiring a pianist on the same basis). Time zone differences are a bigger issue than many people realize -- it can kill the decision loop. And finally, intimacy with the business side (the customer) is invaluable. No offshore developer can overcome the advantage an onshore developer has in that regard because the onshore developer can go to lunch or out for beers with the customer.

President of a small medical software company here. Clearly none of you are involved in software that is under regulatory compliance. Finding software engineers who can do cutting edge image analysis stuff as well as comply with all the arcanities of for example of ISO13845 are *extremely* hard to find.

Those who possess the combination of creativity and smartness for designing good softwaree and also possess the conscientiousness required for complying with ISO and CFR are truly rare and valuable.

There is no "law of price" - only a model approximation of either reality or a more accurate model of reality.

The more accurate model is that all prices are set by specific negotiation and thus that all economic transactions are simply types of negotiations between two or more parties. Once you understand this, the price disparity of the "law" is trivially obvious. This is in reality what/how bourses do as well only in a more automated, systematic fashion.

As any negotiation starts to involve valuation (which is all that a price is) of things that are higher risk to either party or more nebulous and abstract in value, the dynamics of negotiation changes. The first thing that occurs is the variance of price in similar transactions expands. This is primarily what causes this 5:1 price difference. The tasks are not commodified (despite specifications - when compared to actual commodities like wheat or sugar or electronic gadgets or books) in programming so the valuation is harder to define "correctly" (more accurately "uniformly" or "repeatedly" for comparables).

What also changes is the "abruptness" of negotiation power and its perception by either side. Ultra-low wages occur with a power asymmetry which often is due to information asymmetry or resource asymmetry (both of which are typical of countries that do outsourcing - places like India a knowledge and training asymmetry that is being equalized but the equalization has a longer time constant than the speed of outsourcing adoption).

The reality has been for many years now that if you want quality akin US programmers for a leading edge project, you will have to pay $0.90 on the dollar for an Indian programmer. Why? Because they need knowledge and experience comparable to the best US programmers and they know it because that's how they got the knowledge and experience (being connected well enough to have no information asymmetry).

"Why are some programmers paid more than others?"

Why does George Soros pay CBBB per comment to be his Muppet?

This is one of the more distasteful discussions I have seen on MR. Several comments reek of thinly veiled racism. Tyler Cowen, I would be very embarrassed to be a hosting such a discussion. It speaks to a tolerance for bigotry and also says something about the type of people who read your blog. I'm disappointed.

My personal experience with Indian programmers has generally been that they are intelligent and decent people (one or two notable exceptions). But when they are based offshore or have not had time to make the cultural adjustments to the US they operate at severe disadvantages that show up in the product. I can't see how being objective about those issues is racist. It just is. Many Indian programmers will say as much if you get a beer or two into them.

I wasn't one of the people of commenting on India etc., but I must reproach YOU. You're playing the race card in bad faith. Doing things like that spoils the public debate space for everyone.

The people here mostly complained about the hurdles of inter-cultural communication, not about the Indians being brown.

Have YOU ever tried to do business with an entirely different Asian culture? (Indians, Chinese, Japanese).

If you ever try to, you will find that the English words you'll use seem to have different meanings over there. The entire way of thinking is different.

This is already visible if Czechs talk to Americans - the mentality is quite different over here, even if both sides are lily-white blue-eyed blonds.

Racism on Marginal Revolution?!

I'm shocked SHOCKED to see gambling going on in this Casino!

I haven't seen every comment, but tried to read them all, and I don't *think* this one came up.

The vacuum cleaner effect. Why are so many very highly paid people working in the valley, seattle, etc? Because large companies with huge appetites for those people have paid high salaries to attract people - from all over the world. Are there real genius programmers growing up in India? Probably. Where will they end up? Seattle or CA if at all possible.

It's like asking "why do all the high paid movie actors hang out around LA?" or "why are so many of the highest paid investment bankers found in NY or London?" - they followed the money...


Here is Peter Diamond's Nobel Prize presentation. In it he explores a model of search costs which proves the law of one price to be nil. He poses the simple extrapolation from the model: if it costs more to find a lower price than the price itself, agents will simply pay the prevailing (local) cost. You could apply the same logic to computer programming (plus the Balassa-Samuelson effect). SHAZAM!

Why are wages different for programmers?
Quite simple...The mind set or thinking process(es) is/are different due to cultural differences.
Not a racial issue, just differences in the way people think.....no one is inferior or superior either.

All of the above are good points, but i'll add another perspective.

My first experience with Indian programmers was being given a project, 2 months, 5 developers (3 of them Indians on temp visas), and a lab that had been used as a storage closet for the last 4 years for office space. Within an hour 3 of us had started ripping the space apart getting it ready to use, while 3 others stood to the side having an emotional breakdown because they didn't think we had the proper authorization to get started and would be fired. That was pretty representative of the next 2 months.

Technical skills were okay, but the ability to deliver in an unstructured and self-directed environment were limited.

Wow, that sounds aweful!!!

I've seen terrible native-US developers and plenty of incredible Asian developers. I intended to speak more broadly to the cultural and educational differences that might account for the variation. I failed.

Years of living under micromanaging oppressive socialism will do that to you. In much of india, moving equipment from one side of a factory floor to the other without government approval was illegal.

Were Mark Zuckerberg, Larry Page, Sergey Brin, and Bill Gates the best programmers in the world?

Of course not. Their advantage was that they grew up in a free country alive with creativity.

I haven't read all of the comments, and this doesn't fully answer the question, but...

I think a lot of large companies that 1) Have high internal transaction costs and 2) Are large enough to fund multiple projects simultaneously like to outsource an entire team to work on a project by themselves. India is a typical example. You can pay programmers in India as little as 30% of what you pay a programmer here. "Theoretically," says the engineering manager/director "I can get about the same bang for about 50% of the buck; why *not* do it?" Problem is that the lack of co-location, and all the other stuff mentioned in the comments above, drags team performance down (and of course drives up those already-high management/transactions costs within the business), and the project(s) tend(s) to end(s) up sloppily-designed, poorly-executed, and very late. But, this narrow-minded approach to cost-saving is widespread, and thus might account for some sporadic development/testing/requirements engineering salaries across seas.

If only we didn't kick out all of the out-of-country CS students who come here for their education, we'd have a much higher demand for computer science jobs in the US and, comparatively, wage differences between the US and foreign countries would probably be reduced somewhat. But until/if then, weeeeeeeeeee.

My dog has just farted and I have to vacate the code lab until the gas clears. I will send you my thoughts after the dog fart mire clears.

Much has been made recently of the value of innovation based on close proximity and shared experience with other liked minded developers and innovators. So, Silicon Valley and New York/Boston gain much greater network effect from the dispersion of good ideas leading to successful software innovation. It really does seem that there is an issue of critical mass for things like facebook, foursquare, etc to be thought of and successfully developed.

Cheap programmers are very expensive.

I am an american programmer who will try to answer this question.
Very bad programmers don't just have low MVP they often have /negative/ MVP. Its also almost impossible to tell good programmers from bad ones before a project is completed. And management lacks the skills to tell a good programmer from a bad one.

This means that if you are statically likely to be a good programmer, you get paid a lot more.

More importantly, once someone finds a good foreign programmer also, they are likely to become an american one, because bringing them here increases their MVP.

"Its also almost impossible to tell good programmers from bad ones before a project is completed."

This is a task for the senior programmer reviewing the development.

1. See who produces the most bugs.
2. Make sure that it isn't a side-effect of the fact that that person does the most work.
3. Review origin of some of the mentioned bugs, and possibly also some other randomly chosen code of that person.
4. If it exhibits serious code smells or sloppy thinking, reassign that person to some other work like running tests, and do not let him near the code again.

This is time-expensive, though, and does not scale well.
4. If

Review origin of bugs...that's hard to do as a programmer, let alone by a pointy-haired boss.

PHBs are not allowed anywhere near code in my corp :-)

An older programmer is usually good for things like this. Guys who started on the big mainframes 30 years ago etc.

Tyler... why aren't all artists paid the same wages?
The same answer applies here.

It is not just a question of cost... That coding done in South Asia is cheaper... But is it any good? What about social obligations on the part of American companies? If they continue to farm work elsewhere, domestic workers that they denied employment wont have money to consume the goods these companies are selling... You only get what you give... Software is lifeless. put people first. Employ native American workers not people who come from some place on the other side of the world.

I have been told by project managers (OK, just one, but it was a conversation that recurred many times) in computer programming that they are prepared to pay far less to Chinese workers of equivalent technical ability because it is highly unlikely that they will create additional value beyond their contributions to carefully delineated current projects. They don't come up with unique or creative solutions to problems that came up on other projects and don't tend to look for ways to use their skills to push new projects, and are not willing to challenge the status quo (i.e. challenge the boss when necessary) to make a project work better.

All of these things made this project manager hesitate to continue to work with Chinese programmers even though they were so much cheaper because so much effort was needed to make use of their highly proficient technical problem solving skills. Of course, the price did make the different, and he did continue to do so.

I'm only loosely aware of the general nature of problem solving and project operations in computer programming, but between technical skills and overall contributions to project and company viability (and growth), I fully expect to see enormous gaps in pay.

Oh yeah, and cost of living and wage expectations are both lower in China, so they'll work for less ... but that one's obvious.

With apologies to all, I've found these comments so fascinating that I have lifted a number of them for a post on my own blog. Hopefully this is not a gross violation of some sort of netiquette. You can see my post here: http://www.tomnoir.com/2012/01/dark-art-of-computer-progrmaming.html

Joel Spolsky has written about this multiple times for eg go and read http://www.joelonsoftware.com/articles/Craftsmanship.html. Software development is a craft.

Risk of change
if you have a business, what is the risk of going from known american coders at X $/line of code to new, untested asian coders at X-y$/.line of code
there is RISK here; one batch of bad code can ruin your whole day

A great programmer can be more than 10 times as productive as a mediocre one. Some programmers will never be able to accomplish what others can no matter how much time you give them, because programming is primarily a creative endeavor. That's the reason for the range of compensation.

Programming is not a creative endeavor. It is processing rules in the most efficient logical manner to get to an end result.

However you want to describe it, some can do it much faster than others.

American programmers are paid more because they have to turn around and pay for organizations like the EPA, OSHA, social security, national defense, foreign aid, class action law suits, free eduction, unemployment, etc. - i.e. all of the overhead that is required to be a "first world" country. Once these organizations take hold in third world countries like India and China, then programmers in those countries will start demanding the same wages that Americans do. When that finally happens, then we'll be able to find out if one country's programmers are actually more valuable than another country's programmers.

You can't justify the cost of a programmer based on the costs of society. You can, supposebly, justify the cost on how long it takes to put a round peg into a round hole versus how long it takes to put a square peg in a round hole.

Because capitalism is terrible at allocating resources, particularly labour.

*runs away*


The problems in working with offshore resources is comparable to a situation that most all of you have dealt with on a daily basis. Location - fast food restaurant drive thru. My kids (business user) tell me what they want. I (IT analyst) consolidate the orders and prepare to place the order to the drive thu worker (offshore). I can't understand the order taker, who probably can't understnand me, I have to repeat the order several times. Still got the wrong order. Try to hold a discussion at the window explaining what was wrong with the order. Waited 5 more minutes to get the order. Kids are pissed because they didn't get what they ordered. This inefficieny may cost the restaurant missed current and future sales, and the frustration level of the customer is very high. That's what we go through in IT.

How surprising to see narcissistic elitist libertarian programmers with the notion that Americans are Exceptional.

How much of this is due to differences in cost of living? Salaries for computer programmers can vary considerably even in the U.S. I'll assume that even after adjusting for CoL there are significant differences. How much of that is due simply to lack of opportunity? These countries may be producing a glut of "qualified" (so far as that can be assessed) computer programmers who are angling for a small (relative to the supply of labor) number of computer programming jobs. In a completely fluid world they'd just immigrate somewhere they could be paid more (e.g. the US) but political realities (and language barriers) make that difficult for many.

Many people underestimate time zones and communication issues (not time related). Programming requires a high bandwidth of communication, and that's not location independent.

While I agree with most of what's being said making sense, there are a few other angles/connections/considerations that haven't been mentioned. For conversation's sake:

1)Culture of Innovation/Competition: Good bands, for example, tend to come from creative hubs where there are a lot of bands, and where having a good band is valued highly. Thus, it could be said that the Silicon Valley is kinda like the Hollywood of Movies or the Scotland of Bagpipe playing. There's a cultural synergy going on in these places where a greater value is placed on programming, and it brings out the best.

2)Given that software companies could theoretically be based anywhere, it's not suprising that many are based in the state that many people dream of living in. The "California" brand is pretty intense globally, and I'd imagine that Hong Kong is probably similar in East Asia.

3)Culture: California is still an immigrant state. Culture-wise, it's the most flexible and entreprenurial state, and individuality, image (what's sexier than technology?), and creativity are of the highest value. If my Cali friends have an idea, they think "I'm gonna start a business and see where it goes," often without major concern as to whether or not it will make them rich. Someone from the east coast, however, is more likely to think in terms of climbing an established ladder in order to make a set amount. People in the deep South, people who live in poverty, and Asian philosophies in general, tend to be a little more fatalistic. (The heartland tends to have a more progressive pro-active outlook, but has a very different set of values.)

4)Americans, Japanese, and even people from Hong Kong are, on average, much wealthier. This means today's programmers from these countries grew up using computers, and maybe even did some coding in high school. Considering that PCs were only really common in American middle-class homes post-1995, (when a thirty year old programmer today was already a teen) this is a significant advantage in experience.

5)Demand. Who needs major software projects? Large information-based companies. Computer companies. Startup companies. Obviously these are going to be based in areas with growing economies (Cali boomed in the 90s), pre-existing tech companies, high technological norms, and large centers of population. Hence, NY/MA, Japan (which has perhaps the highest level of technological integration into daily life), and Cali. (Another note: these cultures value and even fetishize technology, much more than say rural Oklahoma or Mexico. Although this involves even more factors, someone in California is more likely to make a decision based on a website, whereas your average business in Oklahoma relies more on word of mouth and repeat business.)

6)Telecommuting does happen, but I know some companies prefer to do it small-scale. Two friends of mine work with small international teams (like 5 people or less) on specific programming projects for large corporations. Obviously, for something like creating a major video game or large program, this would be problematic and in-house would be much better, but for programming a toy or tackling a specific problem, it works. It should probably also be noted that these were experienced professionals, closer to retirement age, and I assume their teams were likewise proven workers. Which leads to...

7)Known commodities. The comments have talked all AROUND this idea, but as one person mentioned "in the old days 50% employee success rate was considered par" but I think all companies are a lot more cognizant of the cost of training a new employee in this day and age. Thus, if an MIT or Stanford grad is a 75% success, he'll get picked over the 30% Uni of Iowa guy every time. So perhaps, it isn't that Chinese programmers are worse (although I think they are for many of the reasons stated), it's that when it comes to software, companies want reliability, so they're gonna go with a known. So software programmers in Hong Kong and Japan and California are paid more because they've been successful in the past. (This also ties into the whole convo on soft skills, as people will continue to look to the US and Japan for video games, because they've delivered in the past. A lone programmer in Mexico is unlikely to be able to form the team necessary, or make the necessary connections to financing and once finishing the product, the market, to make the next Call of Duty.)

I'm a software developer from Argentina. I've found that, while some Indian developers were highly skilled, the great majority of them simply were not, and ended up making spaghetti code, near impossible to maintain. Why not outsource to Argentina? I've seen lots of companies started outsourcing here, and the trend continues. Take, for instance, MuleSoft. And the time and cultural differences are very little in fact.

You've got it all wrong. Software is not the ultimate tradeable good at all. At least not across language barriers. Software that nobody understands well enough to maintain is worthless. It's worse than worthless, it's a burden. You need a continuing relationship with the people who write your software and you need to be able to communicate effectively with them. (If my theory is right then the countries with the most fluent, English speaking programmers will have the highest prices)

Comments for this post are closed