What makes geeks tick? A study of Stack Overflow careers

That is the job market paper from Lei Xu, at McGill University, co-authored with Nian and Cabral at NYU’s Stern School.  Here is the abstract:

Many online platforms such as Yahoo! Answers and GitHub rely on users to voluntarily provide content. What motivates users to contribute content for free however is not well understood. In this paper, we use a revealed preference approach to show that career concerns play an important role in user contributions to Stack Overflow, the largest online Q&A community. We investigate how activities that can enhance a user’s reputation vary before and after the user finds a new job. We contrast this with activities that do not help in enhancing a user’s reputation. After finding a new job, users contribute 25% less in reputation-generating activity on Stack Overflow. By contrast, they reduce their non-reputation-generating activity by only 8% after finding a new job. These findings suggest that users contribute to Stack Overflow in part because they perceive this as a way to improve future employment prospects. We provide direct evidence against alternative explanations such as integer constraints, skills mismatch, and dynamic selection effects. The results also suggest that, beyond altruism, career concerns play an important role in explaining voluntary contributions on Stack Overflow.

I think of this as yet another paper on how reputation can (sometimes, not always) enable the private production of public goods.

Comments

"I think of this as yet another paper on how reputation can (sometimes, not always) enable the private production of public goods."

Is that not the entire blogosphere?

But are their Stack Overflow contributions helping them get noticed and hired? Maybe in some specialty niches. But not in most companies..

I think making excellent contributions to open source code projects that big corps have people contributing to might draw more attention of the right sorts of people in those corps.

I agree, contributing to open-source projects is probably a much better career move. I'm pretty sure if you can manage to get a few thousand lines of code with your name in the trunk of the Linux kernel, you're guaranteed a job for the rest of your life.

I took the Coursera/Stanford "start-up" MOOC. I was told by the profs, serial founders, that a good Github reputation was the new standard in the Valley.

I did not test this, taking the course as a hobby, but it made sense for that group. You get to see someone's work.

I've heard some people view answering questions in that sort of forum as interview practice. I'm not sure how common this is.

At least for me, the answer was "yes". My StackOverflow usage totally followed the pattern suggested by this paper to a "T": I didn't like my current job; I posted a lot on StackOverflow to raise my reputation; I got contacted by a head hunter who said he found me through StackOverflow; I eventually got an offer from the company he worked for; after I got my new job, my StackOverflow output dropped dramatically.

I'm with you, we don't hire anyone who has time to contribute, only enough time to copy/paste code from there. Plus you had to be there in the early days to be able to answer questions; it would take even longer to pretend to ask questions you have to solve right away anyway if there is no existing answer just to build up enough to answer or comment.

Of course there are other reasons why someone might contribute less to an online forum after getting a new job, such as increased commitment and motivation, as well as more fear of wasting time at work when just getting started.

The paper tries to control for this by comparing reputation-generating to non-reputation-generating activity. However, I don't think this really works. The problem is that the non-reputation-generating activity (editing and voting) requires significantly less time commitment than writing questions and answers. It is quite likely that if someone reduces their level of involvement on the site, they will spend a higher proportion of their time on non-reputation-generating activities--particularly if the length of each visit to the site reduces.

Bingo. There are new time constraints. High quality comments take time to craft, and sometimes require a bit of research.

I know a fair number of people who have gone through a period where they were always on Stack Overflow, it was because they were procrastinating grad students. As soon as they finished it pretty much stopped.

It's definitely a valid point. I call it "Integer Constraints (IC)" in my paper and provide two tests on this: 1. if the DD result is driven by IC, then analysis using weekend activity should give a zero estimate. However, separate analysis using weekday and weekend data give similar estimates. 2. Assuming jobs with the same job titles have similar work schedule, for workers who switch to new jobs with the same job title as the old one, DD analysis gives very similar results.

I don't know how valid that model is for Stack Overflow, but it's the way that most sports analytics careers started in baseball and it's still important in pro basketball: create a website or blog, or contribute to an existing one, with post after post of original research. Get noticed and get hired. There aren't any academic programs in sports analytics and until the rise of the Sloan Sports Analytics Conference and similar meetings there was no infra-structure, no job market, for finding such jobs so you had to prove your abilities by displaying your work publicly.

I noticed (some time since I used Stack Overflow) that authors use SO to prove their expertise. Basically "I easily answer this, buy my book."

Many in academia also engage in contributing to public goods to signal their ability: referee reports.

This willingness to share information is something that I also observe in Chinese culture. The well-known example is Uri Triesman observing that the Chinese students in his calculus classes at Berkeley studied in groups whereas the non-Chinese students tended to study alone. I've also seen it in a professional association where, faced with a challenging question or project, someone of Chinese background will ask others on the list "how do I do this?" -- and people will answer, whereas in a non-Chinese group the attitude would be more likely to be "why are you asking us? Do your own work."

There's much more willingness to share information and to help others in the group. In some contexts some of this information-sharing could even be considered cheating, whereas among the Chinese it's simply sharing and helping.

So there can be cultural factors, in addition to the reputation-enhancing ones, the job market and book-advertising ones, etc. that have been mentioned.

The sudden flood of job market paper reviews suggests that Tyler is exercising a new tool, one that searches particular corners of the web (econ department sites) for these papers, then evaluates them by some proprietary metric of interest. Presumably, he still culls through the crop of papers kicked out of the machine and makes the final curation and commentary by hand, using the traditional artisan methods.

If true, this clarifies his earlier blogging about the importance of technical skills for future research. Traditional statistics is not enough. You'll need search and AI skills, so you can construct automatic tools that scour the infoverse for items of interest and rank them by your interest metric. There's too much to do it all by hand -- you'll need these robots to prune the vineyard for you and pass only the choicest morsels. Only then will you apply your own intellect and limited time in deep thought on the most worthy work.

I assume it has more to do with the timing of the AEA meeting, even though GMU is not hiring this year.

Yes, but...this is the Linux ecosystem. Reputation is not seen as tool to get something , it's the goal.

Yesterday a guy at the job made a half hour introduction to managing code with git. Having a structured history of code development sold me the idea of learningto use git. Later, the guy told this story about reputation. He bragged that after making improvements to Linux code he received a job offer from someone acknowledging his Python skills. It sounded just like multilevel marketing. Not sure if it helps to land a job or is a thing people desperate for validation do believing it will land a job. Coding is good but the knowledge in the team is valuable to the oil industry....

"this is the Linux ecosystem."

Not really. StackOverflow has a pretty broad user base, lots of Windows developers, etc., on there.

Creepy headhunters have always been a thing.

Most of reputation-generating activities (mostly, answering questions) take much more time than the others. No time to read the paper, but really hope they only look at comparable activities with this respect.

Is this post about Tyler Cowen and MR? Indeed, why does Cowen blog? The simple answer is that we are a media-dominated culture, and the influence of media far surpasses the influence of the academy. If an economist seeks influence, imitate Paul Krugman not an obscure but peer-admired academic. Cowen's musings on this blog are not random thoughts of an intellectual, but purposeful repetitions of a well-thought-out theme he wants to advance. It's not a coincidence that Cowen is always identified as the MR blogger.

Stack Overflow makes explicit a link between hiring and contributions. The board is monetized by a set of job listings which are pitched to the hiring manager as a useful filter for technical skill, and to the board correspondents as a way to get your skill evident. This may be addressed in the paper, but it's clear that not every commenter is familiar with Stack Overflow.

Programmers are expensive, and hiring managers and HR departments often have only a vague idea of what constitutes a "good" programmer. Hence the laundry list of programming language acronyms in many a job post that is the "tell" that the folks doing the initial filtering are flying blind.

Despite the stereotype, programming is rarely a solitary pursuit: in most business settings it's a collaborative team pursuit that requires actual IRL communication skills.

StackOverflow, then, is one of the few venues you can both measure technical expertise + (written) communication skills.

StackOverflow + Github has had a (mostly) unheralded multi-billion dollar impact on coding productivity in the last five years.

BT

I don't think this is true. Any academic publication as well as basically any standards body provide a much better venue.

As an occasional answerer on Stack Overflow, I confirm the general gist of the paper.

Answering questions on Stack Overflow is not a passive activity because: The vast majority of the reputation awarded goes to the top/accepted answer. Due to the high number of users using the site for reputation, an answer usually is written within minutes and is accepted shortly thereafter. There is hardly any incentive to craft detailed answers later. (This is at least true for the large categories, like Javascript. It might be not true for obscure libraries.)

This means that if you're going to answer on Stack Overflow and get any reputation in return, you need to refresh the Unanswered questions page, and answer quickly. That is, a person willing to do this knows that SO is not a platform for providing good, educational answers and is almost by definition not in it for purely altruistic reasons. (By the way, that this encourages low quality answers is a known problem on SO.)

-R

But in terms of Brian's point, fast and good enough answers dramatically change productivity for users. 30 years ago you might have needed to buy a book. If you could get to a technical bookstore, turnaround in days.

SO is so gamified that competition is over minutes.

There is an extensive discussion on this issue on Stack Overflow, titled "Fastest Gun in the West Problem"
http://meta.stackexchange.com/questions/9731/fastest-gun-in-the-west-problem

Or, you know, they do it because they're geeks, and this is their idea of fun.

Look at the effort that goes into some responses on Marginal Revolution. Is everyone here participating because they are trying to build reputation or improve job prospects? I doubt it. Most of the people commenting here are doing so because they are interested in economics and like to talk about it. It's recreation.

I know it may be hard for non-geeks to understand how meticulously crafting a program to illustrate the answer to a programming problem could be 'recreation', but it is. I have friends who think it's the height of fun to design a new network stack for their home network, and no one even sees their work. Likewise, some carpenters love building intricate cabinetry for free for their own homes, and they aren't doing it to enhance their marketable skills or signal their ability to an employer. They're doing it because they like building cabinets, which is why they went into carpentry in the first place.

Next, we need a paper on the financial and career motivations of people who make insanely complex minecraft devices. Because surely they're not just doing it for fun.

I'm in the top single digits of stackoverflow reputation, limited mainly because I'm busy running a tech business. Did I mention I'm running my own company? So no concerns about future hireability. I'm there.

The hacker grail is reputation, pure and not-so-simple. For the same reason LeBron James is probably more concerned about the legacy of his athletic achievements than his accumulated lifetime earnings.

Plus humans are social animals - many like helping others and get reward from that without external reputation building. Maybe it's in part internal reputation building. I help others therefore I am a good person even if no one else knows. :)

Contributing to open-source projects (which anyone can see by looking at your Git commits on your Github repository), DEFINITELY has a significant impact on job prospects for programmers.

It's a very interesting -- and I think, good -- development that we've motivated a bunch of people to get out there and solve real problems for free as a reputation building exercise. Easy access to a solid body of prospective hires' work is both a more authentic and more accurate means of assessment than interviews, resumes, and references. Of course, HR still usually asks for those things, but Github is a great supplement to them.

Overall, HR's reliance on Github is a fascinating manifestation of your idea that "reputation can...enable the private production of public goods."

Along the same lines, one always wonders why people spend time producing incredibly high-quality Wikipedia contributions.

It seems to be partly recreation (as Mr. Hanson suggests above), but that doesn't appear to be sufficient. There's also the need to have an audience.

From a 2010 Financial Times article:
The Chinese government has a habit of blocking access to Wikipedia. Zhang and Zhu study a particular episode in October 2005 during which contributors who lived on the Chinese mainland couldn’t reach the site, but contributors from outside could....When a large number of potential readers were cut off from the site, many writers who could have continued to contribute stopped bothering."
http://www.ft.com/intl/cms/s/2/9b1281be-c06b-11df-8a81-00144feab49a.html

Perhaps this desire for "observed recreation" is the same urge that causes young children on the playground to plague their conversing parents with demands to "watch me, watch me, watch what I'm going to do!" Adults without children are usually confused and surprised by kids' wish to be constantly observed while playing. They think: "why aren't these kids happy to just play? Why do they need us to watch them every second?" Perhaps the overseas Chinese-language Wikipedians know the answer. Recreation alone doesn't appear to be enough to motive public good production.

Comments for this post are closed