The jobs of the future?

Katherine Young, 23, is a Google rater — a contract worker and a college student in Macon, Ga. She is shown an ambiguous search query like “what does king hold,” presented with two sets of Google search results and asked to rate their relevance, accuracy and quality. The current search result for that imprecise phrase starts with links to Web pages saying that kings typically hold ceremonial scepters, a reasonable inference.

Her judgments, Ms. Young said, are “not completely black and white; some of it is subjective.” She added, “You try to put yourself in the shoes of the person who typed in the query.”

How smart do you need to be to do this?  How well-educated?  How is the quality of your work to be judged?  The full article is here, interesting throughout.


We now work for the machines. Maybe they are evaluating her.

The real fun would be if Katherine Young is actually answering using a script that pretends it is a human.

Maybe Watson is looking for someone with a compatible sign.

+1 Although, this could be somewhat recursive. If the machine doesn't know the phrase, and you give it the answer, how does it know that you gave it the right answer so that it could evaluate you. I suppose at some point you could look at how satisfied the customer was with the human input by measuring whether the consumer continued the search using different terms and found a completely different set of data than indicated by the first search term and human interpretation thereof.

I'll have to write an algorithm for this and sell it to Google.

That was supposed to be a reply to Andrew'

Then you better start working on that algorithm sooner rather than later.

Got a friend who's doing this right now.
Pay is OK, but it's real boring.

AFAIK they don't ask for credentials, just a test. A long test.
Don't know how smart you have to be, but you know how some people are really good at finding stuff at Google and others suck? Well that's the skill they're asking for here. I guess it's more experience than anything else.

The new "Common Core" K-12 educational standards being adopted by most states over the next few years make a big deal over not using multiple choice tests. Instead, students will have to write out their answers in prose. Supposedly, this will be machine-gradeable Real Soon Now, but it sounds like this grading Common Core tests will be a major growth category in jobs.

The ultimate Sailerian irony would be if grading that prose got massively outsourced to India etc.

The problem is that the Common Core tests are supposed to be "adaptive" -- get the answer wrong, they serve up an easier question, and vice-versa. How that will work with requiring verbal answers for everything seems unclear. But don't worry, only 40+ states are implementing this K-12 strategy over the next couple of years, so it's not like we're betting the whole country on the Common Core.

Adaptive essays? Real Time Grading! :)

Google has found a way to inshore the World of Warcraft "gold miner" model - take that, China !

Mr. Watson, come here!

I have done this for about two years.

When I was a writing/production freelancer, it was almost always my primary source of income, keeping me afloat during lean months. At my most ferocious I would work six hours every weekday and four hours on the weekend. That was enough for me to gross $1500-2000 a month. Though I have a full-time job I still log a few hours a month to keep my contact active. It's likely I'll freelance again someday and it's an excellent way for me to subsidize my creative pursuits.

The nature of the work is repetitive and often boring, but does speak to the particular skills of the infovore. Rating guidelines can be complex, but as a voracious consumer of the web I am able very quickly discern 'user intent' and rate accordingly. One definitely benefits from having a wide array of interests. I can accurately rate much faster than I'm required to, which allows me to take frequent breaks. As a result the work is not so onerous. Have often caught up on my RSS backlog including MR while rating!

To answer Tyler's questions directly, in my experience:

How smart do you need to be to do this?
Hard to say, but it's easier the smarter you are. Some queries might be scientific or refer to pop culture miscellanea. Without a certain level of knowledge you would not be able to maintain the productivity expected and would eventually be fired. Interestingly some tasks explicitly require local knowledge and are given to you based on your location.

How well-educated?
The more the better, roughly. Some queries require a facility with academic-style publications and the truth-value thereof.

How is the quality of your work to be judged?
By how well you adhere to the guidelines and how well your evaluation agrees with other raters.' If there's a significant deviation in opinions among raters you enter into a moderated side-process where you debate with each other until everyone more or less agrees.

In general rating requires the ability to mentally catalog a lot of disparate information and consistently and logically apply a set of somewhat complex rules to a given scenario. The stickiest wicket though is being good at quickly determining what a user wants, which requires a certain empathy for how people try to find information online.

Good readers make good raters.

I used to work at Google in search quality, and would often end up judging the raters. For judging the quality of work, an engineer will be skimming the ratings in large groups with outliers marked and there are checks for statistical consistency and correlations with other raters. Traditional education is not very important, but familiarity with searching is important.

I would guess that an older (middle aged) person would be better at this work than a college student or recent grad, simply because the older person has had more life experiences and has been exposed to more things.

Older people doing this job are probably lower IQ than younger people doing the same job. The increased dumbness of the old person may make him worse than the young person.

Comments for this post are closed