How the machines will pick the best workers in the future

From Aki Ito, here is a a good discussion of a new innovation, related to some trends I discussed in Average is Over:

To aid that search [for better workers], Juhl this month will begin using an online video game designed to track, record and analyze every millisecond of its players’ behavior. Developed by Knack in Palo AltoCalifornia, Wasabi Waiter places job-seekers in the shoes of a sushi server who must identify the mood of his cartoon customers and bring them the dish labeled with the matching emotion. On a running clock, they must also clear empty dishes into the sink while tending to new customers who take a seat at the bar.

Using about a megabyte of data per candidate, Knack’s software measures a variety of attributes shown in academic studies to relate to job performance, including conscientiousness and the capacity to recognize others’ emotions. Knack’s clients will also see a score estimating each applicant’s likelihood of being a high performer.

As for another company:

…The patterns gleaned since the company’s founding in 2007 have debunked many of the common assumptions held by recruiters, Evolv executives say. For example, a history of job-hopping or long bouts of unemployment has little relationship with how long the candidate will stay at his or her next job, according to Evolv’s analysis of call center agents.

“As human beings, we’re actually pretty bad at evaluating other human beings,” said David Ostberg, vice president of workforce science at Evolv. “We’re making sure people are using the right data, instead of the traditional methods that were previously thought to be valid but big data’s showing are not.”


New York-based ConnectCubed has also developed software to determine the personality and cognitive abilities of job applicants that, at its largest clients, is tailored for that specific company. ConnectCubed has existing workers at those businesses complete its video games and questionnaires so the behavioral profiles of the star employees serve as a benchmark for who managers should hire in the future.

“When new people apply, you can say, wow this guy has all the makings of our top salesmen,” said Michael Tanenbaum, chief executive officer and co-founder of the service. “These are things that are impossible to measure from a resume, especially with educational backgrounds that are often more determined by socioeconomic status than your innate ability.”

To be sure, Knack and ConnectCubed, which say they can predict high-performers across a broad set of workers, haven’t been around for long enough to track, over time, whether their technologies actually are improving the quality of the employees their clients hire or those businesses’ bottom line.

The article is interesting throughout.


"Knack’s software measures a variety of attributes shown in academic studies to relate to job performance, including conscientiousness and the capacity to recognize others’ emotions."

If you can fake conscientiousness and empathy, you've got it made.

Seriously, we're seeing a steady increase in the gaming of standardized tests:

Human ingenuity trumps systems. Do we really think machines would be better at hiring _saleguys_ that people?

Well, as you imply, the ability to game standardized tests correlates with ingenuity, a trait employers are looking for anyway.

Parenthetically, I never get the sense that Asians game their jobs the way they do college entrance (which they have to anyway because of the Asian quota system).

"...the ability to game...tests correlates with ingenuity": Kobayashi Maru!

How much of this is reading more than what the data says? It's not as if we're not champions in exaggerating findings from social data.

Instead of finding the most efficient people, this approach will force everyone to adapt in order to conform to whatever its metrics are. True disruptive innovation, creativity will be gone. And we'll tell ourselves we've improved and become more meritocratic when in reality we would have just changed the definition of merit. I don't know if this has always been true of humanity but the 21st century American sure is intellectually lazy.

If you're not playing videogames at least 6 hours a day, you're going to be left behind, old man! Boom! Whizz! Beep-beep-beep-beep.

A megabyte of data per candidate? Probably just his photo gets you there.

What's sorely missing in this video-game-based-hiring crap is validation. Where's Knack's data on how it's choices are better than old school human hiring?

Big data mining I can understand but this video game strategy sounds nutty to me. Show me data please!

Hey Rahul -

All of our games and surveys meet APA guidelines for validation. It's a pretty high barrier to entry! All of them are also reviewed and approved by our scientific advisory board. There are a couple academic studies coming out in the new year using our data that I think will answer a lot of your questions.

--Michael Tanenbaum, ConnectCubed

I'm sorry, but your website is really light on numbers. Or maybe I was stupid at searching.

I found tons of news stories, interviews, beautiful webpages but really no data at all. Zero. I'm not sure why you'd try to sell a fairly wacky idea without offering any hard data to judge your claims by.

Call me a skeptic. I'll wait for your papers to appear.

Have Evolv executives heard of external validity? Quite a stretch to extrapolate from one study of call center agents to the whole job market, eh?

Related, WSJ article on software making parole assessments. It has been a major factor in reducing prison populations for the last 2 years, which may interest AT:

Fascinating - so standardized testing is now, just like watching classes in front of a screen, worth mentioning because, what, it involves video games?

Man, sometimes I wonder if Jerry Pournelle has wasted decades talking about his fixation on predicting performance based on valid measurements. Though that 'valid measurements' part is difficult - Snowden looked pretty good to a string of employers whose use of various data mining techniques is literally world class.

The really funny thing will be when a company, using its own metrics to select exactly the employees it feels best reflect the opinions of those in charge of designing the algorithms, blames the employees for the failure. I wonder if that will be worth a comment then, too.

I think the words missing from this are "disparate" and "impact"?

A megabyte of data? They are going to have to show it is all race neutral and necessary for the business role?

Good luck with that.

Some of you HBD guys seem only interested in a subject if you can bring it back to your hobby horse.

Gee, Therapsid, ya think?

I imagine the PC crowd will indeed be displeased if algorithmic selections do not produce equal results across the board, whether it be across race/gender/LGBT/etc.

What's up with the shaming attempt, Therapsid?

It may be a hobby horse but it sounds like a real legal problem to me.

It does not matter what my interests are, the first thing you have to ask about any method of selection is whether it is legal or not. The first question. Before "Does it work?" even.

Which in the US means looking at what the Supreme Court said and what the Supreme Court said is that no method, no matter how benign the intent, no matter how fair it is on the surface, can be used if it has a disparate impact on minority candidates.

Now I don't know if this method will disproportionately select Whites or Asians or Native Americans. Although, as you point out, I do have my suspicions. But even if that is so very wrong of me, the first question you have to ask is whether it is legal to use this method or not. They have invested in what looks like a forbidden technology.

Or they'll just tweak it. Didn't the the Supreme Court kinda weaken that decision with the Sotomayor firemen case?

I am not sure they weakened it. They screwed it up royally if that is what you mean. They replaced a simple rule that everyone could understand - promote Blacks - with a much more complicated rule that lawyers will love because it will mean law suits forever. But they left the Disparate Impact rule there.

Especially as a computer game must just be begging for a law suit. If anyone has had lots of experience with computers and computer games it is White suburbanites. As opposed to recent immigrants and inner city residents. They tend to be poor - you seen the price of a computer game these days?

If I hire for a job that needs super tall people and my systems end up selecting a lot of Dutch & Scandinavians would that be disparate impact? I think not.

What @Subtlety forgets is the "business necessity" defense, I think (depends on the quality of their data).

I forgot it? What do you think "A megabyte of data? They are going to have to show it is all race neutral and necessary for the business role?" means?

They are going to have to go through every line of that program and show that everything it tests is necessary. Which in turn will depend on how the Courts interpret necessity. IQ tests for instance are very good at predicting all sorts of things you want in an employee, but the Courts still do not accept them as a valid means of testing.

It is going to be a legal nightmare.

I think there's plenty of wiggle room. Companies can use these programs on a technically "advisory basis" so that no one actually gets hired or fired by the algorithm itself.

You will be required to provide stilts and training to shorter ethnicities.

Big data shows that using this approach is very us.

Tyler Cowen posted about our company. I can die happy...

Long time fan. You are a hero from my days as a lowly grad student. Thanks for linking!

I've played this sushi game at my job, as we were experimenting with it (we invest heavily in recruiting). The game itself was fine, although it was fairly simple (each customer emotion has a corresponding type of sushi, and you give the appropriate sushi to each customer based on his/her facial expression). At least initially, I predict a hesitancy to adopt just because it seems so silly, for lack of a better term. It will be difficult for candidates to accept that they were turned down because they didn't click on the right type of virtual sushi fast enough.

Can you provide more details? Were there different ways of playing the game? Could you see what the game was trying to measure? I've played a bunch of games over my life and can't see my play style revealing much about me other than that I'm a conservative defense oriented player when it comes to real time strategy games. I also like to use area effect spells when I play role playing games - sometimes that causes friendly fire. Would that cost me the job?

If it were up to me, every job candidate would have to play three games of Tetris. The highest score among all candidates gets the job.

Comments for this post are closed