By Jacob Clemente

Senior reporter, AI & work

“When are combinations of humans and AI useful?” That’s the trillion-dollar question researchers at MIT tackle in a new working paper that analyzes 74 recent studies on human and AI collaboration. The answer: Not as often as we might have hoped. 

The authors looked at a series of experiments where humans and AI performed tasks individually and together. They found that, on average, combining the two yielded worse outcomes than using the best of the two alone.

“That's surprising…because many people, including us, have been saying for years that we should be looking at the combination—human augmentation and all that,” says Thomas Malone, a professor of management at the MIT Sloan School of Management and one of the paper’s authors, who wrote a book called Superminds in 2018 with the subtitle “The Surprising Power of People and Computers Thinking Together.” However, Malone adds, “It would be a mistake to interpret our results as meaning…we shouldn't be trying to combine people and computers.”

We spoke with Malone and Michelle Vaccaro, a co-author of the paper and a PhD student at MIT, about their findings and what they mean for businesses looking to augment their employees with AI. Here are highlights from that discussion and the paper:

  • The combination of humans and AI, on average, performs worse than the best of the two alone. Ideally, humans and AI would have what the researchers refer to as “strong synergy,” where the combination of the two outperforms both of them on their own. Imagine, for example, an AI system that predicts successful job candidates with 80% accuracy and a recruiter who can do so with 70% accuracy. If combining the recruiter and the AI yields 85% accuracy, we would say they have strong synergy. Unfortunately, the researchers didn’t find evidence of strong synergy, on average. In other words, if you were to select one of the experiments they analyzed at random, chances are that either the human or the AI performed better alone than they did together. Though, in general, the human-AI combo tended to perform better than the human alone.