WHAT AI IS ACTUALLY GOOD FOR ACCORDING TO SCIENCE

New research reveals Al's strengths-and limitations.

- by LIZ BRODY

As entrepreneurs experiment with generative AI, academic researchers are running their own experiments. They want to know: What works? What doesn't? And why?

The early results are fascinating.

On the following pages, we highlight studies conducted by researchers at Harvard, MIT, Columbia, and other universities, where you'll see a theme: AI cannot effectively replace people at this point, but it can be a powerful collaboration tool-and in ways you may not expect. Here's how.

STUDY NO. 1 Does GenAl Increase Productivity?

SCIENCE SAYS: Yes, but only for certain tasks.

It’s taken as a given: AI boosts your productivity. But...does it really?

A group of researchers at the business schools of Harvard, MIT and Warwick universities along with the University of Pennsylvania’s Wharton School wanted to find out. So they assembled a team of 758 Boston Consulting Group employees, and put them through a few experiments to study genAI's effect on their work.

The research

The investigators separated the employees, all consultants, into two groups for different experiments. In both, the subjects were first asked to solve a problem without using AI to establish a performance baseline. Then their group was given a challenge where some of the consultants were able to use GPT-4, while others were not.

In the first group, the assignment was to come up with footwear ideas for niche markets.

For the second group, the researchers purposely designed a task where the AI would make a mistake in its answer. In this challenge, the employees had to solve a problem for a company by assessing interviews with its insiders and evaluating quantitative data. Would the consultants carefully check the AI’s work?

At the end, researchers measured how productive everybody was.

The results