Knowledge Workers Show 49% Improvement With Gen AI In Tasks Outside Their Skillset: Study

Generative artificial intelligence is creating knowledge workers, who can code faster and summarize documents instantly. But can the tool also enable people to meet the shifting demands of their roles? 

A study conducted by the Boston Consultancy Group's Henderson Institute examined what happens when, instead of using gen AI to improve performance within their current skillset, people use the technology to complete tasks beyond their existing capabilities. 

In the experiment, BCG consultants completed two of three short tasks that mimic the daily activities of a data scientist—writing Python code to merge and clean datasets, building a predictive model, and validating ChatGPT-generated statistical analyses. Their results were compared with those of BCG data scientists who worked without the assistance of gen AI.

Aptitude Expansion

When using gen AI, consultants were able to instantly expand their aptitude for new tasks. Even without experience in coding or statistics, consultants with access to gen AI were able to write code, apply machine learning models, and correct erroneous statistical processes. 

Participants who used gen AI achieved an average score equivalent to 86% of the benchmark set by data scientists, a 49% improvement over participants not using gen AI. The gen AI group also finished the task roughly 10% faster than data scientists.

Brainstorming Partner

Predictive analytics was the task which gen AI-augmented consultants were least likely to perform on par with data scientists, as the gen AI tool is likely to misunderstand the reliability prompt without trial and error or rephrasing of the question.  

Yet, with the support of gen AI, participants were able to brainstorm with the tool, combining their knowledge with gen AI's to discover new modelling techniques and identify steps to solve problems successfully. The gen AI-augmented participants were 15% more likely to select and appropriately apply machine learning methods than their counterparts who did not have access to gen AI.

'Doing Gen AI' Doesn't Mean Learning To Do

The study showed that gen AI-augmented workers gained new capabilities beyond what either the human or gen AI could do alone. But gen AI was only an exoskeleton; humans alone were not intrinsically reskilled, because “doing” with gen AI does not immediately nor inherently mean “learning to do. 

In addition, gen AI-augmented participants with moderate coding experience performed 10–20% better on all three tasks than their peers who self-identified as novices, even when coding was not involved.