A Bank for International Settlements' field study on AI use by Ant Group programmers demontrated a clear boost in the number of lines of code produced by staff exposed to the Large Language Model.
The intention of the study was to conduct empirical research on AI's impact on productivity in tasks requiring cognitive abilities.
The field study was conducted with Ant Group following the September 2023 launch of CodeFuse, a large language model (LLM) designed to assist programming teams. In the experiment, one group of programmers had access to CodeFuse (the treatment group), while another group did not (the control group).
Productivity (measured by the number of lines of code produced) increased by 55% for the group using the LLM. Approximately one third of this increase was directly attributable to code generated by the LLM. The remaining productivity gains were likely due to improved efficiency in other coding tasks, as programmers had more time available.
However, the productivity gains were statistically significant primarily among junior staff, with a less pronounced effect on senior employees.
This difference appears to stem from lower engagement with the LLM by senior programmers, rather than the tool being less useful to them. The rate at which programmers accepted the LLM's suggestions did not vary with experience level, suggesting that the lower impact on senior programmers' productivity was due to less frequent use of the tool.
"Despite this, when senior programmers did use gen AI, they paid close attention to the suggestions and found them useful," the BIS research team conclude. "This suggests that while gen AI can enhance productivity, its adoption and usage patterns vary significantly with experience levels, highlighting the need for targeted strategies to maximize its benefits across different seniority levels.
Discover new challenges and opportunities artificial intelligence brings to the banking sector at Finextra's first NextGenAI conference on November 26 2024. Register your interest here.