
A very deep exploration, conducted by METR with 16 open-source developers and 246 real issues, has looked at perceptions and reality of productivity when using AI coding assistants. Titled Measuring the impact of early-2025 AI on experienced open-source developer productivity, the report tackles something we’ve known for a while, our perception of productivity is no indicator for reality.
We had the same issue with multi-tasking, where people thought they were more productive, but the reality was they were less productive. So, how does this translate to software delivery with AI assistance? The TL;DR is a perception of a 20% less time to complete tasks, but a reality of an additional 19%. Less than half of AI suggestions were accepted by the developers.
A lot of earlier studies looked at artificial problems, things that were self contained, maybe didn’t reflect the messiness or real code, or they relied on metrics that, honestly, AI could game.