Why AI slowed down experienced developers in the METR study
For a week now I’ve been seeing a link to the results of an experiment measuring how AI tools affect development productivity. And almost every time it’s mentioned, the message is: “Ha-ha, AI tools don’t actually speed up development, they slow it down. Checkmate, AI fans!” Indeed, if you skim the article, the conclusion that sticks is “we expected a 20–40% speed-up and got a 19% slowdown”.
But if you read more carefully and get into the essence of the experiment and how it was run, the results become quite understandable and even obvious. Among the most influential factors are:
- The participants had a lot of experience with their repositories. They felt like fish in water with any task and knew exactly how to solve it.
- Large, complex repositories. The average repository was 10 years old and had a million lines of code.
- Weak use of the task’s implicit context and key knowledge about the repository. The developers relied on AI tools to analyse the code and build the context automatically.
These factors show a quite expected reality: if people have spent years accumulating knowledge about something non-trivial in their heads, then without formalising that knowledge and giving it to AI tools, productivity will be very low. That leads to lots of extra time spent correcting prompts or polishing the generated code. And, of course, it’s faster for an expert to just do it by hand.
To get a serious productivity gain from AI tools in a study like this, you first need to invest in formalising and documenting knowledge. That includes the overall structure of the repository, technical design rules, development standards, quirks and implicit rules, historical tech debt and so on. It’s very much like onboarding a new developer onto a project.
That’s why practically all tools support reading this kind of knowledge from special files into the task context, at the level of the whole repository or a specific directory. And then even autonomous task solving by an agent brings much more pleasant results.
Still, the experiment is very interesting, and I recommend reading the full version rather than a short article with conclusions.