An acquaintance asked a question recently that I haven't stopped thinking about. For teams past twenty engineers, where everyone's using two or three or four different AI tools and each one only reports on itself, how do you actually know if AI is working? Not per tool. Across the whole org.

I think the honest answer starts by admitting the question, asked that way, can't be answered - and gets more interesting once you sit with why.

"AI productivity" isn't one number waiting to be discovered. It's three separate questions wearing one trench coat: is anyone using the thing, is that usage actually making anything better, and is what got better still holding up a month later. Bolt four vendor dashboards together and average them, and you get a very precise number that answers none of those three. You've measured usage and quietly called it impact.

Here's the finding that made me stop trusting my own instinct on this. A controlled study - real methodology, control group, the kind of rigor usually reserved for drug trials - timed experienced developers writing code with AI versus without. Going in, they predicted they'd be about a quarter faster. Coming out, they were confident they had been. Measured against the group that worked without AI, they were nineteen percent slower. Not a rounding error. A fully inverted read on their own performance, held with total conviction, in exactly the wrong direction.

I keep coming back to that gap, because it says something uncomfortable that goes beyond individual developers. If the people closest to the keyboard can be that wrong about themselves, nobody averaging four dashboards from a distance is going to stumble onto the truth by accident. Feeling faster and being faster are two different things that happen to share a vocabulary, and most measurement setups quietly assume they're the same.

So I went looking for who's actually solved this, not who has an opinion about it.

Booking.com is the case I keep coming back to. They rolled AI coding tools out to around 3,500 engineers and ran straight into the exact problem in the original question - adoption numbers, but no line of sight into whether any of it was working. Instead of trusting "developers say they feel faster" on its own, they paired usage data with what actually happened downstream: velocity, code quality, satisfaction, checked against each other rather than reported in isolation. What they found reshaped the whole rollout. Developers using AI on twelve or more days a month were measurably more effective - a meaningfully higher PR merge rate - while occasional users showed almost nothing. That one finding changed their internal goal from "get people to try it" to "get people using it daily," and it's the kind of insight you only get by refusing to stop at the adoption number. Their own words on it stuck with me: without that data, they couldn't have had an honest conversation about ROI with the rest of the business. The adoption percentage alone was never going to get them there.

Vercel ran a version of the same instinct and cut cycle times by 43%. Pfizer's engineering org used it to find millions in reclaimable time buried in workflow friction nobody had measured before. Different companies, same underlying move - stop trusting the "we shipped more" headline the moment it shows up, and hold it against a quality or experience counterweight before believing it.

One detail I liked more than the big numbers: 1-800 Contacts didn't throw out what they already trusted to chase something newer. They kept DORA - deployment frequency, lead time, change failure rate, the metrics that don't care which tool touched the code - as their baseline, and layered AI-specific usage data on top of it instead of replacing it. That's a quieter kind of discipline than adopting the newest framework. It's the engineering equivalent of not mistaking a new tool for a new truth.

And plenty of teams never adopted a bespoke AI framework at all. They just refused to let per-tool session counts stand in for delivery health, and kept measuring the same four things they always had - deployment frequency, lead time, failure rate, recovery time - on the logic that a pull request is a pull request no matter which of the four AI assistants touched it first.

Google's own research team, the one that's been tracking engineering performance at industry scale for over a decade, put words to the pattern sitting underneath every one of these examples: AI doesn't fix a team. It amplifies what's already there. Strong engineering cultures get stronger. Weak ones get faster at accumulating the same problems they always had, just with better throughput numbers to hide behind while it happens.

Which, I think, is the real answer to the original question, once you strip away the framework names. Nobody who's actually solved this found the metric. They found the discipline of not believing the first number that looked good. Booking.com didn't trust adoption percentage on its own. 1-800 Contacts didn't discard what already worked for something shinier. The teams doing this well aren't the ones with the best dashboard. They're the ones who decided, early, not to let a single flattering number stand in for the truth - and built something slower and more annoying to check instead.

The tools were never going to tell you the truth about themselves. That part was always going to be the org's job.