A 2025 report from MIT’s Project NANDA found that 95% of enterprise AI pilots showed no measurable ROI (link below). While it’s tempting to view a stat like this as a sign that something is very wrong, I think it’s a sign that the adoption process for such a new technology is working OK. Obviously, companies want value from their AI investments, so anyone selling and implementing AI solutions wants to deliver. Personally, though, I don’t know how else to learn how to really get value out of AI without constantly playing with it, often with no goal beyond seeing what happens when I do X.

Working at Amazon, we use Amazon Quick and Kiro all the time, but I also use Codex, Claude Code and Claude Cowork on Amazon Bedrock. I’ve done a lot of setting up and tearing down experiments with personal context and second brains; built little utilities and dashboards that don’t really get used after the initial build; and finally built applications that I’ve had on a personal backlog for a long time, only to discover they weren’t as cool or helpful as they were when they lived only in my head (aka first brain). Sometimes I give multiple tools the same task just to see what happens.

But with every experiment, mini project, and attempt to “work differently,” I learn something about how these models behave and how these tools work. None of it had measurable ROI in the traditional sense, but there has absolutely been a return in what I’ve learned and in how I use these tools now to do better work more efficiently. I’m just realizing the value in sips, not gulps.

I’ve also learned a lot about what’s just better for me to do the “old way” vs trying to AI-ify it, like using my calculator app or writing a quick Slack to my boss or taking pen to paper at the end of the week to summarize my thoughts. And you know what? This stuff is changing so fast, all the time, that I have no doubt a big part of what I’ve learned will be obsolete or need to be relearned in a matter of months. Maybe sooner.

Jeff Bezos made a now well-cited point in his 2018 shareholder letter that the size of your failed experiments must grow with the company, and that not every good bet pays out. So maybe a 95% “no measurable ROI” rate isn’t necessarily evidence of failure or an over-hyped waste of time. Maybe some of it is just the discovery process working.

I’m not saying that companies (or individuals, for that matter) should go on an unbounded spending spree, lighting tokens on fire for sport. But maybe a small, intentional budget dedicated to pure experimentation is not only a good idea, but necessary if we want to eventually find the big wins.

Report referenced: The GenAI Divide: State of AI in Business 2025, MIT Project NANDA, July 2025.