The case against experimentation

"We should test it!" = 💀

Since The Lean Startup came out, founders have LARPed as scientists.

“We should test it out!” “We need to experiment more!”

These sentences are unserious, and the underlying theory is fatal to startups.

Here’s why:

1. To figure out anything that’s worth knowing, you have to grind your soul down. The answer you’re looking for is on the other side of pain. Experimentation makes you think you’ll just get lucky and get signal immediately. Nope. Figuring your business out is like running a marathon - the answer emerges somewhere between mile 18 and mile 22, when you’ve put your soul into making it happen, and only if you’re paying attention.

2. Experimentation is more difficult than just doing the obvious thing. You’re telling me you can design an experiment that you can really learn something from? Or that you can know exactly what you’re looking for in advance, and design an experiment that can give you conclusive results? This is more difficult than the simple process of “selling & delivering to figure out what you’re selling & delivering.”

3. Experimentation is a reductive trap. More fundamental… you assume that you can split your business into parts, test the most “risky” parts, and the whole will work? This might sound right at first glance… but it ignores literally everything we know about how complex systems emerge & function.

In theory, experimentation makes sense. In practice, experimentation is functionally a LARP - an excuse to have no conviction and waste time instead of committing to figuring the thing out.

But who knows, I could be wrong. There are almost certainly some scenarios in which experimentation makes sense. Let’s test it. :)

—

PS - As I build Reframe & my next software business, I spend zero time thinking about experiments.

• For Reframe, I am committed to growth via content. I am suffering through building an audience on LinkedIn, and have written this newsletter for like 4 years. With nearly zero growth on either… up until this fucking month. Conviction > experimentation.

• For my next startup, there are a million GTM channels that could work. I am going all in on one, and I am going to make it work.

—

PS, v2 - I recorded a free, but very ugly, video on the path to 10 customers. Check it out here.

—

PS, v3 - If your team’s going to be deploying on AWS (or switching to AWS), and want some free CloudFormation templates that will save you weeks, prevent you from needing to become an AWS expert, and set you up to be SOC audit-ready from the start… email me at rob@reframeb2b.com. Happy to set you up!