Guides Design experiments at scale

Updated November 22, 2026

AI-generated from Mike Stilling's work, notes, and writing.

Running design experiments at scale

As a Staff designer at Stripe, Mike Stilling iterates on experiments across high-traffic surfaces like registration, sign-in, and contact sales. Mike is also openly skeptical of how experimentation often gets practiced. This guide is a practical take on running experiments that actually teach you something.

The short answer

An experiment is a way to learn, not a way to avoid making decisions. Big traffic makes it possible to measure small changes, which is useful. But a test is only as good as the idea behind it. The best experiments come from a real understanding of what's getting in people's way.

The designer's role in experiments

In a large company, experiments are a team sport. Data science owns the statistics, engineering owns the implementation, and product owns the roadmap. The designer's job is to make sure the experiment is worth running:

  • Find the real problem. Look at where people drop off, read support tickets, watch session recordings, and talk to researchers.
  • Shape the hypothesis. Turn "this page feels off" into a clear, testable claim.
  • Design variants that are actually different. A variant nobody notices can't teach you anything.
  • Design for all outcomes. Know in advance what you'll do if it wins, loses, or comes back flat.
  • Write it down. Document what you changed, why, and what happened. A record of experiments is how a team stops re-running the same ideas.

Prototyping in code helps here. A working prototype lets partners see exactly what the variant does, including states and edge cases, before anyone builds it for real. See prototyping in code for designers.

Writing a good hypothesis

A useful hypothesis has four parts:

  1. Who the change is for.
  2. What you're changing.
  3. Why you think it will help, based on evidence.
  4. Which metric should move, and which shouldn't get worse.

Weak: "A bigger button will increase sign-ups."

Stronger: "New visitors from paid search are dropping off on the pricing page because they can't tell which plan fits them. Adding a short plan recommendation will increase plan selection without reducing paid conversion."

The "why" is the important part. It's what you learn from, whether the test wins or not.

Choosing metrics

Pick one primary metric before launch and stick with it. Then add guardrails.

  • Primary metric: the one number this change is supposed to move, like completed registrations.
  • Downstream metrics: what happens next. More sign-ups only matter if those people go on to use the product. As Mike has written, measuring more sign-ups doesn't necessarily mean more paying users.
  • Guardrails: things that must not get worse, like support contacts, error rates, page speed, or drop-off later in the funnel.

Be wary of metrics that are easy to move but don't mean much. Clicks and time-on-page can go up for bad reasons.

Common ways experiments go wrong

Mike has written about whether marketing design growth hacks work on B2B SaaS sites. The short version: a lot of them don't, at least not in ways that last. Some of the most common problems:

  • Underpowered tests. Too little traffic or too small a change, so the result is noise. A "lift" you can't reproduce isn't a lift.
  • Peeking and stopping early. Calling a winner the moment the numbers look good. Agree on a duration or sample size up front.
  • Hacks over needs. Urgency banners and dark patterns can move a short-term number while hurting trust. Clarity and less friction usually win over time.
  • Incremental forever. Small tests only explore small changes. They can't tell you whether a completely different approach would be better.
  • Testing in isolation. A win on one page that causes problems for another team isn't a win.

When not to test

Not everything needs an experiment:

  • Fixes. If an error message is confusing or a field is broken, fix it.
  • Low traffic. If you can't detect a meaningful change in a reasonable time, use research and judgment instead.
  • Big rethinks. A new approach often looks worse at first because people are used to the old one, and a short test can't account for that. Use research, prototypes, and staged rollouts instead.
  • Foundations. Accessibility, performance, and clear copy are worth doing regardless of what a test says.

At Rewatch, analytics showed that pages with concise, feature-based copy converted at a significantly higher rate than pages relying on benefit-based jargon. That kind of evidence is valuable, and it came from reading data carefully, not from running dozens of tiny tests. (More on that.)

Frequently asked questions

Should designers run A/B tests?

Designers should help shape them. The designer is often best placed to form the hypothesis, design the variants, and make sure the test actually changes something people will notice. Data science and engineering partners usually own the statistics and setup.

What makes a good design experiment hypothesis?

A good hypothesis names who the change is for, what you're changing, why you believe it will help, and which metric should move. For example: 'New visitors abandon sign-up at the business details step because they don't have the information handy. Making those fields optional at sign-up will increase completed registrations without hurting activation.'

When should you not run an A/B test?

When there isn't enough traffic to detect the change, when the change is clearly a fix (like a broken error state), when the change is too small to matter, or when the right move is a bigger rethink that a test can't evaluate fairly.

What is a guardrail metric?

A guardrail metric is something you don't want to get worse while you try to improve your main metric. If a change increases sign-ups but also increases support tickets, fraud, or drop-off later in onboarding, the guardrail catches it.