Skip to content

Enhance experiment designing documentation#781

Open
jaredlockhart wants to merge 1 commit intoenhance-rollouts-docsfrom
enhance-experiments-docs
Open

Enhance experiment designing documentation#781
jaredlockhart wants to merge 1 commit intoenhance-rollouts-docsfrom
enhance-experiments-docs

Conversation

@jaredlockhart
Copy link
Contributor

@jaredlockhart jaredlockhart commented Feb 9, 2026

Warning

This PR is based on #779 (Rollouts enhancement), which is based on #777 (Firefox Labs docs). Review those PRs first — this one only adds the experiments article changes on top.

Summary

Expands the experiment designing article from ~25 lines to a comprehensive guide, drawing on the experiment brief template to walk users through the full design process. File renamed from designing.md to experiments.md to match its sidebar label.

Changes

Restructured into clear phases:

  • When to use / when not to use — with cross-links to rollouts and Firefox Labs
  • How experiments work — lifecycle and key concepts (branches, enrollment, observation, sizing)
  • Designing your experiment — 7-step process from hypothesis through office hours
  • Build and de-risk — feature enablement, exposure events, QA, risk review
  • Launch, monitor, learn — sizing, launch, monitoring, and recording results/next steps

New content from the experiment brief template:

  • Expanded hypothesis format with the "If we / for / then we will see / because" template
  • Success metrics section (primary, guardrail, feature-specific, segments)
  • Branch design guidance (control vs treatments, what to document for each)
  • Targeting considerations table (application, version, locales, advanced targeting, exposure conditions, exclusions)
  • Build and de-risk checklist (Nimbus API, exposure events, QA, risk review)
  • Results decision framework (ship it, kill it, iterate) with link to promote-to-rollout
  • Note about message consult requirement for messaging experiments

Housekeeping:

  • File renamed: designing.mdexperiments.md
  • Frontmatter id/slug updated: designingexperiments
  • Sidebar and cross-link references updated
  • Workflow diagram moved to end
  • Removed Jetstream name reference
  • Corrected observation period to default 3 weeks (4 weeks total)

fixes #780

🤖 Generated with Claude Code

@jaredlockhart jaredlockhart force-pushed the enhance-experiments-docs branch 7 times, most recently from 2f3d167 to 48920a0 Compare February 9, 2026 21:01
Because

* The designing experiments article was very brief (~25 lines) with
  minimal guidance beyond linking to external resources
* It lacked consistency with the recently enhanced Rollouts and Firefox
  Labs articles
* There was no guidance on when to use an experiment vs alternatives

This commit

* Expands the article with clear sections matching the Rollouts and Labs
  structure: when to use, when not to use, lifecycle, key concepts, and
  step-by-step design process
* Adds cross-links to Rollouts and Firefox Labs as alternatives
* Adds experiment lifecycle documentation (Design → Configure → Review →
  Enrolling → Observation → Analysis → Complete)
* Adds key concepts section covering branches, enrollment, observation,
  and sizing
* Moves the workflow diagram to the end and links to the live Miro board
* Adds a "Next steps" link to the Configuring page

fixes #780

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant