Loading...
Loading...
Rollouts, guardrails, holdouts, flag debt, and kill switches for safe releases.
Module 20 · Experimentation & Feature Flags
Feature flag architecture, gradual rollouts, guardrails, holdouts, flag debt lifecycle, and emergency kill switches for safe releases.
Feature flag architecture, sticky assignment, ring rollouts, metric guardrails, holdouts, flag debt lifecycle, and emergency kill switches.
Feature flags decouple deploy from release — staff engineers design systems with sticky assignment, metric guardrails, holdout groups, and flag lifecycle management.
This module covers flag SDK bootstrap, ring deployments, auto-rollback on regression, holdouts for portfolio measurement, flag debt lint rules, and kill switches with CDN fallback.
Pairs with Frontend Reliability (Module 19) for guardrails and kill switch patterns.
Staff track module — complete Staff Frontend System Design (Module 16) first.
Staff engineers own safe release infrastructure — flags, experiments, and kill switches are how you ship to millions without big-bang risk.
Ship safely — rollouts, guardrails, holdouts, and kill switches.
Feature flags decouple deploy from release. Staff engineers design flag systems with guardrails, statistical holdouts, experimentation loops, and kill switches that prevent flag debt from becoming operational risk.
Two systems hide behind one word. A flag is a runtime switch owned by whoever is on call. An experiment is a measurement instrument owned by whoever will defend the result. Conflating them is how teams end up with 900 permanent toggles and no trustworthy readouts.
| Type | Lifetime | Owner | Removal trigger |
|---|---|---|---|
| Release flag | Days to 8 weeks | Feature team | GA reached |
| Experiment flag | 2–6 weeks | Product + data | Decision recorded |
| Ops kill switch | Permanent | On-call / SRE | Never — tested quarterly |
| Permission flag | Permanent | Platform | Becomes an entitlement |
| Config value | Permanent | Owning service | Moves to config store |
Ownership boundary
Client SDK, evaluation, and caching patterns.
| Layer | Responsibility | Example |
|---|---|---|
| Flag service | Authoring, targeting rules, audit | LaunchDarkly, Unleash, Statsig |
| Edge/CDN | Low-latency bootstrap config | JSON polled every 60s as fallback |
| Client SDK | Evaluate + cache + subscribe | @corp/flags-sdk wrapper |
| App code | FeatureGate component / hook | useFlag('new-checkout') |
const { enabled, variant } = useFlag("checkout-v2", { userId: user.id, tenantId: tenant.id, attributes: { plan: tenant.plan }, });
| Evaluation site | Flicker | Latency added | Cost of choosing it |
|---|---|---|---|
| Client after hydration | Visible flash of control UI | 0ms server, 1 frame | Worst CLS; never for above-the-fold |
| Server component / SSR | None | 5–20ms per request | Response is user-specific, weakens CDN caching |
| Edge middleware | None | 1–5ms at PoP | Rules must fit the edge runtime; cold config risk |
| Build time | None | 0ms | Requires redeploy — not a flag, a constant |
// Deterministic sticky bucketing — no network call, stable across devices. // Same (key, unit) always lands in the same bucket, so a reload cannot reroll. export function assign( flagKey: string, unitId: string, // stable id: userId, or anonymousId for logged-out salt: string, // bump to re-randomise (e.g. quarterly holdout reset) variants: { name: string; weight: number }[], ): string { const bucket = fnv1a(`${flagKey}:${salt}:${unitId}`) % 10_000; // 0–9999 let cursor = 0; for (const v of variants) { cursor += v.weight * 100; // weight expressed in percent if (bucket < cursor) return v.name; } return variants[0].name; // control is always the safe default } // Exposure is logged only when the gate is actually rendered, never on page load — // otherwise the denominator inflates and every experiment looks flat. export function useGate(flagKey: string) { const variant = useFlagVariant(flagKey); useEffect(() => { analytics.track("exposure", { flagKey, variant, unit: "user" }); }, [flagKey, variant]); return variant; }
Failure mode
Percentage, cohort, and ring-based releases.
| Strategy | When | Risk |
|---|---|---|
| Percentage | Low-risk UI changes | Statistical noise at low % |
| Allowlist | Enterprise beta | Manual maintenance |
| Geographic ring | Regulatory rollout | Region-specific bugs hidden |
| Tenant-level | B2B where one account cannot see mixed UI | Coarse — 1 tenant may be 30% of traffic |
| Device tier | Perf-sensitive changes | Low-end regressions surface last |
| Ring | Traffic | Soak time | Automatic abort if |
|---|---|---|---|
| Internal | ~0.1% | 1 day | Any new error signature |
| Canary | 1% | 24h | Client error rate +0.05pp |
| Early | 10% | 48h | INP p75 +25ms or LCP p75 +200ms |
| Broad | 50% | 72h | Conversion −1% at 95% confidence |
| GA | 100% | — | Guardrail breach reverts to previous ring |
// Guardrail watcher: abort the rollout without waking anyone up. const GUARDRAILS = [ { metric: "client_error_rate", max_delta: 0.0005 }, // +0.05pp { metric: "inp_p75_ms", max_delta: 25 }, { metric: "checkout_conversion", min_delta: -0.01, requireSignificance: true }, ]; export async function evaluateRing(flagKey: string, ring: Ring): Promise<Verdict> { const stats = await metrics.compare({ flagKey, since: ring.startedAt }); for (const g of GUARDRAILS) { const s = stats[g.metric]; const breached = (g.max_delta !== undefined && s.delta > g.max_delta) || (g.min_delta !== undefined && s.delta < g.min_delta); if (breached && (!g.requireSignificance || s.pValue < 0.05)) { await flags.setRollout(flagKey, ring.previous); // seconds, no deploy await pager.notify({ flagKey, metric: g.metric, delta: s.delta }); return { action: "rolled_back", metric: g.metric }; } } return stats.soakedFor >= ring.soakMs ? { action: "advance" } : { action: "hold" }; }
Blast radius
Protect core metrics during experiments.
Sample size is the conversation most candidates skip. For a two-sided test at 80% power and 95% confidence, you need roughly 16 × p(1−p) / MDE² users per arm. At a 4% baseline conversion and a 5% relative lift, that is about 154,000 users per arm — two weeks at 22k daily sessions, not two days.
| Baseline | Relative MDE | Users per arm | At 20k/day |
|---|---|---|---|
| 4% conversion | 10% | ~38,000 | 4 days |
| 4% conversion | 5% | ~154,000 | 16 days |
| 4% conversion | 2% | ~960,000 | 96 days — do not run it |
| 30% activation | 5% | ~15,000 | 2 days |
Interview pattern
Measure cumulative impact and avoid over-exposure.
| Holdout | Size / duration | Answers | Cost |
|---|---|---|---|
| Feature holdout | 5% for 4 weeks | Did this one launch hold up past novelty? | 5% miss the feature |
| Team holdout | 2% for a quarter | Did a quarter of shipping move the metric? | Two code paths maintained |
| Company holdout | 1% for a year | Is aggregate product work compounding? | Highest support and QA burden |
| Geo holdout | One matched market | Effect where user-level split leaks | Low power, confounded by market |
Reversibility
Emergency off without redeploying.
| Mechanism | Time to take effect | Depends on | Use when |
|---|---|---|---|
| SDK streaming update | 1–5s | Flag service healthy | Normal incident response |
| CDN JSON poll | ≤ 60s | CDN only | Flag service is the incident |
| Edge middleware rule | ≤ 30s | Edge platform | Block a route entirely |
| Feature redeploy | 8–25 min | CI + CDN purge | Last resort |
| DNS / CDN failover | 1–15 min | TTL propagation | Whole-origin outage |
// Layered read: streamed value first, CDN snapshot second, safe default last. export function killSwitch(key: string): boolean { const streamed = sdk.peek(key); // undefined if SDK not ready if (streamed !== undefined) return streamed; const snapshot = cdnConfig.get(key); // polled every 60s, cached in memory if (snapshot !== undefined) return snapshot; return SAFE_DEFAULTS[key] ?? false; // "off" for anything expensive } // Guard the import, not just the render — a killed feature should cost zero bytes. if (!killSwitch("disable_recommendations")) { const { Recommendations } = await import("./Recommendations"); mount(Recommendations); }
Staff pattern
Sunset flags or they become permanent complexity.
| Phase | Action | Owner |
|---|---|---|
| Launch | RFC + expiry date (90 days) | Feature team |
| Rollout | Weekly % increase with metrics review | Feature + data |
| GA | Remove flag; default on in code | Feature team |
| Cleanup | ESLint rule fails on stale flag keys | Platform |
Flag debt is measurable, so budget it. A useful rule: no more than one active release flag per engineer and zero flags older than 90 days without a written extension. Each surviving flag doubles the paths a test suite must cover.
// Codemod input: flag GA'd as permanently true. // npx jscodeshift -t remove-flag.ts src --flag=checkout-v2 --value=true - const { enabled } = useFlag("checkout-v2"); - return enabled ? <CheckoutV2 /> : <CheckoutLegacy />; + return <CheckoutV2 />; // CI gate: fail the build on expired keys. // flags.registry.json => { "checkout-v2": { owner: "payments", expires: "2026-09-01" } } for (const [key, meta] of Object.entries(registry)) { if (Date.parse(meta.expires) < Date.now()) { fail(`Flag ${key} expired on ${meta.expires} (owner: ${meta.owner})`); } }
| Debt symptom | What users hit | Fix |
|---|---|---|
| Stale flag never removed | Dead code path silently rots and breaks on re-enable | Expiry gate in CI + codemod |
| Interacting flags | Two toggles combine into an untested fourth state | Cap variants; contract-test combinations |
| Flag read in a render loop | INP regression from thousands of evaluations | Memoize per unit; evaluate at the boundary |
| Orphaned owner | Nobody can say what it does, so nobody removes it | Registry requires a live owning team |
Hypothesis → ship → measure → decide.
| Step | Artifact | Typical duration |
|---|---|---|
| Hypothesis | One page: metric, MDE, runtime, guardrails | 1 day |
| Instrument | Exposure event + outcome events, SRM check | 2–3 days |
| Soak | Full weekly cycle minimum | 14 days |
| Readout | Segment cuts, guardrails, decision memo | 2 days |
| Cleanup | Codemod, flag removal, archive doc | Same sprint |
Failure mode
Experimentation and flags at staff depth.
Feature flags and experimentation interview questions.
Test your understanding — 5 questions
You are designing flag infrastructure for 50 apps and expect to reevaluate the vendor within three years. What is the most important structural decision?