AI Features in the Frontend

• •Stream tokens — never block UI on full response
• •Context windows are finite — prioritize what you send
• •Safety filters on input and output
• •Graceful degradation when model or API fails
• •Human-in-the-loop for high-stakes actions

• •Server-Sent Events for one-way token streams
• •Append tokens to message bubble with requestAnimationFrame batching
• •Show typing indicator until first token arrives
• •Cancel button aborts fetch via AbortController
• •Markdown render incrementally — debounce re-parse every 50ms

// Streaming handler: rAF-batched append + abort + resume offset
export async function streamChat(
  prompt: string,
  onFlush: (text: string) => void,
  signal: AbortSignal,
): Promise<{ text: string; finished: boolean }> {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ prompt }),
    signal,
  });
  if (!res.ok || !res.body) throw new Error(`chat_${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let text = "";
  let pending = "";
  let frame = 0;

  const flush = () => {
    frame = 0;
    if (!pending) return;
    text += pending;
    pending = "";
    onFlush(text); // one React state write per animation frame
  };

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      pending += value;
      frame ||= requestAnimationFrame(flush);
    }
    flush();
    return { text, finished: true };
  } catch (err) {
    cancelAnimationFrame(frame);
    flush();
    // Partial text is still useful — render it with a Resume affordance.
    if ((err as Error).name === "AbortError") return { text, finished: false };
    throw err;
  } finally {
    reader.releaseLock();
  }
}

• •Naive per-token setState = 40+ renders/sec and INP > 500ms; batch per frame
• •Reverse proxies buffer by default — set X-Accel-Buffering: no on the SSE route
• •Keep a monotonically increasing offset so Resume replays from the last token
• •Virtualize the transcript above ~50 messages — long chats regress scroll INP

// Deterministic budgeter: never let the request exceed the window
const BUDGET = { system: 1_200, retrieved: 3_000, history: 3_500, reply: 1_024 };

export function buildMessages(thread: Turn[], docs: Chunk[]): Message[] {
  const retrieved = takeWhileUnder(docs, BUDGET.retrieved, (d) => d.tokens);
  const recent = takeFromEndUnder(thread, BUDGET.history, (t) => t.tokens);
  const dropped = thread.length - recent.length;

  return [
    { role: "system", content: SYSTEM_PROMPT }, // stable prefix => cacheable
    ...(dropped > 0
      ? [{ role: "system" as const, content: summaryFor(thread.slice(0, dropped)) }]
      : []),
    { role: "system", content: renderCitations(retrieved) },
    ...recent.map(toMessage),
  ];
}

• •Count tokens client-side before send — reject at the edge, not after a 400
• •Keep the system prompt byte-stable so provider prefix caching hits (50–90% input discount)
• •Summarize on a background turn, not in the user's critical path
• •Emit tokens_in / tokens_out per feature so finance can see AI COGS by surface

• •Route simple queries to smaller/cheaper models
• •Cache identical prompts with semantic similarity threshold
• •Time-to-first-token (TTFT) matters more than total time for UX
• •Prefetch likely follow-up suggestions in background
• •Rate limit per user/tenant — show queue position when throttled

// Three-layer cache in front of the gateway
const exact = await kv.get(`ai:v3:${hash(model + promptPrefix + input)}`);
if (exact) return { ...exact, cache: "exact" };            // ~0ms, ~35% hit

const near = await vectorCache.query(embedding, { minScore: 0.94 });
if (near) return { ...near.value, cache: "semantic" };      // ~25ms, +10% hit

const answer = await gateway.complete({ model: route(intent), messages });
await Promise.all([kv.set(key, answer, { ttl: 3600 }), vectorCache.put(embedding, answer)]);
return { ...answer, cache: "miss" };

• •Perceived latency: render skeleton at 0ms, first token by 800ms, and never a blank pane
• •Cache key must include model id and prompt version — bump the prefix on every prompt edit
• •Semantic cache below ~0.94 similarity starts answering the wrong question; measure it
• •Per-tenant token quotas with a visible remaining balance beat surprise 429s

• •Client-side input length limits and PII redaction before send
• •Server-side moderation API on input and streamed output
• •Block streaming mid-response if policy violation detected
• •Refusal UX — explain why, suggest alternatives
• •Audit log AI interactions for enterprise compliance

// Never dangerouslySetInnerHTML raw model output.
import DOMPurify from "dompurify";

const SAFE = {
  ALLOWED_TAGS: ["p", "ul", "ol", "li", "code", "pre", "strong", "em", "a", "h3"],
  ALLOWED_ATTR: ["href", "class"],
  ALLOWED_URI_REGEXP: /^(https?:|mailto:|#|\/)/i,
};

export function renderAnswer(markdown: string): string {
  return DOMPurify.sanitize(markdownToHtml(markdown), SAFE);
}

// Stream-side: moderate each flushed buffer, not just the final text.
if (verdict.blocked) {
  controller.abort();
  ui.replaceWith(refusal(verdict.category)); // roll back painted tokens
  audit.write({ promptHash, category: verdict.category, action: "blocked" });
}

• •Redact before the network call — once a secret reaches the provider it is an incident
• •Refusals need a next step: rephrase, link the policy, or hand off to support
• •Log prompt hash + category + model version; never the raw prompt for consumer tiers
• •Enterprise contracts often require zero-retention endpoints — verify per provider

// Breaker + provider failover, evaluated per surface
const breaker = getBreaker("ai:assistant"); // opens after 5 failures / 30s

if (breaker.isOpen()) return { mode: "keyword-search", reason: "breaker_open" };

for (const provider of ["primary", "secondary"] as const) {
  try {
    const out = await withTimeout(complete(provider, req), 20_000);
    breaker.recordSuccess();
    return { mode: "ai", out, provider };
  } catch (err) {
    breaker.recordFailure();
    if (!isRetryable(err)) break;
  }
}
return { mode: "keyword-search", reason: "all_providers_failed" };

• •Circuit breaker after N failures — switch to non-AI search rather than spinning
• •Never an infinite spinner: 15s without a token is a failure, show it
• •Offline: queue the prompt in IndexedDB, replay when connectivity returns
• •Retries multiply cost — cap at 2 attempts and never retry a streamed partial blindly

• •Draft → review → approve workflow for legal/finance content
• •Inline edit before send — user owns final output
• •Feedback thumbs up/down feeds fine-tuning pipeline
• •Escalation to human agent when confidence score low
• •Display AI disclaimer and confidence where required by policy

• •Confidence below threshold routes to a human queue — do not hide the uncertainty
• •Capture the edit delta between draft and sent text; it is your best eval signal
• •Every automated action needs an actor field: user, agent, or agent-on-behalf-of-user

• •Log prompt hash (not raw PII), model, token count, latency, outcome
• •RUM on TTFT and stream completion rate
• •Quality evals: sample conversations reviewed weekly
• •Cost per feature dashboard — alert on 2x spike
• •A/B model versions with guardrail on user satisfaction

// One low-cardinality event per turn — joins cost, latency, and quality
rum.track("ai_turn", {
  feature: "support_assistant",
  model: "mid-v4",
  prompt_version: "sys-2026-07-11",
  cache: "semantic",          // exact | semantic | miss
  ttft_ms: 640,
  total_ms: 4_180,
  tokens_in: 3_412,
  tokens_out: 517,
  cost_usd: 0.0091,
  outcome: "completed",       // completed | aborted | blocked | error
  edited_before_send: true,
});

• •Keep dimensions low-cardinality: feature, model, route — never user_id in metric labels
• •Golden-set eval of ~200 prompts runs in CI on every prompt change
• •Correlate spend spikes to a prompt_version so you can name the regressing change

Rollout, cost control, and reversal

• •Hard spend ceiling per tenant per day; degrade to search rather than invoicing a surprise
• •Kill switch is a flag read before the entry point renders — no redeploy, effective in 60s
• •Reversal plan: flag off, gateway route to previous prompt_version, cache prefix rollback
• •TCO = tokens + gateway + vector store + eval labour + on-call, not just the model bill

• •Stream with SSE + AbortController cancel; batch appends per animation frame
• •Context is a budget: sliding window, rolling summary, RAG, stable cacheable prefix
• •Route by intent — rules, small, mid, frontier; cache exact then semantic
• •Safety on input and streamed output; sanitize markdown; refusal UX with a next step
• •Fallbacks: breaker, provider failover, partial stream resume, classic search
• •Human-in-the-loop scales with irreversibility; audit every automated action
• •Observability: TTFT, completion rate, tokens, cost per feature, thumbs-down
• •Reversal: flag off plus gateway config rollback in under a minute