Quality Assurance

Runtime verification that gets smarter every run

The qa-engineer: what it does

  • Verifies the change in a real browser - navigates, clicks, screenshots, captures console errors
  • Falls back to source-reading when browser access is blocked (labels those criteria [source-verified])
  • Returns a structured PASS / FAIL / PARTIAL / BLOCKED / INCONCLUSIVE report with evidence
  • Does not fix anything - reporting is the whole job
Static review (Skeptic) + runtime review (qa-engineer) = the protocol's two-pass safety net.

The qa-engineer: when it runs

  • For UI-visible changes with qa_criteria defined: spawned in parallel with the Skeptic (both background, single message) - sign-off requires both to pass
  • For non-UI or unknown-diff cases: spawned after Skeptic sign-off as a sequential fallback
For UI-visible changes, Skeptic and qa-engineer run concurrently - no sequential delay. Both must pass before the unit is complete.

qa.md is the project's QA memory

Per-project file, seeded by /ds-init-project when a web UI is detected. Two jobs:

1. Config - how to run the app for QA

## Dev server
command: npm run dev
port: 3000
## URLs
local: http://localhost:3000
staging: https://staging.example.com
## Preferences
prefer: local

2. Knowledge - project-specific quirks learned from past runs

qa.md is the only file the qa-engineer is allowed to write to. It is QA infrastructure, not application code.

How qa.md shapes a run - pre-flight

Before touching the browser, the qa-engineer:

  1. Resolves the URL - prompt URL wins, otherwise reads qa.md config
  2. Starts the dev server automatically using the config'd command and port
  3. Waits up to 30s for the port to respond, then curls the URL to confirm
  4. Reads every ## Knowledge entry and applies them as pre-flight adjustments
  5. Only then does it begin testing the acceptance criteria
No URL-hunting, no manual dev server instructions, no re-discovering quirks. The project already told the qa-engineer how to run itself.

Knowledge tags - how quirks get applied

Every knowledge entry is tagged. The tag tells the qa-engineer exactly how to use it.

Tag Effect on the next run
server Adjust the dev server startup (extra flag, different command)
timing Insert the specified delay at the relevant workflow step
port Override the config'd port with the noted alternative
auth Follow the documented login flow instead of discovering it fresh
noise Exclude those console errors from blocking-issue classification
retry Retry that endpoint or action once before marking FAIL
tool Apply specific flags when invoking Playwright or agent-browser

The write-back loop - qa runs get smarter

After each run the qa-engineer reviews what it discovered and appends up to 3 knowledge entries, only if all of these hold:

  • Project-specific quirk, not generic browser behavior
  • Likely to recur on every future QA run
  • Required non-obvious handling (a flag, delay, retry, workaround)
  • Not already captured

Bugs in the app do not go here - those belong in the QA report. qa.md is for workarounds the QA process itself needs to remember.

First run in a new project is slow - qa-engineer is discovering quirks. Every run after pays that cost back: the pre-flight already knows.

Example - one week in the life of qa.md

# QA Config
## Dev server
command: npm run dev
port: 3000
## URLs
local: http://localhost:3000

## Knowledge
- [2026-04-02] timing: Wait 2s after nav to /dashboard - React Query refetch is async
- [2026-04-04] noise: Ignore "Hydration mismatch" warning in dev build - SSR/CSR known gap
- [2026-04-07] auth: Dev login uses demo@example.com / password. Submit, wait for /app redirect.
- [2026-04-09] retry: /api/search 500s once on cold start - retry once before FAIL

Each entry was a surprise the first time. After the entry, the qa-engineer handles it automatically on every run that follows.

When QA is skipped - qa_skip enum

QA fires by default for every Elevated unit. It is skipped only when the architect explicitly sets one of these five qa_skip values:

Value When it applies
pure-backend-library No UI or behavioral surface visible to users
config-only Change is purely configuration with no runtime code path
type-only-refactor Only types/interfaces changed - no runtime effect
dep-bump-no-runtime-change Dependency version bump with no API change
docs-only Documentation files only, no code

No qa.md is NOT a reason to skip QA - its absence only removes supplemental context; the gate still fires.

If none of these five values fit, QA fires. The qa_skip rationale must be logged in the Brief or architect plan.

INCONCLUSIVE - when runtime is unreachable

When the qa-engineer cannot reach a runtime path (preview deploy blocked AND local-env unavailable), the result is INCONCLUSIVE (qa_unverified=true) - not a pass.

  • The conductor MUST NOT auto-promote INCONCLUSIVE to PASS
  • Static source review of an Elevated UI-visible criterion is approximately zero signal - state hooks, conditional rendering, and prop-sync bugs are invisible to source review
  • The conductor surfaces the state to the operator with three options:
    1. Provide the missing env/URL and re-run QA
    2. Accept INCONCLUSIVE - the PR can merge but carries qa_unverified=true
    3. Abandon the ticket
INCONCLUSIVE is not a pass. The operator must explicitly accept the unverified state before merge. The conductor does not proceed silently.

Phase 6b QA loop - a bounded parallel to Phase 6

Phase 6b QA loop (independent 3-pass cap)
─────────────────────────────────────────────────────────
Only runs when Phase 6 exits cleanly (Skeptic sign-off)
    │
qa-engineer verifies acceptance criteria
    │
PASS? ──> Phase 7 (quality gate)
    │
failures? ──> update qa_failures_log ──> Engineer fix pass ──> loop back
    │
cap_reached (iteration == 3) or convergence_failure ──> ESCALATE to human
  • 3-pass cap is independent: exhausting Phase 6 Skeptic cap does not consume Phase 6b QA budget
  • Phase 6b only runs after Phase 6 clean exit - if Phase 6 escalates (cap_reached, convergence_failure, blocked), Phase 6b is skipped entirely
  • qa_failures_log schema: each failure tracked with id, description, first_raised, status, claimed_fix, re_raised - mirrors findings_log structure
  • QA convergence trigger: same failure re-raised unchanged after a claimed fix - no severity qualifier (QA failures are not Critical/Major/Minor; any re-raised failure triggers convergence_failure)
  • Same BLOCKED/NEEDS_CONTEXT handling: Engineer BLOCKED = immediate escalation; NEEDS_CONTEXT = context re-supply without incrementing iteration
Skeptic and QA test orthogonal properties - correctness vs. functional acceptance. The loop budgets reflect this: independent caps, independent state, independent escalation.

QA that compounds

Browser-real verification. Per-project memory. Every run starts smarter.

github.com/Space-Dinosaurs/DinoStack