feat(validate): catch broken references and show findings on the PR

[scott-lowe-vapi]

Value

V.A.L.U.E. tier: small — a behavior change: validate, and so apply and the Validate resources check, now fail on configs they used to pass.

• Problem: a reference that names no file fails in one of three ways, depending on the field (improvements.md docs: document orphan-YAML gate + --allow-new-files in README and AGENTS #31):

  • it's silently dropped: model.toolIds, artifactPlan.structuredOutputIds;
  • it's sent raw and rejected partway through a push: squad members, hook tools, personalityId, scenarioId;
  • or it's deferred to a linking pass.

  validate never checked references, so the Validate resources check from ci: validate every org's resources on every pull request #76 couldn't catch a typo'd tool name either. Warnings were also invisible in CI: they don't fail the check, and nobody reads the job log.

• Who it affects: everyone who edits resource files by hand or with a coding agent, and reviewers of their PRs.

• What changes:

  • New src/validate-refs.ts, run by validate (so by apply and CI) and by push:

    Rule	Severity	Catches
    dangling-reference	error	A name with no local file and no state entry. Uses the shared reference walk, plus scenario judges' evaluations[].structuredOutputId.
    malformed-reference	error	An empty or non-name entry in a reference list (it used to crash validate).
    override-tool-by-name	error	A tool name in toolIds inside assistantOverrides, membersOverrides or targetOverrides, where push never resolves names. The fix it gives is tools:append, because model.tools there replaces the member's tool set.
    unresolved-credential	warning	A credential name missing from the state file. The message names the org's bootstrap pull.
    reference-by-uuid	warning	A UUID for a resource this repo tracks, which breaks promotion; it names the file to use. UUIDs the repo doesn't track (dashboard-owned, stock personalities) aren't reported.

  • validate now also runs reference-to-ignored, as push already did. It reads the committed state file and stays offline.

  • On GitHub Actions, every finding becomes an annotation, so it shows on the file in the PR, warnings included.

  • push reports the new rules alongside its existing validators: warnings by default, blocking under --strict.

  • Docs:

    • a rule-by-rule table in troubleshooting;
    • the validate row in the commands guide;
    • AGENTS.md: never edit the state file to make a reference resolve;
    • improvements.md docs: document orphan-YAML gate + --allow-new-files in README and AGENTS #31 marked resolved by validation.

Evidence of value

The starter example with two typos, scheduler → schedular in the squad and booking-confirmed → booking-confirmd in a judge:

	Before (#76)	After
npm run validate	0 error(s) — ✅ Validation passed	2 error(s), one dangling-reference per typo, naming the file and the missing name

• New tests/validate-refs.test.ts covers:
  • names that resolve to local files and to state;
  • a typo in each reference field;
  • ignored references;
  • UUIDs and stock personalities;
  • all three override keys;
  • credentials as names, as UUIDs, and known to state.
• Annotation format: escaping of %, newlines, : and , is tested in tests/validate.test.ts.
• End to end: tests/ci-validate-workflow.test.ts runs the CI step with GITHUB_ACTIONS=true on the typo'd squad. The step fails, and the ::error points at resources/clinic/squads/front-desk.yml.
• Mutation: dropping the judge collection or the override walk fails two tests.
• Every example org still passes. The cross-org promotion example (which has no state files) now shows an unresolved-credential warning, which is accurate.

Testing plan

• npm test (523 tests) and npx tsc --noEmit pass.
• Behavior to expect: apply validates before it pulls. So a reference to a resource created in the dashboard and never pulled now stops apply; the message says to pull first. Before, apply went on to pull and push, and the reference resolved only if the pull happened to produce that exact name.
• Not tested: a live apply or push. Neither code path changed except for the added findings.
• Not in this PR: a check for secrets committed in resource files. It goes in its own PR, because a false positive there would block a customer's merge.

Refs TEST-141

After review

• Ignored files: validate skips .vapi-ignored files, as push does, so ignored files can't fail CI or apply.
• One collector: referencesCollect in resolver.ts feeds both reference-to-ignored and these rules, so a judge that references an ignored structured output is caught.
• Lookups: use ?.uuid, so constructor and friends don't resolve.
• Troubleshooting: the table gives safe advice for ignored and override references, and AGENTS.md says never to edit the state file or .vapi-ignore to make a reference resolve.
• improvements.md: docs: document orphan-YAML gate + --allow-new-files in README and AGENTS #31 is marked mitigated, with the remaining gaps listed (unchecked override paths, toolRefs, inline members, annotation lines). feat(drift): three-way drift-direction classifier + content-drift audit + canonicalization fixes #38 records the promotion and stock-personality limitation found in feat(promotion): gate promotion out of an org on a passing check #66.

Follow-ups: line numbers on annotations, the wider override walk, and deduplicating the credential walkers.

🤖 Generated with Claude Code

[scott-lowe-vapi]

• feat: identify every gitops API request with a User-Agent #78
• feat(validate): catch broken references and show findings on the PR #77 👈 (View in Graphite)
• ci: validate every org's resources on every pull request #76
• ci: lint workflows and test the PR check's key-sharing decision #75
• test(docs): check that doc links, commands and flags exist #74
• fix(cleanup): never delete resources matched by .vapi-ignore #73
• docs(agents): one compact guide for Claude, Codex and Cursor #72
• docs: edit the guides for new readers and fix inaccuracies #71
• docs: restructure the README into a short landing page and topic guides #70
• docs: prepare for public release: scrub internal names, fix examples, add community files #69
• feat(promotion): gate promotion out of an org on a passing check #66
• fix(promotion): commit the files of transitions that applied when a later one fails #65
• test(promotion): pin the exact files promoting an assistant writes to the target #68
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[chris-garber-vapi]

Aggressive review pass. Each finding was checked against the code, and most were reproduced by running this branch's validators on the input described in the comment.

Severity: 🔴 blocker · 🟠 fix before merge · 🟡 should fix (a stacked PR is fine where noted) · 🟢 nit

• 🟠 ×3: validate checks .vapi-ignored files that push skips, which blocks CI and apply. The override-tool-by-name fix advice (model.tools) replaces the member's tool set (message and docs row).
• 🟡 ×6:
  • Judge references to ignored structured outputs pass both rules.
  • A non-string toolIds entry crashes validate.
  • Override, toolRefs and inline-assistant references aren't checked.
  • reference-by-uuid fires on every PR for dashboard-owned resources.
  • Annotations have no line.
  • The troubleshooting row tells readers to un-ignore resources.
• 🟢 ×3: prototype-key lookup, duplicated credential walk, #31 status wording.

No 🔴. Tests and tsc pass on the branch as-is.

[scott-lowe-vapi]

Merge activity

• Oct 7, 5:47 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Oct 7, 6:09 PM UTC: Graphite rebased this pull request as part of a merge.
• Oct 7, 6:09 PM UTC: @scott-lowe-vapi merged this pull request with Graphite.