ci: validate every org's resources on every pull request

[scott-lowe-vapi]

Value

V.A.L.U.E. tier: small — touches .github/workflows/ (a blast-radius path), and changes what customer forks see on their pull requests.

• Problem: nothing ran npm run validate before merge. apply, and so promotion, refuses to deploy a config with validation errors: a name over 40 characters, or a per-provider voice schema error. (The validator's other rules, structured-output lockstep, duplicated prompts and the maxTokens floor, are warnings and don't fail it.) That refusal came after merge: main held a config that wouldn't deploy, and promotion stopped until someone opened a fix PR. Plain push only warns, then can fail partway with an API 400. Each rule in the validator comes from a real mid-push failure (improvements.md membersOverrides.artifactPlan.structuredOutputIds is requiring UUID #8, Specifying handoff tools in a squad requires UUID to function correctly #9, fix(call): clear wrapped partial transcripts cleanly in npm run call #11, feat: simulation suite runner (npm run sim) #18, refactor: state schema with per-resource content hashes #19).
• Who it affects: every repository on this template, and especially multi-org repos that promote from main.
• What changes:
  • A new Validate resources job in ci.yml runs validate for every folder under resources/ on every pull request. It reports every failing org rather than stopping at the first.
  • No engine change. validate makes no network call; loading the engine's config only requires a key to be set, so the step sets a placeholder key and an unroutable base URL, so nothing can be sent. The job has no secrets, so it runs the same on forks and Dependabot PRs.
  • It's in ci.yml, not the PR check workflow, so every fork gets it without turning on PR checks. On this template, which has no org folders, it does nothing.
  • Docs:
    • the README quick start;
    • the AGENTS.md change loop: if the check fails, fix the errors and don't weaken the check;
    • the workflows guide: make it a required check;
    • the PR checks and troubleshooting guides;
    • improvements.md Add GitOps Support for Pronunciation Dictionaries with Versioned Updates #37.

Heads-up for forks: plain push only warned about these errors, so a repo may already carry some. The first PR after this lands will show them, whatever it changes. The troubleshooting guide covers it. apply already refused those configs, so this moves an existing failure earlier rather than adding a new one.

Evidence of value

tests/ci-validate-workflow.test.ts runs the job's real step, read from ci.yml, against copies of the starter example:

Case	Result
No org folders	passes, "nothing to validate"
Two valid orgs	passes, both validated
One org with a 41+ character assistant name, one valid	fails; both validated, the error names only the bad org and the reason ("Vapi caps at 40")
A folder that isn't a valid org name	fails, naming it
The job's secrets	none; checkout doesn't persist credentials; the only key is the placeholder

Mutation: making the loop ignore validate's exit code fails the two failure-case tests.

Testing plan

• npm test (514 tests) and npx tsc --noEmit pass.
• actionlint, from ci: lint workflows and test the PR check's key-sharing decision #75, lints the new job in this PR's CI.
• Not tested: a customer repo with real resources. The validator itself is unchanged, and apply already runs it on every deploy.

Refs TEST-141

After review

• Install: npm ci --ignore-scripts, which skips the native audio builds validate never loads.
• Reproducing locally: the failure message, the troubleshooting guide and AGENTS.md give a placeholder-key command, so nobody fetches a production key for an offline check.
• Docs:
  • workflows.md says that requiring the check ties every team's PRs to every org's health, and that merge queues need a merge_group: trigger;
  • CI validates .ts resources without .env.<org>;
  • "the same validator apply runs" instead of "the same checks".
• Starter example: its one-sided structured-output link is fixed, and the test asserts it validates with no warnings.
• Tests: the no-secrets test also checks the job's permissions and that pull_request_target isn't a trigger.
• improvements.md Add GitOps Support for Pronunciation Dictionaries with Versioned Updates #37: reworded (errors vs. warnings) and carries the PR number.

Annotations for warnings come from #77, so I didn't add the sed version here.

🤖 Generated with Claude Code

[scott-lowe-vapi]

• feat: identify every gitops API request with a User-Agent #78
• feat(validate): catch broken references and show findings on the PR #77
• ci: validate every org's resources on every pull request #76 👈 (View in Graphite)
• ci: lint workflows and test the PR check's key-sharing decision #75
• test(docs): check that doc links, commands and flags exist #74
• fix(cleanup): never delete resources matched by .vapi-ignore #73
• docs(agents): one compact guide for Claude, Codex and Cursor #72
• docs: edit the guides for new readers and fix inaccuracies #71
• docs: restructure the README into a short landing page and topic guides #70
• docs: prepare for public release: scrub internal names, fix examples, add community files #69
• feat(promotion): gate promotion out of an org on a passing check #66
• fix(promotion): commit the files of transitions that applied when a later one fails #65
• test(promotion): pin the exact files promoting an assistant writes to the target #68
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[chris-garber-vapi]

I think running validate in CI makes sense

[chris-garber-vapi]

Review of the Validate resources job. Nothing here blocks merge (no 🔴 or 🟠). The main themes: the docs overstate what fails (3 of the 5 named rules are warnings, and they stay hidden), the local repro path needs a real org key, env-dependent .ts resources validate differently in CI than under apply, and every org being validated on every PR widens the blast radius. Suggestion blocks were run locally against the PR head unless noted.

[scott-lowe-vapi]

Merge activity

• Oct 7, 5:47 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Oct 7, 6:07 PM UTC: Graphite rebased this pull request as part of a merge.
• Oct 7, 6:07 PM UTC: @scott-lowe-vapi merged this pull request with Graphite.