Skip to content

feat: identify every gitops API request with a User-Agent - #78

Merged
scott-lowe-vapi merged 1 commit into
mainfrom
feat/user-agent-everywhere
Oct 7, 2026
Merged

feat: identify every gitops API request with a User-Agent#78
scott-lowe-vapi merged 1 commit into
mainfrom
feat/user-agent-everywhere

Conversation

@scott-lowe-vapi

@scott-lowe-vapi scott-lowe-vapi commented Oct 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Value

V.A.L.U.E. tier: small — a behavior change: every API request now carries a new header. No blast-radius path.

  • Problem: we can't measure how gitops is used beyond simulations. Only npm run sim and npm run check identify themselves. Setup, pull, push, apply, promote, cleanup, rollback, call and audit send Node's default User-Agent, node, which was 126k of 728k api.vapi.ai requests in one hour this morning. So "how many deploys come from gitops, from CI or from laptops" has no answer.
  • Who it affects: the Vapi team measuring adoption after the public launch. Customers see no change in behavior.
  • What changes:
    • src/user-agent.ts builds vapi-gitops-<command>/<version>, plus (ci) when CI or GITHUB_ACTIONS is set:
      • <command> comes from a fixed list of this repo's commands, so a fork's own npm script names are never sent. The first process pins its label in VAPI_GITOPS_COMMAND, which child processes inherit: npm run apply labels its pull and push apply, a promotion's applies are promote, and the PR check's bindings pull is check. Promotion gate runs are labelled promote, apart from PR check runs.
      • When a script is run directly, as the PR check workflow does, it falls back to the entry script's name.
      • sim and check keep their fixed labels, which existing simulation analytics already counts by.
    • Every fetch to the Vapi API sends it: api.ts (push, pull, apply, promote), cleanup, setup, the interactive pickers, rollback, call, and push's direct fetch.
    • Fixes two tests that called production. cleanup-safety and new-file-gate sent about 12 requests per npm test to api.vapi.ai, with a fake key, getting 401s. That breaks the repo's own rule that tests never call the real API, and with this PR it would have counted every fork's CI run as gitops usage. They now point at a dead local address.
    • Docs: how-it-works.md says exactly what the API sees (and that there is no other telemetry). AGENTS.md says every request must send the header.

Evidence of value

Live, through the Cloudflare request logs in Axiom (cloudflare-logpush): a read-only npm run setup -- ua-check --resources none against the test org, run from a scratch copy:

User-Agent Requests Status
vapi-gitops-setup/1.0.0 13 GET 200
vapi-gitops-test/1.0.0 24 GET 401 — the two leaky tests, from the two npm test runs before the fix

Before this PR, both rows would have been indistinguishable node traffic.

Tests:

  • tests/user-agent-coverage.test.ts reads every fetch( in src/ and requires a User-Agent. On the parent branch it lists 10 call sites without one; here it lists none. It also runs api.ts against a local server with npm_lifecycle_event=apply.
  • tests/user-agent.test.ts pins the format: fixed sim and check labels, npm script vs. entry script vs. cli, label cleaning, and the CI marker for GITHUB_ACTIONS=true, CI=true and CI=1 (but not false, 0 or empty).
  • No more production calls: running the full suite with a fetch trap that records any request to vapi.ai caught 12 requests before the test fix and none after.

Testing plan

  • npm test (527 tests) and npx tsc --noEmit pass.
  • Not covered:
    • Distinct-org counts for non-simulation commands. The request logs carry the User-Agent but not the org. Only simulation runs get an org ID, through PostHog. Joining the two needs an API-side change, outside this repo.
    • The version is still 1.0.0; bumping it is deliberately left out of this PR.
    • The call command's WebSocket audio connection isn't a Vapi REST request and doesn't carry the header.

Refs TEST-141

After review

  • Labels: come only from the known command list, and are pinned for child processes (Chris's 🟠: the PR check's bindings pull was counted as a CI pull). npx and fork script names fall through to the entry script, then to cli.
  • GitHub status call: now sends the User-Agent too, so every fetch in src/ does. The coverage test checks each call on its own and scans subfolders.
  • tests/no-vapi-api.ts: loads before every test file, with an unroutable base URL and no inherited real key. With a pretend key exported and a fetch trap on, the suite makes zero requests to vapi.ai.
  • Docs: how-it-works.md matches the code on (ci) and <command>.
  • Formatting: the unrelated Prettier changes are reverted.

🤖 Generated with Claude Code

This was referenced Oct 3, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
@scott-lowe-vapi Graphite App

scott-lowe-vapi commented Oct 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@scott-lowe-vapi scott-lowe-vapi mentioned this pull request Oct 3, 2026
Merged
@scott-lowe-vapi
scott-lowe-vapi marked this pull request as ready for review October 3, 2026 07:59
chris-garber-vapi
chris-garber-vapi reviewed Oct 6, 2026
View reviewed changes

@chris-garber-vapi chris-garber-vapi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aggressive review: 1 🟠, 3 🟡, 4 🟢, no blockers.

The main problem: the label doesn't reach child processes when a command isn't started through npm run, so PR check runs are counted as CI pulls (🟠 on user-agent.ts:61). The fix for that also keeps the label set bounded (🟡 on :63).

Every comment says what I checked. Every fix was tried on a copy of 674d8a1: tsc and npm test pass.

Not part of this diff, so no inline comment: under npm run promote, the promotion gate's simulation runs still send vapi-gitops-check/… (promotion-gate.ts:90). That means check-run counts include promotion gates.

Comment thread src/user-agent.ts
Comment thread src/user-agent.ts Outdated
Comment thread src/user-agent.ts Outdated
Comment thread tests/user-agent-coverage.test.ts Outdated
Comment thread tests/user-agent-coverage.test.ts Outdated
Comment thread tests/cleanup-safety.test.ts
Comment thread docs/guides/how-it-works.md Outdated
Comment thread src/call.ts Outdated
@scott-lowe-vapi
scott-lowe-vapi force-pushed the fix/validate-references branch from 1f982af to 908d125 Compare October 6, 2026 22:50
@scott-lowe-vapi
scott-lowe-vapi force-pushed the feat/user-agent-everywhere branch from 674d8a1 to 5d90e98 Compare October 6, 2026 22:50
@scott-lowe-vapi

scott-lowe-vapi commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

@chris-garber-vapi on the gate label from your summary: done. Promotion gate runs now send vapi-gitops-promote/…, so PR check counts no longer include gate runs. The sim and check prefixes are unchanged.

🤖 Generated with Claude Code

chris-garber-vapi
chris-garber-vapi reviewed Oct 7, 2026
View reviewed changes
Comment thread src/cleanup.ts
chris-garber-vapi
chris-garber-vapi approved these changes Oct 7, 2026
View reviewed changes

@chris-garber-vapi chris-garber-vapi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but it might be worth looking into if we should use X-Client-Source instead. I'm not sure how much value we lose from a debugging standpoint by overriding this field. Right now, the only values accepted by the api for X-Client-Source our dashboard and composer: I think it's worth extending that enum to include get apps, but also further extending it to parse x-client-version and x-client-action to capture some of the data that you've got here.

chris-garber-vapi
chris-garber-vapi reviewed Oct 7, 2026
View reviewed changes

@chris-garber-vapi chris-garber-vapi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing a little more, I like these changes, I think we should do the X-Client-Source (and other headers) as a follow up.

@scott-lowe-vapi Graphite App

scott-lowe-vapi commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Merge activity

  • Oct 7, 5:47 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Oct 7, 6:11 PM UTC: Graphite rebased this pull request as part of a merge.
  • Oct 7, 6:11 PM UTC: @scott-lowe-vapi merged this pull request with Graphite.

scott-lowe-vapi added a commit that referenced this pull request Oct 7, 2026
@scott-lowe-vapi
feat(promotion): gate promotion out of an org on a passing check (#66)
b8115c0 
## Value

**V.A.L.U.E. tier:** project — PR 10 of 10 for inline simulation PR checks ([TEST-141](https://linear.app/vapi/issue/TEST-141/gitops-run-simulation-suites-against-pr-changes-inline-as-ci-checks)), the "check before deploy" step in promotion.

> Stacked on #65 (the promotion partial-failure fix), now that #57–#64 have merged. This PR is the single gate commit.

- **Problem:** promotion copies staging's reviewed files into production, but nothing checks that staging's agents still behave before they move on. Teams promoting dev → staging → prod need a behaviour gate between orgs, without new infrastructure.
- **Who it affects:** multi-org gitops users (the promotion pipeline), who get a "check before deploy" step with one line of `promotion.yml`. Single-org users are unaffected.
- **What changes:**
  - **`promotion.yml`** accepts `orgs.<slug>.check: <name>` (a slug), naming a `vapi-checks.yml` check.
  - **New `src/promotion-gate.ts`:**
    - **Validation before any transition:** the check must exist, and its `org` and `runOrg` must be the gated org, otherwise the run errors. `vapi-checks.yml` is required once any org is gated.
    - **Plan line:** what the gate would run, built offline.
    - **Live gate:** the check runs live and reduces to the worst target result.
  - **`src/promote-cmd.ts`:** in each transition, after the plan is built:
    - no changes skips the gate;
    - plan-only prints `check  would run <name> in <org> (<n> simulations × <t> targets)`;
    - `--apply` runs the check (after the bindings refresh, before `promotionPlanApply` writes anything). Any non-pass throws `Promotion out of <org> blocked: check <name> <outcome> (<run url>)`.
    - A pass is cached per source org and dropped once a transition applies into that org.
    - `promotionCommandRun(args, overrides)` now takes `Partial<PromotionDeps>` (`childRun`, `checkRun`).
  - **`.github/workflows/promotion.yml`:** `timeout-minutes: 90` on the "Reconcile configured promotions" **step**, not the job, so the `if: always()` commit step (fixed in #65) still runs after a blocked or slow gate.
  - **Docs:** `promotion.example.yml` (a commented `check:`), a README "Check before promoting" section, and a pointer from "PR Checks".

## Evidence of value

**The real gate, run live** in the owner's test org on the TEST-141 parity squad.
- **Setup:** a scratch repo whose `promotion.yml` gates `parity` on check `core`, with pipeline `parity → parity-prod`.
- **The run:** `promote --pipeline release --from parity --to parity-prod --apply`.
- **The fake:** the child runner was faked, so bindings pulls were no-ops and the downstream `apply.ts` was recorded but not run. No second org was needed or touched.

| Variant | Gate run | Result | Downstream apply | `resources/parity-prod/` |
|---|---|---|---|---|
| Degraded scheduler prompt | [7ed19587](https://dashboard.vapi.ai/simulations/run/7ed19587-d1ca-4d44-9232-7cdd15a50d67): 2 of 3 failed | `Promotion out of parity blocked: check core failed (https://dashboard.vapi.ai/simulations/run/7ed19587-…)` | **none** | **empty** (nothing written) |
| Fixture as-is | [95470670](https://dashboard.vapi.ai/simulations/run/95470670-2861-45d3-a483-7a220fc3591a): 3 of 3 passed | promoted | `["parity-prod"]` | written; 20 applied paths recorded |

The test org's resource counts were identical before and after both gate runs.

**Tests:** `npm test` goes from 484 (#65) to 492 passing, and #68's golden promotion test passes unchanged.

## Testing plan

- **`tests/promotion-gate.test.ts`** (6 tests, real git fixture, injected `childRun` / `checkRun`):
  - a pass applies;
  - failed and incomplete both block with the exact message, with no apply and the target untouched;
  - plan-only prints the line and runs nothing;
  - no changes skips the gate;
  - the three config errors (no `vapi-checks.yml`, unknown check, check in another org) stop before anything applies;
  - the pass cache: reused for two pipelines out of one org, and re-run after a transition applies into the gated org.
- **No gate configured, no change:** with no `check:` in `promotion.yml` and an **invalid** `vapi-checks.yml` present, plan and `--apply` both succeed, `checkRun` is never called, and the plan output equals a pinned string. That string is exactly what #65's code (before the gate existed) prints for the same fixture, which I confirmed by running #65's `promote-cmd` on it. So the gate is invisible unless someone opts in.
- **`tests/promotion.test.ts`:** `orgs.<slug>.check` is parsed, and a non-slug is rejected.
- **Not tested:**
  - **A real two-org promotion:** only one test org was available. The downstream apply was faked, so the blocked case shows nothing written, and the pass case shows the apply was called.
  - **A GitHub Actions promotion run with a gate**, including the step timeout firing.
- **Found while testing (pre-existing, out of scope):** promotion's dependency check rejects simulations that reference a **stock personality by UUID**, with "Referenced managed dependency is missing from source: personalities/a0000000-…". So a gated org's tests need local personality files until that's fixed.

Stacked on #65.

Refs TEST-141

## After review

The gate now refuses, when the config loads and before anything applies:

- an unknown key under an org in `promotion.yml`, so a misspelled `check:` can't silently drop the gate;
- `toolMocks: off` and `stripWebhooks: false`, because a gate runs in the real org, never a CI org;
- a check `baseUrl` that differs from the org's `baseUrl` in `promotion.yml`, so the org's key only goes to the host promotion uses (the gate always uses that host);
- a gate on an org that is last in every pipeline, where it would never run;
- gated checks whose combined budget is over 300 minutes.

It also fixes:

- **Deadline:** each batch of 3 targets gets a full `timeoutMinutes`, so a check with more than 3 targets is no longer falsely blocked as incomplete.
- **Step timeout:** raised from 90 to 330 minutes, as a safety net that no longer cuts short long ungated promotions.
- **Tests:** the deadline and the worst-target rule are pure helpers with their own tests.

The guide changes (blocks stop the whole run, simulation cost, the stock-personality limitation, accurate wording) are in #71. The `promote` User-Agent for gate runs is in #78. The block-report detail, fetch-stubbed gate test and deduplication are follow-ups.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@scott-lowe-vapi
scott-lowe-vapi changed the base branch from fix/validate-references to graphite-base/78 October 7, 2026 18:08
@scott-lowe-vapi
scott-lowe-vapi changed the base branch from graphite-base/78 to main October 7, 2026 18:09
@scott-lowe-vapi @claude
feat: identify every gitops API request with a User-Agent
6f0c59f 
Only `npm run sim` and `npm run check` identified themselves. Setup,
pull, push, apply, promote, cleanup, rollback, call and audit sent Node's
default `node` User-Agent, about a sixth of all api.vapi.ai traffic, so
gitops usage beyond simulations couldn't be counted.

- src/user-agent.ts: `vapi-gitops-<command>/<version>`, plus ` (ci)`
  when GITHUB_ACTIONS=true or CI is set (not false or 0). The command
  comes from a fixed list of this repo's commands, so a fork's own npm
  script names are never sent and npx is not a label; otherwise the entry
  script names it, else `cli`. The first process pins its label in
  VAPI_GITOPS_COMMAND, which spawned processes inherit, so the PR check's
  bindings pull is labelled check and `npm run apply`'s pull and push are
  labelled apply. sim and check keep their labels; promotion gate runs
  are labelled promote, apart from PR check runs.
- Every fetch sends it, the GitHub status call included.
  tests/user-agent-coverage.test.ts checks each call on its own, scans
  src/ recursively, and checks api.ts against a local server.
- cleanup-safety and new-file-gate tests sent about a dozen requests to
  the real api.vapi.ai per `npm test` (fake key, 401s), which the new
  User-Agent made visible in the request logs. tests/no-vapi-api.ts now
  loads before every test file: an unroutable base URL and no inherited
  real key, for the tests and every CLI they spawn.
- how-it-works.md says exactly what the API sees; AGENTS.md says every
  request sends the header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scott-lowe-vapi
scott-lowe-vapi force-pushed the feat/user-agent-everywhere branch from 5d90e98 to 6f0c59f Compare October 7, 2026 18:10
@scott-lowe-vapi
scott-lowe-vapi merged commit a67a4b1 into main Oct 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chris-garber-vapi chris-garber-vapi chris-garber-vapi approved these changes
@dhruva-vapi dhruva-vapi Awaiting requested review from dhruva-vapi
@vtkovapi vtkovapi Awaiting requested review from vtkovapi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@scott-lowe-vapi @chris-garber-vapi