Skip to content

feat(ids): use queue-scoped decimal resource IDs - #770

Merged
behinddwalls merged 4 commits into
preetam/id-uri-rfcfrom
preetam/numeric-resource-ids
Oct 6, 2026
Merged

feat(ids): use queue-scoped decimal resource IDs#770
behinddwalls merged 4 commits into
preetam/id-uri-rfcfrom
preetam/numeric-resource-ids

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Why?

Queue-prefixed resource IDs contain URL separators and duplicate scope already carried by the queue, route, and typed field. Resource IDs should remain flexible string contracts rather than forcing Go, protobuf, or SQL resource fields to integer types.

What?

  • Store canonical positive decimal strings such as "42" for SubmitQueue request IDs, SubmitQueue batch IDs, and Stovepipe request IDs while keeping resource and reference fields as strings and VARCHAR columns.
  • Keep the existing counter schema and (queue, domain) key unchanged; domain names the sequence (request or batch), not an application. SubmitQueue and Stovepipe have separate storage backends, so no ownerDomain dimension is introduced. Stovepipe uses the durable MySQL counter.
  • Put the shared formatting, validation, and numeric comparison helpers in platform/base/id. Keep domain naming consistent across the counter contract, implementation, callers, mocks, tests, and docs.
  • Validate direct resource-ID inputs and retain the queue-scoped test fixes and cross-queue ID regression coverage from the prior review pass. Provider IDs, URIs, hashes, and derived event IDs keep their contracts.

Test Plan

  • ✅ make build
  • ✅ Latest stack validation: make test (130 targets passed)
  • ✅ make check-gazelle, make check-mocks, make check-tidy, and make lint
  • ✅ git diff --check
  • ✅ git diff --quiet main -- ':(glob)**/schema/*.sql' (no schema differences from main)
  • ✅ Rebased onto main (5b6af68f), preserving the RFC → implementation stack and all seven commits. Runtime code is unchanged by the RFC table update.
  • ✅ Fresh MySQL counter integration run with test-result caching disabled.
  • ✅ The preceding comment pass also ran SubmitQueue gateway integration, Stovepipe integration, and Stovepipe e2e successfully (4/4 targets including the counter suite).
  • ⚠️ aifx verify could not complete its monorepo coverage, generic Go lint, and UReview API checks; repository-native checks above passed.

Stack

  1. docs: define scoped sequential resource IDs #762
  2. @ feat(ids): use queue-scoped decimal resource IDs #770
  3. fix(id): always compare legacy IDs below newly formatted IDs #778

@behinddwalls behinddwalls mentioned this pull request Oct 2, 2026
Merged
@behinddwalls
behinddwalls added this pull request to stack #771 October 2, 2026 20:48
@behinddwalls
behinddwalls marked this pull request as ready for review October 2, 2026 22:15
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners October 2, 2026 22:15
@behinddwalls
behinddwalls force-pushed the preetam/numeric-resource-ids branch from 7bb69d3 to fedc7f9 Compare October 2, 2026 22:41
@behinddwalls
behinddwalls force-pushed the preetam/numeric-resource-ids branch from fedc7f9 to 4ab3cdc Compare October 5, 2026 19:05
@behinddwalls
behinddwalls force-pushed the preetam/numeric-resource-ids branch 2 times, most recently from 567e269 to d1c4280 Compare October 5, 2026 21:41
@behinddwalls
behinddwalls force-pushed the preetam/numeric-resource-ids branch from d1c4280 to 64f60a8 Compare October 5, 2026 23:21
@behinddwalls
feat(ids): use queue-scoped decimal resource IDs
5b79c54 
## Summary

### Why?

Queue-prefixed resource IDs contain URL separators and duplicate scope already carried by the queue, route, and typed field. Resource IDs should remain flexible string contracts rather than forcing Go, protobuf, or SQL resource fields to integer types.

### What?

- Store canonical positive decimal strings such as "42" for SubmitQueue request IDs, SubmitQueue batch IDs, and Stovepipe request IDs while keeping resource and reference fields as strings and VARCHAR columns.
- Persist counter high-water marks by owner domain, queue, and resource kind; wire Stovepipe to the durable MySQL counter so allocation survives restarts and remains isolated across domains.
- Validate direct resource-ID inputs, compare decimal strings numerically where allocation order matters, and update queue contracts, documentation, fixtures, and generated bindings without changing provider IDs, URIs, hashes, or derived event IDs.

## Test Plan

- ✅ make build
- ✅ make test (127 targets passed)
- ✅ make fmt, make gazelle, make mocks, and make tidy; repeated runs were byte-for-byte stable
- ✅ make lint-binary lint-license lint-message-id lint-queue-shard
- ✅ git diff --check
- ⚠️ Docker-backed integration and end-to-end tests were not run because the Docker daemon socket is unavailable.
@behinddwalls
fix(tests): scope resource ID fixtures and queries by queue
2192f75 
## Summary

### Why?

Queue-prefixed resource IDs contain URL separators and duplicate scope already carried by the queue, route, and typed field. Resource IDs should remain flexible string contracts rather than forcing Go, protobuf, or SQL resource fields to integer types.

### What?

- Store canonical positive decimal strings such as "42" for SubmitQueue request IDs, SubmitQueue batch IDs, and Stovepipe request IDs while keeping resource and reference fields as strings and VARCHAR columns.
- Persist counter high-water marks by owner domain, queue, and resource kind; wire Stovepipe to the durable MySQL counter so allocation survives restarts and remains isolated across domains.
- Validate direct resource-ID inputs, compare decimal strings numerically where allocation order matters, and update queue contracts, documentation, fixtures, and generated bindings without changing provider IDs, URIs, hashes, or derived event IDs.
- Address the verified autoreview findings: scope Stovepipe test queries by queue/tenant and topic, use non-colliding decimal gateway fixtures, fail fast on unexpected request-summary RPC errors, repair stale storage assertions, and cover independent queues reusing the same ID.

## Test Plan

- ✅ make build
- ✅ make test (127 targets passed)
- ✅ make check-gazelle, make check-mocks, make check-tidy, and make lint
- ✅ git diff --check
- ✅ Fresh Docker run of all 9 integration targets, Runway e2e, and Stovepipe e2e, including the cross-queue ID regression.
- ⚠️ The complete 12-target integration/e2e run has 11 passing targets. SubmitQueue real-Git e2e still fails because Git rejects the mounted /srv/git/sandbox.git repository as having dubious ownership in this macOS container environment; the non-Git SubmitQueue e2e suite passes.
- ⚠️ aifx verify could not complete its checks: coverage requires a monorepo .arcconfig, the UReview API failed, and the generic Go lint verifier exited without reporting issues. Repository-native checks above passed.
@behinddwalls
refactor(ids): simplify shared helpers and counter scope
857b445 
## Summary

### Why?

Queue-prefixed resource IDs contain URL separators and duplicate scope already carried by the queue, route, and typed field. Resource IDs should remain flexible string contracts rather than forcing Go, protobuf, or SQL resource fields to integer types.

### What?

- Store canonical positive decimal strings such as "42" for SubmitQueue request IDs, SubmitQueue batch IDs, and Stovepipe request IDs while keeping resource and reference fields as strings and VARCHAR columns.
- Keep the existing counter schema and (queue, domain) key unchanged; domain stores the resource type (request or batch). SubmitQueue and Stovepipe have separate storage backends, so no ownerDomain dimension is introduced. Stovepipe uses the durable MySQL counter.
- Put the shared formatting, validation, and numeric comparison helpers in platform/base/id. Use resourceType consistently in Go contracts, mocks, tests, and docs without renaming the existing SQL domain column.
- Validate direct resource-ID inputs and retain the queue-scoped test fixes and cross-queue ID regression coverage from the prior review pass. Provider IDs, URIs, hashes, and derived event IDs keep their contracts.

## Test Plan

- ✅ make build
- ✅ make test (127 targets passed)
- ✅ make check-gazelle, make check-mocks, make check-tidy, and make lint
- ✅ git diff --check
- ✅ git diff --quiet 343875c -- ':(glob)**/schema/*.sql' (no schema differences from the RFC branch)
- ✅ Fresh Docker tests: MySQL counter integration, SubmitQueue gateway integration, Stovepipe integration, and Stovepipe e2e (4/4 targets passed).
- ⚠️ The prior full Docker run passed all 9 integration targets and Runway/Stovepipe e2e; SubmitQueue real-Git e2e remains blocked by the mounted sandbox repository's dubious-ownership check. That unrelated Git configuration was not changed or rerun in this comment pass.
- ⚠️ The prior aifx verify run could not complete its monorepo coverage, generic Go lint, and UReview API checks; repository-native checks above passed.
@behinddwalls
refactor(counter): keep domain as the sequence name
12fca80 
## Summary

### Why?

Queue-prefixed resource IDs contain URL separators and duplicate scope already carried by the queue, route, and typed field. Resource IDs should remain flexible string contracts rather than forcing Go, protobuf, or SQL resource fields to integer types.

### What?

- Store canonical positive decimal strings such as "42" for SubmitQueue request IDs, SubmitQueue batch IDs, and Stovepipe request IDs while keeping resource and reference fields as strings and VARCHAR columns.
- Keep the existing counter schema and (queue, domain) key unchanged; domain names the sequence (request or batch), not an application. SubmitQueue and Stovepipe have separate storage backends, so no ownerDomain dimension is introduced. Stovepipe uses the durable MySQL counter.
- Put the shared formatting, validation, and numeric comparison helpers in platform/base/id. Keep domain naming consistent across the counter contract, implementation, callers, mocks, tests, and docs.
- Validate direct resource-ID inputs and retain the queue-scoped test fixes and cross-queue ID regression coverage from the prior review pass. Provider IDs, URIs, hashes, and derived event IDs keep their contracts.

## Test Plan

- ✅ make build
- ✅ make test (129 targets passed)
- ✅ make check-gazelle, make check-mocks, make check-tidy, and make lint
- ✅ git diff --check
- ✅ git diff --quiet main -- ':(glob)**/schema/*.sql' (no schema differences from main)
- ✅ Rebased onto main (`6884180e`), preserving the RFC → implementation stack and all seven commits. Runtime code is unchanged by the RFC table update.
- ✅ Fresh MySQL counter integration run with test-result caching disabled.
- ✅ The preceding comment pass also ran SubmitQueue gateway integration, Stovepipe integration, and Stovepipe e2e successfully (4/4 targets including the counter suite).
- ⚠️ aifx verify could not complete its monorepo coverage, generic Go lint, and UReview API checks; repository-native checks above passed.
@behinddwalls
behinddwalls force-pushed the preetam/numeric-resource-ids branch from 64f60a8 to 12fca80 Compare October 5, 2026 23:40
@behinddwalls behinddwalls mentioned this pull request Oct 5, 2026
Merged
@behinddwalls
behinddwalls added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit 52e3fc0 Oct 6, 2026
16 of 29 checks passed
@behinddwalls
behinddwalls deployed to stack-rebase October 6, 2026 17:10 — with GitHub Actions Active
@behinddwalls
behinddwalls deleted the preetam/numeric-resource-ids branch October 6, 2026 17:11
Tmwakalasya pushed a commit to Tmwakalasya/submitqueue that referenced this pull request Oct 6, 2026
@behinddwalls
docs: define scoped sequential resource IDs (uber#762)
79d9421 
## Summary

### Why?

Resource identity should stay flexible at storage and API boundaries
even when the current allocator produces sequential numbers.

### What?

Define generated IDs as canonical decimal strings, keep resource and
reference columns as VARCHAR, and limit integer storage to counter
high-water marks.

Replace the URLs and display examples with a before/after table showing
base64url, percent-encoded, and readable queue-scoped routes; leave the
rest of the proposal intact.

## Stack

1. @ uber#762
1. uber#770
1. uber#778
Tmwakalasya pushed a commit to Tmwakalasya/submitqueue that referenced this pull request Oct 6, 2026
@mnoah1
fix(id): always compare legacy IDs below newly formatted IDs (uber#778)
e6e5251 
Update stovepipe's ID comparisons - place all legacy requests before the
new numerical only value. This will avoid existing queues getting wedged
when they hit the boundary between the last legacy request, and first
new one.

## Why?

Without this change, queues would have stalled at the boundary between
the last legacy request bookmark, and the next incoming request in the
new format. The comparison would have continually failed, and never
drained out, because we are trying to compare a request in the old
format, with a request in the new format.

## What?

Adjust the request ID comparator to place all legacy requests before new
ones in ordering.

Within each format, IDs retain numeric ordering. Decimal IDs sort after
all valid legacy IDs; this assumes a one-way writer switch from legacy
to decimal IDs.

## Test Plan

- Updated tests
- During deployment, monitor that new requests continue to advance
beyond the last legacy-formatted request.
- ✅ Rebased the RFC → ID implementation → compatibility stack onto main
(`5b6af68f`), preserving all eight commits and the compatibility patch's
five-file scope.
- ✅ make lint, make check-gazelle, make check-mocks, make check-tidy,
make build, and make test (130 targets passed)
- ✅ git diff --check
- ⚠️ aifx verify could not complete its monorepo coverage, generic Go
lint, and UReview API checks; repository-native checks above passed.

## Stack

1. uber#762
1. uber#770
1. @ uber#778

This branch was successfully deployed

1 active deployment
stack-rebase — 12fca80d Deployed Oct 6, 2026 by behinddwalls via Rebase Stack #563
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mnoah1 mnoah1 mnoah1 approved these changes
@sbalabanov sbalabanov Awaiting requested review from sbalabanov sbalabanov is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@behinddwalls @mnoah1