From 928eb0fe202a405d3397046e0fa8a267dad99f19 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Wed, 30 Sep 2026 13:58:57 -0700 Subject: [PATCH 1/3] docs: define scoped sequential resource IDs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary ### Why? Slash-delimited resource IDs repeat queue and resource-kind context that stores, APIs, and messages already carry separately, while making a single ID span multiple browser path segments. ### What? Define counter-generated resource IDs as positive numeric values scoped by owner domain, queue, and resource kind. Persist durable counter high-water marks, store numeric IDs directly under queue-leading keys, keep queue and kind out of the ID, and reserve prefixes such as `request.42` for presentation only. ## Test Plan ✅ `make fmt` ✅ `make lint-binary lint-license lint-message-id lint-queue-shard` ✅ `git diff --check` --- doc/rfc/index.md | 1 + doc/rfc/scoped-resource-ids.md | 73 ++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) create mode 100644 doc/rfc/scoped-resource-ids.md diff --git a/doc/rfc/index.md b/doc/rfc/index.md index 4122aff7..f27eaaf0 100644 --- a/doc/rfc/index.md +++ b/doc/rfc/index.md @@ -10,6 +10,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting - [Consumer Gate](consumer-gate.md) - Stopping and starting individual queue controllers at runtime via a consumer-side check: blocked deliveries are recorded as parked and postponed back to the queue (re-checked on redelivery), gate state as a separate extension with a file-based first implementation shared by tests and operators - [Consumer Hold](consumer-hold.md) - Fourth delivery outcome letting a controller postpone its delivery: the message becomes a partition barrier that pauses consumption for a chosen delay, redelivers in order, and does not count as a failure toward dead-lettering - [Change URIs](change-uri.md) - Identity of a code change: `scheme://{host[:port]}/{path}` per provider (GitHub PR, Phabricator Diff, git ref/commit) and canonical-form rules +- [Scoped Sequential Resource IDs](scoped-resource-ids.md) - Queue-scoped positive numeric IDs allocated by durable per-domain, per-kind counters, stored without queue/kind prefixes, and rendered directly in resource URL segments - [Hooks Framework](hook-framework.md) - Implemented fire-and-forget side effects: one shared `HookEvent` contract (`api/base/hook/`) on a durable per-domain hook topic, dispatched by `platform/hook` to `platform/extension/hook`. Stovepipe `process` and `record` publish repository events; the SubmitQueue orchestrator registers the stage and does not publish events yet - [Service-Scoped Extensions](service-scoped-extensions.md) - Implemented for SubmitQueue storage: gateway and orchestrator aggregates, schemas, and the core packages that serve one service have moved, while store contracts stay at `submitqueue/extension/storage`. Domain-level `buildrunner`, `conflict`, and `speculation` have not moved, and `changeset` still declares its own store slice diff --git a/doc/rfc/scoped-resource-ids.md b/doc/rfc/scoped-resource-ids.md new file mode 100644 index 00000000..af745253 --- /dev/null +++ b/doc/rfc/scoped-resource-ids.md @@ -0,0 +1,73 @@ +# Scoped Sequential Resource IDs + +## Status + +Proposed. + +## Decision + +A generated resource ID is the positive `int64` returned by a durable counter scoped to `(owner domain, queue, resource kind)`. + +| Resource | Current ID | Proposed ID | Complete identity | +|---|---:|---:|---| +| SubmitQueue request | `demo-queue/42` | `42` | `(submitqueue, demo-queue, request, 42)` | +| SubmitQueue batch | `demo-queue/batch/7` | `7` | `(submitqueue, demo-queue, batch, 7)` | +| Stovepipe request | `request/monorepo/main/42` | `42` | `(stovepipe, monorepo/main, request, 42)` | + +The numeric ID is unique only within its scope. The same value may appear in another queue, resource kind, or domain. APIs and messages therefore carry the queue separately; their typed field or message type supplies the resource kind. + +Do not embed scope into the ID. Forms such as `demo-queue/42`, `demo-queue/batch/7`, `request.42`, and ARN-like resource names are not stored or accepted as IDs. + +## Counter + +The counter backend persists one high-water mark per `(owner domain, queue, resource kind)`. For example: + +```text +(submitqueue, demo-queue, request) -> 42 +(submitqueue, demo-queue, batch) -> 7 +(stovepipe, demo-queue, request) -> 11 +``` + +Controllers allocate an ID before creating the resource; stores accept the caller-supplied ID and never generate one. + +- The first ID is `1`; `0` is the unset value. +- Allocation is atomic across replicas and durable across restarts. +- Allocated values are never reused. Failed writes may leave gaps. +- Overflow fails instead of wrapping. +- Numeric order is allocation order only within the same scope. + +The counter contract requires an atomic durable increment, not MySQL specifically. MySQL remains the initial implementation. + +## Storage and contracts + +Resource tables store the numeric value directly. Queue remains the leading key: + +```text +request(queue, id BIGINT, ...) PRIMARY KEY (queue, id) +batch(queue, id BIGINT, ...) PRIMARY KEY (queue, id) +``` + +Reference columns use the same numeric type. Domain entities use distinct named types such as `RequestID` and `BatchID`, and protobuf resource fields use `int64`. + +This proposal applies only to counter-generated resources. Provider build IDs, message and hook IDs, change URIs, and content hashes keep their existing contracts. + +## URLs and display + +The decimal ID is used directly as one path segment: + +```text +/requests/42 +/batches/7 +``` + +The route supplies the resource kind; the request context supplies the queue. Queue URL design is separate. + +A UI may display `request.42`, `batch.7`, or `#42`, but those are derived labels, not identities. + +## Rejected alternatives + +- **Queue or kind prefixes:** duplicate explicit context, lengthen keys, require parsing, and introduce URL separators. +- **ARN-like names:** solve global lookup, which current APIs neither provide nor require. +- **UUIDs or a global counter:** provide global uniqueness at the cost of unnecessary encoding or coordination. +- **SQL auto-increment or `MAX(id) + 1`:** move allocation into one storage implementation or fail under concurrency. +- **Process-local counters:** reuse IDs after restart and collide across replicas. From 79882506b5d2163773a6390a8fb73c005a809a78 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Fri, 2 Oct 2026 12:58:05 -0700 Subject: [PATCH 2/3] docs: keep scoped resource IDs string-valued ## Summary ### Why? Resource identity should stay flexible at storage and API boundaries even when the current allocator produces sequential numbers. ### What? Define generated IDs as canonical decimal strings, keep resource and reference columns as VARCHAR, and limit integer storage to counter high-water marks. --- doc/rfc/scoped-resource-ids.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/doc/rfc/scoped-resource-ids.md b/doc/rfc/scoped-resource-ids.md index af745253..99122361 100644 --- a/doc/rfc/scoped-resource-ids.md +++ b/doc/rfc/scoped-resource-ids.md @@ -6,15 +6,15 @@ Proposed. ## Decision -A generated resource ID is the positive `int64` returned by a durable counter scoped to `(owner domain, queue, resource kind)`. +A generated resource ID is the canonical decimal string for a positive value returned by a durable counter scoped to `(owner domain, queue, resource kind)`. | Resource | Current ID | Proposed ID | Complete identity | |---|---:|---:|---| -| SubmitQueue request | `demo-queue/42` | `42` | `(submitqueue, demo-queue, request, 42)` | -| SubmitQueue batch | `demo-queue/batch/7` | `7` | `(submitqueue, demo-queue, batch, 7)` | -| Stovepipe request | `request/monorepo/main/42` | `42` | `(stovepipe, monorepo/main, request, 42)` | +| SubmitQueue request | `demo-queue/42` | `"42"` | `(submitqueue, demo-queue, request, "42")` | +| SubmitQueue batch | `demo-queue/batch/7` | `"7"` | `(submitqueue, demo-queue, batch, "7")` | +| Stovepipe request | `request/monorepo/main/42` | `"42"` | `(stovepipe, monorepo/main, request, "42")` | -The numeric ID is unique only within its scope. The same value may appear in another queue, resource kind, or domain. APIs and messages therefore carry the queue separately; their typed field or message type supplies the resource kind. +The decimal ID is unique only within its scope. The same value may appear in another queue, resource kind, or domain. APIs and messages therefore carry the queue separately; their typed field or message type supplies the resource kind. Do not embed scope into the ID. Forms such as `demo-queue/42`, `demo-queue/batch/7`, `request.42`, and ARN-like resource names are not stored or accepted as IDs. @@ -40,14 +40,14 @@ The counter contract requires an atomic durable increment, not MySQL specificall ## Storage and contracts -Resource tables store the numeric value directly. Queue remains the leading key: +Resource tables keep IDs as strings. Queue remains the leading key: ```text -request(queue, id BIGINT, ...) PRIMARY KEY (queue, id) -batch(queue, id BIGINT, ...) PRIMARY KEY (queue, id) +request(queue, id VARCHAR(...), ...) PRIMARY KEY (queue, id) +batch(queue, id VARCHAR(...), ...) PRIMARY KEY (queue, id) ``` -Reference columns use the same numeric type. Domain entities use distinct named types such as `RequestID` and `BatchID`, and protobuf resource fields use `int64`. +Reference columns use the same string type. Domain entities may use distinct named string types such as `RequestID` and `BatchID`; protobuf resource fields remain `string`. The counter backend may store its high-water marks as integers, and controllers convert allocated values to canonical decimal strings before creating resources. This proposal applies only to counter-generated resources. Provider build IDs, message and hook IDs, change URIs, and content hashes keep their existing contracts. @@ -69,5 +69,6 @@ A UI may display `request.42`, `batch.7`, or `#42`, but those are derived labels - **Queue or kind prefixes:** duplicate explicit context, lengthen keys, require parsing, and introduce URL separators. - **ARN-like names:** solve global lookup, which current APIs neither provide nor require. - **UUIDs or a global counter:** provide global uniqueness at the cost of unnecessary encoding or coordination. +- **Integer resource fields:** couple the persisted and wire contracts to the current counter representation without adding identity semantics. - **SQL auto-increment or `MAX(id) + 1`:** move allocation into one storage implementation or fail under concurrency. - **Process-local counters:** reuse IDs after restart and collide across replicas. From e6bfaa0cc0a7018bb1c05b091619b1234990c702 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Mon, 5 Oct 2026 14:34:35 -0700 Subject: [PATCH 3/3] docs: show before-and-after resource URLs ## Summary ### Why? Resource identity should stay flexible at storage and API boundaries even when the current allocator produces sequential numbers. ### What? Define generated IDs as canonical decimal strings, keep resource and reference columns as VARCHAR, and limit integer storage to counter high-water marks. Replace the URLs and display examples with a before/after table showing base64url, percent-encoded, and readable queue-scoped routes; leave the rest of the proposal intact. --- doc/rfc/scoped-resource-ids.md | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/doc/rfc/scoped-resource-ids.md b/doc/rfc/scoped-resource-ids.md index 99122361..e408459e 100644 --- a/doc/rfc/scoped-resource-ids.md +++ b/doc/rfc/scoped-resource-ids.md @@ -1,9 +1,5 @@ # Scoped Sequential Resource IDs -## Status - -Proposed. - ## Decision A generated resource ID is the canonical decimal string for a positive value returned by a durable counter scoped to `(owner domain, queue, resource kind)`. @@ -53,16 +49,14 @@ This proposal applies only to counter-generated resources. Provider build IDs, m ## URLs and display -The decimal ID is used directly as one path segment: - -```text -/requests/42 -/batches/7 -``` - -The route supplies the resource kind; the request context supplies the queue. Queue URL design is separate. +| Before — base64url or percent-encoded | After — readable | +|---|---| +| `/queues/demo-queue/requests/ZGVtby1xdWV1ZS80Mg`
or `/queues/demo-queue/requests/demo-queue%2F42` | `/queues/demo-queue/requests/42` | +| `/queues/demo-queue/batches/ZGVtby1xdWV1ZS9iYXRjaC83`
or `/queues/demo-queue/batches/demo-queue%2Fbatch%2F7` | `/queues/demo-queue/batches/7` | +| `/queues/demo-queue/changes/Z2l0aHViOi8v…`
or `/queues/demo-queue/changes/github%3A%2F%2Fgithub.com%2Fuber%2Frepo%2Fpull%2F123%2F{sha}` | `/queues/demo-queue/changes/github/github.com/uber/repo/pull/123` | +| `/queues/demo-queue/changes/cGhhYjovL3BoYWIuZXhhbXBsZS5jb20vRDEyMzQ1LzY3ODkw`
or `/queues/demo-queue/changes/phab%3A%2F%2Fphab.example.com%2FD12345%2F67890` | `/queues/demo-queue/changes/phab/phab.example.com/D12345` | -A UI may display `request.42`, `batch.7`, or `#42`, but those are derived labels, not identities. +Both columns use the same queue prefix for comparison. The before batch/change routes and percent-encoded alternatives are illustrative, not implemented pages; GitHub base64url is abbreviated, and `{sha}` stands for a full lowercase commit SHA. ## Rejected alternatives