gh-153569: Simplify tokenizer source views and reader ownership - #158849

Draft

pablogsal wants to merge 6 commits into

python:mainfrom

pablogsal:tokenizer-source-api-cleanup

Commits on Oct 5, 2026

gh-153569: Move tokenizer source access into the source API
pablogsal
committed
8f7e822
gh-153569: Use logical spans in tokenizer views and column calculations
pablogsal
committed
2b46f42
gh-153569: Move tokenizer input state and encoding ownership into the reader
pablogsal
committed
1cfbc93
gh-153569: Keep tokenizer source scans and copies on offsets and spans
pablogsal
committed
e60d602
gh-153569: Remove redundant tokenizer views and simplify reader branching
pablogsal
committed
ffb781d
gh-153569: Reuse the incremental decoder factory helper
pablogsal
committed
33d219c