feat: add direct async iteration on Dataset and KeyValueStore by Pijukatel · Pull Request #2267 · apify/crawlee-python

Pijukatel · 2026-10-02T07:00:11Z

Description

Add Dataset.__aiter__, so async for item in dataset yields the items, like iterating over a list.
Add KeyValueStore.iterate_values, iterate_entries and __aiter__. async for key in kvs yields the keys, like iterating over a dict; iterate_values and iterate_entries give the values and (key, value) pairs. This follows Python conventions and differs from JS, where iterating a KVS yields [key, value] pairs.
Add overridable KeyValueStoreClient.iterate_entries with a key-by-key default; the SQL and Redis clients read in batches bounded by record count and size.
Explicitly state that the Redis client created with decode_responses=True is not supported for KVS and fix related typing.

Issues

Closes: Add direct async iteration on Dataset and KeyValueStore #1745

Testing

Unit tests for the new frontend methods on all storage clients, plus client-level tests for the SQL and Redis overrides.

Checklist

CI passed

Add `__aiter__` to `Dataset` (delegating to `iterate_items`) and to `KeyValueStore` (delegating to the new `iterate_entries`), plus `KeyValueStore.iterate_values` and `KeyValueStore.iterate_entries`. Values are fetched one at a time as the iteration advances, so only a single value is held in memory regardless of the record sizes. On the Apify platform this means one request per record on top of the paginated key listing, which is the only way the API offers to read values. The memory key-value store client now skips keys deleted while a key iteration is suspended instead of raising `KeyError`, matching the other backends. Closes #1745 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

Add a non-abstract `KeyValueStoreClient.iterate_entries` to the storage client base class, yielding `KeyValueStoreRecord`s. The default implementation lists the keys with `iterate_keys` and reads each value with `get_value` as the iteration advances, so existing and third-party clients need no changes. Backends that can read keys together with their values more efficiently can override it. `KeyValueStore.iterate_entries` and `iterate_values` now delegate to the client method instead of combining `iterate_keys` and `get_value` themselves. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

Override `iterate_entries` where the backend can do better than one value read per key: - The SQL client selects keys, metadata and values in a single streamed query, so a store with N records costs one query instead of N+1 and still holds only one row in memory at a time. - The Redis client fetches values with one HMGET per batch instead of two round trips per record. Batches are bounded by key count and by the record sizes known from the metadata hash, so large values do not pile up in memory. A record larger than the byte limit is fetched alone. Both clients share the value decoding with `get_value` through a new `_build_record` helper. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

codecov · 2026-10-02T07:04:26Z

Codecov Report

❌ Patch coverage is 99.10714% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.97%. Comparing base (df1178c) to head (2c3dd78).
⚠️ Report is 19 commits behind head on master.

Files with missing lines	Patch %	Lines
...e/storage_clients/_base/_key_value_store_client.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2267      +/-   ##
==========================================
+ Coverage   93.87%   93.97%   +0.10%     
==========================================
  Files         182      183       +1     
  Lines       13130    13252     +122     
==========================================
+ Hits        12326    12454     +128     
+ Misses        804      798       -6

Flag	Coverage Δ
unit	`93.97% <99.10%> (+0.10%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Whether a record deleted mid-iteration is yielded depends on how the storage client reads values, which is not part of the contract. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

…e client `set_value` stores an empty byte string for `None`, so the batched read can fetch those records like any other and let `_build_record` map the `application/x-none` content type to `None`. This drops the key filtering and the per-key lookup dict from the batch fetch. Also reword the base `iterate_entries` docstring so it does not present the handling of records deleted mid-iteration as a contract. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

…ype checker redis-py types every reply as `bytes | str | None` because a client created with `decode_responses=True` returns strings. The key-value store client stores binary values and needs the raw bytes back, so it narrowed the type with `ty: ignore` and would fail with an `AttributeError` on such a client. Add `expect_bytes`, which narrows a reply to bytes and raises a `TypeError` naming the unsupported option, use it at both value-reading sites, and document the requirement on `RedisStorageClient`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

`expect_bytes` now handles a single reply, which `isinstance` narrows on its own, and the HMGET result is narrowed with a comprehension. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

…ient Datasets use RedisJSON and request queues parse JSON strings, so both work with a decoding Redis client. Only key-value store values are binary. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

`stream_results` buffers up to 1000 rows client side, so iterating a store of large values held hundreds of megabytes before the first yield, and the single open transaction lasted for the whole iteration. The SQL client now reads record metadata in keyset-paginated pages, splits each page into batches bounded by the record sizes, and fetches every batch's values with one query. Each query runs in its own short session. The batching rule is shared with the Redis client through a new `batch_records_by_size` helper. Measured with 300 records of 1 MiB: peak memory drops from 300 MiB to 9 MiB. With 2000 small records the iteration issues 41 selects instead of 2001 and is about 40 times faster than the default implementation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

The default iteration retries every read through `get_value`, but the batched HMGET in the Redis override was not retried. It now returns a list under the same `retry_on_error` as the other Redis reads, matching the SQL client. Add a direct unit test for `batch_records_by_size`, covering the count bound, the size bound, an oversized record alone in its batch and records of unknown size. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

The test directories have no `__init__.py`, so pytest cannot import two modules named `test_utils.py` and aborted the whole unit test collection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

Mantisus

LGTM

… warning

vdusek

I think this needs a little more thought before we merge it.

The main question is: should we mirror the JS API, or follow Python conventions?

IMO:

async for item in dataset should behave like iterating over a list. And it currently does.
async for ... in kvs should behave like iterating over a dict (or tuple?). And now, it doesn't. Iterating over a Python dict yields keys, and you explicitly call .items() when you want (key, value) pairs. Our __aiter__, however, yields (key, value) tuples.

That means someone writing async for key in kvs would silently get a tuple instead of a string, which feels surprising in Python.

Also, I think both storages should support both styles: the __aiter__ dunder for the natural/default async for behavior. And the explicit methods when you want something more specific/explicit.

Dataset: __aiter__ yields items, while iterate_items() supports the filtering options.
KVS: __aiter__ yields keys (dict-like), with iterate_keys(), iterate_values(), and iterate_entries() for keys, values, and (key, value) pairs respectively.

`async for key in kvs` now yields the keys, matching how iterating over a `dict` behaves. Values and `(key, value)` pairs stay available through `iterate_values` and `iterate_entries`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

…ataset does

vdusek

LGTM

Pijukatel and others added 3 commits October 1, 2026 08:19

Pijukatel and others added 7 commits October 2, 2026 07:09

docs: describe the bulk iterate_entries overrides in docstrings

0e255a5

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV

Pijukatel requested a balanced review from Copilot October 2, 2026 09:22

Copilot started reviewing on behalf of Pijukatel October 2, 2026 09:23 View session

This comment was marked as resolved.

Sign in to view

Pijukatel and others added 2 commits October 2, 2026 11:26

Pijukatel marked this pull request as ready for review October 2, 2026 11:38

Pijukatel requested review from Mantisus and vdusek October 2, 2026 11:38

apify-service-account added the tested Temporary label used only programatically for some analytics. label Oct 2, 2026

apify-service-account assigned Pijukatel Oct 2, 2026

apify-service-account added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 2, 2026

Mantisus approved these changes Oct 4, 2026

View reviewed changes

vdusek added 4 commits October 6, 2026 10:36

fix: skip Redis records deleted mid-iteration without a missing-value…

f7db136

… warning

docs: correct memory and round-trip claims in KVS iteration docstrings

5918899

test: cover SQL iterate_entries limit ending inside a later page

eb6a2f3

test: rename batching test helpers and add missing test docstrings

a967540

vdusek requested changes Oct 6, 2026

View reviewed changes

Pijukatel requested a review from vdusek October 6, 2026 08:51

vdusek added 4 commits October 6, 2026 11:48

refactor: return the key iterator from KeyValueStore.__aiter__ like D…

0b9af81

…ataset does

docs: fix RedisStorageClient and KVS iteration docstrings

4905199

test: record Redis and SQL batch calls with wrapping mocks

1c66c3b

test: cover records deleted mid-iteration in memory and SQL KVS clients

2c3dd78

vdusek approved these changes Oct 6, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat: add direct async iteration on Dataset and KeyValueStore - #2267

feat: add direct async iteration on Dataset and KeyValueStore#2267
Pijukatel wants to merge 21 commits into
masterfrom
claude/sharp-hopper-yidrmf

Pijukatel commented Oct 2, 2026 •

edited by vdusek

Loading

Uh oh!

codecov Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

Mantisus left a comment

Uh oh!

vdusek left a comment

Uh oh!

vdusek left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Pijukatel commented Oct 2, 2026 • edited by vdusek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Testing

Checklist

Uh oh!

codecov Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as resolved.

Uh oh!

Mantisus left a comment

Choose a reason for hiding this comment

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Pijukatel commented Oct 2, 2026 •

edited by vdusek

Loading

codecov Bot commented Oct 2, 2026 •

edited

Loading