Skip to content

feat: batched request addition in RequestQueue#186

Merged
vdusek merged 11 commits into
masterfrom
improve-add-requests
Jun 17, 2024
Merged

feat: batched request addition in RequestQueue#186
vdusek merged 11 commits into
masterfrom
improve-add-requests

Conversation

@vdusek

@vdusek vdusek commented Jun 14, 2024

Copy link
Copy Markdown
Collaborator

Description

  • RequestQueue supports adding requests in batches.
  • The RequestQueueClient in StorageClient has the same interface as in the apify-python-client.
  • MemoryStorageClient implements this interface as well.

Related issues

Testing

  • A new unit test was implemented.

Checklist

  • Changes are described in the CHANGELOG.md
  • CI passed

@vdusek vdusek requested a review from janbuchar June 14, 2024 09:16
@vdusek vdusek self-assigned this Jun 14, 2024
@github-actions github-actions Bot added this to the 91st sprint - Tooling team milestone Jun 14, 2024
@github-actions github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Jun 14, 2024
@vdusek vdusek marked this pull request as ready for review June 14, 2024 09:20
Comment thread src/crawlee/models.py
@vdusek vdusek requested a review from janbuchar June 14, 2024 09:35
Comment thread src/crawlee/basic_crawler/basic_crawler.py
Comment thread src/crawlee/storages/request_queue.py Outdated
Comment thread src/crawlee/storages/request_queue.py Outdated
@vdusek vdusek requested a review from janbuchar June 14, 2024 12:31
Comment thread src/crawlee/_utils/wait.py
Comment thread tests/unit/storages/test_request_queue.py
Comment thread src/crawlee/storages/request_queue.py Outdated
Comment thread src/crawlee/storages/request_queue.py
@vdusek vdusek requested a review from janbuchar June 17, 2024 11:51
@vdusek vdusek merged commit f48c806 into master Jun 17, 2024
@vdusek vdusek deleted the improve-add-requests branch June 17, 2024 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants