Currently, crawlee-js does this in two separate ways. What you ported over is the "regular" one. Apart from that, we have adaptive crawling, where useState bypasses get_auto_saved_value. See https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts for details.

The general idea behind this is that we don't want to "save" the changed state before we know that the request handler succeeded.

We should probably discuss what would be the best way forward. Try to give the javascript adaptive crawler a glance, but if it doesn't make any sense, just say so so that we don't waste any time. Then we'll talk more 🙂

I looked up adaptive-playwright-crawler, if I understand its workings correctly, it completely overrides the default useState

We could do the same thing when we need to implement this in Python. Given that this behavior is specific to one particular implementation of crawler.

Yes, that's basically how it works. The reason is that in adaptive crawling, we routinely run request handler functions twice and we do not want to introduce duplicities to the storages (or inconsistent state).

However, re-running the request handler is common enough to warrant only "commiting" storage changes after a request handler is finished in all cases, not just the adaptive one. In the javascript version, this functionality was not made default 1. because it would break BC and 2. because it was a bit of an experiment.

What I want to say is, we should think this through. Once we implement this, it will be much harder to change. By the way, adaptive crawling for the python version is very much on the roadmap 🙂 (#249)