This option automatically fetches the robots.txt file based on the current request and adheres to the disallow directives.

JS version was implemented via the following PRs:

• feat: add respectRobotsTxtFile crawler option crawlee#2910
• feat: add onSkippedRequest option crawlee#2916
• fix: rename RobotsFile to RobotsTxtFile crawlee#2913

We will first need to implement the RobotsTxtFile and Sitemap classes:

• https://github.com/apify/crawlee/blob/master/packages/utils/src/internals/robots.ts
• https://github.com/apify/crawlee/blob/master/packages/utils/src/internals/sitemap.ts