add `Sitemap` parser utility #1161

Closed

Assignees

Labels

Milestone

opened

on Apr 17, 2025

Sitemap parser is closely related to RobotsTxtFile. But it is not used for support `respect_robots_txt_file'

Related: Implement respectRobotsTxtFile crawler option #1144

Metadata

Assignees

Mantisus

Labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

1.0

Relationships

None yet

Development

No branches or pull requests