Build a crawler policy

Choose whether to allow the listed training crawlers while keeping search crawlers available. Robots rules are voluntary and do not protect private data. Keep authentication on private pages. Review the generated file before replacing an existing policy.

Build a crawler policy · Estimate a marketing scenario · Plan customer questions · Structured data · Site overview · Readiness checklist · Free course