Fill in these first
- Current robots.txt: [paste the full file]
- Site type and platform: [e.g. WordPress blog, Shopify store, custom web app]
- Sections that must be crawled: [e.g. /products/, /blog/, images, CSS and JS]
- Sections that should not be crawled: [e.g. /cart/, /checkout/, internal search results, staging paths]
- Sitemap URL: [URL]
- Symptoms, if any: [e.g. "Blocked by robots.txt" in Search Console, pages missing from search]
Task
Explain what the current file does, find the rules that cause harm or do nothing, and write a corrected file.
Output format
- | Line | Rule | What it does | Verdict: keep, fix or remove | Reason |
- The corrected robots.txt in one code block, with a short comment line above each group.
- Five URLs to test against the new file, with the expected result: allowed or blocked.
- Anything robots.txt cannot do here, and the right tool instead.
Rules
- Blocking a URL in robots.txt stops crawling, not indexing. If a page must stay out of search results, recommend a noindex tag and say the page must stay crawlable for the tag to be seen.
- Never block CSS, JavaScript or image files a page needs to render.
- Google ignores noindex and crawl-delay lines in robots.txt; say so if the file uses them.
- Paths are case sensitive and match from the start; check each wildcard does not catch more than intended.
- Do not change rules for sections the inputs do not mention; flag them as questions.
Example
Line 4: Disallow: /*? blocks every URL with a parameter, including paginated category pages. Verdict: fix.






