wowprompt

Robots.txt review with safe crawl rules for a website

Reviews a robots.txt file line by line, flags rules that block pages or resources search engines need, and returns a corrected file with a comment on every change. Use it after a redesign, a platform move, or when pages vanish from search for no clear reason.

A line-by-line table of the current robots.txt with what each rule does and a verdict, followed by a corrected file in a code block and a test list
The output this prompt produced, unedited.

The prompt

Fill in these first

  • Current robots.txt: [paste the full file]
  • Site type and platform: [e.g. WordPress blog, Shopify store, custom web app]
  • Sections that must be crawled: [e.g. /products/, /blog/, images, CSS and JS]
  • Sections that should not be crawled: [e.g. /cart/, /checkout/, internal search results, staging paths]
  • Sitemap URL: [URL]
  • Symptoms, if any: [e.g. "Blocked by robots.txt" in Search Console, pages missing from search]

Task

Explain what the current file does, find the rules that cause harm or do nothing, and write a corrected file.

Output format

  1. | Line | Rule | What it does | Verdict: keep, fix or remove | Reason |
  2. The corrected robots.txt in one code block, with a short comment line above each group.
  3. Five URLs to test against the new file, with the expected result: allowed or blocked.
  4. Anything robots.txt cannot do here, and the right tool instead.

Rules

  • Blocking a URL in robots.txt stops crawling, not indexing. If a page must stay out of search results, recommend a noindex tag and say the page must stay crawlable for the tag to be seen.
  • Never block CSS, JavaScript or image files a page needs to render.
  • Google ignores noindex and crawl-delay lines in robots.txt; say so if the file uses them.
  • Paths are case sensitive and match from the start; check each wildcard does not catch more than intended.
  • Do not change rules for sections the inputs do not mention; flag them as questions.

Example

Line 4: Disallow: /*? blocks every URL with a parameter, including paginated category pages. Verdict: fix.

robots txtcrawl rulesdisallow rulesrobots txt checkercrawl budgettechnical seo

How to run this

Copy the prompt and paste it into Gemini, then set the aspect ratio to 3:4. No reference photo is needed for this one.

The prompt is written in Markdown with each part under its own heading, which is what the copy action hands over. Change one section at a time, keeping the rest intact, so you can tell which change did the work.

Related prompts

All productivity