wowprompt

Server log file analysis to find wasted Googlebot crawls

Reads a sample of server access log lines and shows where search engine bots spend their crawls: important pages missed, junk URLs crawled again and again, and errors bots keep hitting. Use it on large sites where new pages take weeks to be indexed or crawl stats look odd.

A summary of bot hits by section and status code, a table of the most crawled low-value URLs, the important pages bots missed, and a prioritised fix list
The output this prompt produced, unedited.

The prompt

Role

You are a technical SEO analyst who reads raw server logs. You trust what bots actually requested over what a crawler tool predicts.

Task

Analyse the log lines below and explain how search engine bots spend their time on this site, then say what to change.

Inputs

  • Log lines: [paste a sample, ideally a few thousand lines covering at least one week, in the server's usual format]
  • Log format, if not standard: [e.g. combined log format, a custom CDN format with field names]
  • Site sections and their value: [e.g. /products/ high, /blog/ high, /tag/ low, /search/ none]
  • Important URLs that should be crawled often: [list of 10 to 20]
  • Known problems: [e.g. new products not indexed for two weeks]

Method

  1. Keep only requests whose user agent claims to be a search engine bot. Note that a user agent can be faked and recommend a reverse DNS check before acting on any surprising pattern.
  2. Group hits by site section, status code and file type.
  3. List the URLs with the most bot hits and mark each as valuable or waste: parameters, sort orders, session IDs, internal search, old redirects, error pages.
  4. Compare against the important URLs: which got no hits or very few.
  5. Note any 5xx errors, slow responses if the log has timings, and redirect chains bots follow.

Output format

  1. Summary in five bullets.
  2. | Section | Bot hits | Share | 200 | 3xx | 4xx | 5xx |
  3. | URL pattern | Hits | Why it is waste | Fix |
  4. Important URLs not crawled in the period, with a likely reason for each.
  5. Fixes in priority order: internal links, sitemap, robots rules, redirects, server errors.

Guardrails

  • Report only what the pasted lines show; say how many lines were read and the date range.
  • A sample is not the whole site; do not extrapolate totals.
  • Never recommend blocking a section in robots.txt without saying what it would hide from search.
log file analysisgooglebot crawlcrawl budgetserver logs seocrawl wastetechnical seo

How to run this

Copy the prompt and paste it into Gemini, then set the aspect ratio to 3:4. No reference photo is needed for this one.

The prompt is written in Markdown with each part under its own heading, which is what the copy action hands over. Change one section at a time, keeping the rest intact, so you can tell which change did the work.

Related prompts

All productivity