What Is Googlebot? Diagnose Discovery, Crawling, and Indexing
Published:
Last Updated:
Category: SEO & Content
Authors: Shusaku Yosa
When a published page does not appear in Google, first separate four questions: does Google know the URL, can it fetch the page, has it indexed the content, and has it shown the page for relevant searches? Those questions require different evidence. A request to crawl a page is not proof that any later stage has succeeded.
What Googlebot does and how it differs from other Google fetchers
Googlebot retrieves web content for Google Search. Search systems then process that content, select canonical URLs, and decide what to index. A visit from a crawler therefore does not establish that the page is indexed or that it will rank for a particular query.
Google also operates other crawlers and user-triggered fetchers. If you inspect server logs, identify the actual requester rather than treating every Google-related request as a Search indexing event. The official crawler overview explains these categories.
Discovery, crawling, and indexing are separate stages
Keep discovery, retrieval, indexing, and search exposure in separate columns. Google explains the processing stages in its guide to how Search works.
Stage | Useful evidence | What remains unknown |
|---|---|---|
Discovery | A known URL and recorded referring sources | Whether Google fetched the main content |
Crawling | Last crawl time and fetch result | Whether the fetched page was indexed |
Indexing | URL Inspection status and selected canonical | Whether it appears for a particular search |
Search exposure | Performance data filtered to the exact page | The page’s current indexing state |
A missing Performance row is not an indexing diagnosis. An indexed URL can have no recorded exposure in the selected period. Conversely, historical impressions do not establish its present status. Use URL Inspection when current indexing status matters.
Check robots.txt and noindex for different problems
Robots.txt generally controls crawling; noindex controls indexing. A crawler must be able to retrieve a noindex rule to act on it. Blocking the URL in robots.txt can prevent that rule from being read. A blocked URL may also be indexed from information elsewhere without its content being fetched.
Inspect both HTML robots metadata and the X-Robots-Tag response header. Google documents the distinction in its robots.txt introduction and noindex guidance.
Consider a hypothetical restricted page with both noindex and a crawl block. Ask the owner to check the intended access policy, authentication, and response rules together. Do not remove access controls simply to make a test pass.
Read URL Inspection and indexing states in Search Console
Inspection state | First investigation | Record next |
|---|---|---|
Unknown URL | Published links, URL spelling, and discovery routes | Sources checked and sitemap evidence |
Discovered, not indexed | Fetch scheduling and site availability | Response checks and observation date |
Crawled, not indexed | Fetched content, duplication, directives, and canonical selection | Last crawl and selected canonical |
Indexed | Relevant query, country, device, and date filters | Visible exposure and clicks |
“Crawled, not indexed” does not identify content quality as the sole cause. Retain other hypotheses. Also distinguish stored inspection information from a live test: a successful live test is not confirmation of inclusion in the index.
Inspect links, sitemaps, and page responses in order
- Fix the exact URL and language; note any redirects.
- Check whether relevant published pages link to it normally.
- Check individual sitemap inclusion when the XML is accessible.
- Inspect HTTP responses, robots rules, and canonicals.
- Read the retrievable main content and assess duplication or missing user value.
Links and sitemaps help discovery. Neither guarantees indexing or rankings. Google’s technical requirements provide baseline checks, not a promise of search exposure.
When crawl budget deserves investigation
For a small group of missing articles, start with URL-level evidence. Crawl-budget analysis becomes more relevant for large URL populations, frequent changes, or extensive discovery backlogs. Investigate unnecessary URL variants and server availability before proposing broad changes.
The crawl-budget management guide describes these use cases. Do not change dates or publish low-value pages merely to attract bots. Better response times also do not guarantee a specified crawl increase.
Keep an evidence log before checking again
In another hypothetical case, an indexed article never appears for “analytics tools.” Repeated submission is unlikely to answer the real question: does the page satisfy that query, and is there any exposure under matched filters? Technical eligibility and competitive relevance are different investigations.
Log field | What to capture |
|---|---|
Target | Exact URL, language, owner, observation date |
Processing | Inspection state, last crawl, response, selected canonical |
Controls | Robots.txt, HTML robots, and header findings |
Change | Published version, specific edits, next review |
Outcome | Separate request acceptance, indexing, and impressions |
Use the same evidence at the next review. When the cause is unresolved, record the missing check instead of assigning blame to the text.
