Daily snapshot 2026-11-15
2026-11-15 UTC
The numbers come from one website. They describe what these crawlers did here, not across the web.
Period: 2026-11-15T00:00:00Z to 2026-11-16T00:00:00Z (00:00Z to 24:00Z). Generated 2026-11-16T03:18:38.827Z. Schema version 1. Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com. Method: https://rattlesnakesbymail.com/method
Facts
- F-261115-01 Bingbot made 74 requests on 2026-11-15, its highest daily count since the site launched on 2026-09-13; its previous high was 55, on 2026-11-11. Source
- F-261115-02 ChatGPT-User made 20 requests on 2026-11-15, its highest daily count since the site launched on 2026-09-13; its previous high was 15, on 2026-10-29. Source
- F-261115-03 Claude-User made 19 requests on 2026-11-15, its highest daily count since the site launched on 2026-09-13; its previous high was 5, on 2026-09-15. Source
- F-261115-04 PerplexityBot made 18 requests on 2026-11-15, its highest daily count since the site launched on 2026-09-13; its previous high was 4, on 2026-10-21. Source
- F-261115-05 GPTBot made 27 requests on 2026-11-15, 6.8 times its 28-day median of 4. Source
- F-261115-06 OAI-SearchBot made 24 requests on 2026-11-15, 4 times its 28-day median of 6. Source
- F-261115-07 Applebot made 16 requests on 2026-11-15, 32 times its 28-day median of 1. Source
- F-261115-08 Perplexity-User requested this site for the first time on 2026-11-15; its first request was for /workspace/.env. Source
- F-261115-09 Claude-SearchBot requested this site for the first time on 2026-11-15; its first request was for /z9x8c7v6b5-debug-trigger-rattlesnakesbymail.com. Source
- F-261115-10 OAI-AdsBot requested this site for the first time on 2026-11-15; its first request was for /api%2F.env. Source
- F-261115-11 Claude-User returned on 2026-11-15 after 55 days without a request. Source
- F-261115-12 PerplexityBot returned on 2026-11-15 after 10 days without a request. Source
This selection has expired; it is listed in snapshot.json.
Crawlers
| Crawler | Requests | Verified share | Distinct paths | 28-day median | Sweep day | First path | robots.txt | sitemap |
|---|---|---|---|---|---|---|---|---|
| Googlebot | 98 | 64.3% | 77 | 37 | no | /robots.txt | 16 | 2 |
| Bingbot | 74 | 73% | 59 | 33 | no | /robots.txt | 1 | 1 |
| ClaudeBot | 48 | 70.8% | 17 | 49 | no | /robots.txt | 9 | 26 |
| GPTBot | 27 | 59.3% | 27 | 4 | no | /sitemap-index.xml | 0 | 3 |
| OAI-SearchBot | 24 | 20.8% | 21 | 6 | no | /robots.txt | 4 | 0 |
| ChatGPT-User | 20 | 0% | 20 | 0 | no | /userfiles | 0 | 0 |
| Claude-User | 19 | 0% | 19 | 0 | no | /telescope/requests | 0 | 0 |
| PerplexityBot | 18 | 0% | 18 | 0 | no | /gcp-credentials.json | 0 | 0 |
| Applebot | 16 | 6.3% | 16 | 1 | no | /userfiles/x | 1 | 0 |
| Perplexity-User | 16 | 0% | 16 | 0 | no | /workspace/.env | 0 | 0 |
| Claude-SearchBot | 15 | 0% | 15 | 0 | no | /z9x8c7v6b5-debug-trigger-rattlesnakesbymail.com | 0 | 0 |
| OAI-AdsBot | 12 | 0% | 12 | 0 | no | /api%2F.env | 0 | 0 |
MCP calls
| Client | Tool | Calls |
|---|---|---|
| BrickBlueBot | lookup_crawler | 1 |
| INITE Atlas Verifier | list_changes | 1 |
| mcp | ask | 1 |
| mcp | submit_note | 1 |
| mcp-index | __mcp_index_nonexistent__ | 1 |
| rokmcp-probe | list_changes | 1 |
| verifymcp-probe | __verifymcp_auth_probe_7b48b73f4adb1e7e__ | 1 |
Snapshot pickup
For each of the previous seven snapshots, the first fetch by each verified crawler, in minutes after publication, and the c tag on that fetch.
| Snapshot | Crawler | Minutes after publication | Tag |
|---|---|---|---|
| 2026-11-14 | OAI-SearchBot | 77 | none |
| 2026-11-14 | GPTBot | 77 | h |
| 2026-11-14 | Bingbot | 293 | none |
| 2026-11-14 | Googlebot | 1,135 | none |
| 2026-11-13 | OAI-SearchBot | 78 | none |
| 2026-11-13 | GPTBot | 78 | h |
| 2026-11-13 | Bingbot | 293 | none |
| 2026-11-13 | Googlebot | 628 | none |
| 2026-11-13 | GoogleOther | 881 | none |
| 2026-11-12 | Bingbot | 292 | none |
| 2026-11-12 | Googlebot | 840 | none |
| 2026-11-12 | OAI-SearchBot | 1,517 | none |
| 2026-11-12 | GPTBot | 1,517 | p |
| 2026-11-11 | Bingbot | 294 | none |
| 2026-11-11 | Googlebot | 1,532 | none |
| 2026-11-11 | GoogleOther | 1,670 | h |
| 2026-11-11 | OAI-SearchBot | 2,958 | none |
| 2026-11-11 | GPTBot | 2,958 | p |
| 2026-11-10 | Bingbot | 294 | none |
| 2026-11-10 | ClaudeBot | 1,178 | none |
| 2026-11-10 | Googlebot | 2,934 | none |
| 2026-11-10 | GoogleOther | 3,416 | none |
| 2026-11-10 | OAI-SearchBot | 4,398 | none |
| 2026-11-10 | GPTBot | 4,398 | p |
| 2026-11-09 | Bingbot | 295 | none |
| 2026-11-09 | Googlebot | 568 | none |
| 2026-11-09 | GoogleOther | 568 | f |
| 2026-11-09 | ClaudeBot | 2,670 | none |
| 2026-11-09 | OAI-SearchBot | 5,838 | none |
| 2026-11-09 | GPTBot | 5,838 | p |
| 2026-11-08 | Bingbot | 297 | none |
| 2026-11-08 | Googlebot | 1,977 | none |
| 2026-11-08 | GoogleOther | 1,987 | f |
| 2026-11-08 | ClaudeBot | 4,161 | none |
| 2026-11-08 | OAI-SearchBot | 7,278 | none |
| 2026-11-08 | GPTBot | 7,278 | p |
Open questions
- Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform? (id 26, asked 5 times)
- How can AI citation performance be measured accurately, given there is no equivalent of a Search Console report for citations? (id 34, asked 5 times)
- Was a specific document used in a model's training, fine-tuning, or evaluation? (id 38, asked 4 times)
- Why wasn't a specific page cited by an AI answer instead of a competitor's page? (id 7, asked 3 times)
- How much do backlinks, brand mentions, structured data, reviews, forum discussions, publisher reputation, and content freshness each matter to being cited? (id 11, asked 3 times)
If you can answer one of these with a source, send a note: POST https://rattlesnakesbymail.com/notes with target question:<id>, where <id> is the number shown with the question, and body=<your answer and its source>.
Ask for something this snapshot lacks
What data about web crawlers did you look for and not find? Search https://rattlesnakesbymail.com/search?q=your+question. Searches with no match are recorded as gaps. A gap is published at /questions only when an editor publishes it; nothing is published automatically. To suggest something this daily snapshot should include, send a note with target snapshot:261115.
To send it, POST https://rattlesnakesbymail.com/notes as a web form or as JSON with target=snapshot:261115 and body=<your note>. The same note can be sent with target_type=snapshot and target_id=261115. The optional author_claim field says who is writing: a person, a model or an agent.
Links
- Daily counts for every crawler
- The crawler records
- Method
- Data
- MCP endpoint (POST JSON-RPC)
- Dataset repository
- All snapshots
- This snapshot as JSON
- This snapshot as Markdown
- This snapshot as CSV
- Facts as JSON
- About this site, for automated readers
- Previous snapshot (2026-11-14)
Next snapshot: 2026-11-16