Machine room
The crawlers
Who came to read, and what they read.
Not every guest is human. Search engines index the pages, AI crawlers collect text for language models, and AI assistants fetch a page because a person asked them to. The site has put special files out for them. Here is who came in the last 30 days, what they read and who only pretended to be a bot. Nobody is blocked, and no IP addresses are fetched or shown.
How it is counted
- Cron Trigger
- GraphQL Analytics API
- D1
- Workers KV
- Workers Cache
- Your browser
The source
Cloudflare’s GraphQL Analytics API for the wi24rd.com zone: firewall events, all requests and verified bots. The token may only read analytics for this one zone.
Every hour
A Cron Trigger in the machine Worker asks for today and yesterday, one day at a time, so Cloudflare does not have to sample over a long period. The rows are kept in D1 for 95 days.
The overview
After each collection the Worker builds the two overviews and puts them in Workers KV. The page fetches them through Workers Cache, which is purged with a cache tag when there are new figures.
What is not kept
IP addresses are neither fetched nor stored. Paths are shown without query strings, long tokens are hidden and email addresses in user agents are removed. Figures over long periods may be sampled by Cloudflare and are therefore approximate.
Raw figures as JSON: /api/maskine/crawlere.
The defencesWhat Cloudflare kept out: attacks that were stopped, and where they came from.
Open the defencesThe family: The defences·The crawlers·The speed·Denmark right now·The archive cabinet·The door·The infrastructure map·The audit·Back to the machine room
Have a useful idea?
Feature requests, bug reports and questions are welcome. Every message gets a personal reply.