Website Monitor
If you found this page through your server logs: requests with the User-Agent
sven.guru website monitor - https://sven.guru/monitoring come from a small, personal,
non-commercial uptime monitor run by me, Sven Koschnicke. It checks whether a handful of websites I care
about are up. It isn’t a search engine, an archive, an SEO tool or a crawler collecting training
data.
How to opt out
Add this to your robots.txt:
User-agent: SvenGuruMonitor
Disallow: /
The monitor caches robots.txt for up to 24 hours, so it stops within about a day. After that it only re-reads your robots.txt once a day to see whether it has changed, and requests nothing else from your site.
What it requests
-
Only the one configured URL per site, usually the homepage, with
GET. It doesn’t crawl or follow links, doesn’t load images, CSS or JavaScript, and doesn’t execute anything. - It follows up to 5 redirects, reads at most the first 64 KB of the response and gives up after 15 seconds.
- It records only the status code, the response time and the outcome (ok, error, rate-limited). Page content isn’t stored. Detailed results are kept for 30 days, daily uptime totals after that.
- The results only show up on a private dashboard on my home network. Nothing is published.
How often
- While a site is healthy: once every 5 minutes (±10% jitter), so about 12 requests an hour, or 290 a day.
- After a failed request: one retry after about 1 minute. While a site is down: retries after 1, 2 and 4 minutes, then every 5 minutes.
Rate limiting
- A
429 Too Many Requestsis never treated as an outage. The monitor just backs off. -
Retry-Afteris honoured on any response, as seconds or as an HTTP date. If it asks for a longer wait than the normal schedule, the monitor waits that long (up to 1 hour per instruction). It never makes the monitor check more often.
robots.txt details
The monitor follows RFC 9309:
-
Product token:
SvenGuruMonitor. The monitor obeys the group for that token if there is one, otherwise the*group. - robots.txt is fetched per origin and cached for up to 24 hours. Every redirect hop is checked against its own origin’s robots.txt before it’s requested.
- If robots.txt disallows the URL, the monitor stops requesting the site entirely and only re-reads robots.txt once a day.
- A missing robots.txt (404 or any other 4xx) means “allowed”, as the RFC says. If robots.txt can’t be fetched (server error, timeout) and the monitor has no earlier copy, it doesn’t request the site at all.
Contact
If the monitor misbehaves, or you can’t edit your robots.txt, write to me at s.koschnicke@gfxpro.com. I’ll then adjust the monitor for your site.