# Website Monitor

> What the SvenGuruMonitor bot is, what it requests, and how to block it.

- Canonical URL: https://sven.guru/monitoring/
- Translation (Deutsch): https://sven.guru/de/monitoring/index.md


If you found this page through your server logs: requests with the User-Agent
`sven.guru website monitor - https://sven.guru/monitoring` come from a small,
personal, non-commercial uptime monitor run by me, Sven Koschnicke. It checks
whether a handful of websites I care about are up. It isn't a search engine,
an archive, an SEO tool or a crawler collecting training data.

## How to opt out {#how-to-opt-out}

Add this to your robots.txt:

```text
User-agent: SvenGuruMonitor
Disallow: /
```

The monitor caches robots.txt for up to 24 hours, so it stops within about a
day. After that it only re-reads your robots.txt once a day to see whether it
has changed, and requests nothing else from your site.

## What it requests {#what-it-requests}

-   Only the one configured URL per site, usually the homepage, with `GET`. It
    doesn't crawl or follow links, doesn't load images, CSS or JavaScript, and
    doesn't execute anything.
-   It follows up to 5 redirects, reads at most the first 64 KB of the response
    and gives up after 15 seconds.
-   It records only the status code, the response time and the outcome (ok,
    error, rate-limited). Page content isn't stored. Detailed results are kept
    for 30 days, daily uptime totals after that.
-   The results only show up on a private dashboard on my home network. Nothing
    is published.

## How often {#how-often}

-   While a site is healthy: once every 5 minutes (±10% jitter), so about 12
    requests an hour, or 290 a day.
-   After a failed request: one retry after about 1 minute. While a site is
    down: retries after 1, 2 and 4 minutes, then every 5 minutes.

## Rate limiting {#rate-limiting}

-   A `429 Too Many Requests` is never treated as an outage. The monitor just
    backs off.
-   `Retry-After` is honoured on any response, as seconds or as an HTTP date. If
    it asks for a longer wait than the normal schedule, the monitor waits that
    long (up to 1 hour per instruction). It never makes the monitor check more
    often.

## robots.txt details {#robots-dot-txt-details}

The monitor follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309):

-   Product token: `SvenGuruMonitor`. The monitor obeys the group for that token
    if there is one, otherwise the `*` group.
-   robots.txt is fetched per origin and cached for up to 24 hours. Every
    redirect hop is checked against its own origin's robots.txt before it's
    requested.
-   If robots.txt disallows the URL, the monitor stops requesting the site
    entirely and only re-reads robots.txt once a day.
-   A missing robots.txt (404 or any other 4xx) means "allowed", as the RFC
    says. If robots.txt can't be fetched (server error, timeout) and the monitor
    has no earlier copy, it doesn't request the site at all.

## Contact {#contact}

If the monitor misbehaves, or you can't edit your robots.txt, write to me at
[s.koschnicke@gfxpro.com](mailto:s.koschnicke@gfxpro.com). I'll then adjust the monitor for your site.
