A small site does not need an elaborate robots policy. The useful part is consistency: the robots file should say what a crawler can read, where the sitemap lives, and whether temporary material is intentionally absent.
This notebook keeps the robots file short so it can be read by a person as easily as by an automated crawler. That makes later changes easier to review because policy changes are not mixed with unrelated presentation work.
The sitemap reference is the important link. It gives crawlers a bounded list of canonical pages and avoids guessing around generated paths, drafts, or ad hoc files.
Every public page carries a compact title, description and canonical URL. The goal is not to make the page loud; the goal is to make the page unambiguous when it appears in a reader, crawler cache or link preview.
Canonical URLs are kept aligned with the visible navigation. A page that exists in the sitemap should also be reachable from the site itself, and a page that is not meant to be browsed should not be advertised as a primary document.
The custom not-found page follows the same rule. It is a normal page with navigation, not a raw server message, so people and crawlers see the same public document language.
A not-found page should explain that a document is unavailable and give a few useful exits. It should not expose framework messages, stack traces or implementation labels.
The recovery page here uses the same layout as the rest of the notebook. That makes a missing path feel like part of the maintained site instead of a separate server error.
The linked destinations are intentionally conservative: home, indexing notes, policy notes, API notes and status. They are enough to continue reading without making the error page noisy.
