Skip to main content

ADR-0012: Fetch and archive web sources via a focused fetcher, not a general crawler

  • Status: Accepted
  • Date: 2026-06-18
  • Deciders: Michel Gillet
  • Specializes: ADR-0011

Context​

ADR-0011 introduced [web:NNN] shorthand tags and docs/web-source-mapping.md as the central registry for external web sources in a2c-managed ADRs and docs. That creates a natural opportunity to:

  • fetch mapped URLs,
  • archive content locally,
  • and make sources available for offline audit, AI workflows, and future analysis.

We need a strategy for how to retrieve content behind these URLs, how much of a surrounding site to crawl, and where to store archived content.

Broadly, there are two options:

  1. Build or adopt a general-purpose web crawler that traverses arbitrary domains, follows links with careful depth/breadth configuration, and is designed for large-scale discovery [web:003, web:007, web:008, web:009, web:010].

  2. Implement a focused web source fetcher that reads the mapping file, fetches only listed URLs (plus an optional small, explicitly configured neighborhood), stores content locally, and uses mature HTTP/crawler libraries under the hood (for example Scrapy, Crawlee, Crawl4AI, or requests with an HTML parser) [web:003, web:004, web:005, web:008, web:009, web:010].

Given our current needs — a curated list of URLs from ADRs and docs rather than open-ended crawling — the second option is more appropriate and less complex.

Decision​

We will implement a focused web source fetcher driven by the [web:*] mapping file, instead of building a general web crawler.

Concretely:

  • a2c will include a web source fetch task that:

    • reads docs/web-source-mapping.md,
    • extracts (tag, label, url) entries,
    • fetches each URL using existing HTTP/crawler libraries,
    • stores content under a local sources/ namespace, and
    • maintains an index file (for example sources/web-index.json) mapping [web:*] tags to local files and metadata (url, fetched_at, http_status, etc.).

  • The fetcher may optionally follow a small number of in-domain links, but only when explicitly configured per URL or domain.

  • The fetcher must respect ethical crawling practices (robots.txt, crawl delays, rate limiting) [web:002, web:006].

Storage​

  • Crawled web content must not be stored in the main code/ADR repository by default, to avoid bloating history and slowing clones.
  • The fetcher should support writing to:
    • a separate sources Git repository (for example project-name-sources/ with sources/web/ inside), or
    • an external object store (S3/GCS/MinIO) using a simple naming convention (for example web/<tag>-<slug>.html).
  • The main repo contains only:
    • the mapping file (docs/web-source-mapping.md), and
    • index metadata (sources/web-index.json or equivalent) referencing archive locations without embedding full content.

We explicitly do not commit here to building or configuring a full general web crawler for broader site-wide crawling. If that need arises (for example indexing all pages under a documentation subtree), we will document it in a separate ADR and likely adopt an open-source crawler such as Scrapy, Crawlee, or Crawl4AI with appropriate configuration [web:003, web:007, web:008, web:009, web:010].

Consequences​

Positive​

  • Implementation is simpler: focus on a curated URL list, not arbitrary web discovery.
  • Mature open-source tools can handle fetching and parsing [web:003, web:004, web:005, web:008, web:009, web:010].
  • Local archives make ADRs and decisions more auditable and resilient to link rot.
  • AI agents can use local archives for offline context, reducing dependence on live web access.
  • Separating archives into a dedicated repo or object store keeps the main workflow repo small.

Negative​

  • The fetcher does not automatically discover related pages beyond mapped URLs unless explicitly configured.
  • A separate sources repo or object store adds operational overhead.
  • Broad site-wide crawling would require additional infrastructure and another ADR.

Follow-up​

  • Optional in-domain link following per URL/domain — not implemented in v1; configure in a future ADR or fetcher revision if needed.
  • If deeper crawling is required later, create a new ADR selecting and configuring a general crawler [web:003, web:007, web:008, web:009, web:010].

Implemented (v1)​

  • sources/web-index.json schema and scripts/fetch_web_sources.py (local, separate sources Git repo, and S3/MinIO backends).
  • robots.txt checks, rate limiting via --delay, and prompts/maintenance/web-source-fetch.md.
  • a2c web-sources check, fetch, and recover CLI commands.

References​

Supersedes​

  • (none)

Superseded by​

  • (none)