General Retrieval

GeneralRetrievalBot: the General Retrieval search crawler

GeneralRetrievalBot collects public web pages for General Retrieval, a search service for software agents. It is a search crawler. Every use of the pages it collects: (1) they are indexed for search; (2) the search service returns links, titles and short excerpts from them to software agents, including AI agents, that query it on behalf of their users (AI-agent delivery), with the limits each page sets (below); (3) GeneralRetrieval-Fetcher reads a single page when such a request names it. General Retrieval is operated by AxionOrbital Space Inc., 95 3rd Street, 2nd floor 3916, San Francisco, CA 94103, USA. AxionOrbital Space Inc. does not use pages collected by GeneralRetrievalBot to train AI models.

During our first, invitation-only crawl, collected pages are stored for indexing and are not yet shown in search results.

How to recognise it

  • User-agent: GeneralRetrievalBot/1.0 (+https://generalretrieval.com/bot; crawler@generalretrieval.com; AxionOrbital Space Inc.; search index for AI-agent search). A second agent, GeneralRetrieval-Fetcher, fetches a single page when a user's request asks for it; its user-agent ends in user-requested fetch for an AI agent, and it follows robots.txt rules for GeneralRetrieval-Fetcher (or *).
  • Signed requests: every request carries an HTTP Message Signature (RFC 9421, Ed25519) under the IETF Web Bot Auth scheme: its Signature-Agent header names https://keys.generalretrieval.com, where /.well-known/http-message-signatures-directory lists our public keys, and the signature covers the request's method, host and path. A request that claims to be GeneralRetrievalBot without a valid signature is not from us.
  • Addresses: we crawl only from the IP ranges listed at https://keys.generalretrieval.com/crawler-ips.json (JSON); their reverse DNS names end in crawl.generalretrieval.com and resolve back to the same address.
  • We send a From: header with crawler@generalretrieval.com.

How to control it

  • robots.txt (RFC 9309). Use the token GeneralRetrievalBot (or *). Example: 
    User-agent: GeneralRetrievalBot
Disallow: /

    Earlier names. We also honour robots.txt rules written for the names SextantBot (for GeneralRetrievalBot) and Sextant-Fetcher (for GeneralRetrieval-Fetcher). When rules for both names apply, the stricter one wins: a page is not fetched if either group disallows it, and the longer Crawl-delay applies. We use a cached copy of your robots.txt for at most 24 hours (less if your Cache-Control: max-age asks for it, but not less than one hour); after that we read it again before we request anything else from your site. If your robots.txt returns 401, 402 or 403, or a challenge page, or cannot be reached and we have no copy that we read successfully in the last 7 days, we do not crawl your site (an outage never widens what we fetch). We follow at most five redirects of a robots.txt; if it still does not resolve, we do not crawl your site. We read the first 512 KiB of a robots.txt and record when a file was longer; we never claim to have honoured rules we could not read. When a new robots.txt disallows pages that we hold, we delete them when we read it.

  • Crawl rate: we send one request at a time per host name (and at most two at a time per IP address), leave at least 5 seconds between the end of one request and the start of the next (10 seconds during our first, invitation-only crawl), slow down when your server slows down or returns errors, and honour Crawl-delay. When you return 429, or 503 with Retry-After, we send nothing to that host until your Retry-After time has passed, however far ahead it is (a time beyond our seven-day schedule parks the host until then).
  • Refusals at your edge: if your site answers us with 401, 402 or 403, a bot challenge, a CAPTCHA, or pages that look like generated junk, we stop requesting anything from that host for up to 24 hours and never retry with another name, address or identity. We never pay for access (we send no payment headers).
  • Pages: noindex, none and nofollow are honoured in both <meta name="robots"> (or <meta name="GeneralRetrievalBot">) and the X-Robots-Tag header (also when addressed to SextantBot): we do not keep pages marked noindex or none, and we do not follow links on pages marked nofollow or none. unavailable_after is honoured: after that date we no longer use our copy, and we delete it at our next refresh pass. nosnippet, max-snippet and data-nosnippet are recorded with the page and applied when results are shown.
  • AI-usage preferences: we read Cloudflare Content Signals (search, ai-input, ai-train), IETF AI-preference Content-Usage signals, TDMRep (tdm-reservation), RSL licences and ai.txt, and record them with every page. Content-Signal: search=no keeps your pages out of our index: we may still request them, but we do not keep them. A TDMRep reservation means we keep only metadata. ai-input=no limits our results to a link and title.
  • Removal: to remove pages already collected, email crawler@generalretrieval.com from an address at the site's domain (or tell us how to verify that you control it). We acknowledge every request within 24 hours. A request we cannot verify yet stops our crawling of the pages it names at once while we check it. As soon as a request is verified, the pages stop being shown in our results, are deleted from our crawl store and hidden from our index within one hour, and are not requested again; our crawl store erases the deleted copy physically within 24 hours, and backups within 7 days. Copyright notices go to our designated DMCA agent: Copyright Agent, AxionOrbital Space, 95 3rd Street, 2nd floor 3916, San Francisco, CA 94103; (415) 212-3061; support@generalretrieval.com.

What it never does

  • Log in, bypass paywalls, solve CAPTCHAs, or use proxies or disguised identities.
  • Download files declared as images, video, audio, fonts or PDF, or request URLs that end in such a file extension: such responses are dropped before their content is read. A media file mislabelled as a web page is discarded, not kept, when it starts with a known media file signature, and embedded data: images are removed from the pages we keep.
  • Submit forms, run scripts, or store or send cookies.
  • Use full page text more than 30 days after we last fetched the whole page (a "not modified" answer does not extend this; such copies are deleted at our next refresh pass), or keep pages you opt out of: they are deleted when we next read your robots.txt (robots.txt rules, Content-Signal: search=no), when we next fetch the page (noindex, also in the headers of a "not modified" answer), or at once when your removal request is verified (see Removal above).
  • Index the text of a page larger than 2 MiB as transferred, or one cut off before its end: we keep only its address and title.

Accountability

Each request is recorded in an append-only, cryptographically signed log (URL, time, content hash; never page text). Site owners may ask us whether, and when, we fetched a page: crawler@generalretrieval.com.

Contact

crawler@generalretrieval.com - we answer within one business day.