The AudacianResearch crawler

AudacianResearch/1.0 is the crawler Audacian uses for its research on how ready websites are for AI search. It makes at most 6 requests per site, follows robots.txt, and we publish only totals: no site is ever named.

What it is

It runs the same eight checks as our free AI search readiness checker on the homepage of each site in a study sample: AI crawler access in robots.txt, indexability, the canonical tag, structured data (JSON-LD), llms.txt, the title and meta description, sameAs links and hreflang.

It sends this user agent on every request:

AudacianResearch/1.0 (+https://audacian.eu/research/crawler; hello@audacian.eu)

Exactly what it requests

  • /robots.txt first. If robots.txt disallows a path for AudacianResearch or for all crawlers (User-agent: *), that path is not requested.
  • The homepage.
  • /llms.txt.
  • Only when the homepage points to them, and only on the same host: the canonical URL (one HEAD request) and up to three language versions listed in hreflang.
  • At most 6 requests per site in a run, counting every redirect. If the limit is reached, the remaining checks are recorded as not measured.
  • It never requests pages on other sites, such as the social profiles in sameAs, and never submits forms, logs in, runs JavaScript or stores cookies.

How often and how fast

  • One site at a time, with at least 10 seconds between sites.
  • Each request stops after 5 seconds or 2 MB.
  • At most one retry per site, and only after a network error or a timeout.
  • A site is checked once per study run. Before a full run, a test run of about 20 sites checks our own rules; those sites can be visited up to three times, and a person may open them once to check the results by hand.

What we publish

  • Only totals and percentages, each with the number of sites behind it. Groups of fewer than 10 sites are merged or left out.
  • No site is named, linked, quoted or shown, in the study page, its charts or its downloadable data.
  • Per-site results stay on our machine. They are not published and are deleted 90 days after the study is published.

How to opt out

Email hello@audacian.eu with your domain. We add it to our opt-out list, which is applied before every run, and remove the site from the next update of any study it was part of.

Or block the crawler in robots.txt. It reads the file before anything else and stops if the homepage is disallowed:

User-agent: AudacianResearch Disallow: /