Projects with this topic
Sort by:
-
Engine-aware robots.txt policy for R with Google's matcher algorithm.
Updated -
Open dataset: robots.txt and llms.txt of 285 major websites checked against 14 AI user agents plus Googlebot (Oct 2026). CC BY 4.0. Full report: https://airanked.ai/research/ai-crawler-access-report
Updated -
Install and load the seor family of SEO and developer tools for R.
Updated -
Mirror of github.com/BeeRanked/ai-crawler-checker. See which AI crawlers a site's robots.txt allows or blocks (answers vs training). Node library, CLI and Worker.
Updated
