ai-crawler-access-study

Project information

Open dataset: robots.txt and llms.txt of 285 major websites checked against 14 AI user agents plus Googlebot (Oct 2026). CC BY 4.0. Full report: https://airanked.ai/research/ai-crawler-access-report

robots-txt llms-txt ai-crawlers dataset seo

2 Commits
1 Branch
0 Tags

README

Created on

October 11, 2026

Loading