L

llms-txt

Энциклопедия SOTA реверс-инжиниринга: 9 разделов + верификация 96 URL и 58 репозиториев (GitHub REST API, 2026-09-26)

Open dataset: robots.txt and llms.txt of 285 major websites checked against 14 AI user agents plus Googlebot (Oct 2026). CC BY 4.0. Full report: https://airanked.ai/research/ai-crawler-access-report

MkDocs plugin for SEO metadata, social cards, JSON-LD, Git authors, Copy for LLM, and llms.txt generation.

GitHub: https://github.com/ultralytics/mkdocs

https://pypi.org/project/mkdocs-ultralytics-plugin/

Mirror of github.com/BeeRanked/ai-crawler-checker. See which AI crawlers a site's robots.txt allows or blocks (answers vs training). Node library, CLI and Worker.