12 lines
No EOL
218 B
Text
12 lines
No EOL
218 B
Text
# Robots.txt for OpenRAG Documentation
|
|
|
|
# Block all crawlers by default
|
|
User-agent: *
|
|
Disallow: /
|
|
|
|
# Allow specific crawlers
|
|
User-agent: Googlebot
|
|
Allow: /
|
|
|
|
# Sitemap location
|
|
Sitemap: https://docs.openr.ag/sitemap.xml |