gmakstutis/openrag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add-robots-block-crawlers

Browse source
This commit is contained in:
Mendon Kissling 2025-10-22 09:06:19 -04:00
parent 8c73d49a06
commit 4473447843
1 changed files with 12 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
docs/static/robots.txt vendored Normal file
View file

@ -0,0 +1,12 @@
# Robots.txt for OpenRAG Documentation
# Block all crawlers by default
User-agent: *
Disallow: /
# Allow specific crawlers if needed (uncomment when ready for launch)
# User-agent: Googlebot
# Allow: /
# Sitemap location (uncomment when ready for launch)
# Sitemap: https://docs.openr.ag/sitemap.xml