gmakstutis/openrag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #683 from langflow-ai/docs-robots-public-preview

Browse source 
Docs: Prepare robots.txt for public site
This commit is contained in:
Sebastián Estévez 2025-12-22 21:12:47 -05:00 • committed by GitHub
commit 900e5f9795
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 4 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
docs/static/robots.txt vendored
View file

@ -1,12 +1,8 @@
# Robots.txt for OpenRAG Documentation
# Block all crawlers by default
# Allow all crawlers
User-agent: *
Disallow: /
Allow: /
# Allow specific crawlers if needed (uncomment when ready for launch)
# User-agent: Googlebot
# Allow: /
# Sitemap location (uncomment when ready for launch)
# Sitemap: https://docs.openr.ag/sitemap.xml
# Sitemap location
Sitemap: https://docs.openr.ag/sitemap.xml