gmakstutis/openrag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

Merge pull request #284 from langflow-ai/docs-add-robots-txt

Browse source 
docs: add robots.txt and noIndex
This commit is contained in:
Mendon Kissling 2025-10-22 14:18:29 -04:00 • committed by GitHub
commit 2f3b7b69bd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 15 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
docs/docusaurus.config.js
View file

@ -25,6 +25,9 @@ const config = {
// For GitHub pages deployment, it is often '/<projectName>/' // For GitHub pages deployment, it is often '/<projectName>/'
baseUrl: process.env.BASE_URL ? process.env.BASE_URL : '/', baseUrl: process.env.BASE_URL ? process.env.BASE_URL : '/',
// Control search engine indexing - set to true to prevent indexing
noIndex: true,
// GitHub pages deployment config. // GitHub pages deployment config.
// If you aren't using GitHub pages, you don't need these. // If you aren't using GitHub pages, you don't need these.
organizationName: 'langflow-ai', // Usually your GitHub org/user name. organizationName: 'langflow-ai', // Usually your GitHub org/user name.

12
docs/static/robots.txt vendored Normal file
View file

@ -0,0 +1,12 @@
# Robots.txt for OpenRAG Documentation
# Block all crawlers by default
User-agent: *
Disallow: /
# Allow specific crawlers if needed (uncomment when ready for launch)
# User-agent: Googlebot
# Allow: /
# Sitemap location (uncomment when ready for launch)
# Sitemap: https://docs.openr.ag/sitemap.xml