gmakstutis/openrag
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'main' into docs-installation

Browse source
This commit is contained in:
Mendon Kissling 2025-09-24 14:40:30 -04:00 • committed by GitHub
commit 12e6341c91
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
54 changed files with 7824 additions and 2515 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.env.example
View file

@ -8,6 +8,8 @@ LANGFLOW_SECRET_KEY=
# flow ids for chat and ingestion flows
LANGFLOW_CHAT_FLOW_ID=1098eea1-6649-4e1d-aed1-b77249fb8dd0
LANGFLOW_INGEST_FLOW_ID=5488df7c-b93f-4f87-a446-b67028bc0813
# Ingest flow using docling
LANGFLOW_INGEST_FLOW_ID=1402618b-e6d1-4ff2-9a11-d6ce71186915
NUDGES_FLOW_ID=ebc01d31-1976-46ce-a385-b0240327226c
# Set a strong admin password for OpenSearch; a bcrypt hash is generated at

48
docs/docs/get-started/intro.mdx
View file

@ -1,48 +0,0 @@
---
title: What is OpenRAG?
slug: /
---
# OpenRAG Introduction
Let's discover **Docusaurus in less than 5 minutes**.
## Getting Started
Get started by **creating a new site**.
Or **try Docusaurus immediately** with **[docusaurus.new](https://docusaurus.new)**.
### What you'll need
- [Node.js](https://nodejs.org/en/download/) version 18.0 or above:
- When installing Node.js, you are recommended to check all checkboxes related to dependencies.
## Generate a new site
Generate a new Docusaurus site using the **classic template**.
The classic template will automatically be added to your project after you run the command:
```bash
npm init docusaurus@latest my-website classic
```
You can type this command into Command Prompt, Powershell, Terminal, or any other integrated terminal of your code editor.
The command also installs all necessary dependencies you need to run Docusaurus.
## Start your site
Run the development server:
```bash
cd my-website
npm run start
```
The `cd` command changes the directory you're working with. In order to work with your newly created Docusaurus site, you'll need to navigate the terminal there.
The `npm run start` command builds your website locally and serves it through a development server, ready for you to view at http://localhost:3000/.
Open `docs/intro.md` (this page) and edit some lines: the site **reloads automatically** and displays your changes.

390
docs/docs/get-started/quickstart.mdx Normal file
View file

@ -0,0 +1,390 @@
---
title: Quickstart
slug: /quickstart
---
import Icon from "@site/src/components/icon/icon";
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
Get started with OpenRAG by loading your knowledge, swapping out your language model, and then chatting with the OpenRAG API.
## Prerequisites
- Install and start OpenRAG
## Find your way around
1. In OpenRAG, click <Icon name="MessageSquare" aria-hidden="true"/> **Chat**.
2. Ask `What documents are available to you?`
The agent responds with a message summarizing the documents that OpenRAG loads by default, which are PDFs about evaluating data quality when using LLMs in health care.
3. To confirm the agent is correct, click <Icon name="Library" aria-hidden="true"/> **Knowledge**.
The **Knowledge** page lists the documents OpenRAG has ingested into the OpenSearch vector database. Click on a document to display the chunks derived from splitting the default documents into the vector database.
## Add your own knowledge
1. To add documents to your knowledge base, click <Icon name="Plus" aria-hidden="true"/> **Add Knowledge**.
* Select **Add File** to add a single file from your local machine (mapped with the Docker volume mount).
* Select **Process Folder** to process an entire folder of documents from your local machine (mapped with the Docker volume mount).
2. Return to the Chat window and ask a question about your loaded data.
For example, with a manual about a PC tablet loaded, ask `How do I connect this device to WiFI?`
The agent responds with a message indicating it now has your knowledge as context for answering questions.
3. Click the <Icon name="Gear" aria-hidden="true"/> **Function Call: search_documents (tool_call)** that is printed in the Playground.
These events log the agent's request to the tool and the tool's response, so you have direct visibility into your agent's functionality.
If you aren't getting the results you need, you can further tune the knowledge ingestion and agent behavior in the next section.
## Swap out the language model to modify agent behavior
To modify the knowledge ingestion or Agent behavior, click <Icon name="Settings" aria-hidden="true"/> **Settings**.
In this example, you'll try a different LLM to demonstrate how the Agent's response changes.
1. To edit the Agent's behavior, click **Edit in Langflow**.
2. OpenRAG warns you that you're entering Langflow. Click **Proceed**.
3. The OpenRAG Open Search Agent flow appears.
![OpenRAG Open Search Agent Flow](/img/opensearch-agent-flow.png)
4. In the **Language Model** component, under **Model Provider**, select **Anthropic**.
:::note
This guide uses an Anthropic model for demonstration purposes. If you want to use a different provider, change the **Model Provider** and **Model Name** fields, and then provide credentials for your selected provider.
:::
5. Save your flow with <kbd>Command+S</kbd>.
6. In OpenRAG, start a new conversation by clicking the <Icon name="Plus" aria-hidden="true"/> in the **Conversations** tab.
7. Ask the same question as before to demonstrate how a different language model changes the results.
## Integrate OpenRAG into your application
:::tip
Ensure the `openrag-backend` container has port 8000 exposed in your `docker-compose.yml`:
```yaml
openrag-backend:
ports:
- "8000:8000"
```
:::
OpenRAG provides a REST API that you can call from Python, TypeScript, or any HTTP client to chat with your documents.
These example requests are run assuming OpenRAG is in "no-auth" mode.
For complete API documentation, including authentication, request and response parameters, and example requests, see the API documentation.
### Chat with your documents
Prompt OpenRAG at the `/chat` API endpoint.
<Tabs>
<TabItem value="python" label="Python">
```python
import requests
url = "http://localhost:8000/chat"
payload = {
"prompt": "What documents are available to you?",
"previous_response_id": None
}
response = requests.post(url, json=payload)
print("OpenRAG Response:", response.json())
```
</TabItem>
<TabItem value="typescript" label="TypeScript">
```typescript
import fetch from 'node-fetch';
const response = await fetch("http://localhost:8000/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
prompt: "What documents are available to you?",
previous_response_id: null
})
});
const data = await response.json();
console.log("OpenRAG Response:", data);
```
</TabItem>
<TabItem value="curl" label="curl">
```bash
curl -X POST "http://localhost:8000/chat" \
-H "Content-Type: application/json" \
-d '{
"prompt": "What documents are available to you?",
"previous_response_id": null
}'
```
</TabItem>
</Tabs>
<details closed>
<summary>Response</summary>
```
{
"response": "I have access to a wide range of documents depending on the context and the tools enabled in this environment. Specifically, I can search for and retrieve documents related to various topics such as technical papers, articles, manuals, guides, knowledge base entries, and other text-based resources. If you specify a particular subject or type of document you're interested in, I can try to locate relevant materials for you. Let me know what you need!",
"response_id": "resp_68d3fdbac93081958b8781b97919fe7007f98bd83932fa1a"
}
```
</details>
### Search your documents
Search your document knowledge base at the `/search` endpoint.
<Tabs>
<TabItem value="python" label="Python">
```python
import requests
url = "http://localhost:8000/search"
payload = {"query": "healthcare data quality", "limit": 5}
response = requests.post(url, json=payload)
results = response.json()
print("Search Results:")
for result in results.get("results", []):
print(f"- {result.get('filename')}: {result.get('text', '')[:100]}...")
```
</TabItem>
<TabItem value="typescript" label="TypeScript">
```typescript
const response = await fetch("http://localhost:8000/search", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
query: "healthcare data quality",
limit: 5
})
});
const results = await response.json();
console.log("Search Results:");
results.results?.forEach((result, index) => {
const filename = result.filename || 'Unknown';
const text = result.text?.substring(0, 100) || '';
console.log(`${index + 1}. ${filename}: ${text}...`);
});
```
</TabItem>
<TabItem value="curl" label="curl">
```bash
curl -X POST "http://localhost:8000/search" \
-H "Content-Type: application/json" \
-d '{"query": "healthcare data quality", "limit": 5}'
```
</TabItem>
</Tabs>
<details closed>
<summary>Example response</summary>
```
Found 5 results
1. 2506.08231v1.pdf: variables with high performance metrics. These variables might also require fewer replication analys...
2. 2506.08231v1.pdf: on EHR data and may lack the clinical domain knowledge needed to perform well on the tasks where EHR...
3. 2506.08231v1.pdf: Abstract Large language models (LLMs) are increasingly used to extract clinical data from electronic...
4. 2506.08231v1.pdf: these multidimensional assessments, the framework not only quantifies accuracy, but can also be appl...
5. 2506.08231v1.pdf: observed in only the model metrics, but not the abstractor metrics, it indicates that model errors m...
```
</details>
### Use chat and search together
Create a complete chat application that combines an interactive terminal chat with session continuity and search functionality.
<Tabs>
<TabItem value="python" label="Python">
```python
import requests
# Configuration
OPENRAG_BASE_URL = "http://localhost:8000"
CHAT_URL = f"{OPENRAG_BASE_URL}/chat"
SEARCH_URL = f"{OPENRAG_BASE_URL}/search"
DEFAULT_SEARCH_LIMIT = 5
def chat_with_openrag(message, previous_response_id=None):
try:
response = requests.post(CHAT_URL, json={
"prompt": message,
"previous_response_id": previous_response_id
})
response.raise_for_status()
data = response.json()
return data.get("response"), data.get("response_id")
except Exception as e:
return f"Error: {str(e)}", None
def search_documents(query, limit=DEFAULT_SEARCH_LIMIT):
try:
response = requests.post(SEARCH_URL, json={
"query": query,
"limit": limit
})
response.raise_for_status()
data = response.json()
return data.get("results", [])
except Exception as e:
return []
# Interactive chat with session continuity and search
previous_response_id = None
while True:
question = input("Your question (or 'search <query>' to search): ").strip()
if question.lower() in ['quit', 'exit', 'q']:
break
if not question:
continue
if question.lower().startswith('search '):
query = question[7:].strip()
print("Searching documents...")
results = search_documents(query)
print(f"\nFound {len(results)} results:")
for i, result in enumerate(results, 1):
filename = result.get('filename', 'Unknown')
text = result.get('text', '')[:100]
print(f"{i}. {filename}: {text}...")
print()
else:
print("OpenRAG is thinking...")
result, response_id = chat_with_openrag(question, previous_response_id)
print(f"OpenRAG: {result}\n")
previous_response_id = response_id
```
</TabItem>
<TabItem value="typescript" label="TypeScript">
```ts
import fetch from 'node-fetch';
// Configuration
const OPENRAG_BASE_URL = "http://localhost:8000";
const CHAT_URL = `${OPENRAG_BASE_URL}/chat`;
const SEARCH_URL = `${OPENRAG_BASE_URL}/search`;
const DEFAULT_SEARCH_LIMIT = 5;
async function chatWithOpenRAG(message: string, previousResponseId?: string | null) {
try {
const response = await fetch(CHAT_URL, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
prompt: message,
previous_response_id: previousResponseId
})
});
const data = await response.json();
return [data.response || "No response received", data.response_id || null];
} catch (error) {
return [`Error: ${error}`, null];
}
}
async function searchDocuments(query: string, limit: number = DEFAULT_SEARCH_LIMIT) {
try {
const response = await fetch(SEARCH_URL, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ query, limit })
});
const data = await response.json();
return data.results || [];
} catch (error) {
return [];
}
}
// Interactive chat with session continuity and search
let previousResponseId = null;
const readline = require('readline');
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
const askQuestion = () => {
rl.question("Your question (or 'search <query>' to search): ", async (question) => {
if (question.toLowerCase() === 'quit' || question.toLowerCase() === 'exit' || question.toLowerCase() === 'q') {
console.log("Goodbye!");
rl.close();
return;
}
if (!question.trim()) {
askQuestion();
return;
}
if (question.toLowerCase().startsWith('search ')) {
const query = question.substring(7).trim();
console.log("Searching documents...");
const results = await searchDocuments(query);
console.log(`\nFound ${results.length} results:`);
results.forEach((result, i) => {
const filename = result.filename || 'Unknown';
const text = result.text?.substring(0, 100) || '';
console.log(`${i + 1}. ${filename}: ${text}...`);
});
console.log();
} else {
console.log("OpenRAG is thinking...");
const [result, responseId] = await chatWithOpenRAG(question, previousResponseId);
console.log(`\nOpenRAG: ${result}\n`);
previousResponseId = responseId;
}
askQuestion();
});
};
console.log("OpenRAG Chat Interface");
console.log("Ask questions about your documents. Type 'quit' to exit.");
console.log("Use 'search <query>' to search documents directly.\n");
askQuestion();
```
</TabItem>
</Tabs>
<details closed>
<summary>Example response</summary>
```
Your question (or 'search <query>' to search): search healthcare
Searching documents...
Found 5 results:
1. 2506.08231v1.pdf: variables with high performance metrics. These variables might also require fewer replication analys...
2. 2506.08231v1.pdf: on EHR data and may lack the clinical domain knowledge needed to perform well on the tasks where EHR...
3. 2506.08231v1.pdf: Abstract Large language models (LLMs) are increasingly used to extract clinical data from electronic...
4. 2506.08231v1.pdf: Acknowledgements Darren Johnson for support in publication planning and management. The authors used...
5. 2506.08231v1.pdf: Ensuring Reliability of Curated EHR-Derived Data: The Validation of Accuracy for LLM/ML-Extracted In...
Your question (or 'search <query>' to search): what's the weather today?
OpenRAG is thinking...
OpenRAG: I don't have access to real-time weather data. Could you please provide me with your location? Then I can help you find the weather information.
Your question (or 'search <query>' to search): newark nj
OpenRAG is thinking...
```
</details>
## Next steps
TBD

19
docs/docs/get-started/what-is-openrag.mdx Normal file
View file

@ -0,0 +1,19 @@
---
title: What is OpenRAG?
slug: /
---
OpenRAG is an open-source package for building agentic RAG systems.
It supports integration with a wide range of orchestration tools, vector databases, and LLM providers.
OpenRAG connects and amplifies three popular, proven open-source projects into one powerful platform:
* [Langflow](https://docs.langflow.org) - Langflow is a powerful tool to build and deploy AI agents and MCP servers. It supports all major LLMs, vector databases and a growing library of AI tools.
* [OpenSearch](https://docs.opensearch.org/latest/) - OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.
* [Docling](https://docling-project.github.io/docling/) - Docling simplifies document processing, parsing diverse formats — including advanced PDF understanding — and providing seamless integrations with the gen AI ecosystem.
OpenRAG builds on Langflow's familiar interface while adding OpenSearch for vector storage and Docling for simplified document parsing, with opinionated flows that serve as ready-to-use recipes for ingestion, retrieval, and generation from popular sources like OneDrive, Google Drive, and AWS. And don't fear: every part of the stack is swappable. Write your own custom components in Python, try different language models, and customize your flows to build an agentic RAG system that solves problems.
Ready to get started? Install OpenRAG and then run the Quickstart to create a powerful RAG pipeline.

10
docs/package-lock.json generated
View file

@ -12,6 +12,7 @@
"@docusaurus/preset-classic": "3.8.1",
"@mdx-js/react": "^3.0.0",
"clsx": "^2.0.0",
"lucide-react": "^0.544.0",
"prism-react-renderer": "^2.3.0",
"react": "^19.0.0",
"react-dom": "^19.0.0"
@ -9801,6 +9802,15 @@
"yallist": "^3.0.2"
}
},
"node_modules/lucide-react": {
"version": "0.544.0",
"resolved": "https://registry.npmjs.org/lucide-react/-/lucide-react-0.544.0.tgz",
"integrity": "sha512-t5tS44bqd825zAW45UQxpG2CvcC4urOwn2TrwSH8u+MjeE+1NnWl6QqeQ/6NdjMqdOygyiT9p3Ev0p1NJykxjw==",
"license": "ISC",
"peerDependencies": {
"react": "^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0"
}
},
"node_modules/markdown-extensions": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/markdown-extensions/-/markdown-extensions-2.0.0.tgz",

1
docs/package.json
View file

@ -19,6 +19,7 @@
"@docusaurus/preset-classic": "3.8.1",
"@mdx-js/react": "^3.0.0",
"clsx": "^2.0.0",
"lucide-react": "^0.544.0",
"prism-react-renderer": "^2.3.0",
"react": "^19.0.0",
"react-dom": "^19.0.0"

9
docs/sidebars.js
View file

@ -22,14 +22,19 @@ const sidebars = {
items: [
{
type: "doc",
id: "get-started/intro",
label: "Introduction"
id: "get-started/what-is-openrag",
label: "About OpenRAG"
},
{
type: "doc",
id: "get-started/install",
label: "Installation"
},
{
type: "doc",
id: "get-started/quickstart",
label: "Quickstart"
},
{
type: "doc",
id: "get-started/docker",

19
docs/src/components/icon/icon.tsx Normal file
View file

@ -0,0 +1,19 @@
import React from "react";
import * as LucideIcons from "lucide-react";
/*
How to use this component:
import Icon from "@site/src/components/icon";
<Icon name="AlertCircle" size={24} color="red" />
*/
type IconProps = {
name: string;
};
export default function Icon({ name, ...props }: IconProps) {
const Icon = LucideIcons[name];
return Icon ? <Icon {...props} /> : null;
}

BIN
docs/static/img/opensearch-agent-flow.png vendored Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 951 KiB

2534
docs/yarn.lock
View file

File diff suppressed because it is too large Load diff

40
flows/components/ollama_embedding.json
View file

@ -2,7 +2,9 @@
"data": {
"id": "OllamaEmbeddings-4ah5Q",
"node": {
"base_classes": ["Embeddings"],
"base_classes": [
"Embeddings"
],
"beta": false,
"conditional_paths": [],
"custom_fields": {},
@ -10,10 +12,13 @@
"display_name": "Ollama Embeddings",
"documentation": "https://python.langchain.com/docs/integrations/text_embedding/ollama",
"edited": false,
"field_order": ["model_name", "base_url"],
"field_order": [
"model_name",
"base_url"
],
"frozen": false,
"icon": "Ollama",
"last_updated": "2025-09-17T20:01:59.954Z",
"last_updated": "2025-09-22T20:18:27.128Z",
"legacy": false,
"metadata": {
"code_hash": "0db0f99e91e9",
@ -29,12 +34,17 @@
},
{
"name": "langflow",
"version": "1.5.0.post2"
"version": null
}
],
"total_dependencies": 3
},
"keywords": ["model", "llm", "language model", "large language model"],
"keywords": [
"model",
"llm",
"language model",
"large language model"
],
"module": "langflow.components.ollama.ollama_embeddings.OllamaEmbeddingsComponent"
},
"minimized": false,
@ -51,7 +61,9 @@
"required_inputs": null,
"selected": "Embeddings",
"tool_mode": true,
"types": ["Embeddings"],
"types": [
"Embeddings"
],
"value": "__UNDEFINED__"
}
],
@ -64,7 +76,9 @@
"display_name": "Ollama Base URL",
"dynamic": false,
"info": "",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": true,
@ -106,7 +120,9 @@
"dynamic": false,
"info": "",
"name": "model_name",
"options": [],
"options": [
"all-minilm:latest"
],
"options_metadata": [],
"placeholder": "",
"real_time_refresh": true,
@ -118,7 +134,7 @@
"tool_mode": false,
"trace_as_metadata": true,
"type": "str",
"value": ""
"value": "all-minilm:latest"
}
},
"tool_mode": false
@ -133,9 +149,9 @@
"width": 320
},
"position": {
"x": 964,
"y": 248
"x": 282.29416840859585,
"y": 279.4218065717267
},
"selected": false,
"type": "genericNode"
}
}

75
flows/components/ollama_llm.json
View file

@ -2,7 +2,10 @@
"data": {
"id": "OllamaModel-eCsJx",
"node": {
"base_classes": ["LanguageModel", "Message"],
"base_classes": [
"LanguageModel",
"Message"
],
"beta": false,
"conditional_paths": [],
"custom_fields": {},
@ -40,7 +43,7 @@
],
"frozen": false,
"icon": "Ollama",
"last_updated": "2025-09-17T20:01:59.191Z",
"last_updated": "2025-09-22T20:14:45.057Z",
"legacy": false,
"metadata": {
"code_hash": "af399d429d23",
@ -56,12 +59,17 @@
},
{
"name": "langflow",
"version": "1.5.0.post2"
"version": null
}
],
"total_dependencies": 3
},
"keywords": ["model", "llm", "language model", "large language model"],
"keywords": [
"model",
"llm",
"language model",
"large language model"
],
"module": "langflow.components.ollama.ollama.ChatOllamaComponent"
},
"minimized": false,
@ -77,7 +85,9 @@
"options": null,
"required_inputs": null,
"tool_mode": true,
"types": ["Message"],
"types": [
"Message"
],
"value": "__UNDEFINED__"
},
{
@ -91,7 +101,9 @@
"required_inputs": null,
"selected": "LanguageModel",
"tool_mode": true,
"types": ["LanguageModel"],
"types": [
"LanguageModel"
],
"value": "__UNDEFINED__"
}
],
@ -104,7 +116,9 @@
"display_name": "Base URL",
"dynamic": false,
"info": "Endpoint of the Ollama API.",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": true,
@ -144,7 +158,9 @@
"display_name": "Format",
"dynamic": false,
"info": "Specify the format of the output (e.g., json).",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -165,7 +181,9 @@
"display_name": "Input",
"dynamic": false,
"info": "",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -207,7 +225,11 @@
"dynamic": false,
"info": "Enable/disable Mirostat sampling for controlling perplexity.",
"name": "mirostat",
"options": ["Disabled", "Mirostat", "Mirostat 2.0"],
"options": [
"Disabled",
"Mirostat",
"Mirostat 2.0"
],
"options_metadata": [],
"placeholder": "",
"real_time_refresh": true,
@ -265,7 +287,9 @@
"dynamic": false,
"info": "Refer to https://ollama.com/library for more models.",
"name": "model_name",
"options": [],
"options": [
"qwen3:4b"
],
"options_metadata": [],
"placeholder": "",
"real_time_refresh": true,
@ -277,7 +301,7 @@
"tool_mode": false,
"trace_as_metadata": true,
"type": "str",
"value": ""
"value": "qwen3:4b"
},
"num_ctx": {
"_input_type": "IntInput",
@ -375,7 +399,9 @@
"display_name": "Stop Tokens",
"dynamic": false,
"info": "Comma-separated list of tokens to signal the model to stop generating text.",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -414,7 +440,9 @@
"display_name": "System",
"dynamic": false,
"info": "System to use for generating text.",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -436,7 +464,9 @@
"display_name": "System Message",
"dynamic": false,
"info": "System message to pass to the model.",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -458,7 +488,9 @@
"display_name": "Tags",
"dynamic": false,
"info": "Comma-separated list of tags to add to the run trace.",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -507,7 +539,9 @@
"display_name": "Template",
"dynamic": false,
"info": "Template to use for generating text.",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -638,15 +672,16 @@
"showNode": true,
"type": "OllamaModel"
},
"dragging": false,
"id": "OllamaModel-eCsJx",
"measured": {
"height": 494,
"width": 320
},
"position": {
"x": 554,
"y": 225
"x": 248.08287272472313,
"y": 216.98088326271431
},
"selected": false,
"type": "genericNode"
}
}

687
flows/components/ollama_llm_text.json Normal file
View file

File diff suppressed because one or more lines are too long

37
flows/components/watsonx_embedding.json
View file

@ -2,7 +2,9 @@
"data": {
"id": "WatsonxEmbeddingsComponent-pJfXI",
"node": {
"base_classes": ["Embeddings"],
"base_classes": [
"Embeddings"
],
"beta": false,
"conditional_paths": [],
"custom_fields": {},
@ -20,6 +22,7 @@
],
"frozen": false,
"icon": "WatsonxAI",
"last_updated": "2025-09-22T20:11:38.181Z",
"legacy": false,
"metadata": {
"code_hash": "b6c6d50cc7ed",
@ -43,7 +46,7 @@
},
{
"name": "langflow",
"version": "1.5.0.post2"
"version": null
}
],
"total_dependencies": 5
@ -60,9 +63,13 @@
"group_outputs": false,
"method": "build_embeddings",
"name": "embeddings",
"options": null,
"required_inputs": null,
"selected": "Embeddings",
"tool_mode": true,
"types": ["Embeddings"],
"types": [
"Embeddings"
],
"value": "__UNDEFINED__"
}
],
@ -131,7 +138,16 @@
"dynamic": true,
"info": "",
"name": "model_name",
"options": [],
"options": [
"ibm/granite-embedding-107m-multilingual",
"ibm/granite-embedding-278m-multilingual",
"ibm/slate-125m-english-rtrvr",
"ibm/slate-125m-english-rtrvr-v2",
"ibm/slate-30m-english-rtrvr",
"ibm/slate-30m-english-rtrvr-v2",
"intfloat/multilingual-e5-large",
"sentence-transformers/all-minilm-l6-v2"
],
"options_metadata": [],
"placeholder": "",
"required": true,
@ -140,7 +156,8 @@
"toggle": false,
"tool_mode": false,
"trace_as_metadata": true,
"type": "str"
"type": "str",
"value": "ibm/granite-embedding-107m-multilingual"
},
"project_id": {
"_input_type": "StrInput",
@ -205,7 +222,8 @@
"toggle": false,
"tool_mode": false,
"trace_as_metadata": true,
"type": "str"
"type": "str",
"value": "https://us-south.ml.cloud.ibm.com"
}
},
"tool_mode": false
@ -213,15 +231,16 @@
"showNode": true,
"type": "WatsonxEmbeddingsComponent"
},
"dragging": false,
"id": "WatsonxEmbeddingsComponent-pJfXI",
"measured": {
"height": 467,
"width": 320
},
"position": {
"x": 999.129592360849,
"y": 753.2332292351236
"x": 364.4406919374723,
"y": 282.29319267029086
},
"selected": false,
"type": "genericNode"
}
}

69
flows/components/watsonx_llm.json
View file

@ -2,7 +2,10 @@
"data": {
"id": "IBMwatsonxModel-jA4Nw",
"node": {
"base_classes": ["LanguageModel", "Message"],
"base_classes": [
"LanguageModel",
"Message"
],
"beta": false,
"conditional_paths": [],
"custom_fields": {},
@ -31,6 +34,7 @@
],
"frozen": false,
"icon": "WatsonxAI",
"last_updated": "2025-09-22T20:03:31.248Z",
"legacy": false,
"metadata": {
"code_hash": "7767fd69a954",
@ -50,12 +54,17 @@
},
{
"name": "langflow",
"version": "1.5.0.post2"
"version": null
}
],
"total_dependencies": 4
},
"keywords": ["model", "llm", "language model", "large language model"],
"keywords": [
"model",
"llm",
"language model",
"large language model"
],
"module": "langflow.components.ibm.watsonx.WatsonxAIComponent"
},
"minimized": false,
@ -68,8 +77,12 @@
"group_outputs": false,
"method": "text_response",
"name": "text_output",
"options": null,
"required_inputs": null,
"tool_mode": true,
"types": ["Message"],
"types": [
"Message"
],
"value": "__UNDEFINED__"
},
{
@ -79,9 +92,13 @@
"group_outputs": false,
"method": "build_model",
"name": "model_output",
"options": null,
"required_inputs": null,
"selected": "LanguageModel",
"tool_mode": true,
"types": ["LanguageModel"],
"types": [
"LanguageModel"
],
"value": "__UNDEFINED__"
}
],
@ -157,7 +174,9 @@
"display_name": "Input",
"dynamic": false,
"info": "",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -242,7 +261,26 @@
"dynamic": true,
"info": "",
"name": "model_name",
"options": [],
"options": [
"ibm/granite-3-2-8b-instruct",
"ibm/granite-3-2b-instruct",
"ibm/granite-3-3-8b-instruct",
"ibm/granite-3-8b-instruct",
"ibm/granite-guardian-3-2b",
"ibm/granite-guardian-3-8b",
"ibm/granite-vision-3-2-2b",
"meta-llama/llama-3-2-11b-vision-instruct",
"meta-llama/llama-3-2-90b-vision-instruct",
"meta-llama/llama-3-3-70b-instruct",
"meta-llama/llama-3-405b-instruct",
"meta-llama/llama-4-maverick-17b-128e-instruct-fp8",
"meta-llama/llama-guard-3-11b-vision",
"mistralai/mistral-large",
"mistralai/mistral-medium-2505",
"mistralai/mistral-small-3-1-24b-instruct-2503",
"mistralai/pixtral-12b",
"openai/gpt-oss-120b"
],
"options_metadata": [],
"placeholder": "",
"required": true,
@ -251,7 +289,8 @@
"toggle": false,
"tool_mode": false,
"trace_as_metadata": true,
"type": "str"
"type": "str",
"value": "ibm/granite-3-2-8b-instruct"
},
"presence_penalty": {
"_input_type": "SliderInput",
@ -362,7 +401,9 @@
"display_name": "System Message",
"dynamic": false,
"info": "System message to pass to the model.",
"input_types": ["Message"],
"input_types": [
"Message"
],
"list": false,
"list_add_label": "Add More",
"load_from_db": false,
@ -484,7 +525,8 @@
"toggle": false,
"tool_mode": false,
"trace_as_metadata": true,
"type": "str"
"type": "str",
"value": "https://us-south.ml.cloud.ibm.com"
}
},
"tool_mode": false
@ -493,15 +535,16 @@
"showNode": true,
"type": "IBMwatsonxModel"
},
"dragging": false,
"id": "IBMwatsonxModel-jA4Nw",
"measured": {
"height": 632,
"width": 320
},
"position": {
"x": 562.2658900512183,
"y": 895.3455179382565
"x": 371.93566807042805,
"y": 197.47711431325635
},
"selected": false,
"type": "genericNode"
}
}

551
flows/components/watsonx_llm_text.json Normal file
View file

File diff suppressed because one or more lines are too long

2220
flows/openrag_ingest_docling.json Normal file
View file

File diff suppressed because one or more lines are too long

887
flows/openrag_nudges.json
View file

File diff suppressed because one or more lines are too long

14
frontend/components/knowledge-actions-dropdown.tsx
View file

@ -12,6 +12,7 @@ import { Button } from "./ui/button";
import { DeleteConfirmationDialog } from "./confirmation-dialog";
import { useDeleteDocument } from "@/app/api/mutations/useDeleteDocument";
import { toast } from "sonner";
import { useRouter } from "next/navigation";
interface KnowledgeActionsDropdownProps {
filename: string;
@ -22,6 +23,7 @@ export const KnowledgeActionsDropdown = ({
}: KnowledgeActionsDropdownProps) => {
const [showDeleteDialog, setShowDeleteDialog] = useState(false);
const deleteDocumentMutation = useDeleteDocument();
const router = useRouter();
const handleDelete = async () => {
try {
@ -43,7 +45,17 @@ export const KnowledgeActionsDropdown = ({
<EllipsisVertical className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent side="right" sideOffset={-10}>
<DropdownMenuContent side="right" align="start" sideOffset={-10}>
<DropdownMenuItem
className="text-primary focus:text-primary"
onClick={() => {
router.push(
`/knowledge/chunks?filename=${encodeURIComponent(filename)}`
);
}}
>
View chunks
</DropdownMenuItem>
{/* //TODO: Implement rename and sync */}
{/* <DropdownMenuItem
className="text-primary focus:text-primary"

271
frontend/components/knowledge-filter-list.tsx Normal file
View file

@ -0,0 +1,271 @@
"use client";
import { useState } from "react";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Textarea } from "@/components/ui/textarea";
import { Filter, Loader2, Plus, Save, X } from "lucide-react";
import { cn } from "@/lib/utils";
import {
useGetFiltersSearchQuery,
type KnowledgeFilter,
} from "@/src/app/api/queries/useGetFiltersSearchQuery";
import { useCreateFilter } from "@/src/app/api/mutations/useCreateFilter";
import {
Dialog,
DialogContent,
DialogDescription,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
interface ParsedQueryData {
query: string;
filters: {
data_sources: string[];
document_types: string[];
owners: string[];
};
limit: number;
scoreThreshold: number;
}
interface KnowledgeFilterListProps {
selectedFilter: KnowledgeFilter | null;
onFilterSelect: (filter: KnowledgeFilter | null) => void;
}
export function KnowledgeFilterList({
selectedFilter,
onFilterSelect,
}: KnowledgeFilterListProps) {
const [searchQuery] = useState("");
const [showCreateModal, setShowCreateModal] = useState(false);
const [createName, setCreateName] = useState("");
const [createDescription, setCreateDescription] = useState("");
const [creating, setCreating] = useState(false);
const { data, isFetching: loading } = useGetFiltersSearchQuery(
searchQuery,
20
);
const filters = data || [];
const createFilterMutation = useCreateFilter();
const handleFilterSelect = (filter: KnowledgeFilter) => {
onFilterSelect(filter);
};
const handleCreateNew = () => {
setShowCreateModal(true);
};
const handleCreateFilter = async () => {
if (!createName.trim()) return;
setCreating(true);
try {
// Create a basic filter with wildcards (match everything by default)
const defaultFilterData = {
query: "",
filters: {
data_sources: ["*"],
document_types: ["*"],
owners: ["*"],
},
limit: 10,
scoreThreshold: 0,
};
const result = await createFilterMutation.mutateAsync({
name: createName.trim(),
description: createDescription.trim(),
queryData: JSON.stringify(defaultFilterData),
});
// Select the new filter from API response
onFilterSelect(result.filter);
// Close modal and reset form
setShowCreateModal(false);
setCreateName("");
setCreateDescription("");
} catch (error) {
console.error("Error creating knowledge filter:", error);
} finally {
setCreating(false);
}
};
const handleCancelCreate = () => {
setShowCreateModal(false);
setCreateName("");
setCreateDescription("");
};
const parseQueryData = (queryData: string): ParsedQueryData => {
return JSON.parse(queryData) as ParsedQueryData;
};
return (
<>
<div className="flex flex-col items-center gap-1 px-3 !mb-12 mt-0 h-full overflow-y-auto">
<div className="flex items-center w-full justify-between pl-3">
<div className="text-sm font-medium text-muted-foreground">
Knowledge Filters
</div>
<Button
variant="ghost"
size="sm"
onClick={handleCreateNew}
title="Create New Filter"
className="h-8 px-3 text-muted-foreground"
>
<Plus className="h-3 w-3" />
</Button>
</div>
{loading ? (
<div className="flex items-center justify-center p-4">
<Loader2 className="h-4 w-4 animate-spin" />
<span className="ml-2 text-sm text-muted-foreground">
Loading...
</span>
</div>
) : filters.length === 0 ? (
<div className="p-4 text-center text-sm text-muted-foreground">
{searchQuery ? "No filters found" : "No saved filters"}
</div>
) : (
filters.map((filter) => (
<div
key={filter.id}
onClick={() => handleFilterSelect(filter)}
className={cn(
"flex items-center gap-3 px-3 py-2 w-full rounded-lg hover:bg-accent hover:text-accent-foreground cursor-pointer group transition-colors",
selectedFilter?.id === filter.id &&
"bg-accent text-accent-foreground"
)}
>
<div className="flex flex-col gap-1 flex-1 min-w-0">
<div className="flex items-center gap-2">
<div className="flex items-center justify-center bg-blue-500/20 w-5 h-5 rounded">
<Filter className="h-3 w-3 text-blue-400" />
</div>
<div className="text-sm font-medium truncate group-hover:text-accent-foreground">
{filter.name}
</div>
</div>
{filter.description && (
<div className="text-xs text-muted-foreground group-hover:text-accent-foreground/70 line-clamp-2">
{filter.description}
</div>
)}
<div className="flex items-center gap-2">
<div className="text-xs text-muted-foreground group-hover:text-accent-foreground/70">
{new Date(filter.created_at).toLocaleDateString(undefined, {
month: "short",
day: "numeric",
year: "numeric",
})}
</div>
<span className="text-xs bg-muted text-muted-foreground px-1 py-0.5 rounded-sm">
{(() => {
const dataSources = parseQueryData(filter.query_data)
.filters.data_sources;
if (dataSources[0] === "*") return "All sources";
const count = dataSources.length;
return `${count} ${count === 1 ? "source" : "sources"}`;
})()}
</span>
</div>
</div>
{selectedFilter?.id === filter.id && (
<Button
variant="ghost"
size="sm"
className="px-0"
onClick={(e) => {
e.stopPropagation();
onFilterSelect(null);
}}
>
<X className="h-4 w-4 flex-shrink-0 opacity-0 group-hover:opacity-100 text-muted-foreground" />
</Button>
)}
</div>
))
)}
</div>
{/* Create Filter Dialog */}
<Dialog open={showCreateModal} onOpenChange={setShowCreateModal}>
<DialogContent>
<DialogHeader>
<DialogTitle>Create a new knowledge filter</DialogTitle>
<DialogDescription>
Save a reusable filter to quickly scope searches across your
knowledge base.
</DialogDescription>
</DialogHeader>
<div className="flex flex-col gap-2 space-y-2">
<div>
<Label htmlFor="filter-name" className="font-medium mb-2 gap-1">
Name<span className="text-red-400">*</span>
</Label>
<Input
id="filter-name"
type="text"
placeholder="Enter filter name"
value={createName}
onChange={(e) => setCreateName(e.target.value)}
className="mt-1"
/>
</div>
<div>
<Label htmlFor="filter-description" className="font-medium mb-2">
Description (optional)
</Label>
<Textarea
id="filter-description"
placeholder="Brief description of this filter"
value={createDescription}
onChange={(e) => setCreateDescription(e.target.value)}
className="mt-1"
rows={3}
/>
</div>
</div>
<div className="flex justify-end gap-2">
<Button
variant="outline"
onClick={handleCancelCreate}
disabled={creating}
>
Cancel
</Button>
<Button
onClick={handleCreateFilter}
disabled={!createName.trim() || creating}
className="flex items-center gap-2"
>
{creating ? (
<>
<Loader2 className="h-4 w-4 animate-spin" />
Creating...
</>
) : (
<>
<Save className="h-4 w-4" />
Create Filter
</>
)}
</Button>
</div>
</DialogContent>
</Dialog>
</>
);
}

431
frontend/components/knowledge-filter-panel.tsx
View file

@ -1,133 +1,115 @@
"use client"
import { useState, useEffect } from 'react'
import { X, Edit3, Save, Settings, RefreshCw } from 'lucide-react'
import { Button } from '@/components/ui/button'
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card'
import { Input } from '@/components/ui/input'
import { Label } from '@/components/ui/label'
import { Textarea } from '@/components/ui/textarea'
import { MultiSelect } from '@/components/ui/multi-select'
import { Slider } from '@/components/ui/slider'
import { useKnowledgeFilter } from '@/contexts/knowledge-filter-context'
"use client";
import { useState, useEffect } from "react";
import { X, Edit3, Save, RefreshCw } from "lucide-react";
import { Button } from "@/components/ui/button";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Textarea } from "@/components/ui/textarea";
import { MultiSelect } from "@/components/ui/multi-select";
import { Slider } from "@/components/ui/slider";
import { useKnowledgeFilter } from "@/contexts/knowledge-filter-context";
import { useDeleteFilter } from "@/app/api/mutations/useDeleteFilter";
import { useUpdateFilter } from "@/app/api/mutations/useUpdateFilter";
import { useGetSearchAggregations } from "@/src/app/api/queries/useGetSearchAggregations";
interface FacetBucket {
key: string
count: number
key: string;
count: number;
}
interface AvailableFacets {
data_sources: FacetBucket[]
document_types: FacetBucket[]
owners: FacetBucket[]
connector_types: FacetBucket[]
data_sources: FacetBucket[];
document_types: FacetBucket[];
owners: FacetBucket[];
connector_types: FacetBucket[];
}
export function KnowledgeFilterPanel() {
const { selectedFilter, parsedFilterData, setSelectedFilter, isPanelOpen, closePanelOnly } = useKnowledgeFilter()
const {
selectedFilter,
parsedFilterData,
setSelectedFilter,
isPanelOpen,
closePanelOnly,
} = useKnowledgeFilter();
const deleteFilterMutation = useDeleteFilter();
const updateFilterMutation = useUpdateFilter();
// Edit mode states
const [isEditingMeta, setIsEditingMeta] = useState(false)
const [editingName, setEditingName] = useState('')
const [editingDescription, setEditingDescription] = useState('')
const [isSaving, setIsSaving] = useState(false)
const [isEditingMeta, setIsEditingMeta] = useState(false);
const [editingName, setEditingName] = useState("");
const [editingDescription, setEditingDescription] = useState("");
const [isSaving, setIsSaving] = useState(false);
// Filter configuration states (mirror search page exactly)
const [query, setQuery] = useState('')
const [query, setQuery] = useState("");
const [selectedFilters, setSelectedFilters] = useState({
data_sources: ["*"] as string[], // Default to wildcard
document_types: ["*"] as string[], // Default to wildcard
owners: ["*"] as string[], // Default to wildcard
connector_types: ["*"] as string[] // Default to wildcard
})
const [resultLimit, setResultLimit] = useState(10)
const [scoreThreshold, setScoreThreshold] = useState(0)
connector_types: ["*"] as string[], // Default to wildcard
});
const [resultLimit, setResultLimit] = useState(10);
const [scoreThreshold, setScoreThreshold] = useState(0);
// Available facets (loaded from API)
const [availableFacets, setAvailableFacets] = useState<AvailableFacets>({
data_sources: [],
document_types: [],
owners: [],
connector_types: []
})
connector_types: [],
});
// Load current filter data into controls
useEffect(() => {
if (selectedFilter && parsedFilterData) {
setQuery(parsedFilterData.query || '')
setQuery(parsedFilterData.query || "");
// Set the actual filter selections from the saved knowledge filter
const filters = parsedFilterData.filters
const filters = parsedFilterData.filters;
// Use the exact selections from the saved filter
// Empty arrays mean "none selected" not "all selected"
const processedFilters = {
data_sources: filters.data_sources,
document_types: filters.document_types,
owners: filters.owners,
connector_types: filters.connector_types || ["*"]
}
console.log("[DEBUG] Loading filter selections:", processedFilters)
setSelectedFilters(processedFilters)
setResultLimit(parsedFilterData.limit || 10)
setScoreThreshold(parsedFilterData.scoreThreshold || 0)
setEditingName(selectedFilter.name)
setEditingDescription(selectedFilter.description || '')
}
}, [selectedFilter, parsedFilterData])
connector_types: filters.connector_types || ["*"],
};
console.log("[DEBUG] Loading filter selections:", processedFilters);
setSelectedFilters(processedFilters);
setResultLimit(parsedFilterData.limit || 10);
setScoreThreshold(parsedFilterData.scoreThreshold || 0);
setEditingName(selectedFilter.name);
setEditingDescription(selectedFilter.description || "");
}
}, [selectedFilter, parsedFilterData]);
// Load available facets using search aggregations hook
const { data: aggregations } = useGetSearchAggregations("*", 1, 0, {
enabled: isPanelOpen,
placeholderData: (prev) => prev,
staleTime: 60_000,
gcTime: 5 * 60_000,
});
// Load available facets from API
useEffect(() => {
if (isPanelOpen) {
loadAvailableFacets()
}
}, [isPanelOpen])
const loadAvailableFacets = async () => {
console.log("[DEBUG] Loading available facets...")
try {
// Do a search to get facets (similar to search page)
const response = await fetch("/api/search", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
query: "*", // Use wildcard like search page to get all documents/facets
limit: 1,
scoreThreshold: 0
// Omit filters entirely to get all available facets
}),
})
const result = await response.json()
console.log("[DEBUG] Search API response:", result)
if (response.ok && result.aggregations) {
const facets = {
data_sources: result.aggregations.data_sources?.buckets || [],
document_types: result.aggregations.document_types?.buckets || [],
owners: result.aggregations.owners?.buckets || [],
connector_types: result.aggregations.connector_types?.buckets || []
}
console.log("[DEBUG] Setting facets:", facets)
setAvailableFacets(facets)
} else {
console.log("[DEBUG] No aggregations in response or response not ok")
}
} catch (error) {
console.error("Failed to load available facets:", error)
}
}
if (!aggregations) return;
const facets = {
data_sources: aggregations.data_sources?.buckets || [],
document_types: aggregations.document_types?.buckets || [],
owners: aggregations.owners?.buckets || [],
connector_types: aggregations.connector_types?.buckets || [],
};
setAvailableFacets(facets);
}, [aggregations]);
// Don't render if panel is closed or no filter selected
if (!isPanelOpen || !selectedFilter || !parsedFilterData) return null
if (!isPanelOpen || !selectedFilter || !parsedFilterData) return null;
const selectAllFilters = () => {
// Use wildcards instead of listing all specific items
@ -135,116 +117,105 @@ export function KnowledgeFilterPanel() {
data_sources: ["*"],
document_types: ["*"],
owners: ["*"],
connector_types: ["*"]
})
}
connector_types: ["*"],
});
};
const clearAllFilters = () => {
setSelectedFilters({
data_sources: [],
document_types: [],
owners: [],
connector_types: []
})
}
connector_types: [],
});
};
const handleEditMeta = () => {
setIsEditingMeta(true)
}
setIsEditingMeta(true);
};
const handleCancelEdit = () => {
setIsEditingMeta(false)
setEditingName(selectedFilter.name)
setEditingDescription(selectedFilter.description || '')
}
setIsEditingMeta(false);
setEditingName(selectedFilter.name);
setEditingDescription(selectedFilter.description || "");
};
const handleSaveMeta = async () => {
if (!editingName.trim()) return
if (!editingName.trim()) return;
setIsSaving(true)
setIsSaving(true);
try {
const response = await fetch(`/api/knowledge-filter/${selectedFilter.id}`, {
method: 'PUT',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
name: editingName.trim(),
description: editingDescription.trim(),
}),
})
const result = await updateFilterMutation.mutateAsync({
id: selectedFilter.id,
name: editingName.trim(),
description: editingDescription.trim(),
});
const result = await response.json()
if (response.ok && result.success) {
const updatedFilter = {
...selectedFilter,
name: editingName.trim(),
description: editingDescription.trim(),
updated_at: new Date().toISOString(),
}
setSelectedFilter(updatedFilter)
setIsEditingMeta(false)
if (result.success && result.filter) {
setSelectedFilter(result.filter);
setIsEditingMeta(false);
}
} catch (error) {
console.error('Error updating filter:', error)
console.error("Error updating filter:", error);
} finally {
setIsSaving(false)
setIsSaving(false);
}
}
};
const handleSaveConfiguration = async () => {
const filterData = {
query,
filters: selectedFilters,
limit: resultLimit,
scoreThreshold
}
scoreThreshold,
};
setIsSaving(true)
setIsSaving(true);
try {
const response = await fetch(`/api/knowledge-filter/${selectedFilter.id}`, {
method: 'PUT',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
queryData: JSON.stringify(filterData)
}),
})
const result = await updateFilterMutation.mutateAsync({
id: selectedFilter.id,
queryData: JSON.stringify(filterData),
});
const result = await response.json()
if (response.ok && result.success) {
// Update the filter in context
const updatedFilter = {
...selectedFilter,
query_data: JSON.stringify(filterData),
updated_at: new Date().toISOString(),
}
setSelectedFilter(updatedFilter)
if (result.success && result.filter) {
setSelectedFilter(result.filter);
}
} catch (error) {
console.error('Error updating filter configuration:', error)
console.error("Error updating filter configuration:", error);
} finally {
setIsSaving(false)
setIsSaving(false);
}
}
};
const formatDate = (dateString: string) => {
return new Date(dateString).toLocaleDateString('en-US', {
year: 'numeric',
month: 'short',
day: 'numeric',
hour: '2-digit',
minute: '2-digit'
})
}
return new Date(dateString).toLocaleDateString("en-US", {
year: "numeric",
month: "short",
day: "numeric",
hour: "2-digit",
minute: "2-digit",
});
};
const handleFilterChange = (facetType: keyof typeof selectedFilters, newValues: string[]) => {
setSelectedFilters(prev => ({
const handleFilterChange = (
facetType: keyof typeof selectedFilters,
newValues: string[]
) => {
setSelectedFilters((prev) => ({
...prev,
[facetType]: newValues
}))
}
[facetType]: newValues,
}));
};
const handleDeleteFilter = async () => {
const result = await deleteFilterMutation.mutateAsync({
id: selectedFilter.id,
});
if (result.success) {
setSelectedFilter(null);
closePanelOnly();
}
};
return (
<div className="fixed right-0 top-14 bottom-0 w-80 bg-background border-l border-border/40 z-40 overflow-y-auto">
@ -252,7 +223,6 @@ export function KnowledgeFilterPanel() {
<CardHeader className="pb-3">
<div className="flex items-center justify-between">
<CardTitle className="text-lg flex items-center gap-2">
<Settings className="h-5 w-5" />
Knowledge Filter
</CardTitle>
<Button
@ -264,9 +234,6 @@ export function KnowledgeFilterPanel() {
<X className="h-4 w-4" />
</Button>
</div>
<CardDescription>
Configure your knowledge filter settings
</CardDescription>
</CardHeader>
<CardContent className="space-y-6">
@ -301,7 +268,7 @@ export function KnowledgeFilterPanel() {
className="flex-1"
>
<Save className="h-3 w-3 mr-1" />
{isSaving ? 'Saving...' : 'Save'}
{isSaving ? "Saving..." : "Save"}
</Button>
<Button
onClick={handleCancelEdit}
@ -315,9 +282,11 @@ export function KnowledgeFilterPanel() {
</div>
) : (
<div className="space-y-3">
<div className="flex items-start justify-between">
<div className="flex items-center justify-between">
<div className="flex-1">
<h3 className="font-semibold text-lg">{selectedFilter.name}</h3>
<h3 className="font-semibold text-lg">
{selectedFilter.name}
</h3>
{selectedFilter.description && (
<p className="text-sm text-muted-foreground mt-1">
{selectedFilter.description}
@ -336,7 +305,10 @@ export function KnowledgeFilterPanel() {
<div className="text-xs text-muted-foreground">
Created {formatDate(selectedFilter.created_at)}
{selectedFilter.updated_at !== selectedFilter.created_at && (
<span> • Updated {formatDate(selectedFilter.updated_at)}</span>
<span>
{" "}
• Updated {formatDate(selectedFilter.updated_at)}
</span>
)}
</div>
</div>
@ -345,14 +317,15 @@ export function KnowledgeFilterPanel() {
{/* Search Query */}
<div className="space-y-2">
<Label htmlFor="search-query" className="text-sm font-medium">Search Query</Label>
<Input
<Label htmlFor="search-query" className="text-sm font-medium">
Search Query
</Label>
<Textarea
id="search-query"
type="text"
placeholder="e.g., 'financial reports from Q4'"
value={query}
onChange={(e) => setQuery(e.target.value)}
className="bg-background/50 border-border/50"
rows={3}
/>
</div>
@ -361,13 +334,15 @@ export function KnowledgeFilterPanel() {
<div className="space-y-2">
<Label className="text-sm font-medium">Data Sources</Label>
<MultiSelect
options={(availableFacets.data_sources || []).map(bucket => ({
options={(availableFacets.data_sources || []).map((bucket) => ({
value: bucket.key,
label: bucket.key,
count: bucket.count
count: bucket.count,
}))}
value={selectedFilters.data_sources}
onValueChange={(values) => handleFilterChange('data_sources', values)}
onValueChange={(values) =>
handleFilterChange("data_sources", values)
}
placeholder="Select data sources..."
allOptionLabel="All Data Sources"
/>
@ -376,13 +351,17 @@ export function KnowledgeFilterPanel() {
<div className="space-y-2">
<Label className="text-sm font-medium">Document Types</Label>
<MultiSelect
options={(availableFacets.document_types || []).map(bucket => ({
value: bucket.key,
label: bucket.key,
count: bucket.count
}))}
options={(availableFacets.document_types || []).map(
(bucket) => ({
value: bucket.key,
label: bucket.key,
count: bucket.count,
})
)}
value={selectedFilters.document_types}
onValueChange={(values) => handleFilterChange('document_types', values)}
onValueChange={(values) =>
handleFilterChange("document_types", values)
}
placeholder="Select document types..."
allOptionLabel="All Document Types"
/>
@ -391,13 +370,13 @@ export function KnowledgeFilterPanel() {
<div className="space-y-2">
<Label className="text-sm font-medium">Owners</Label>
<MultiSelect
options={(availableFacets.owners || []).map(bucket => ({
options={(availableFacets.owners || []).map((bucket) => ({
value: bucket.key,
label: bucket.key,
count: bucket.count
count: bucket.count,
}))}
value={selectedFilters.owners}
onValueChange={(values) => handleFilterChange('owners', values)}
onValueChange={(values) => handleFilterChange("owners", values)}
placeholder="Select owners..."
allOptionLabel="All Owners"
/>
@ -406,13 +385,17 @@ export function KnowledgeFilterPanel() {
<div className="space-y-2">
<Label className="text-sm font-medium">Sources</Label>
<MultiSelect
options={(availableFacets.connector_types || []).map(bucket => ({
value: bucket.key,
label: bucket.key,
count: bucket.count
}))}
options={(availableFacets.connector_types || []).map(
(bucket) => ({
value: bucket.key,
label: bucket.key,
count: bucket.count,
})
)}
value={selectedFilters.connector_types}
onValueChange={(values) => handleFilterChange('connector_types', values)}
onValueChange={(values) =>
handleFilterChange("connector_types", values)
}
placeholder="Select sources..."
allOptionLabel="All Sources"
/>
@ -420,18 +403,18 @@ export function KnowledgeFilterPanel() {
{/* All/None buttons */}
<div className="flex gap-2">
<Button
variant="outline"
size="sm"
onClick={selectAllFilters}
<Button
variant="outline"
size="sm"
onClick={selectAllFilters}
className="h-auto px-3 py-1.5 text-xs text-muted-foreground hover:text-foreground hover:bg-muted/50 border-border/50"
>
All
</Button>
<Button
variant="outline"
size="sm"
onClick={clearAllFilters}
<Button
variant="outline"
size="sm"
onClick={clearAllFilters}
className="h-auto px-3 py-1.5 text-xs text-muted-foreground hover:text-foreground hover:bg-muted/50 border-border/50"
>
None
@ -442,18 +425,23 @@ export function KnowledgeFilterPanel() {
<div className="space-y-4 pt-4 border-t border-border/50">
<div className="space-y-2">
<div className="flex items-center justify-between">
<Label className="text-sm font-medium">Limit</Label>
<Label className="text-sm font-medium text-nowrap">
Response limit
</Label>
<Input
type="number"
min="1"
max="1000"
value={resultLimit}
onChange={(e) => {
const newLimit = Math.max(1, Math.min(1000, parseInt(e.target.value) || 1))
setResultLimit(newLimit)
const newLimit = Math.max(
1,
Math.min(1000, parseInt(e.target.value) || 1)
);
setResultLimit(newLimit);
}}
className="h-6 text-xs text-right px-2 bg-muted/30 !border-0 rounded ml-auto focus:ring-0 focus:outline-none"
style={{ width: '70px' }}
style={{ width: "70px" }}
/>
</div>
<Slider
@ -469,16 +457,20 @@ export function KnowledgeFilterPanel() {
{/* Score Threshold Control - exactly like search page */}
<div className="space-y-2">
<div className="flex items-center justify-between">
<Label className="text-sm font-medium">Score Threshold</Label>
<Label className="text-sm font-medium text-nowrap">
Score threshold
</Label>
<Input
type="number"
min="0"
max="5"
step="0.1"
value={scoreThreshold}
onChange={(e) => setScoreThreshold(parseFloat(e.target.value) || 0)}
onChange={(e) =>
setScoreThreshold(parseFloat(e.target.value) || 0)
}
className="h-6 text-xs text-right px-2 bg-muted/30 !border-0 rounded ml-auto focus:ring-0 focus:outline-none"
style={{ width: '70px' }}
style={{ width: "70px" }}
/>
</div>
<Slider
@ -493,7 +485,7 @@ export function KnowledgeFilterPanel() {
</div>
{/* Save Configuration Button */}
<div className="pt-4 border-t border-border/50">
<div className="flex flex-col gap-3 pt-4 border-t border-border/50">
<Button
onClick={handleSaveConfiguration}
disabled={isSaving}
@ -512,10 +504,17 @@ export function KnowledgeFilterPanel() {
</>
)}
</Button>
<Button
variant="destructive"
className="w-full"
onClick={handleDeleteFilter}
>
Delete Filter
</Button>
</div>
</div>
</CardContent>
</Card>
</div>
)
}
);
}

14
frontend/components/navigation.tsx
View file

@ -15,6 +15,8 @@ import { useCallback, useEffect, useRef, useState } from "react";
import { EndpointType } from "@/contexts/chat-context";
import { useLoadingStore } from "@/stores/loadingStore";
import { KnowledgeFilterList } from "./knowledge-filter-list";
import { useKnowledgeFilter } from "@/contexts/knowledge-filter-context";
interface RawConversation {
response_id: string;
@ -74,6 +76,8 @@ export function Navigation() {
const [previousConversationCount, setPreviousConversationCount] = useState(0);
const fileInputRef = useRef<HTMLInputElement>(null);
const { selectedFilter, setSelectedFilter } = useKnowledgeFilter();
const handleNewConversation = () => {
setLoadingNewConversation(true);
refreshConversations();
@ -194,6 +198,7 @@ export function Navigation() {
];
const isOnChatPage = pathname === "/" || pathname === "/chat";
const isOnKnowledgePage = pathname.startsWith("/knowledge");
const createDefaultPlaceholder = useCallback(() => {
return {
@ -310,7 +315,7 @@ export function Navigation() {
]);
return (
<div className="space-y-4 py-4 flex flex-col h-full bg-background">
<div className="flex flex-col h-full bg-background">
<div className="px-3 py-2 flex-shrink-0">
<div className="space-y-1">
{routes.map((route) => (
@ -344,6 +349,13 @@ export function Navigation() {
</div>
</div>
{isOnKnowledgePage && (
<KnowledgeFilterList
selectedFilter={selectedFilter}
onFilterSelect={setSelectedFilter}
/>
)}
{/* Chat Page Specific Sections */}
{isOnChatPage && (
<div className="flex-1 min-h-0 flex flex-col">

10
frontend/components/ui/textarea.tsx
View file

@ -1,18 +1,18 @@
import * as React from "react"
import * as React from "react";
import { cn } from "@/lib/utils"
import { cn } from "@/lib/utils";
function Textarea({ className, ...props }: React.ComponentProps<"textarea">) {
return (
<textarea
data-slot="textarea"
className={cn(
"border-input placeholder:text-muted-foreground focus-visible:border-ring focus-visible:ring-ring/50 aria-invalid:ring-destructive/20 dark:aria-invalid:ring-destructive/40 aria-invalid:border-destructive flex field-sizing-content min-h-16 w-full rounded-md border bg-background px-3 py-2 text-base shadow-xs transition-[color,box-shadow] outline-none focus-visible:ring-[3px] disabled:cursor-not-allowed disabled:opacity-50 md:text-sm",
"primary-input placeholder:font-mono placeholder:text-placeholder-foreground min-h-fit",
className
)}
{...props}
/>
)
);
}
export { Textarea }
export { Textarea };

9
frontend/package-lock.json generated
View file

@ -28,6 +28,7 @@
"@radix-ui/react-tabs": "^1.1.13",
"@radix-ui/react-tooltip": "^1.2.8",
"@tailwindcss/forms": "^0.5.10",
"@tailwindcss/line-clamp": "^0.4.4",
"@tailwindcss/typography": "^0.5.16",
"@tanstack/react-query": "^5.86.0",
"ag-grid-community": "^34.2.0",
@ -2317,6 +2318,14 @@
"tailwindcss": ">=3.0.0 || >= 3.0.0-alpha.1 || >= 4.0.0-alpha.20 || >= 4.0.0-beta.1"
}
},
"node_modules/@tailwindcss/line-clamp": {
"version": "0.4.4",
"resolved": "https://registry.npmjs.org/@tailwindcss/line-clamp/-/line-clamp-0.4.4.tgz",
"integrity": "sha512-5U6SY5z8N42VtrCrKlsTAA35gy2VSyYtHWCsg1H87NU1SXnEfekTVlrga9fzUDrrHcGi2Lb5KenUWb4lRQT5/g==",
"peerDependencies": {
"tailwindcss": ">=2.0.0 || >=3.0.0 || >=3.0.0-alpha.1"
}
},
"node_modules/@tailwindcss/typography": {
"version": "0.5.16",
"resolved": "https://registry.npmjs.org/@tailwindcss/typography/-/typography-0.5.16.tgz",

1
frontend/package.json
View file

@ -29,6 +29,7 @@
"@radix-ui/react-tabs": "^1.1.13",
"@radix-ui/react-tooltip": "^1.2.8",
"@tailwindcss/forms": "^0.5.10",
"@tailwindcss/line-clamp": "^0.4.4",
"@tailwindcss/typography": "^0.5.16",
"@tanstack/react-query": "^5.86.0",
"ag-grid-community": "^34.2.0",

50
frontend/src/app/api/[...path]/route.ts
View file

@ -50,35 +50,61 @@ async function proxyRequest(
try {
let body: string | ArrayBuffer | undefined = undefined;
let willSendBody = false;
if (request.method !== 'GET' && request.method !== 'HEAD') {
const contentType = request.headers.get('content-type') || '';
const contentLength = request.headers.get('content-length');
// For file uploads (multipart/form-data), preserve binary data
if (contentType.includes('multipart/form-data')) {
body = await request.arrayBuffer();
const buf = await request.arrayBuffer();
if (buf && buf.byteLength > 0) {
body = buf;
willSendBody = true;
}
} else {
// For JSON and other text-based content, use text
body = await request.text();
const text = await request.text();
if (text && text.length > 0) {
body = text;
willSendBody = true;
}
}
// Guard against incorrect non-zero content-length when there is no body
if (!willSendBody && contentLength) {
// We'll drop content-length/header below
}
}
const headers = new Headers();
// Copy relevant headers from the original request
for (const [key, value] of request.headers.entries()) {
if (!key.toLowerCase().startsWith('host') &&
!key.toLowerCase().startsWith('x-forwarded') &&
!key.toLowerCase().startsWith('x-real-ip')) {
headers.set(key, value);
const lower = key.toLowerCase();
if (
lower.startsWith('host') ||
lower.startsWith('x-forwarded') ||
lower.startsWith('x-real-ip') ||
lower === 'content-length' ||
(!willSendBody && lower === 'content-type')
) {
continue;
}
headers.set(key, value);
}
const response = await fetch(backendUrl, {
const init: RequestInit = {
method: request.method,
headers,
body,
});
};
if (willSendBody) {
// Convert ArrayBuffer to Uint8Array to satisfy BodyInit in all environments
const bodyInit: BodyInit = typeof body === 'string' ? body : new Uint8Array(body as ArrayBuffer);
init.body = bodyInit;
}
const response = await fetch(backendUrl, init);
const responseBody = await response.text();
const responseHeaders = new Headers();

50
frontend/src/app/api/mutations/useCreateFilter.ts Normal file
View file

@ -0,0 +1,50 @@
import { useMutation, useQueryClient } from "@tanstack/react-query";
import { KnowledgeFilter } from "../queries/useGetFiltersSearchQuery";
export interface CreateFilterRequest {
name: string;
description?: string;
queryData: string; // stringified ParsedQueryData
}
export interface CreateFilterResponse {
success: boolean;
filter: KnowledgeFilter;
message?: string;
}
async function createFilter(
data: CreateFilterRequest,
): Promise<CreateFilterResponse> {
const response = await fetch("/api/knowledge-filter", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
name: data.name,
description: data.description ?? "",
queryData: data.queryData,
}),
});
const json = await response.json().catch(() => ({}));
if (!response.ok) {
const errorMessage = (json && (json.error as string)) || "Failed to create knowledge filter";
throw new Error(errorMessage);
}
return json as CreateFilterResponse;
}
export const useCreateFilter = () => {
const queryClient = useQueryClient();
return useMutation({
mutationFn: createFilter,
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ["knowledge-filters"]});
},
});
};

8
frontend/src/app/api/mutations/useDeleteDocument.ts
View file

@ -14,7 +14,7 @@ interface DeleteDocumentResponse {
}
const deleteDocument = async (
data: DeleteDocumentRequest
data: DeleteDocumentRequest,
): Promise<DeleteDocumentResponse> => {
const response = await fetch("/api/documents/delete-by-filename", {
method: "POST",
@ -37,9 +37,11 @@ export const useDeleteDocument = () => {
return useMutation({
mutationFn: deleteDocument,
onSuccess: () => {
onSettled: () => {
// Invalidate and refetch search queries to update the UI
queryClient.invalidateQueries({ queryKey: ["search"] });
setTimeout(() => {
queryClient.invalidateQueries({ queryKey: ["search"] });
}, 1000);
},
});
};

39
frontend/src/app/api/mutations/useDeleteFilter.ts Normal file
View file

@ -0,0 +1,39 @@
import { useMutation, useQueryClient } from "@tanstack/react-query";
export interface DeleteFilterRequest {
id: string;
}
export interface DeleteFilterResponse {
success: boolean;
message?: string;
}
async function deleteFilter(
data: DeleteFilterRequest,
): Promise<DeleteFilterResponse> {
const response = await fetch(`/api/knowledge-filter/${data.id}`, {
method: "DELETE",
});
const json = await response.json().catch(() => ({}));
if (!response.ok) {
const errorMessage = (json && (json.error as string)) || "Failed to delete knowledge filter";
throw new Error(errorMessage);
}
return (json as DeleteFilterResponse) || { success: true };
}
export const useDeleteFilter = () => {
const queryClient = useQueryClient();
return useMutation({
mutationFn: deleteFilter,
onSuccess: () => {
// Invalidate filters queries so UI refreshes automatically
queryClient.invalidateQueries({ queryKey: ["knowledge-filters"] });
},
});
};

52
frontend/src/app/api/mutations/useUpdateFilter.ts Normal file
View file

@ -0,0 +1,52 @@
import { useMutation, useQueryClient } from "@tanstack/react-query";
import { KnowledgeFilter } from "../queries/useGetFiltersSearchQuery";
export interface UpdateFilterRequest {
id: string;
name?: string;
description?: string;
queryData?: string; // stringified ParsedQueryData
}
export interface UpdateFilterResponse {
success: boolean;
filter: KnowledgeFilter;
message?: string;
}
async function updateFilter(data: UpdateFilterRequest): Promise<UpdateFilterResponse> {
// Build a body with only provided fields
const body: Record<string, unknown> = {};
if (typeof data.name !== "undefined") body.name = data.name;
if (typeof data.description !== "undefined") body.description = data.description;
if (typeof data.queryData !== "undefined") body.queryData = data.queryData;
const response = await fetch(`/api/knowledge-filter/${data.id}`, {
method: "PUT",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
const json = await response.json().catch(() => ({}));
if (!response.ok) {
const errorMessage = (json && (json.error as string)) || "Failed to update knowledge filter";
throw new Error(errorMessage);
}
return json as UpdateFilterResponse;
}
export const useUpdateFilter = () => {
const queryClient = useQueryClient();
return useMutation({
mutationFn: updateFilter,
onSuccess: () => {
// Refresh any knowledge filter lists/searches
queryClient.invalidateQueries({ queryKey: ["knowledge-filters"] });
},
});
};

47
frontend/src/app/api/queries/useGetFiltersSearchQuery.ts Normal file
View file

@ -0,0 +1,47 @@
import {
useQuery,
useQueryClient,
type UseQueryOptions,
} from "@tanstack/react-query";
export interface KnowledgeFilter {
id: string;
name: string;
description: string;
query_data: string;
owner: string;
created_at: string;
updated_at: string;
}
export const useGetFiltersSearchQuery = (
search: string,
limit = 20,
options?: Omit<UseQueryOptions<KnowledgeFilter[]>, "queryKey" | "queryFn">
) => {
const queryClient = useQueryClient();
async function getFilters(): Promise<KnowledgeFilter[]> {
const response = await fetch("/api/knowledge-filter/search", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ query: search, limit }),
});
const json = await response.json();
if (!response.ok || !json.success) {
// ensure we always return a KnowledgeFilter[] to satisfy the return type
return [];
}
return (json.filters || []) as KnowledgeFilter[];
}
return useQuery<KnowledgeFilter[]>(
{
queryKey: ["knowledge-filters", search, limit],
queryFn: getFilters,
...options,
},
queryClient
);
};

2
frontend/src/app/api/queries/useGetModelsQuery.ts
View file

@ -54,7 +54,7 @@ export const useGetOpenAIModelsQuery = (
queryKey: ["models", "openai", params],
queryFn: getOpenAIModels,
retry: 2,
enabled: options?.enabled !== false, // Allow enabling/disabling from options
enabled: !!params?.apiKey,
staleTime: 0, // Always fetch fresh data
gcTime: 0, // Don't cache results
...options,

47
frontend/src/app/api/queries/useGetSearchAggregations.ts Normal file
View file

@ -0,0 +1,47 @@
import { useQuery, useQueryClient, type UseQueryOptions } from "@tanstack/react-query";
export interface FacetBucket {
key: string;
count: number;
}
export interface SearchAggregations {
data_sources?: { buckets: FacetBucket[] };
document_types?: { buckets: FacetBucket[] };
owners?: { buckets: FacetBucket[] };
connector_types?: { buckets: FacetBucket[] };
}
type Options = Omit<UseQueryOptions<SearchAggregations>, "queryKey" | "queryFn">;
export const useGetSearchAggregations = (
query: string,
limit: number,
scoreThreshold: number,
options?: Options
) => {
const queryClient = useQueryClient();
async function fetchAggregations(): Promise<SearchAggregations> {
const response = await fetch("/api/search", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ query, limit, scoreThreshold }),
});
const json = await response.json().catch(() => ({}));
if (!response.ok) {
throw new Error((json && json.error) || "Failed to load search aggregations");
}
return (json.aggregations || {}) as SearchAggregations;
}
return useQuery<SearchAggregations>({
queryKey: ["search-aggregations", query, limit, scoreThreshold],
queryFn: fetchAggregations,
placeholderData: prev => prev,
...options,
}, queryClient);
};

27
frontend/src/app/api/queries/useGetSearchQuery.ts
View file

@ -34,21 +34,28 @@ export interface ChunkResult {
export interface File {
filename: string;
mimetype: string;
chunkCount: number;
avgScore: number;
chunkCount?: number;
avgScore?: number;
source_url: string;
owner: string;
owner_name: string;
owner_email: string;
owner?: string;
owner_name?: string;
owner_email?: string;
size: number;
connector_type: string;
chunks: ChunkResult[];
status?:
| "processing"
| "active"
| "unavailable"
| "failed"
| "hidden"
| "sync";
chunks?: ChunkResult[];
}
export const useGetSearchQuery = (
query: string,
queryData?: ParsedQueryData | null,
options?: Omit<UseQueryOptions, "queryKey" | "queryFn">
options?: Omit<UseQueryOptions, "queryKey" | "queryFn">,
) => {
const queryClient = useQueryClient();
@ -149,7 +156,7 @@ export const useGetSearchQuery = (
}
});
const files: File[] = Array.from(fileMap.values()).map(file => ({
const files: File[] = Array.from(fileMap.values()).map((file) => ({
filename: file.filename,
mimetype: file.mimetype,
chunkCount: file.chunks.length,
@ -173,11 +180,11 @@ export const useGetSearchQuery = (
const queryResult = useQuery(
{
queryKey: ["search", effectiveQuery],
placeholderData: prev => prev,
placeholderData: (prev) => prev,
queryFn: getFiles,
...options,
},
queryClient
queryClient,
);
return queryResult;

2
frontend/src/app/globals.css
View file

@ -162,7 +162,7 @@
}
.side-bar-arrangement {
@apply flex h-full w-[14.5rem] flex-col overflow-hidden border-r scrollbar-hide;
@apply flex h-full w-[18rem] flex-col overflow-hidden border-r scrollbar-hide;
}
.side-bar-search-div-placement {

294
frontend/src/app/knowledge/chunks/page.tsx
View file

@ -1,17 +1,14 @@
"use client";
import {
Building2,
Cloud,
FileText,
HardDrive,
ArrowLeft,
Copy,
File as FileIcon,
Loader2,
Search,
} from "lucide-react";
import { Suspense, useCallback, useEffect, useState } from "react";
import { Suspense, useCallback, useEffect, useMemo, useState } from "react";
import { useRouter, useSearchParams } from "next/navigation";
import { SiGoogledrive } from "react-icons/si";
import { TbBrandOnedrive } from "react-icons/tb";
import { ProtectedRoute } from "@/components/protected-route";
import { Button } from "@/components/ui/button";
import { useKnowledgeFilter } from "@/contexts/knowledge-filter-context";
@ -21,22 +18,16 @@ import {
type File,
useGetSearchQuery,
} from "../../api/queries/useGetSearchQuery";
import { Label } from "@/components/ui/label";
import { Checkbox } from "@/components/ui/checkbox";
import { Input } from "@/components/ui/input";
// Function to get the appropriate icon for a connector type
function getSourceIcon(connectorType?: string) {
switch (connectorType) {
case "google_drive":
return <SiGoogledrive className="h-4 w-4 text-foreground" />;
case "onedrive":
return <TbBrandOnedrive className="h-4 w-4 text-foreground" />;
case "sharepoint":
return <Building2 className="h-4 w-4 text-foreground" />;
case "s3":
return <Cloud className="h-4 w-4 text-foreground" />;
default:
return <HardDrive className="h-4 w-4 text-muted-foreground" />;
}
}
const getFileTypeLabel = (mimetype: string) => {
if (mimetype === "application/pdf") return "PDF";
if (mimetype === "text/plain") return "Text";
if (mimetype === "application/msword") return "Word Document";
return "Unknown";
};
function ChunksPageContent() {
const router = useRouter();
@ -46,10 +37,47 @@ function ChunksPageContent() {
const filename = searchParams.get("filename");
const [chunks, setChunks] = useState<ChunkResult[]>([]);
const [chunksFilteredByQuery, setChunksFilteredByQuery] = useState<
ChunkResult[]
>([]);
const [selectedChunks, setSelectedChunks] = useState<Set<number>>(new Set());
// Calculate average chunk length
const averageChunkLength = useMemo(
() =>
chunks.reduce((acc, chunk) => acc + chunk.text.length, 0) /
chunks.length || 0,
[chunks]
);
const [selectAll, setSelectAll] = useState(false);
const [queryInputText, setQueryInputText] = useState(
parsedFilterData?.query ?? ""
);
// Use the same search query as the knowledge page, but we'll filter for the specific file
const { data = [], isFetching } = useGetSearchQuery("*", parsedFilterData);
useEffect(() => {
if (queryInputText === "") {
setChunksFilteredByQuery(chunks);
} else {
setChunksFilteredByQuery(
chunks.filter((chunk) =>
chunk.text.toLowerCase().includes(queryInputText.toLowerCase())
)
);
}
}, [queryInputText, chunks]);
const handleCopy = useCallback((text: string) => {
navigator.clipboard.writeText(text);
}, []);
const fileData = (data as File[]).find(
(file: File) => file.filename === filename
);
// Extract chunks for the specific file
useEffect(() => {
if (!filename || !(data as File[]).length) {
@ -57,16 +85,37 @@ function ChunksPageContent() {
return;
}
const fileData = (data as File[]).find(
(file: File) => file.filename === filename
);
setChunks(fileData?.chunks || []);
}, [data, filename]);
// Set selected state for all checkboxes when selectAll changes
useEffect(() => {
if (selectAll) {
setSelectedChunks(new Set(chunks.map((_, index) => index)));
} else {
setSelectedChunks(new Set());
}
}, [selectAll, setSelectedChunks, chunks]);
const handleBack = useCallback(() => {
router.back();
router.push("/knowledge");
}, [router]);
const handleChunkCardCheckboxChange = useCallback(
(index: number) => {
setSelectedChunks((prevSelected) => {
const newSelected = new Set(prevSelected);
if (newSelected.has(index)) {
newSelected.delete(index);
} else {
newSelected.add(index);
}
return newSelected;
});
},
[setSelectedChunks]
);
if (!filename) {
return (
<div className="flex items-center justify-center h-64">
@ -83,7 +132,7 @@ function ChunksPageContent() {
return (
<div
className={`fixed inset-0 md:left-72 top-[53px] flex flex-col transition-all duration-300 ${
className={`fixed inset-0 md:left-72 top-[53px] flex flex-row transition-all duration-300 ${
isMenuOpen && isPanelOpen
? "md:right-[704px]"
: // Both open: 384px (menu) + 320px (KF panel)
@ -98,29 +147,47 @@ function ChunksPageContent() {
>
<div className="flex-1 flex flex-col min-h-0 px-6 py-6">
{/* Header */}
<div className="flex items-center justify-between mb-6">
<div className="flex items-center gap-3">
<Button
variant="ghost"
size="sm"
onClick={handleBack}
className="text-muted-foreground hover:text-foreground px-2"
>
← Back
<div className="flex flex-col mb-6">
<div className="flex items-center gap-3 mb-2">
<Button variant="ghost" onClick={handleBack}>
<ArrowLeft size={18} />
<FileIcon className="text-muted-foreground" size={18} />
<h1 className="text-lg font-semibold">
{filename.replace(/\.[^/.]+$/, "")}
</h1>
</Button>
<div className="flex flex-col">
<h2 className="text-lg font-semibold">Document Chunks</h2>
<p className="text-sm text-muted-foreground truncate max-w-md">
{decodeURIComponent(filename)}
</p>
</div>
</div>
<div className="text-sm text-muted-foreground">
{!isFetching && chunks.length > 0 && (
<span>
{chunks.length} chunk{chunks.length !== 1 ? "s" : ""} found
</span>
)}
<div className="flex items-center gap-3 pl-4 mt-2">
<div className="flex items-center gap-2">
<Checkbox
id="selectAllChunks"
checked={selectAll}
onCheckedChange={(handleSelectAll) =>
setSelectAll(!!handleSelectAll)
}
/>
<Label
htmlFor="selectAllChunks"
className="font-medium text-muted-foreground whitespace-nowrap cursor-pointer"
>
Select all
</Label>
</div>
<div className="flex-1 flex items-center gap-2">
<Input
name="search-query"
id="search-query"
type="text"
defaultValue={parsedFilterData?.query}
value={queryInputText}
onChange={(e) => setQueryInputText(e.target.value)}
placeholder="Search chunks..."
className="flex-1 bg-muted/20 rounded-lg border border-border/50 px-4 py-3 focus-visible:ring-1 focus-visible:ring-ring"
/>
<Button variant="outline" size="sm">
<Search />
</Button>
</div>
</div>
</div>
@ -147,41 +214,130 @@ function ChunksPageContent() {
</div>
) : (
<div className="space-y-4 pb-6">
{chunks.map((chunk, index) => (
{chunksFilteredByQuery.map((chunk, index) => (
<div
key={chunk.filename + index}
className="bg-muted/20 rounded-lg p-4 border border-border/50"
className="bg-muted rounded-lg p-4 border border-border/50"
>
<div className="flex items-center justify-between mb-2">
<div className="flex items-center gap-2">
<FileText className="h-4 w-4 text-blue-400" />
<span className="font-medium truncate">
{chunk.filename}
<div className="flex items-center gap-3">
<div>
<Checkbox
checked={selectedChunks.has(index)}
onCheckedChange={() =>
handleChunkCardCheckboxChange(index)
}
/>
</div>
<span className="text-sm font-bold">
Chunk {chunk.page}
</span>
{chunk.connector_type && (
<div className="ml-2">
{getSourceIcon(chunk.connector_type)}
</div>
)}
<span className="bg-background p-1 rounded text-xs text-muted-foreground/70">
{chunk.text.length} chars
</span>
<div className="py-1">
<Button
className="p-1"
onClick={() => handleCopy(chunk.text)}
variant="ghost"
size="xs"
>
<Copy className="text-muted-foreground" />
</Button>
</div>
</div>
<span className="text-xs text-green-400 bg-green-400/20 px-2 py-1 rounded">
{chunk.score.toFixed(2)}
</span>
{/* TODO: Update to use active toggle */}
{/* <span className="px-2 py-1 text-green-500">
<Switch
className="ml-2 bg-green-500"
checked={true}
/>
Active
</span> */}
</div>
<div className="flex items-center gap-4 text-sm text-muted-foreground mb-3">
<span>{chunk.mimetype}</span>
<span>Page {chunk.page}</span>
{chunk.owner_name && <span>Owner: {chunk.owner_name}</span>}
</div>
<p className="text-sm text-foreground/90 leading-relaxed">
<blockquote className="text-sm text-muted-foreground leading-relaxed border-l-2 border-input ml-1.5 pl-4">
{chunk.text}
</p>
</blockquote>
</div>
))}
</div>
)}
</div>
</div>
{/* Right panel - Summary (TODO), Technical details, */}
<div className="w-[320px] py-20 px-2">
<div className="mb-8">
<h2 className="text-xl font-semibold mt-3 mb-4">Technical details</h2>
<dl>
<div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Total chunks</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
{chunks.length}
</dd>
</div>
<div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Avg length</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
{averageChunkLength.toFixed(0)} chars
</dd>
</div>
{/* TODO: Uncomment after data is available */}
{/* <div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Process time</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
</dd>
</div>
<div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Model</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
</dd>
</div> */}
</dl>
</div>
<div className="mb-8">
<h2 className="text-xl font-semibold mt-2 mb-3">Original document</h2>
<dl>
<div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Name</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
{fileData?.filename}
</dd>
</div>
<div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Type</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
{fileData ? getFileTypeLabel(fileData.mimetype) : "Unknown"}
</dd>
</div>
<div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Size</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
{fileData?.size
? `${Math.round(fileData.size / 1024)} KB`
: "Unknown"}
</dd>
</div>
<div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Uploaded</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
N/A
</dd>
</div>
{/* TODO: Uncomment after data is available */}
{/* <div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Source</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0"></dd>
</div> */}
<div className="sm:grid sm:grid-cols-3 sm:gap-4 sm:px-0 mb-2.5">
<dt className="text-sm/6 text-muted-foreground">Updated</dt>
<dd className="mt-1 text-sm/6 text-gray-100 sm:col-span-2 sm:mt-0">
N/A
</dd>
</div>
</dl>
</div>
</div>
</div>
);
}

182
frontend/src/app/knowledge/page.tsx
View file

@ -1,38 +1,25 @@
"use client";
import {
Building2,
Cloud,
HardDrive,
Loader2,
Search,
Trash2,
} from "lucide-react";
import { AgGridReact, CustomCellRendererProps } from "ag-grid-react";
import {
type FormEvent,
useCallback,
useEffect,
useState,
useRef,
} from "react";
import type { ColDef } from "ag-grid-community";
import { AgGridReact, type CustomCellRendererProps } from "ag-grid-react";
import { Building2, Cloud, HardDrive, Search, Trash2, X } from "lucide-react";
import { useRouter } from "next/navigation";
import { type ChangeEvent, useCallback, useRef, useState } from "react";
import { SiGoogledrive } from "react-icons/si";
import { TbBrandOnedrive } from "react-icons/tb";
import { KnowledgeDropdown } from "@/components/knowledge-dropdown";
import { ProtectedRoute } from "@/components/protected-route";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { useKnowledgeFilter } from "@/contexts/knowledge-filter-context";
import { useTask } from "@/contexts/task-context";
import { type File, useGetSearchQuery } from "../api/queries/useGetSearchQuery";
import { ColDef } from "ag-grid-community";
import "@/components/AgGrid/registerAgGridModules";
import "@/components/AgGrid/agGridStyles.css";
import { toast } from "sonner";
import { KnowledgeActionsDropdown } from "@/components/knowledge-actions-dropdown";
import { StatusBadge } from "@/components/ui/status-badge";
import { DeleteConfirmationDialog } from "../../../components/confirmation-dialog";
import { useDeleteDocument } from "../api/mutations/useDeleteDocument";
import { toast } from "sonner";
// Function to get the appropriate icon for a connector type
function getSourceIcon(connectorType?: string) {
@ -58,41 +45,48 @@ function getSourceIcon(connectorType?: string) {
function SearchPage() {
const router = useRouter();
const { isMenuOpen } = useTask();
const { parsedFilterData, isPanelOpen } = useKnowledgeFilter();
const [query, setQuery] = useState("");
const [queryInputText, setQueryInputText] = useState("");
const { isMenuOpen, files: taskFiles } = useTask();
const { selectedFilter, setSelectedFilter, parsedFilterData, isPanelOpen } =
useKnowledgeFilter();
const [selectedRows, setSelectedRows] = useState<File[]>([]);
const [showBulkDeleteDialog, setShowBulkDeleteDialog] = useState(false);
const deleteDocumentMutation = useDeleteDocument();
const {
data = [],
isFetching,
refetch: refetchSearch,
} = useGetSearchQuery(query, parsedFilterData);
// Update query when global filter changes
useEffect(() => {
if (parsedFilterData?.query) {
setQueryInputText(parsedFilterData.query);
}
}, [parsedFilterData]);
const handleSearch = useCallback(
(e?: FormEvent<HTMLFormElement>) => {
if (e) e.preventDefault();
if (query.trim() === queryInputText.trim()) {
refetchSearch();
return;
}
setQuery(queryInputText);
},
[queryInputText, refetchSearch, query]
const { data = [], isFetching } = useGetSearchQuery(
parsedFilterData?.query || "*",
parsedFilterData,
);
const fileResults = data as File[];
const handleTableSearch = (e: ChangeEvent<HTMLInputElement>) => {
gridRef.current?.api.setGridOption("quickFilterText", e.target.value);
};
// Convert TaskFiles to File format and merge with backend results
const taskFilesAsFiles: File[] = taskFiles.map((taskFile) => {
return {
filename: taskFile.filename,
mimetype: taskFile.mimetype,
source_url: taskFile.source_url,
size: taskFile.size,
connector_type: taskFile.connector_type,
status: taskFile.status,
};
});
const backendFiles = data as File[];
const filteredTaskFiles = taskFilesAsFiles.filter((taskFile) => {
return (
taskFile.status !== "active" &&
!backendFiles.some(
(backendFile) => backendFile.filename === taskFile.filename,
)
);
});
// Combine task files first, then backend files
const fileResults = [...backendFiles, ...filteredTaskFiles];
const gridRef = useRef<AgGridReact>(null);
@ -106,13 +100,14 @@ function SearchPage() {
minWidth: 220,
cellRenderer: ({ data, value }: CustomCellRendererProps<File>) => {
return (
<div
className="flex items-center gap-2 cursor-pointer hover:text-blue-600 transition-colors"
<button
type="button"
className="flex items-center gap-2 cursor-pointer hover:text-blue-600 transition-colors text-left w-full"
onClick={() => {
router.push(
`/knowledge/chunks?filename=${encodeURIComponent(
data?.filename ?? ""
)}`
data?.filename ?? "",
)}`,
);
}}
>
@ -120,7 +115,7 @@ function SearchPage() {
<span className="font-medium text-foreground truncate">
{value}
</span>
</div>
</button>
);
},
},
@ -143,18 +138,29 @@ function SearchPage() {
{
field: "chunkCount",
headerName: "Chunks",
valueFormatter: (params) => params.data?.chunkCount?.toString() || "-",
},
{
field: "avgScore",
headerName: "Avg score",
initialFlex: 0.5,
cellRenderer: ({ value }: CustomCellRendererProps<File>) => {
return (
<span className="text-xs text-green-400 bg-green-400/20 px-2 py-1 rounded">
{value.toFixed(2)}
{value?.toFixed(2) ?? "-"}
</span>
);
},
},
{
field: "status",
headerName: "Status",
cellRenderer: ({ data }: CustomCellRendererProps<File>) => {
// Default to 'active' status if no status is provided
const status = data?.status || "active";
return <StatusBadge status={status} />;
},
},
{
cellRenderer: ({ data }: CustomCellRendererProps<File>) => {
return <KnowledgeActionsDropdown filename={data?.filename || ""} />;
@ -167,9 +173,8 @@ function SearchPage() {
},
colId: "actions",
filter: false,
width: 60,
minWidth: 60,
maxWidth: 60,
minWidth: 0,
width: 40,
resizable: false,
sortable: false,
initialFlex: 0,
@ -196,7 +201,7 @@ function SearchPage() {
try {
// Delete each file individually since the API expects one filename at a time
const deletePromises = selectedRows.map((row) =>
deleteDocumentMutation.mutateAsync({ filename: row.filename })
deleteDocumentMutation.mutateAsync({ filename: row.filename }),
);
await Promise.all(deletePromises);
@ -204,7 +209,7 @@ function SearchPage() {
toast.success(
`Successfully deleted ${selectedRows.length} document${
selectedRows.length > 1 ? "s" : ""
}`
}`,
);
setSelectedRows([]);
setShowBulkDeleteDialog(false);
@ -217,7 +222,7 @@ function SearchPage() {
toast.error(
error instanceof Error
? error.message
: "Failed to delete some documents"
: "Failed to delete some documents",
);
}
};
@ -244,19 +249,29 @@ function SearchPage() {
</div>
{/* Search Input Area */}
<div className="flex-shrink-0 mb-6 lg:max-w-[75%] xl:max-w-[50%]">
<form onSubmit={handleSearch} className="flex gap-3">
<Input
name="search-query"
id="search-query"
type="text"
defaultValue={parsedFilterData?.query}
value={queryInputText}
onChange={(e) => setQueryInputText(e.target.value)}
placeholder="Search your documents..."
className="flex-1 bg-muted/20 rounded-lg border border-border/50 px-4 py-3 focus-visible:ring-1 focus-visible:ring-ring"
/>
<Button
<div className="flex-shrink-0 mb-6 xl:max-w-[75%]">
<form className="flex gap-3">
<div className="primary-input min-h-10 !flex items-center flex-nowrap gap-2 focus-within:border-foreground transition-colors !py-0">
{selectedFilter?.name && (
<div className="flex items-center gap-1 bg-blue-500/20 text-blue-400 px-1.5 py-0.5 rounded max-w-[300px]">
<span className="truncate">{selectedFilter?.name}</span>
<X
aria-label="Remove filter"
className="h-4 w-4 flex-shrink-0 cursor-pointer"
onClick={() => setSelectedFilter(null)}
/>
</div>
)}
<input
className="bg-transparent w-full h-full focus:outline-none focus-visible:outline-none placeholder:font-mono"
name="search-query"
id="search-query"
type="text"
placeholder="Search your documents..."
onChange={handleTableSearch}
/>
</div>
{/* <Button
type="submit"
variant="outline"
className="rounded-lg p-0 flex-shrink-0"
@ -266,7 +281,7 @@ function SearchPage() {
) : (
<Search className="h-4 w-4" />
)}
</Button>
</Button> */}
{/* //TODO: Implement sync button */}
{/* <Button
type="button"
@ -276,15 +291,16 @@ function SearchPage() {
>
Sync
</Button> */}
<Button
type="button"
variant="destructive"
className="rounded-lg flex-shrink-0"
onClick={() => setShowBulkDeleteDialog(true)}
disabled={selectedRows.length === 0}
>
<Trash2 className="h-4 w-4" /> Delete
</Button>
{selectedRows.length > 0 && (
<Button
type="button"
variant="destructive"
className="rounded-lg flex-shrink-0"
onClick={() => setShowBulkDeleteDialog(true)}
>
<Trash2 className="h-4 w-4" /> Delete
</Button>
)}
</form>
</div>
<AgGridReact
@ -298,8 +314,8 @@ function SearchPage() {
rowMultiSelectWithClick={false}
suppressRowClickSelection={true}
getRowId={(params) => params.data.filename}
domLayout="autoHeight"
onSelectionChanged={onSelectionChanged}
suppressHorizontalScroll={false}
noRowsOverlayComponent={() => (
<div className="text-center">
<Search className="h-12 w-12 mx-auto mb-4 text-muted-foreground/50" />

341
frontend/src/app/onboarding/page.tsx
View file

@ -1,10 +1,11 @@
"use client";
import { useRouter } from "next/navigation";
import { Suspense, useEffect, useState } from "react";
import { toast } from "sonner";
import {
useOnboardingMutation,
type OnboardingVariables,
type OnboardingVariables,
useOnboardingMutation,
} from "@/app/api/mutations/useOnboardingMutation";
import IBMLogo from "@/components/logo/ibm-logo";
import OllamaLogo from "@/components/logo/ollama-logo";
@ -12,198 +13,198 @@ import OpenAILogo from "@/components/logo/openai-logo";
import { ProtectedRoute } from "@/components/protected-route";
import { Button } from "@/components/ui/button";
import {
Card,
CardContent,
CardFooter,
CardHeader,
Card,
CardContent,
CardFooter,
CardHeader,
} from "@/components/ui/card";
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
import {
Tooltip,
TooltipContent,
TooltipTrigger,
} from "@/components/ui/tooltip";
import { useGetSettingsQuery } from "../api/queries/useGetSettingsQuery";
import { IBMOnboarding } from "./components/ibm-onboarding";
import { OllamaOnboarding } from "./components/ollama-onboarding";
import { OpenAIOnboarding } from "./components/openai-onboarding";
import {
Tooltip,
TooltipContent,
TooltipTrigger,
} from "@/components/ui/tooltip";
import { useGetSettingsQuery } from "../api/queries/useGetSettingsQuery";
import { useRouter } from "next/navigation";
function OnboardingPage() {
const { data: settingsDb, isLoading: isSettingsLoading } =
useGetSettingsQuery();
const { data: settingsDb, isLoading: isSettingsLoading } =
useGetSettingsQuery();
const redirect = "/";
const redirect = "/";
const router = useRouter();
const router = useRouter();
// Redirect if already authenticated or in no-auth mode
useEffect(() => {
if (!isSettingsLoading && settingsDb && settingsDb.edited) {
router.push(redirect);
}
}, [isSettingsLoading, redirect]);
// Redirect if already authenticated or in no-auth mode
useEffect(() => {
if (!isSettingsLoading && settingsDb && settingsDb.edited) {
router.push(redirect);
}
}, [isSettingsLoading, settingsDb, router]);
const [modelProvider, setModelProvider] = useState<string>("openai");
const [modelProvider, setModelProvider] = useState<string>("openai");
const [sampleDataset, setSampleDataset] = useState<boolean>(true);
const [sampleDataset, setSampleDataset] = useState<boolean>(true);
const handleSetModelProvider = (provider: string) => {
setModelProvider(provider);
setSettings({
model_provider: provider,
embedding_model: "",
llm_model: "",
});
};
const handleSetModelProvider = (provider: string) => {
setModelProvider(provider);
setSettings({
model_provider: provider,
embedding_model: "",
llm_model: "",
});
};
const [settings, setSettings] = useState<OnboardingVariables>({
model_provider: modelProvider,
embedding_model: "",
llm_model: "",
});
const [settings, setSettings] = useState<OnboardingVariables>({
model_provider: modelProvider,
embedding_model: "",
llm_model: "",
});
// Mutations
const onboardingMutation = useOnboardingMutation({
onSuccess: (data) => {
toast.success("Onboarding completed successfully!");
console.log("Onboarding completed successfully", data);
},
onError: (error) => {
toast.error("Failed to complete onboarding", {
description: error.message,
});
},
});
// Mutations
const onboardingMutation = useOnboardingMutation({
onSuccess: (data) => {
toast.success("Onboarding completed successfully!");
console.log("Onboarding completed successfully", data);
router.push(redirect);
},
onError: (error) => {
toast.error("Failed to complete onboarding", {
description: error.message,
});
},
});
const handleComplete = () => {
if (
!settings.model_provider ||
!settings.llm_model ||
!settings.embedding_model
) {
toast.error("Please complete all required fields");
return;
}
const handleComplete = () => {
if (
!settings.model_provider ||
!settings.llm_model ||
!settings.embedding_model
) {
toast.error("Please complete all required fields");
return;
}
// Prepare onboarding data
const onboardingData: OnboardingVariables = {
model_provider: settings.model_provider,
llm_model: settings.llm_model,
embedding_model: settings.embedding_model,
sample_data: sampleDataset,
};
// Prepare onboarding data
const onboardingData: OnboardingVariables = {
model_provider: settings.model_provider,
llm_model: settings.llm_model,
embedding_model: settings.embedding_model,
sample_data: sampleDataset,
};
// Add API key if available
if (settings.api_key) {
onboardingData.api_key = settings.api_key;
}
// Add API key if available
if (settings.api_key) {
onboardingData.api_key = settings.api_key;
}
// Add endpoint if available
if (settings.endpoint) {
onboardingData.endpoint = settings.endpoint;
}
// Add endpoint if available
if (settings.endpoint) {
onboardingData.endpoint = settings.endpoint;
}
// Add project_id if available
if (settings.project_id) {
onboardingData.project_id = settings.project_id;
}
// Add project_id if available
if (settings.project_id) {
onboardingData.project_id = settings.project_id;
}
onboardingMutation.mutate(onboardingData);
};
onboardingMutation.mutate(onboardingData);
};
const isComplete = !!settings.llm_model && !!settings.embedding_model;
const isComplete = !!settings.llm_model && !!settings.embedding_model;
return (
<div
className="min-h-dvh w-full flex gap-5 flex-col items-center justify-center bg-background p-4"
style={{
backgroundImage: "url('/images/background.png')",
backgroundSize: "cover",
backgroundPosition: "center",
}}
>
<div className="flex flex-col items-center gap-5 min-h-[550px] w-full">
<div className="flex flex-col items-center justify-center gap-4">
<h1 className="text-2xl font-medium font-chivo">
Configure your models
</h1>
<p className="text-sm text-muted-foreground">[description of task]</p>
</div>
<Card className="w-full max-w-[580px]">
<Tabs
defaultValue={modelProvider}
onValueChange={handleSetModelProvider}
>
<CardHeader>
<TabsList>
<TabsTrigger value="openai">
<OpenAILogo className="w-4 h-4" />
OpenAI
</TabsTrigger>
<TabsTrigger value="watsonx">
<IBMLogo className="w-4 h-4" />
IBM
</TabsTrigger>
<TabsTrigger value="ollama">
<OllamaLogo className="w-4 h-4" />
Ollama
</TabsTrigger>
</TabsList>
</CardHeader>
<CardContent>
<TabsContent value="openai">
<OpenAIOnboarding
setSettings={setSettings}
sampleDataset={sampleDataset}
setSampleDataset={setSampleDataset}
/>
</TabsContent>
<TabsContent value="watsonx">
<IBMOnboarding
setSettings={setSettings}
sampleDataset={sampleDataset}
setSampleDataset={setSampleDataset}
/>
</TabsContent>
<TabsContent value="ollama">
<OllamaOnboarding
setSettings={setSettings}
sampleDataset={sampleDataset}
setSampleDataset={setSampleDataset}
/>
</TabsContent>
</CardContent>
</Tabs>
<CardFooter className="flex justify-end">
<Tooltip>
<TooltipTrigger asChild>
<Button
size="sm"
onClick={handleComplete}
disabled={!isComplete}
loading={onboardingMutation.isPending}
>
Complete
</Button>
</TooltipTrigger>
<TooltipContent>
{!isComplete ? "Please fill in all required fields" : ""}
</TooltipContent>
</Tooltip>
</CardFooter>
</Card>
</div>
</div>
);
return (
<div
className="min-h-dvh w-full flex gap-5 flex-col items-center justify-center bg-background p-4"
style={{
backgroundImage: "url('/images/background.png')",
backgroundSize: "cover",
backgroundPosition: "center",
}}
>
<div className="flex flex-col items-center gap-5 min-h-[550px] w-full">
<div className="flex flex-col items-center justify-center gap-4">
<h1 className="text-2xl font-medium font-chivo">
Configure your models
</h1>
<p className="text-sm text-muted-foreground">[description of task]</p>
</div>
<Card className="w-full max-w-[580px]">
<Tabs
defaultValue={modelProvider}
onValueChange={handleSetModelProvider}
>
<CardHeader>
<TabsList>
<TabsTrigger value="openai">
<OpenAILogo className="w-4 h-4" />
OpenAI
</TabsTrigger>
<TabsTrigger value="watsonx">
<IBMLogo className="w-4 h-4" />
IBM
</TabsTrigger>
<TabsTrigger value="ollama">
<OllamaLogo className="w-4 h-4" />
Ollama
</TabsTrigger>
</TabsList>
</CardHeader>
<CardContent>
<TabsContent value="openai">
<OpenAIOnboarding
setSettings={setSettings}
sampleDataset={sampleDataset}
setSampleDataset={setSampleDataset}
/>
</TabsContent>
<TabsContent value="watsonx">
<IBMOnboarding
setSettings={setSettings}
sampleDataset={sampleDataset}
setSampleDataset={setSampleDataset}
/>
</TabsContent>
<TabsContent value="ollama">
<OllamaOnboarding
setSettings={setSettings}
sampleDataset={sampleDataset}
setSampleDataset={setSampleDataset}
/>
</TabsContent>
</CardContent>
</Tabs>
<CardFooter className="flex justify-end">
<Tooltip>
<TooltipTrigger asChild>
<Button
size="sm"
onClick={handleComplete}
disabled={!isComplete}
loading={onboardingMutation.isPending}
>
Complete
</Button>
</TooltipTrigger>
<TooltipContent>
{!isComplete ? "Please fill in all required fields" : ""}
</TooltipContent>
</Tooltip>
</CardFooter>
</Card>
</div>
</div>
);
}
export default function ProtectedOnboardingPage() {
return (
<ProtectedRoute>
<Suspense fallback={<div>Loading onboarding...</div>}>
<OnboardingPage />
</Suspense>
</ProtectedRoute>
);
return (
<ProtectedRoute>
<Suspense fallback={<div>Loading onboarding...</div>}>
<OnboardingPage />
</Suspense>
</ProtectedRoute>
);
}

110
frontend/src/app/settings/page.tsx
View file

@ -4,11 +4,13 @@ import { Loader2, PlugZap, RefreshCw } from "lucide-react";
import { useSearchParams } from "next/navigation";
import { Suspense, useCallback, useEffect, useState } from "react";
import { useUpdateFlowSettingMutation } from "@/app/api/mutations/useUpdateFlowSettingMutation";
import {
useGetIBMModelsQuery,
useGetOllamaModelsQuery,
useGetOpenAIModelsQuery,
} from "@/app/api/queries/useGetModelsQuery";
import { useGetSettingsQuery } from "@/app/api/queries/useGetSettingsQuery";
import { useGetOpenAIModelsQuery, useGetOllamaModelsQuery, useGetIBMModelsQuery } from "@/app/api/queries/useGetModelsQuery";
import { ConfirmationDialog } from "@/components/confirmation-dialog";
import { ModelSelectItems } from "./helpers/model-select-item";
import { getFallbackModels, type ModelProvider } from "./helpers/model-helpers";
import { ProtectedRoute } from "@/components/protected-route";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
@ -33,6 +35,8 @@ import { Textarea } from "@/components/ui/textarea";
import { useAuth } from "@/contexts/auth-context";
import { useTask } from "@/contexts/task-context";
import { useDebounce } from "@/lib/debounce";
import { getFallbackModels, type ModelProvider } from "./helpers/model-helpers";
import { ModelSelectItems } from "./helpers/model-select-item";
const MAX_SYSTEM_PROMPT_CHARS = 2000;
@ -105,42 +109,46 @@ function KnowledgeSourcesPage() {
// Fetch settings using React Query
const { data: settings = {} } = useGetSettingsQuery({
enabled: isAuthenticated,
enabled: isAuthenticated || isNoAuthMode,
});
// Get the current provider from settings
const currentProvider = (settings.provider?.model_provider || 'openai') as ModelProvider;
const currentProvider = (settings.provider?.model_provider ||
"openai") as ModelProvider;
// Fetch available models based on provider
const { data: openaiModelsData } = useGetOpenAIModelsQuery(
undefined, // Let backend use stored API key from configuration
{
enabled: isAuthenticated && currentProvider === 'openai',
}
enabled:
(isAuthenticated || isNoAuthMode) && currentProvider === "openai",
},
);
const { data: ollamaModelsData } = useGetOllamaModelsQuery(
undefined, // No params for now, could be extended later
{
enabled: isAuthenticated && currentProvider === 'ollama',
}
enabled:
(isAuthenticated || isNoAuthMode) && currentProvider === "ollama",
},
);
const { data: ibmModelsData } = useGetIBMModelsQuery(
undefined, // No params for now, could be extended later
{
enabled: isAuthenticated && currentProvider === 'ibm',
}
enabled: (isAuthenticated || isNoAuthMode) && currentProvider === "ibm",
},
);
// Select the appropriate models data based on provider
const modelsData = currentProvider === 'openai'
? openaiModelsData
: currentProvider === 'ollama'
? ollamaModelsData
: currentProvider === 'ibm'
? ibmModelsData
: openaiModelsData; // fallback to openai
const modelsData =
currentProvider === "openai"
? openaiModelsData
: currentProvider === "ollama"
? ollamaModelsData
: currentProvider === "ibm"
? ibmModelsData
: openaiModelsData; // fallback to openai
// Mutations
const updateFlowSettingMutation = useUpdateFlowSettingMutation({
@ -219,10 +227,10 @@ function KnowledgeSourcesPage() {
// Update processing mode
const handleProcessingModeChange = (mode: string) => {
setProcessingMode(mode);
// Update the configuration setting (backend will also update the flow automatically)
debouncedUpdate({ doclingPresets: mode });
};
// Helper function to get connector icon
const getConnectorIcon = useCallback((iconName: string) => {
const iconMap: { [key: string]: React.ReactElement } = {
@ -611,7 +619,11 @@ function KnowledgeSourcesPage() {
Language Model
</Label>
<Select
value={settings.agent?.llm_model || modelsData?.language_models?.find(m => m.default)?.value || "gpt-4"}
value={
settings.agent?.llm_model ||
modelsData?.language_models?.find((m) => m.default)?.value ||
"gpt-4"
}
onValueChange={handleModelChange}
>
<SelectTrigger id="model-select">
@ -636,10 +648,20 @@ function KnowledgeSourcesPage() {
value={systemPrompt}
onChange={(e) => setSystemPrompt(e.target.value)}
rows={6}
className={`resize-none ${systemPrompt.length > MAX_SYSTEM_PROMPT_CHARS ? 'border-red-500 focus:border-red-500' : ''}`}
className={`resize-none ${
systemPrompt.length > MAX_SYSTEM_PROMPT_CHARS
? "border-red-500 focus:border-red-500"
: ""
}`}
/>
<div className="flex justify-start">
<span className={`text-xs ${systemPrompt.length > MAX_SYSTEM_PROMPT_CHARS ? 'text-red-500' : 'text-muted-foreground'}`}>
<span
className={`text-xs ${
systemPrompt.length > MAX_SYSTEM_PROMPT_CHARS
? "text-red-500"
: "text-muted-foreground"
}`}
>
{systemPrompt.length}/{MAX_SYSTEM_PROMPT_CHARS} characters
</span>
</div>
@ -647,7 +669,10 @@ function KnowledgeSourcesPage() {
<div className="flex justify-end pt-2">
<Button
onClick={handleSystemPromptSave}
disabled={updateFlowSettingMutation.isPending || systemPrompt.length > MAX_SYSTEM_PROMPT_CHARS}
disabled={
updateFlowSettingMutation.isPending ||
systemPrompt.length > MAX_SYSTEM_PROMPT_CHARS
}
className="min-w-[120px]"
size="sm"
variant="outline"
@ -734,7 +759,9 @@ function KnowledgeSourcesPage() {
</Label>
<Select
value={
settings.knowledge?.embedding_model || modelsData?.embedding_models?.find(m => m.default)?.value || "text-embedding-ada-002"
settings.knowledge?.embedding_model ||
modelsData?.embedding_models?.find((m) => m.default)?.value ||
"text-embedding-ada-002"
}
onValueChange={handleEmbeddingModelChange}
>
@ -744,7 +771,9 @@ function KnowledgeSourcesPage() {
<SelectContent>
<ModelSelectItems
models={modelsData?.embedding_models}
fallbackModels={getFallbackModels(currentProvider).embedding}
fallbackModels={
getFallbackModels(currentProvider).embedding
}
provider={currentProvider}
/>
</SelectContent>
@ -805,7 +834,10 @@ function KnowledgeSourcesPage() {
<div className="flex items-center space-x-3">
<RadioGroupItem value="standard" id="standard" />
<div className="flex-1">
<Label htmlFor="standard" className="text-base font-medium cursor-pointer">
<Label
htmlFor="standard"
className="text-base font-medium cursor-pointer"
>
Standard
</Label>
<div className="text-sm text-muted-foreground">
@ -816,18 +848,28 @@ function KnowledgeSourcesPage() {
<div className="flex items-center space-x-3">
<RadioGroupItem value="ocr" id="ocr" />
<div className="flex-1">
<Label htmlFor="ocr" className="text-base font-medium cursor-pointer">
<Label
htmlFor="ocr"
className="text-base font-medium cursor-pointer"
>
Extract text from images
</Label>
<div className="text-sm text-muted-foreground">
Uses OCR to extract text from images/PDFs. Ingest is slower when enabled
Uses OCR to extract text from images/PDFs. Ingest is
slower when enabled
</div>
</div>
</div>
<div className="flex items-center space-x-3">
<RadioGroupItem value="picture_description" id="picture_description" />
<RadioGroupItem
value="picture_description"
id="picture_description"
/>
<div className="flex-1">
<Label htmlFor="picture_description" className="text-base font-medium cursor-pointer">
<Label
htmlFor="picture_description"
className="text-base font-medium cursor-pointer"
>
Generate Description
</Label>
<div className="text-sm text-muted-foreground">
@ -838,11 +880,15 @@ function KnowledgeSourcesPage() {
<div className="flex items-center space-x-3">
<RadioGroupItem value="VLM" id="VLM" />
<div className="flex-1">
<Label htmlFor="VLM" className="text-base font-medium cursor-pointer">
<Label
htmlFor="VLM"
className="text-base font-medium cursor-pointer"
>
AI Vision
</Label>
<div className="text-sm text-muted-foreground">
Advanced processing with vision language models. Highest quality but most expensive
Advanced processing with vision language models. Highest
quality but most expensive
</div>
</div>
</div>

15
frontend/src/components/layout-wrapper.tsx
View file

@ -1,10 +1,8 @@
"use client";
import { Bell, Loader2 } from "lucide-react";
import Image from "next/image";
import { usePathname } from "next/navigation";
import { useGetSettingsQuery } from "@/app/api/queries/useGetSettingsQuery";
import { KnowledgeFilterDropdown } from "@/components/knowledge-filter-dropdown";
import { KnowledgeFilterPanel } from "@/components/knowledge-filter-panel";
import { Navigation } from "@/components/navigation";
import { TaskNotificationMenu } from "@/components/task-notification-menu";
@ -20,8 +18,7 @@ import Logo from "@/components/logo/logo";
export function LayoutWrapper({ children }: { children: React.ReactNode }) {
const pathname = usePathname();
const { tasks, isMenuOpen, toggleMenu } = useTask();
const { selectedFilter, setSelectedFilter, isPanelOpen } =
useKnowledgeFilter();
const { isPanelOpen } = useKnowledgeFilter();
const { isLoading, isAuthenticated, isNoAuthMode } = useAuth();
const { isLoading: isSettingsLoading, data: settings } = useGetSettingsQuery({
enabled: isAuthenticated || isNoAuthMode,
@ -36,7 +33,7 @@ export function LayoutWrapper({ children }: { children: React.ReactNode }) {
(task) =>
task.status === "pending" ||
task.status === "running" ||
task.status === "processing",
task.status === "processing"
);
// Show loading state when backend isn't ready
@ -70,10 +67,10 @@ export function LayoutWrapper({ children }: { children: React.ReactNode }) {
<div className="header-end-division">
<div className="header-end-display">
{/* Knowledge Filter Dropdown */}
<KnowledgeFilterDropdown
{/* <KnowledgeFilterDropdown
selectedFilter={selectedFilter}
onFilterSelect={setSelectedFilter}
/>
/> */}
{/* GitHub Star Button */}
{/* <GitHubStarButton repo="phact/openrag" /> */}
@ -115,10 +112,10 @@ export function LayoutWrapper({ children }: { children: React.ReactNode }) {
isPanelOpen
? "md:pr-80"
: // Only KF panel open: 320px
"md:pr-6" // Neither open: 24px
"md:pr-0" // Neither open: 24px
}`}
>
<div className="container py-6 lg:py-8">{children}</div>
<div className="container py-6 lg:py-8 px-4 lg:px-6">{children}</div>
</main>
<TaskNotificationMenu />
<KnowledgeFilterPanel />

49
frontend/src/components/ui/animated-processing-icon.tsx Normal file
View file

@ -0,0 +1,49 @@
interface AnimatedProcessingIconProps {
className?: string;
size?: number;
}
export const AnimatedProcessingIcon = ({
className = "",
size = 10,
}: AnimatedProcessingIconProps) => {
const width = Math.round((size * 6) / 10);
const height = size;
return (
<svg
width={width}
height={height}
viewBox="0 0 6 10"
fill="none"
xmlns="http://www.w3.org/2000/svg"
className={className}
>
<style>
{`
.dot-1 { animation: pulse-wave 1.5s infinite; animation-delay: 0s; }
.dot-2 { animation: pulse-wave 1.5s infinite; animation-delay: 0.1s; }
.dot-3 { animation: pulse-wave 1.5s infinite; animation-delay: 0.2s; }
.dot-4 { animation: pulse-wave 1.5s infinite; animation-delay: 0.3s; }
.dot-5 { animation: pulse-wave 1.5s infinite; animation-delay: 0.4s; }
@keyframes pulse-wave {
0%, 60%, 100% {
opacity: 0.25;
transform: scale(1);
}
30% {
opacity: 1;
transform: scale(1.2);
}
}
`}
</style>
<circle className="dot-1" cx="1" cy="5" r="1" fill="currentColor" />
<circle className="dot-2" cx="1" cy="9" r="1" fill="currentColor" />
<circle className="dot-3" cx="5" cy="1" r="1" fill="currentColor" />
<circle className="dot-4" cx="5" cy="5" r="1" fill="currentColor" />
<circle className="dot-5" cx="5" cy="9" r="1" fill="currentColor" />
</svg>
);
};

58
frontend/src/components/ui/status-badge.tsx Normal file
View file

@ -0,0 +1,58 @@
import { AnimatedProcessingIcon } from "./animated-processing-icon";
export type Status =
| "processing"
| "active"
| "unavailable"
| "hidden"
| "sync"
| "failed";
interface StatusBadgeProps {
status: Status;
className?: string;
}
const statusConfig = {
processing: {
label: "Processing",
className: "text-muted-foreground dark:text-muted-foreground ",
},
active: {
label: "Active",
className: "text-emerald-600 dark:text-emerald-400 ",
},
unavailable: {
label: "Unavailable",
className: "text-red-600 dark:text-red-400 ",
},
failed: {
label: "Failed",
className: "text-red-600 dark:text-red-400 ",
},
hidden: {
label: "Hidden",
className: "text-zinc-400 dark:text-zinc-500 ",
},
sync: {
label: "Sync",
className: "text-amber-700 dark:text-amber-300 underline",
},
};
export const StatusBadge = ({ status, className }: StatusBadgeProps) => {
const config = statusConfig[status];
return (
<div
className={`inline-flex items-center gap-1 ${config.className} ${
className || ""
}`}
>
{status === "processing" && (
<AnimatedProcessingIcon className="text-current mr-2" size={10} />
)}
{config.label}
</div>
);
};

145
frontend/src/contexts/task-context.tsx
View file

@ -35,9 +35,22 @@ export interface Task {
files?: Record<string, Record<string, unknown>>;
}
export interface TaskFile {
filename: string;
mimetype: string;
source_url: string;
size: number;
connector_type: string;
status: "active" | "failed" | "processing";
task_id: string;
created_at: string;
updated_at: string;
}
interface TaskContextType {
tasks: Task[];
files: TaskFile[];
addTask: (taskId: string) => void;
addFiles: (files: Partial<TaskFile>[], taskId: string) => void;
removeTask: (taskId: string) => void;
refreshTasks: () => Promise<void>;
cancelTask: (taskId: string) => Promise<void>;
@ -51,6 +64,7 @@ const TaskContext = createContext<TaskContextType | undefined>(undefined);
export function TaskProvider({ children }: { children: React.ReactNode }) {
const [tasks, setTasks] = useState<Task[]>([]);
const [files, setFiles] = useState<TaskFile[]>([]);
const [isPolling, setIsPolling] = useState(false);
const [isFetching, setIsFetching] = useState(false);
const [isMenuOpen, setIsMenuOpen] = useState(false);
@ -58,12 +72,32 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
const queryClient = useQueryClient();
const refetchSearch = () => {
const refetchSearch = useCallback(() => {
queryClient.invalidateQueries({
queryKey: ["search"],
exact: false,
});
};
}, [queryClient]);
const addFiles = useCallback(
(newFiles: Partial<TaskFile>[], taskId: string) => {
const now = new Date().toISOString();
const filesToAdd: TaskFile[] = newFiles.map((file) => ({
filename: file.filename || "",
mimetype: file.mimetype || "",
source_url: file.source_url || "",
size: file.size || 0,
connector_type: file.connector_type || "local",
status: "processing",
task_id: taskId,
created_at: now,
updated_at: now,
}));
setFiles((prevFiles) => [...prevFiles, ...filesToAdd]);
},
[],
);
const fetchTasks = useCallback(async () => {
if (!isAuthenticated && !isNoAuthMode) return;
@ -76,13 +110,87 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
const newTasks = data.tasks || [];
// Update tasks and check for status changes in the same state update
setTasks(prevTasks => {
setTasks((prevTasks) => {
// Check for newly completed tasks to show toasts
if (prevTasks.length > 0) {
newTasks.forEach((newTask: Task) => {
const oldTask = prevTasks.find(
t => t.task_id === newTask.task_id
(t) => t.task_id === newTask.task_id,
);
// Update or add files from task.files if available
if (newTask.files && typeof newTask.files === "object") {
const taskFileEntries = Object.entries(newTask.files);
const now = new Date().toISOString();
taskFileEntries.forEach(([filePath, fileInfo]) => {
if (typeof fileInfo === "object" && fileInfo) {
const fileName = filePath.split("/").pop() || filePath;
const fileStatus = fileInfo.status as string;
// Map backend file status to our TaskFile status
let mappedStatus: TaskFile["status"];
switch (fileStatus) {
case "pending":
case "running":
mappedStatus = "processing";
break;
case "completed":
mappedStatus = "active";
break;
case "failed":
mappedStatus = "failed";
break;
default:
mappedStatus = "processing";
}
setFiles((prevFiles) => {
const existingFileIndex = prevFiles.findIndex(
(f) =>
f.source_url === filePath &&
f.task_id === newTask.task_id,
);
// Detect connector type based on file path or other indicators
let connectorType = "local";
if (filePath.includes("/") && !filePath.startsWith("/")) {
// Likely S3 key format (bucket/path/file.ext)
connectorType = "s3";
}
const fileEntry: TaskFile = {
filename: fileName,
mimetype: "", // We don't have this info from the task
source_url: filePath,
size: 0, // We don't have this info from the task
connector_type: connectorType,
status: mappedStatus,
task_id: newTask.task_id,
created_at:
typeof fileInfo.created_at === "string"
? fileInfo.created_at
: now,
updated_at:
typeof fileInfo.updated_at === "string"
? fileInfo.updated_at
: now,
};
if (existingFileIndex >= 0) {
// Update existing file
const updatedFiles = [...prevFiles];
updatedFiles[existingFileIndex] = fileEntry;
return updatedFiles;
} else {
// Add new file
return [...prevFiles, fileEntry];
}
});
}
});
}
if (
oldTask &&
oldTask.status !== "completed" &&
@ -99,9 +207,14 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
refetchSearch();
// Dispatch knowledge updated event for all knowledge-related pages
console.log(
"Task completed successfully, dispatching knowledgeUpdated event"
"Task completed successfully, dispatching knowledgeUpdated event",
);
window.dispatchEvent(new CustomEvent("knowledgeUpdated"));
// Remove files for this completed task from the files list
setFiles((prevFiles) =>
prevFiles.filter((file) => file.task_id !== newTask.task_id),
);
} else if (
oldTask &&
oldTask.status !== "failed" &&
@ -114,6 +227,8 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
newTask.error || "Unknown error"
}`,
});
// Files will be updated to failed status by the file parsing logic above
}
});
}
@ -126,7 +241,7 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
} finally {
setIsFetching(false);
}
}, [isAuthenticated, isNoAuthMode]); // Removed 'tasks' from dependencies to prevent infinite loop!
}, [isAuthenticated, isNoAuthMode, refetchSearch]); // Removed 'tasks' from dependencies to prevent infinite loop!
const addTask = useCallback((taskId: string) => {
// Immediately start aggressive polling for the new task
@ -140,19 +255,21 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
const data = await response.json();
const newTasks = data.tasks || [];
const foundTask = newTasks.find(
(task: Task) => task.task_id === taskId
(task: Task) => task.task_id === taskId,
);
if (foundTask) {
// Task found! Update the tasks state
setTasks(prevTasks => {
setTasks((prevTasks) => {
// Check if task is already in the list
const exists = prevTasks.some(t => t.task_id === taskId);
const exists = prevTasks.some((t) => t.task_id === taskId);
if (!exists) {
return [...prevTasks, foundTask];
}
// Update existing task
return prevTasks.map(t => (t.task_id === taskId ? foundTask : t));
return prevTasks.map((t) =>
t.task_id === taskId ? foundTask : t,
);
});
return; // Stop polling, we found it
}
@ -177,7 +294,7 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
}, [fetchTasks]);
const removeTask = useCallback((taskId: string) => {
setTasks(prev => prev.filter(task => task.task_id !== taskId));
setTasks((prev) => prev.filter((task) => task.task_id !== taskId));
}, []);
const cancelTask = useCallback(
@ -204,11 +321,11 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
});
}
},
[fetchTasks]
[fetchTasks],
);
const toggleMenu = useCallback(() => {
setIsMenuOpen(prev => !prev);
setIsMenuOpen((prev) => !prev);
}, []);
// Periodic polling for task updates
@ -231,7 +348,9 @@ export function TaskProvider({ children }: { children: React.ReactNode }) {
const value: TaskContextType = {
tasks,
files,
addTask,
addFiles,
removeTask,
refreshTasks,
cancelTask,

2
frontend/tailwind.config.ts
View file

@ -4,6 +4,7 @@ import tailwindcssTypography from "@tailwindcss/typography";
import { fontFamily } from "tailwindcss/defaultTheme";
import plugin from "tailwindcss/plugin";
import tailwindcssAnimate from "tailwindcss-animate";
import tailwindcssLineClamp from "@tailwindcss/line-clamp";
const config = {
darkMode: ["class"],
@ -175,6 +176,7 @@ const config = {
},
plugins: [
tailwindcssAnimate,
tailwindcssLineClamp,
tailwindcssForms({
strategy: "class",
}),

18
src/agent.py
View file

@ -106,7 +106,6 @@ async def async_response_stream(
model: str,
extra_headers: dict = None,
previous_response_id: str = None,
tweaks: dict = None,
log_prefix: str = "response",
):
logger.info("User prompt received", prompt=prompt)
@ -121,8 +120,6 @@ async def async_response_stream(
}
if previous_response_id is not None:
request_params["previous_response_id"] = previous_response_id
if tweaks:
request_params["tweaks"] = tweaks
if "x-api-key" not in client.default_headers:
if hasattr(client, "api_key") and extra_headers is not None:
@ -199,7 +196,6 @@ async def async_response(
model: str,
extra_headers: dict = None,
previous_response_id: str = None,
tweaks: dict = None,
log_prefix: str = "response",
):
try:
@ -214,8 +210,6 @@ async def async_response(
}
if previous_response_id is not None:
request_params["previous_response_id"] = previous_response_id
if tweaks:
request_params["tweaks"] = tweaks
if extra_headers:
request_params["extra_headers"] = extra_headers
@ -249,7 +243,6 @@ async def async_stream(
model: str,
extra_headers: dict = None,
previous_response_id: str = None,
tweaks: dict = None,
log_prefix: str = "response",
):
async for chunk in async_response_stream(
@ -258,7 +251,6 @@ async def async_stream(
model,
extra_headers=extra_headers,
previous_response_id=previous_response_id,
tweaks=tweaks,
log_prefix=log_prefix,
):
yield chunk
@ -271,7 +263,6 @@ async def async_langflow(
prompt: str,
extra_headers: dict = None,
previous_response_id: str = None,
tweaks: dict = None,
):
response_text, response_id, response_obj = await async_response(
langflow_client,
@ -279,7 +270,6 @@ async def async_langflow(
flow_id,
extra_headers=extra_headers,
previous_response_id=previous_response_id,
tweaks=tweaks,
log_prefix="langflow",
)
return response_text, response_id
@ -292,7 +282,6 @@ async def async_langflow_stream(
prompt: str,
extra_headers: dict = None,
previous_response_id: str = None,
tweaks: dict = None,
):
logger.debug("Starting langflow stream", prompt=prompt)
try:
@ -302,8 +291,7 @@ async def async_langflow_stream(
flow_id,
extra_headers=extra_headers,
previous_response_id=previous_response_id,
tweaks=tweaks,
log_prefix="langflow",
log_prefix="langflow",
):
logger.debug(
"Yielding chunk from langflow stream",
@ -463,7 +451,6 @@ async def async_langflow_chat(
user_id: str,
extra_headers: dict = None,
previous_response_id: str = None,
tweaks: dict = None,
store_conversation: bool = True,
):
logger.debug(
@ -497,7 +484,6 @@ async def async_langflow_chat(
flow_id,
extra_headers=extra_headers,
previous_response_id=previous_response_id,
tweaks=tweaks,
log_prefix="langflow",
)
logger.debug(
@ -576,7 +562,6 @@ async def async_langflow_chat_stream(
user_id: str,
extra_headers: dict = None,
previous_response_id: str = None,
tweaks: dict = None,
):
logger.debug(
"async_langflow_chat_stream called",
@ -603,7 +588,6 @@ async def async_langflow_chat_stream(
flow_id,
extra_headers=extra_headers,
previous_response_id=previous_response_id,
tweaks=tweaks,
log_prefix="langflow",
):
# Extract text content to build full response for history

97
src/api/models.py
View file

@ -17,14 +17,18 @@ async def get_openai_models(request, models_service, session_manager):
try:
config = get_openrag_config()
api_key = config.provider.api_key
logger.info(f"Retrieved API key from config: {'yes' if api_key else 'no'}")
logger.info(
f"Retrieved API key from config: {'yes' if api_key else 'no'}"
)
except Exception as e:
logger.error(f"Failed to get config: {e}")
if not api_key:
return JSONResponse(
{"error": "OpenAI API key is required either as query parameter or in configuration"},
status_code=400
{
"error": "OpenAI API key is required either as query parameter or in configuration"
},
status_code=400,
)
models = await models_service.get_openai_models(api_key=api_key)
@ -32,8 +36,7 @@ async def get_openai_models(request, models_service, session_manager):
except Exception as e:
logger.error(f"Failed to get OpenAI models: {str(e)}")
return JSONResponse(
{"error": f"Failed to retrieve OpenAI models: {str(e)}"},
status_code=500
{"error": f"Failed to retrieve OpenAI models: {str(e)}"}, status_code=500
)
@ -44,13 +47,31 @@ async def get_ollama_models(request, models_service, session_manager):
query_params = dict(request.query_params)
endpoint = query_params.get("endpoint")
# If no API key provided, try to get it from stored configuration
if not endpoint:
try:
config = get_openrag_config()
endpoint = config.provider.endpoint
logger.info(
f"Retrieved endpoint from config: {'yes' if endpoint else 'no'}"
)
except Exception as e:
logger.error(f"Failed to get config: {e}")
if not endpoint:
return JSONResponse(
{
"error": "Endpoint is required either as query parameter or in configuration"
},
status_code=400,
)
models = await models_service.get_ollama_models(endpoint=endpoint)
return JSONResponse(models)
except Exception as e:
logger.error(f"Failed to get Ollama models: {str(e)}")
return JSONResponse(
{"error": f"Failed to retrieve Ollama models: {str(e)}"},
status_code=500
{"error": f"Failed to retrieve Ollama models: {str(e)}"}, status_code=500
)
@ -63,15 +84,65 @@ async def get_ibm_models(request, models_service, session_manager):
api_key = query_params.get("api_key")
project_id = query_params.get("project_id")
config = get_openrag_config()
# If no API key provided, try to get it from stored configuration
if not api_key:
try:
api_key = config.provider.api_key
logger.info(
f"Retrieved API key from config: {'yes' if api_key else 'no'}"
)
except Exception as e:
logger.error(f"Failed to get config: {e}")
if not api_key:
return JSONResponse(
{
"error": "OpenAI API key is required either as query parameter or in configuration"
},
status_code=400,
)
if not endpoint:
try:
endpoint = config.provider.endpoint
logger.info(
f"Retrieved endpoint from config: {'yes' if endpoint else 'no'}"
)
except Exception as e:
logger.error(f"Failed to get config: {e}")
if not endpoint:
return JSONResponse(
{
"error": "Endpoint is required either as query parameter or in configuration"
},
status_code=400,
)
if not project_id:
try:
project_id = config.provider.project_id
logger.info(
f"Retrieved project ID from config: {'yes' if project_id else 'no'}"
)
except Exception as e:
logger.error(f"Failed to get config: {e}")
if not project_id:
return JSONResponse(
{
"error": "Project ID is required either as query parameter or in configuration"
},
status_code=400,
)
models = await models_service.get_ibm_models(
endpoint=endpoint,
api_key=api_key,
project_id=project_id
endpoint=endpoint, api_key=api_key, project_id=project_id
)
return JSONResponse(models)
except Exception as e:
logger.error(f"Failed to get IBM models: {str(e)}")
return JSONResponse(
{"error": f"Failed to retrieve IBM models: {str(e)}"},
status_code=500
)
{"error": f"Failed to retrieve IBM models: {str(e)}"}, status_code=500
)

115
src/api/settings.py
View file

@ -7,6 +7,7 @@ from config.settings import (
LANGFLOW_CHAT_FLOW_ID,
LANGFLOW_INGEST_FLOW_ID,
LANGFLOW_PUBLIC_URL,
DOCLING_COMPONENT_ID,
clients,
get_openrag_config,
config_manager,
@ -46,22 +47,7 @@ def get_docling_preset_configs():
}
def get_docling_tweaks(docling_preset: str = None) -> dict:
"""Get Langflow tweaks for docling component based on preset"""
if not docling_preset:
# Get current preset from config
openrag_config = get_openrag_config()
docling_preset = openrag_config.knowledge.doclingPresets
preset_configs = get_docling_preset_configs()
if docling_preset not in preset_configs:
docling_preset = "standard" # fallback
preset_config = preset_configs[docling_preset]
docling_serve_opts = json.dumps(preset_config)
return {"DoclingRemote-ayRdw": {"docling_serve_opts": docling_serve_opts}}
async def get_settings(request, session_manager):
@ -234,6 +220,15 @@ async def update_settings(request, session_manager):
current_config.knowledge.doclingPresets = body["doclingPresets"]
config_updated = True
# Also update the flow with the new docling preset
try:
await _update_flow_docling_preset(body["doclingPresets"], preset_configs[body["doclingPresets"]])
logger.info(f"Successfully updated docling preset in flow to '{body['doclingPresets']}'")
except Exception as e:
logger.error(f"Failed to update docling preset in flow: {str(e)}")
# Don't fail the entire settings update if flow update fails
# The config will still be saved
if "chunk_size" in body:
if not isinstance(body["chunk_size"], int) or body["chunk_size"] <= 0:
return JSONResponse(
@ -527,3 +522,93 @@ async def onboarding(request, flows_service):
{"error": f"Failed to update onboarding settings: {str(e)}"},
status_code=500,
)
async def _update_flow_docling_preset(preset: str, preset_config: dict):
"""Helper function to update docling preset in the ingest flow"""
if not LANGFLOW_INGEST_FLOW_ID:
raise ValueError("LANGFLOW_INGEST_FLOW_ID is not configured")
# Get the current flow data from Langflow
response = await clients.langflow_request(
"GET", f"/api/v1/flows/{LANGFLOW_INGEST_FLOW_ID}"
)
if response.status_code != 200:
raise Exception(f"Failed to get ingest flow: HTTP {response.status_code} - {response.text}")
flow_data = response.json()
# Find the target node in the flow using environment variable
nodes = flow_data.get("data", {}).get("nodes", [])
target_node = None
target_node_index = None
for i, node in enumerate(nodes):
if node.get("id") == DOCLING_COMPONENT_ID:
target_node = node
target_node_index = i
break
if target_node is None:
raise Exception(f"Docling component '{DOCLING_COMPONENT_ID}' not found in ingest flow")
# Update the docling_serve_opts value directly in the existing node
if (target_node.get("data", {}).get("node", {}).get("template", {}).get("docling_serve_opts")):
flow_data["data"]["nodes"][target_node_index]["data"]["node"]["template"]["docling_serve_opts"]["value"] = preset_config
else:
raise Exception(f"docling_serve_opts field not found in node '{DOCLING_COMPONENT_ID}'")
# Update the flow via PATCH request
patch_response = await clients.langflow_request(
"PATCH", f"/api/v1/flows/{LANGFLOW_INGEST_FLOW_ID}", json=flow_data
)
if patch_response.status_code != 200:
raise Exception(f"Failed to update ingest flow: HTTP {patch_response.status_code} - {patch_response.text}")
async def update_docling_preset(request, session_manager):
"""Update docling preset in the ingest flow"""
try:
# Parse request body
body = await request.json()
# Validate preset parameter
if "preset" not in body:
return JSONResponse(
{"error": "preset parameter is required"},
status_code=400
)
preset = body["preset"]
preset_configs = get_docling_preset_configs()
if preset not in preset_configs:
valid_presets = list(preset_configs.keys())
return JSONResponse(
{"error": f"Invalid preset '{preset}'. Valid presets: {', '.join(valid_presets)}"},
status_code=400
)
# Get the preset configuration
preset_config = preset_configs[preset]
# Use the helper function to update the flow
await _update_flow_docling_preset(preset, preset_config)
logger.info(f"Successfully updated docling preset to '{preset}' in ingest flow")
return JSONResponse({
"message": f"Successfully updated docling preset to '{preset}'",
"preset": preset,
"preset_config": preset_config
})
except Exception as e:
logger.error("Failed to update docling preset", error=str(e))
return JSONResponse(
{"error": f"Failed to update docling preset: {str(e)}"},
status_code=500
)

19
src/config/settings.py
View file

@ -496,12 +496,18 @@ class AppClients:
WATSONX_LLM_COMPONENT_PATH = os.getenv(
"WATSONX_LLM_COMPONENT_PATH", "flows/components/watsonx_llm.json"
)
WATSONX_LLM_TEXT_COMPONENT_PATH = os.getenv(
"WATSONX_LLM_TEXT_COMPONENT_PATH", "flows/components/watsonx_llm_text.json"
)
WATSONX_EMBEDDING_COMPONENT_PATH = os.getenv(
"WATSONX_EMBEDDING_COMPONENT_PATH", "flows/components/watsonx_embedding.json"
)
OLLAMA_LLM_COMPONENT_PATH = os.getenv(
"OLLAMA_LLM_COMPONENT_PATH", "flows/components/ollama_llm.json"
)
OLLAMA_LLM_TEXT_COMPONENT_PATH = os.getenv(
"OLLAMA_LLM_TEXT_COMPONENT_PATH", "flows/components/ollama_llm_text.json"
)
OLLAMA_EMBEDDING_COMPONENT_PATH = os.getenv(
"OLLAMA_EMBEDDING_COMPONENT_PATH", "flows/components/ollama_embedding.json"
)
@ -514,6 +520,9 @@ OPENAI_EMBEDDING_COMPONENT_ID = os.getenv(
OPENAI_LLM_COMPONENT_ID = os.getenv(
"OPENAI_LLM_COMPONENT_ID", "LanguageModelComponent-0YME7"
)
OPENAI_LLM_TEXT_COMPONENT_ID = os.getenv(
"OPENAI_LLM_TEXT_COMPONENT_ID", "LanguageModelComponent-NSTA6"
)
# Provider-specific component IDs
WATSONX_EMBEDDING_COMPONENT_ID = os.getenv(
@ -522,11 +531,21 @@ WATSONX_EMBEDDING_COMPONENT_ID = os.getenv(
WATSONX_LLM_COMPONENT_ID = os.getenv(
"WATSONX_LLM_COMPONENT_ID", "IBMwatsonxModel-jA4Nw"
)
WATSONX_LLM_TEXT_COMPONENT_ID = os.getenv(
"WATSONX_LLM_TEXT_COMPONENT_ID", "IBMwatsonxModel-18kmA"
)
OLLAMA_EMBEDDING_COMPONENT_ID = os.getenv(
"OLLAMA_EMBEDDING_COMPONENT_ID", "OllamaEmbeddings-4ah5Q"
)
OLLAMA_LLM_COMPONENT_ID = os.getenv("OLLAMA_LLM_COMPONENT_ID", "OllamaModel-eCsJx")
OLLAMA_LLM_TEXT_COMPONENT_ID = os.getenv(
"OLLAMA_LLM_TEXT_COMPONENT_ID", "OllamaModel-XDGqZ"
)
# Docling component ID for ingest flow
DOCLING_COMPONENT_ID = os.getenv("DOCLING_COMPONENT_ID", "DoclingRemote-78KoX")
# Global clients instance
clients = AppClients()

13
src/main.py
View file

@ -971,12 +971,23 @@ async def create_app():
"/onboarding",
require_auth(services["session_manager"])(
partial(
settings.onboarding,
settings.onboarding,
flows_service=services["flows_service"]
)
),
methods=["POST"],
),
# Docling preset update endpoint
Route(
"/settings/docling-preset",
require_auth(services["session_manager"])(
partial(
settings.update_docling_preset,
session_manager=services["session_manager"]
)
),
methods=["PATCH"],
),
Route(
"/nudges",
require_auth(services["session_manager"])(

11
src/services/chat_service.py
View file

@ -2,7 +2,6 @@ import json
from config.settings import NUDGES_FLOW_ID, clients, LANGFLOW_URL, LANGFLOW_CHAT_FLOW_ID
from agent import async_chat, async_langflow, async_chat_stream
from auth_context import set_auth_context
from api.settings import get_docling_tweaks
from utils.logging_config import get_logger
logger = get_logger(__name__)
@ -127,8 +126,6 @@ class ChatService:
"Langflow client not initialized. Ensure LANGFLOW is reachable or set LANGFLOW_KEY."
)
# Get docling tweaks based on current configuration
docling_tweaks = get_docling_tweaks()
if stream:
from agent import async_langflow_chat_stream
@ -140,7 +137,6 @@ class ChatService:
user_id,
extra_headers=extra_headers,
previous_response_id=previous_response_id,
tweaks=docling_tweaks,
)
else:
from agent import async_langflow_chat
@ -152,7 +148,6 @@ class ChatService:
user_id,
extra_headers=extra_headers,
previous_response_id=previous_response_id,
tweaks=docling_tweaks,
)
response_data = {"response": response_text}
if response_id:
@ -202,8 +197,6 @@ class ChatService:
from agent import async_langflow_chat
# Get docling tweaks (might not be used by nudges flow, but keeping consistent)
docling_tweaks = get_docling_tweaks()
response_text, response_id = await async_langflow_chat(
langflow_client,
@ -211,7 +204,6 @@ class ChatService:
prompt,
user_id,
extra_headers=extra_headers,
tweaks=docling_tweaks,
store_conversation=False,
)
response_data = {"response": response_text}
@ -242,8 +234,6 @@ class ChatService:
raise ValueError(
"Langflow client not initialized. Ensure LANGFLOW is reachable or set LANGFLOW_KEY."
)
# Get docling tweaks based on current configuration
docling_tweaks = get_docling_tweaks()
response_text, response_id = await async_langflow(
langflow_client=langflow_client,
@ -251,7 +241,6 @@ class ChatService:
prompt=document_prompt,
extra_headers=extra_headers,
previous_response_id=previous_response_id,
tweaks=docling_tweaks,
)
else: # chat
# Set auth context for chat tools and provide user_id

82
src/services/flows_service.py
View file

@ -3,8 +3,13 @@ from config.settings import (
LANGFLOW_URL,
LANGFLOW_CHAT_FLOW_ID,
LANGFLOW_INGEST_FLOW_ID,
OLLAMA_LLM_TEXT_COMPONENT_ID,
OLLAMA_LLM_TEXT_COMPONENT_PATH,
OPENAI_EMBEDDING_COMPONENT_ID,
OPENAI_LLM_COMPONENT_ID,
OPENAI_LLM_TEXT_COMPONENT_ID,
WATSONX_LLM_TEXT_COMPONENT_ID,
WATSONX_LLM_TEXT_COMPONENT_PATH,
clients,
WATSONX_LLM_COMPONENT_PATH,
WATSONX_EMBEDDING_COMPONENT_PATH,
@ -146,7 +151,7 @@ class FlowsService:
try:
# Load component templates based on provider
llm_template, embedding_template = self._load_component_templates(provider)
llm_template, embedding_template, llm_text_template = self._load_component_templates(provider)
logger.info(f"Assigning {provider} components")
@ -158,6 +163,7 @@ class FlowsService:
"flow_id": NUDGES_FLOW_ID,
"embedding_id": OPENAI_EMBEDDING_COMPONENT_ID,
"llm_id": OPENAI_LLM_COMPONENT_ID,
"llm_text_id": OPENAI_LLM_TEXT_COMPONENT_ID,
},
{
"name": "retrieval",
@ -165,6 +171,7 @@ class FlowsService:
"flow_id": LANGFLOW_CHAT_FLOW_ID,
"embedding_id": OPENAI_EMBEDDING_COMPONENT_ID,
"llm_id": OPENAI_LLM_COMPONENT_ID,
"llm_text_id": None,
},
{
"name": "ingest",
@ -172,6 +179,7 @@ class FlowsService:
"flow_id": LANGFLOW_INGEST_FLOW_ID,
"embedding_id": OPENAI_EMBEDDING_COMPONENT_ID,
"llm_id": None, # Ingestion flow might not have LLM
"llm_text_id": None, # Ingestion flow might not have LLM Text
},
]
@ -181,7 +189,7 @@ class FlowsService:
for config in flow_configs:
try:
result = await self._update_flow_components(
config, llm_template, embedding_template
config, llm_template, embedding_template, llm_text_template
)
results.append(result)
logger.info(f"Successfully updated {config['name']} flow")
@ -215,9 +223,11 @@ class FlowsService:
if provider == "watsonx":
llm_path = WATSONX_LLM_COMPONENT_PATH
embedding_path = WATSONX_EMBEDDING_COMPONENT_PATH
llm_text_path = WATSONX_LLM_TEXT_COMPONENT_PATH
elif provider == "ollama":
llm_path = OLLAMA_LLM_COMPONENT_PATH
embedding_path = OLLAMA_EMBEDDING_COMPONENT_PATH
llm_text_path = OLLAMA_LLM_TEXT_COMPONENT_PATH
else:
raise ValueError(f"Unsupported provider: {provider}")
@ -246,21 +256,31 @@ class FlowsService:
with open(embedding_full_path, "r") as f:
embedding_template = json.load(f)
logger.info(f"Loaded component templates for {provider}")
return llm_template, embedding_template
# Load LLM Text template
llm_text_full_path = os.path.join(project_root, llm_text_path)
if not os.path.exists(llm_text_full_path):
raise FileNotFoundError(
f"LLM Text component template not found at: {llm_text_full_path}"
)
async def _update_flow_components(self, config, llm_template, embedding_template):
with open(llm_text_full_path, "r") as f:
llm_text_template = json.load(f)
logger.info(f"Loaded component templates for {provider}")
return llm_template, embedding_template, llm_text_template
async def _update_flow_components(self, config, llm_template, embedding_template, llm_text_template):
"""Update components in a specific flow"""
flow_name = config["name"]
flow_file = config["file"]
flow_id = config["flow_id"]
old_embedding_id = config["embedding_id"]
old_llm_id = config["llm_id"]
old_llm_text_id = config["llm_text_id"]
# Extract IDs from templates
new_llm_id = llm_template["data"]["id"]
new_embedding_id = embedding_template["data"]["id"]
new_llm_text_id = llm_text_template["data"]["id"]
# Get the project root directory
current_file_dir = os.path.dirname(os.path.abspath(__file__))
src_dir = os.path.dirname(current_file_dir)
@ -308,6 +328,21 @@ class FlowsService:
self._replace_node_in_flow(flow_data, old_llm_id, new_llm_node)
components_updated.append(f"llm: {old_llm_id} -> {new_llm_id}")
# Replace LLM component (if exists in this flow)
if old_llm_text_id:
llm_text_node = self._find_node_by_id(flow_data, old_llm_text_id)
if llm_text_node:
# Preserve position
original_position = llm_text_node.get("position", {})
# Replace with new template
new_llm_text_node = llm_text_template.copy()
new_llm_text_node["position"] = original_position
# Replace in flow
self._replace_node_in_flow(flow_data, old_llm_text_id, new_llm_text_node)
components_updated.append(f"llm: {old_llm_text_id} -> {new_llm_text_id}")
# Update all edge references using regex replacement
flow_json_str = json.dumps(flow_data)
@ -326,6 +361,11 @@ class FlowsService:
flow_json_str = re.sub(
re.escape(old_llm_id), new_llm_id, flow_json_str
)
if old_llm_text_id:
flow_json_str = re.sub(
re.escape(old_llm_text_id), new_llm_text_id, flow_json_str
)
flow_json_str = re.sub(
re.escape(old_llm_id.split("-")[0]),
new_llm_id.split("-")[0],
@ -415,7 +455,7 @@ class FlowsService:
]
# Determine target component IDs based on provider
target_embedding_id, target_llm_id = self._get_provider_component_ids(
target_embedding_id, target_llm_id, target_llm_text_id = self._get_provider_component_ids(
provider
)
@ -429,6 +469,7 @@ class FlowsService:
provider,
target_embedding_id,
target_llm_id,
target_llm_text_id,
embedding_model,
llm_model,
endpoint,
@ -471,12 +512,12 @@ class FlowsService:
def _get_provider_component_ids(self, provider: str):
"""Get the component IDs for a specific provider"""
if provider == "watsonx":
return WATSONX_EMBEDDING_COMPONENT_ID, WATSONX_LLM_COMPONENT_ID
return WATSONX_EMBEDDING_COMPONENT_ID, WATSONX_LLM_COMPONENT_ID, WATSONX_LLM_TEXT_COMPONENT_ID
elif provider == "ollama":
return OLLAMA_EMBEDDING_COMPONENT_ID, OLLAMA_LLM_COMPONENT_ID
return OLLAMA_EMBEDDING_COMPONENT_ID, OLLAMA_LLM_COMPONENT_ID, OLLAMA_LLM_TEXT_COMPONENT_ID
elif provider == "openai":
# OpenAI components are the default ones
return OPENAI_EMBEDDING_COMPONENT_ID, OPENAI_LLM_COMPONENT_ID
return OPENAI_EMBEDDING_COMPONENT_ID, OPENAI_LLM_COMPONENT_ID, OPENAI_LLM_TEXT_COMPONENT_ID
else:
raise ValueError(f"Unsupported provider: {provider}")
@ -486,6 +527,7 @@ class FlowsService:
provider: str,
target_embedding_id: str,
target_llm_id: str,
target_llm_text_id: str,
embedding_model: str,
llm_model: str,
endpoint: str = None,
@ -512,7 +554,7 @@ class FlowsService:
embedding_node = self._find_node_by_id(flow_data, target_embedding_id)
if embedding_node:
if self._update_component_fields(
embedding_node, provider, "embedding", embedding_model, endpoint
embedding_node, provider, embedding_model, endpoint
):
updates_made.append(f"embedding model: {embedding_model}")
@ -521,7 +563,15 @@ class FlowsService:
llm_node = self._find_node_by_id(flow_data, target_llm_id)
if llm_node:
if self._update_component_fields(
llm_node, provider, "llm", llm_model, endpoint
llm_node, provider, llm_model, endpoint
):
updates_made.append(f"llm model: {llm_model}")
if target_llm_text_id:
llm_text_node = self._find_node_by_id(flow_data, target_llm_text_id)
if llm_text_node:
if self._update_component_fields(
llm_text_node, provider, llm_model, endpoint
):
updates_made.append(f"llm model: {llm_model}")
@ -569,7 +619,11 @@ class FlowsService:
updated = False
# Update model_name field (common to all providers)
if "model_name" in template:
if provider == "openai" and "model" in template:
template["model"]["value"] = model_value
template["model"]["options"] = [model_value]
updated = True
elif "model_name" in template:
template["model_name"]["value"] = model_value
template["model_name"]["options"] = [model_value]
updated = True

34
src/services/knowledge_filter_service.py
View file

@ -19,10 +19,18 @@ class KnowledgeFilterService:
# Index the knowledge filter document
result = await opensearch_client.index(
index=KNOWLEDGE_FILTERS_INDEX_NAME, id=filter_doc["id"], body=filter_doc
index=KNOWLEDGE_FILTERS_INDEX_NAME,
id=filter_doc["id"],
body=filter_doc,
refresh="wait_for",
)
if result.get("result") == "created":
# Extra safety: ensure visibility in subsequent searches
try:
await opensearch_client.indices.refresh(index=KNOWLEDGE_FILTERS_INDEX_NAME)
except Exception:
pass
return {"success": True, "id": filter_doc["id"], "filter": filter_doc}
else:
return {"success": False, "error": "Failed to create knowledge filter"}
@ -138,11 +146,19 @@ class KnowledgeFilterService:
# Update the document
result = await opensearch_client.update(
index=KNOWLEDGE_FILTERS_INDEX_NAME, id=filter_id, body={"doc": updates}
index=KNOWLEDGE_FILTERS_INDEX_NAME,
id=filter_id,
body={"doc": updates},
refresh="wait_for",
)
if result.get("result") in ["updated", "noop"]:
# Get the updated document
# Ensure visibility before fetching/returning
try:
await opensearch_client.indices.refresh(index=KNOWLEDGE_FILTERS_INDEX_NAME)
except Exception:
pass
updated_doc = await opensearch_client.get(
index=KNOWLEDGE_FILTERS_INDEX_NAME, id=filter_id
)
@ -164,10 +180,17 @@ class KnowledgeFilterService:
)
result = await opensearch_client.delete(
index=KNOWLEDGE_FILTERS_INDEX_NAME, id=filter_id
index=KNOWLEDGE_FILTERS_INDEX_NAME,
id=filter_id,
refresh="wait_for",
)
if result.get("result") == "deleted":
# Extra safety: ensure visibility in subsequent searches
try:
await opensearch_client.indices.refresh(index=KNOWLEDGE_FILTERS_INDEX_NAME)
except Exception:
pass
return {
"success": True,
"message": "Knowledge filter deleted successfully",
@ -230,7 +253,10 @@ class KnowledgeFilterService:
}
result = await opensearch_client.update(
index=KNOWLEDGE_FILTERS_INDEX_NAME, id=filter_id, body=update_body
index=KNOWLEDGE_FILTERS_INDEX_NAME,
id=filter_id,
body=update_body,
refresh="wait_for",
)
if result.get("result") in ["updated", "noop"]:

16
src/services/models_service.py
View file

@ -124,10 +124,8 @@ class ModelsService:
for model in models:
model_name = model.get(JSON_NAME_KEY, "")
# Remove tag if present (e.g., "llama3:latest" -> "llama3")
clean_model_name = model_name.split(":")[0] if model_name else ""
if not clean_model_name:
if not model_name:
continue
logger.debug(f"Checking model: {model_name}")
@ -152,7 +150,7 @@ class ModelsService:
# Check if it's an embedding model
is_embedding = any(
embed_model in clean_model_name.lower()
embed_model in model_name.lower()
for embed_model in self.OLLAMA_EMBEDDING_MODELS
)
@ -160,8 +158,8 @@ class ModelsService:
# Embedding models only need completion capability
embedding_models.append(
{
"value": clean_model_name,
"label": clean_model_name,
"value": model_name,
"label": model_name,
"default": False,
}
)
@ -169,9 +167,9 @@ class ModelsService:
# Language models need both completion and tool calling
language_models.append(
{
"value": clean_model_name,
"label": clean_model_name,
"default": "llama3" in clean_model_name.lower(),
"value": model_name,
"label": model_name,
"default": "llama3" in model_name.lower(),
}
)
except Exception as e:

66
src/services/task_service.py
View file

@ -17,7 +17,9 @@ class TaskService:
def __init__(self, document_service=None, process_pool=None):
self.document_service = document_service
self.process_pool = process_pool
self.task_store: dict[str, dict[str, UploadTask]] = {} # user_id -> {task_id -> UploadTask}
self.task_store: dict[
str, dict[str, UploadTask]
] = {} # user_id -> {task_id -> UploadTask}
self.background_tasks = set()
if self.process_pool is None:
@ -122,18 +124,27 @@ class TaskService:
# Process files with limited concurrency to avoid overwhelming the system
max_workers = get_worker_count()
semaphore = asyncio.Semaphore(max_workers * 2) # Allow 2x process pool size for async I/O
semaphore = asyncio.Semaphore(
max_workers * 2
) # Allow 2x process pool size for async I/O
async def process_with_semaphore(file_path: str):
async with semaphore:
await self.document_service.process_single_file_task(upload_task, file_path)
await self.document_service.process_single_file_task(
upload_task, file_path
)
tasks = [process_with_semaphore(file_path) for file_path in upload_task.file_tasks.keys()]
tasks = [
process_with_semaphore(file_path)
for file_path in upload_task.file_tasks.keys()
]
await asyncio.gather(*tasks, return_exceptions=True)
except Exception as e:
logger.error("Background upload processor failed", task_id=task_id, error=str(e))
logger.error(
"Background upload processor failed", task_id=task_id, error=str(e)
)
import traceback
traceback.print_exc()
@ -141,7 +152,9 @@ class TaskService:
self.task_store[user_id][task_id].status = TaskStatus.FAILED
self.task_store[user_id][task_id].updated_at = time.time()
async def background_custom_processor(self, user_id: str, task_id: str, items: list) -> None:
async def background_custom_processor(
self, user_id: str, task_id: str, items: list
) -> None:
"""Background task to process items using custom processor"""
try:
upload_task = self.task_store[user_id][task_id]
@ -163,7 +176,9 @@ class TaskService:
try:
await processor.process_item(upload_task, item, file_task)
except Exception as e:
logger.error("Failed to process item", item=str(item), error=str(e))
logger.error(
"Failed to process item", item=str(item), error=str(e)
)
import traceback
traceback.print_exc()
@ -190,7 +205,9 @@ class TaskService:
pass
raise # Re-raise to properly handle cancellation
except Exception as e:
logger.error("Background custom processor failed", task_id=task_id, error=str(e))
logger.error(
"Background custom processor failed", task_id=task_id, error=str(e)
)
import traceback
traceback.print_exc()
@ -212,7 +229,10 @@ class TaskService:
upload_task = None
for candidate_user_id in candidate_user_ids:
if candidate_user_id in self.task_store and task_id in self.task_store[candidate_user_id]:
if (
candidate_user_id in self.task_store
and task_id in self.task_store[candidate_user_id]
):
upload_task = self.task_store[candidate_user_id][task_id]
break
@ -271,10 +291,23 @@ class TaskService:
if task_id in tasks_by_id:
continue
# Calculate running and pending counts
# Calculate running and pending counts and build file statuses
running_files_count = 0
pending_files_count = 0
for file_task in upload_task.file_tasks.values():
file_statuses = {}
for file_path, file_task in upload_task.file_tasks.items():
if file_task.status.value != "completed":
file_statuses[file_path] = {
"status": file_task.status.value,
"result": file_task.result,
"error": file_task.error,
"retry_count": file_task.retry_count,
"created_at": file_task.created_at,
"updated_at": file_task.updated_at,
"duration_seconds": file_task.duration_seconds,
}
if file_task.status.value == "running":
running_files_count += 1
elif file_task.status.value == "pending":
@ -292,6 +325,7 @@ class TaskService:
"created_at": upload_task.created_at,
"updated_at": upload_task.updated_at,
"duration_seconds": upload_task.duration_seconds,
"files": file_statuses,
}
# First, add user-owned tasks; then shared anonymous;
@ -312,7 +346,10 @@ class TaskService:
store_user_id = None
for candidate_user_id in candidate_user_ids:
if candidate_user_id in self.task_store and task_id in self.task_store[candidate_user_id]:
if (
candidate_user_id in self.task_store
and task_id in self.task_store[candidate_user_id]
):
store_user_id = candidate_user_id
break
@ -326,7 +363,10 @@ class TaskService:
return False
# Cancel the background task to stop scheduling new work
if hasattr(upload_task, "background_task") and not upload_task.background_task.done():
if (
hasattr(upload_task, "background_task")
and not upload_task.background_task.done()
):
upload_task.background_task.cancel()
# Mark task as failed (cancelled)