{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d35ac8ce-0f92-46f5-9ba4-a46970f0ce19",
   "metadata": {},
   "source": [
    "# cognee - Get Started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd981778-0c84-4542-8e6f-1a7712184873",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Let's talk about the problem first\n",
    "\n",
    "###  -  Since LLMs appeared, people have tried to personalize them.\n",
    "###  -  People do \"prompt engineering\" and add specific instructions to the LLM\n",
    "###  - \"Become a sales agent\" or \"Become a programmer\""
   ]
  },
  {
   "attachments": {
    "82bccd2a-f1ec-4ddf-afc0-15586ce81d9b.png": {
     "image/png": "[encoded data, 455680 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "4a9159db-1422-4756-b2e7-f0dc28e918e1",
   "metadata": {},
   "source": [
    "![1_GG8LmLk1vgxYW4QkivDE1w.png](attachment:82bccd2a-f1ec-4ddf-afc0-15586ce81d9b.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8e606b1-94d3-43ce-bb4b-dbadff7f4ca6",
   "metadata": {},
   "source": [
    "## The next solution was RAGs \n",
    "\n",
    "RAGs (Retrieval Augmented Generation) are systems that connect to a vector store and search for similar data so they can enrich LLM response."
   ]
  },
  {
   "attachments": {
    "df72c97a-cb3b-4e3c-bd68-d7bc986353c6.png": {
     "image/png": "[encoded data, 165520 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "23e74f22-f43c-4f03-afe0-b423cbaa412a",
   "metadata": {},
   "source": [
    "![1_Jq9bEbitg1Pv4oASwEQwJg.png](attachment:df72c97a-cb3b-4e3c-bd68-d7bc986353c6.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3406b8f-cb9a-46ce-9029-2cbae2755795",
   "metadata": {},
   "source": [
    "## If you search for \"I like apple\", what will you get:\n",
    "###  1. I dislike fruits\n",
    "### 2. I am a vegan\n",
    "### 3. I don't like apples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6a98710-a14b-4a14-bb56-d3ae055e94d9",
   "metadata": {},
   "source": [
    "## Semantic similarity search is not magic\n",
    "### If you search for an apple, the closest thing you get is that you don't like apples.\n",
    "### Would it be nice to have a semantic model LLMs could use?\n"
   ]
  },
  {
   "attachments": {
    "ebdc90d6-be38-481a-aee8-3ecbc5f54bbc.png": {
     "image/png": "[encoded data, 1062484 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "cbc06e81-c540-4933-8fbe-4f35dcb34884",
   "metadata": {},
   "source": [
    "![Screenshot 2024-03-13 at 10.02.16.png](attachment:ebdc90d6-be38-481a-aee8-3ecbc5f54bbc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b900f830-8e9e-4272-b198-594606da4457",
   "metadata": {},
   "source": [
    "# That is where Cognee comes in"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3ae099a-1bbb-4f13-9bcb-c0f778d50e91",
   "metadata": {},
   "source": [
    "### Our goal is to:\n",
    "- create a semantic representation of the data \n",
    "- split the data into a multilayer graph network containing propositions"
   ]
  },
  {
   "attachments": {
    "5962bf80-e424-438a-b7e3-50c13810ecf4.png": {
     "image/png": "[encoded data, 100520 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "e9928ff6-4d82-4d87-9c5a-bf80a50bfa71",
   "metadata": {},
   "source": [
    "![Screenshot 2024-03-08 at 12.06.13.png](attachment:5962bf80-e424-438a-b7e3-50c13810ecf4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "785383b0-87b5-4a0a-be3f-e809aa284e30",
   "metadata": {},
   "source": [
    "## What is a semantic layer and what are propositions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3540ce30-2b22-4ece-8516-8d5ff2a405fe",
   "metadata": {},
   "source": [
    "- Multilayer network is cognitive multilayer networks as a quantitative and interpretative framework for investigating the mental lexicon. \n",
    "- The mental lexicon is a complex cognitive system representing information about the words/concepts that one knows\n",
    "Article 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a21f1cf-ff91-43e6-bd05-39e5ee885790",
   "metadata": {},
   "source": [
    "\n",
    "- Propositions are defined as atomic expressions within text, each encapsulating a distinct factoid and presented in a concise, self-contained natural language format.\n",
    "Article 1"
   ]
  },
  {
   "attachments": {
    "313b90cc-03f2-4c01-acb9-382f6b1d41c8.png": {
     "image/png": "[encoded data, 110536 characters]"
    },
    "af3ff267-9245-4a36-b5cc-53b6eaf7def3.png": {
     "image/png": "[encoded data, 282476 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "fe0bfa57-dca7-40aa-9ead-c6852b155878",
   "metadata": {},
   "source": [
    "![Screenshot 2024-03-08 at 12.17.24.png](attachment:313b90cc-03f2-4c01-acb9-382f6b1d41c8.png) ![Screenshot 2024-03-08 at 12.24.38.png](attachment:af3ff267-9245-4a36-b5cc-53b6eaf7def3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e47bae4-d27d-4430-a134-e1b381378f5c",
   "metadata": {},
   "source": [
    "### We combine the concepts of Multilayer networks with the propositions to create a semantic knowledge graph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f9c9376-8c68-4397-9081-d260cddcbd25",
   "metadata": {},
   "source": [
    "Relevant articles are: https://arxiv.org/pdf/2312.06648.pdf and   https://link.springer.com/article/10.3758/s13423-024-02473-9"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "074f0ea8-c659-4736-be26-be4b0e5ac665",
   "metadata": {},
   "source": [
    "# Demo time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2a41de33",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want to reset the cognee system, you can use the following command:\n",
    "\n",
    "import cognee\n",
    "\n",
    "await cognee.prune.prune_system()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0c89b0e-2455-48a3-bdfe-2009a3811f22",
   "metadata": {},
   "source": [
    "We load the data from a local folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7417ae9b-9322-4eba-b9ea-13721cdd1574",
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import path\n",
    "import cognee\n",
    "\n",
    "# Absolute path to the data directory\n",
    "data_dir = path.abspath(\"../.data\")\n",
    "\n",
    "cognee.config.data_root_directory(data_dir)\n",
    "\n",
    "datasets = cognee.datasets.discover_datasets(data_dir)\n",
    "print(datasets)\n",
    "\n",
    "results = await cognee.add(f\"data://{data_dir}\", datasets[0])\n",
    "\n",
    "for result in results:\n",
    "    print(result)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e39296d",
   "metadata": {},
   "source": [
    "After the data has been added, we can check the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39df49ca-06f0-4b86-ae27-93c68ddceac3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cognee\n",
    "\n",
    "datasets = cognee.datasets.list_datasets()\n",
    "\n",
    "for dataset in datasets:\n",
    "    print(dataset)\n",
    "    data_from_dataset = cognee.datasets.list_data(dataset)\n",
    "    for file_info in data_from_dataset:\n",
    "        print(file_info) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb617b76",
   "metadata": {},
   "source": [
    "All good, let's cognify it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97e8647a-052c-4689-b1de-8d81765462e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cognee\n",
    "from cognee.shared.utils import render_graph\n",
    "\n",
    "graph = await cognee.cognify()\n",
    "\n",
    "graph_url = await render_graph(graph)\n",
    "print(graph_url)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe9e9217-35bc-4c59-9a13-750f8a8ae970",
   "metadata": {},
   "outputs": [],
   "source": [
    "We get the url to the graph on graphistry, showing nodes and connections made by the cognify process.\n",
    "\n",
    "We can also use the cognee.search module to explore the knowledge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ee34d29-f75d-4766-8906-b851b57bd5ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cognee import search\n",
    "\n",
    "\n",
    "from cognee.api.v1.search.search import SearchType\n",
    "\n",
    "search_type = 'SIMILARITY'\n",
    "params = {'query': 'Ministarstvo', 'other_param': {\"node_id\": \"LLM_LAYER_SUMMARY:DOCUMENT:881ecb36-2819-54c3-8147-ed80293084d6\"}}\n",
    "\n",
    "results = await search(search_type, params)\n",
    "\n",
    "for result in results[0]:\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b149c4df-1665-408b-8fb9-735ac2a3770f",
   "metadata": {},
   "source": [
    "# Some common questions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d57a4ea-4157-46cd-834d-d0341be0bdba",
   "metadata": {},
   "source": [
    "- This tool is not a replacement for vector databases or Langchain, it's an extension \n",
    "- We want to help map the old data world to the new one\n",
    "- Local models and evals are on the roadmap\n",
    "- The relationships in the graph are customizable, so is the graph model. It is something we will continue exploring"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed599dcd-c65a-4721-bffe-777a2e632293",
   "metadata": {},
   "source": [
    "# Give us a star if you like it!\n",
    "https://github.com/topoteretes/cognee"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffaf695f",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
