From 8dd575e0043d65a873544102c39d4c336bf44d90 Mon Sep 17 00:00:00 2001 From: alekszievr <44192193+alekszievr@users.noreply.github.com> Date: Wed, 19 Mar 2025 15:02:55 +0100 Subject: [PATCH] chore: move ec2 setup file and remove extra steps [cog-1585] (#653) ## Description This .sh file can be used for EC2 deployment as explained in https://github.com/topoteretes/cognee-docs/pull/58 ## DCO Affirmation I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin ## Summary by CodeRabbit - **Documentation** - Removed outdated guidance for setting up evaluation environments, streamlining the visible instructions. - **Chores** - Updated the Ubuntu setup process to install Python 3.12, ensuring the virtual environment uses the latest version and enhancing overall performance. --- .../setup_ubuntu_instance.sh | 7 +- evals/EC2_README.md | 64 ------------------- 2 files changed, 2 insertions(+), 69 deletions(-) rename {evals/cloud => deployment}/setup_ubuntu_instance.sh (83%) delete mode 100644 evals/EC2_README.md diff --git a/evals/cloud/setup_ubuntu_instance.sh b/deployment/setup_ubuntu_instance.sh similarity index 83% rename from evals/cloud/setup_ubuntu_instance.sh rename to deployment/setup_ubuntu_instance.sh index e05b761e2..854cd1c9f 100644 --- a/evals/cloud/setup_ubuntu_instance.sh +++ b/deployment/setup_ubuntu_instance.sh @@ -15,19 +15,16 @@ sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plug sudo docker run hello-world -sudo apt install -y unzip - sudo apt-get install -y python3-virtualenv sudo add-apt-repository -y ppa:deadsnakes/ppa sudo apt update -y -sudo apt install -y python3.11 +sudo apt install -y python3.12 -virtualenv venv --python=python3.11 +virtualenv venv --python=python3.12 source venv/bin/activate pip install poetry poetry install -pip install swebench transformers sentencepiece datasets tiktoken protobuf diff --git a/evals/EC2_README.md b/evals/EC2_README.md deleted file mode 100644 index d6a937ed7..000000000 --- a/evals/EC2_README.md +++ /dev/null @@ -1,64 +0,0 @@ -## Creating the EC2 Instance - -Create an EC2 Instance with the - -`Ubuntu Image` - -Many instance types will work, we used: - -`m7a.2xlarge` # more than 8 parallel processes doesn't seem to speed up overall process. Maybe to do with docker parallelism? - -DON'T FORGET TO ADD - -`500 GB storage` - -Or the evaluation run will run out of space - -Add a key pair login where you have access to the corresponding key file (*.pem) - -## Accessing your instance and setup - -To ssh into the instance, you have to save your key pair file (*.pem) to an appropriate location, such as ~/.aws. After launching the instance, you can access the Instance Summary, and retrieve "Public IPv4 DNS" address. Then run - -`ssh -i PATH_TO_KEY ubuntu@IPv4ADDRESS` - -to gain command line access to the instance. - -To copy your current state of cognee, go to the folder that contains "cognee" on your local machine, zip it to cognee.zip and run: - -`zip -r cognee.zip cognee` -`scp -i PATH_TO_KEY cognee.zip ubuntu@IPv4ADDRESS:cognee.zip` - -And unzip cognee.zip in your SSH session: - -`sudo apt install unzip` -`unzip cognee.zip` - -Then run: -`cd cognee` -`source evals/cloud/setup_ubuntu_instance.sh` - -`sudo usermod -aG docker $USER` - -disconnect, and reconnect. - -Confirm that `ubuntu` has been added to the docker user group with - -`groups | grep docker` - -## Running SWE-bench - -Then enter a `screen` and activate the virtual env - -`screen` -`source venv/bin/activate` - -then, from cognee, you can run swe_bench: - -`cd cognee` - -`python evals/eval_swe_bench.py --cognee_off --max_workers=N_CPUS` - -Building the environment images should take roughly 17 minutes - -If the virtual env wasn't set up correctly for some reason, just run the last few lines of `setup_ubuntu_instance.sh` manually \ No newline at end of file