[SOLVED] Caching Python venv folder with CodeBuild

Issue

I am trying to cache the Virtual environment folder (.venv) for my Python project with CodeBuild.

Here's my buildspec.yml file:

version: 0.2
env: 
  shell: bash
phases:
  install:
    commands:
      - python3 -m venv .venv && source .venv/bin/activate
      - pip3 install -r requirements.txt
  build:
    commands:
      - pytest -v tests/
cache:
  paths:
    - .venv/**/*

This is the error I get:

[Container] 2022/11/10 11:33:20 MkdirAll: /codebuild/local-cache/custom/11615810f/.venv
[Container] 2022/11/10 11:33:20 Symlinking: /codebuild/output/src682375820/src/hello.org/demo/.venv => /codebuild/local-cache/custom/11615810f/.venv

[Container] 2022/11/10 11:34:01 Running command python3 -m venv .venv && source .venv/bin/activate
Error: Unable to create directory '/codebuild/output/src682375820/src/hello.org/demo/.venv'

[Container] 2022/11/10 11:34:01 Command did not exit successfully python3 -m venv .venv && source .venv/bin/activate exit status 1
[Container] 2022/11/10 11:34:01 Phase complete: INSTALL State: FAILED
[Container] 2022/11/10 11:34:01 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: python3 -m venv .venv && source .venv/bin/activate. Reason: exit status 1

The problem is that you can't instantiate a Python virtual environment into a symlinked folder.

Any ideas?

Solution

CodeBuild uses symlinks to link cached directories and python -m venv tries to create a new directory over a symlink, which isn't possible.

Try to run python3 -m venv command inside the .venv directory:

install:
    commands:
      - cd .venv && python3 -m venv . && cd -
      - source .venv/bin/activate
      - pip3 install -r requirements.txt

If you check the CodeBuild log, you'll see that symlink is created at the very beginning:

# Example when /opt/poetry is cached
[Container] 2023/12/11 08:10:45.046132 Expanded cache path /opt/poetry/
[Container] 2023/12/11 08:10:45.495963 MkdirAll: /codebuild/local-cache/custom/5694e39f8fd7f99453b01828b7946ba4ef802b635c8c9ec0be9b56824a16d47c/opt/poetry
[Container] 2023/12/11 08:10:45.496095 Symlinking: /opt/poetry => /codebuild/local-cache/custom/5694e39f8fd7f99453b01828b7946ba4ef802b635c8c9ec0be9b56824a16d47c/opt/poetry

Answered By - Tom Ehrlich

Answer Checked By - Gilberto Lyons (WPSolving Admin)