[SOLVED] How can I install specific python library in virtual environment in Dockerfile(airflow)?

Issue

I have a Dockerfile:

FROM apache/airflow:2.4.3-python3.8
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         vim \
         gcc \
         g++ \
         libsasl2-dev \
         libc6-dev \
         git \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

USER airflow
RUN python -m virtualenv dbt_venv && source dbt_venv/bin/activate && \
  pip install dbt-core

When I built an image based on this, it raised error: ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

I think this is kinda USER problem but have no idea how to handle this.

I don't need persistence of virtualenv, which mean it doesn't have to be activated whenever container run. I'd like to just install some packges in virtualenv.

Any suggestion?

Solution

It's because apache/airflow Dockerfile sets env var PIP_USER=true. Toggle it it to work with virtualenvs:

ENV PIP_USER=false

before your last RUN.

Another approach is to clear it right in the RUN:

RUN python -m virtualenv dbt_venv && source dbt_venv/bin/activate && \
  PIP_USER=false pip install dbt-core

Answered By - phd

Answer Checked By - Timothy Miller (WPSolving Admin)