[SOLVED] Reading django_settings from Google Cloud Platform's Secret Manager does not work

Issue

When running the command python manage.py makemigrations locally on my laptop, I get the following error on my console:

(mywebsite) C:\Users\Sander\PycharmProjects\mywebsite>python manage.py makemigrations
Invalid line: echo DATABASE_URL=postgres://myuser:mypassword@//cloudsql/mywebsite:europe-west6:mywebsite-db/mydb > .env
Invalid line: echo GS_BUCKET_NAME=mybucket >> .env
Invalid line: echo SECRET_KEY=$(cat /dev/urandom | LC_ALL=C tr -dc '[:alpha:]'| fold -w 50 | head -n

Note that the echo [... etc. ...] > .env instructions are actually the content of a secret I configured on Google Cloud Platform's Secret Manager, when following Google Cloud Platform's instructions for deploying my Django website on Google Cloud Run.

Now I do know these echo [... etc. ...] > .env instructions are supposed to create a file .env with the variables DATABASE_URL, GS_BUCKET_NAME and SECRET_KEY in it, but it doesn't, of course, since it reports the error "Invalid line: ..." instead.

I found this StackOverflow answer, stating that these bash instructions (echo [... etc. ...] > .env) simply can't be executed by python and that I can simply execute them locally by running them from a shell script instead, so executing this create-env-file.sh works:

DATABASE_URL=postgres://myuser:mypassword@//cloudsql/mywebsite:europe-we
st6:mywebsite-db/mydb > .env
GS_BUCKET_NAME=mybucket >> .env
echo SECRET_KEY=$(cat /dev/urandom | LC_ALL=C tr -dc '[:alpha:]'| fold -w 50 | head -n

However, this generates a local .env file and obviously I would like to confirm that the flow with the Secret Manager works instead, before moving the website to production, since if I keep the .env file and upload it to production, there's no point in still using the Secret Manager, it is simply not required anymore.

This also becomes very clear when watching the code that is actually run in the back:

import io
import os
from pathlib import Path

import environ
import google.auth
from google.cloud import secretmanager

if os.path.isfile(env_file):
    # Use a local secret file, if provided

    env.read_env(env_file)
elif os.environ.get("GOOGLE_CLOUD_PROJECT", None):
    # Pull secrets from Secret Manager
    project_id = os.environ.get("GOOGLE_CLOUD_PROJECT")

    client = secretmanager.SecretManagerServiceClient()
    settings_name = os.environ.get("SETTINGS_NAME", "django_settings")
    name = f"projects/{project_id}/secrets/{settings_name}/versions/latest"
    payload = client.access_secret_version(name=name).payload.data.decode("UTF-8")

    env.read_env(io.StringIO(payload))
else:
    raise Exception("No local .env or GOOGLE_CLOUD_PROJECT detected. No secrets found.")

I debugged that code and the problem is in the execution of env.read_env(io.StringIO(payload)), since the payload contains the expected content (the echo [... etc. ...] > .env lines).

I see the advantage of using Google's Secret Manager, but I'm starting to think there's either something wrong with the content of the secret that the Django Cloud Run tutorial provided, or with the code parsing and saving it as environment variables.

I'm running locally on Windows, with the following libraries (content of my requirements.txt):

django

# For integrating with Google Cloud Platform:
django-storages[google]>=1.12
django-environ>=0.8.1
psycopg2-binary>=2.9.1
gunicorn>=20.1.0
google-cloud-secret-manager>=2.7.2

Solution

This is apparently caused by two things:

• In settings.py, the secret content is loaded into environment variables with env.read_env(io.StringIO(payload)), as mentioned in the question. That read_env() function apparently does the following:

  # ...
  for line in content.splitlines():
      m1 = re.match(r'\A(?:export )?([A-Za-z_0-9]+)=(.*)\Z', line)
      if m1:
          # ...
      elif not line or line.startswith('#'):
          # ignore warnings for empty line-breaks or comments
          pass
      else:
          logger.warning('Invalid line: %s', line)
  # ...
  

  Because the secret content's lines start with "echo ", the lines are marked as invalid (you can try this out on https://regex101.com) and the command python manage.py makemigrations therefore crashes with that error.

• When editing payload before it enters env.read_env(io.StringIO(payload)) such that it no longer starts with echo (to solve the problem described in the bullet above) and, also because of the regex matching, also removing > .env or >> .env at the end, the 3rd line also causes problems: SECRET_KEY=$(cat /dev/urandom | LC_ALL=C tr -dc '[:alpha:]'| fold -w 50 | head -n. The reading of /dev/urandom to generate a pseudo-random SECRET_KEY is a problem on Windows, /dev/urandom only exists in Linux-based operating systems.

So it seems like the secret should work on Google Cloud Run's production Linux-based servers, but not on a local computer running Windows, if executed like in the instructions for deploying the Django website.

Instead, for local runs, do create an .env file, either by indeed running the create-env-file.sh script mentioned in the question (*), or by creating the .env file manually in the same folder as settings.py:

DATABASE_URL=postgres://myuser:mypassword@//cloudsql/mywebsite:europe-west6:mywebsite-db/mydb
GS_BUCKET_NAME=mybucket
SECRET_KEY=some1279Long30String5You7Create2Yourself8136

(*) /dev/urandom did work for me then, because when I execute create-env_file.sh on a terminal window in PyCharm, a new terminal of Git Bash opens on my computer, probably because on my computer that is the registered program to execute shell files.

Answered By - Sander Vanden Hautte

Answer Checked By - Katrina (WPSolving Volunteer)