Issue
I have a project that used virtualenv to package a python 2.7 project with 3 dependencies I found in a requirements.txt file: boto3, botocore and pypdftk.
Boto3 is an Amazon Web Services SDK for Python, and so is botocore (I believe.) Pypdftk is some external library used for transforming PDFs in python.
Now I am supposed to get this project compressed to a zip and uploaded to AWS Lambda, a service for server less computing.
Additionally, AWS Lambda only supports the standard python 3.9 library & runtime. So because my project has these external libraries and dependencies, it seems the past developer used a virtualenv to:
- package a deprecated version of python 2.7
- package the dependencies listed in the requirements.txt file
AWS Lambda has a feature called Layers, where you can upload zipped binaries to extend the standard core python3 library.
In summary:
I am failing to understand how to upload my compressed python3.9 project.
Do I upload these dependencies separately in the AWS Lambda Layer?
OR does compressing my file inside a virtualenv take care of the dependencies?
Much like a Docker Container? The virtualenv ships with the compiler/interpreter & dependencies?
Solution
Upgrade your Python code/dependencies to 3.9, following the "how-to" here: https://docs.python.org/3/howto/pyporting.html
While it's possible to deploy 2.7 code using a Docker image (which, at least for now , is still provided by AWS), that's not a long-term solution and you'll almost certainly put in more work to make it happen.
For your other questions
package a deprecated version of python 2.7
Virtual environments won't let you do this. There are tools such as pyenv that do, but they won't work for a Lambda because the version of Python that's used to run your Lambda is part of the Lambda configuration, and cannot be replaced.
package the dependencies listed in the requirements.txt file
Yes, this is what a virtual environment can be used for. When you activate the virtual environment and run pip install, it will install the packages into the lib directory in the virtual environment.
To produce a Lambda deployment bundle you must ZIP your source code together with the installed packages. Making this a little more challenging, the packages are actually installed in lib/python3.9/site-packages/, and Lambda doesn't want to see that path; change into the directory while building the ZIP.
Here's is the official doc for producing Python bundles: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html
Do I upload these dependencies separately in the AWS Lambda Layer?
You can, but that's only useful if you're going to share the dependencies between Lambdas. Otherwise it increases the complexity of your deployment with little benefit.
Answered By - Parsifal