[SOLVED] numpy, pandas modules not found when deploying flask app behind apache with pyenv and virtualenv

Issue

I built a flask frontend for a rather detailed app and it runs perfectly locally after a lot of coding. So now I want to make it visible to the public, so have read that it needs an apache or nginx frontend. I worked out the size of instance I needed and switched it on on AWS, and ran through these two tutorials which prove that the base case is working. Once the Apache setup is done, the Apache welcome page shows when I navigate to the IP of the AWS instance. Then when I temporarily substitute the "Hello World" code for my flask app and refresh the browser with the IP, the hello world appears rendered. Also, if I substitute the flasskapp.py code from here it perfectly serves across apache to the IP typed into the browser a list of modules found by pip including Numpy and Pandas which it otherwise complains about not finding! This script, could however easily be killed by adding import numpy at the top, but it gives me the following error messages:

ImportError: cannot import name 'multiarray'

and

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` 
(removes all files not under version control).  
Otherwise reinstall numpy.

both in the apache log.

The trouble happens when I put my app back in instead of hello world. It just refuses to import pandas here is the last section of the Apache log:

Mon Jul 03 19:42:14.691081 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560] Traceback (most recent call last):
[Mon Jul 03 19:42:14.691126 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]   File "/var/www/html/flaskapp/flaskapp.wsgi", line 30, in <module>
[Mon Jul 03 19:42:14.691130 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]     from similar_items_frontend import app as application
[Mon Jul 03 19:42:14.691137 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]   File "/var/www/html/flaskapp/similar_items_frontend.py", line 4, in <module>
[Mon Jul 03 19:42:14.691140 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]     from item_similar_to_doc import similar_to_doc
[Mon Jul 03 19:42:14.691145 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]   File "/home/ubuntu/find-similar/frontend/../item_similar_to_doc.py", line 1, in <module>
[Mon Jul 03 19:42:14.691148 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]     from PullVectors import p
[Mon Jul 03 19:42:14.691153 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]   File "/home/ubuntu/find-similar/frontend/../RunSimilarity.py", line 5, in <module>
[Mon Jul 03 19:42:14.691156 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]     import pandas as pd
[Mon Jul 03 19:42:14.691161 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]   File "/home/ubuntu/.pyenv/versions/miniconda3-latest/envs/find_similarProject/lib/python3.6/site-packages/pandas/__init__.py", line 19, in <module>
[Mon Jul 03 19:42:14.691164 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560]     "Missing required dependencies {0}".format(missing_dependencies))
[Mon Jul 03 19:42:14.691180 2017] [wsgi:error] [pid 12300:tid 140131159766784] [client 93.21.05.132:52560] ImportError: Missing required dependencies ['numpy']

I feel like I've tried everything it's been many painful hours, hair pulling time..

Here is 000-default.conf :

WSGIPythonPath /home/ubuntu/.pyenv/versions/miniconda3-latest/envs/vtenv4YTproject/lib/python3.6/site-packages/

<VirtualHost *:80>
        # The ServerName directive sets the request scheme, hostname and port that
        # the server uses to identify itself. This is used when creating
        # redirection URLs. In the context of virtual hosts, the ServerName
        # specifies what hostname must appear in the request's Host: header to
        # match this virtual host. For the default virtual host (this file) this
        # value is not decisive as it is used as a last resort host regardless.
        # However, you must set it for any further virtual host explicitly.
        #ServerName www.example.com

        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html

        WSGIDaemonProcess flaskapp threads=5
        WSGIScriptAlias / /var/www/html/flaskapp/flaskapp.wsgi

        <Directory flaskapp>
              WSGIScriptReloading On
              WSGIProcessGroup %{GLOBAL}
              WSGIApplicationGroup %{GLOBAL}
              Order allow,deny
              Allow from all
        </Directory>



        # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
        # error, crit, alert, emerg.
        # It is also possible to configure the loglevel for particular
        # modules, e.g.
        #LogLevel info ssl:warn

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        # For most configuration files from conf-available/, which are
        # enabled or disabled at a global level, it is possible to
        # include a line for only one particular virtual host. For example the
        # following line enables the CGI configuration for this host only
        # after it has been globally disabled with "a2disconf".
        #Include conf-available/serve-cgi-bin.conf
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

I should mention that my flask app runs perfectly (to the point of flask serving the page) in its virtualenv when I am on ssh into the AWS instance, of course was a bit of tinkering to get that to work, but it finds all its dependencies, not crashing at the pandas import.

To try and fix it I looked here, here, added some code from here, went through the extensive docs here, my flaskapp.wsgi file looks like this:

import sys
import logging
import os.path
sys.path.append(os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir))
import config
import site

prev_sys_path = list(sys.path) 

site.addsitedir('/home/ubuntu/.pyenv/versions/miniconda3-latest/envs/find_similarProject/lib/python3.6/site-packages')

new_sys_path = []

for item in list(sys.path): 
    if item not in prev_sys_path:
        new_sys_path.append(item) 
        sys.path.remove(item)
sys.path[:0] = new_sys_path 

logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, "/var/www/html/flaskapp")

from similar_vids_frontend import app as application
application.secret_key = config.WSGI_KEY

There is just no reason I can find why pandas / numpy imports are failing with apache but working fine when just run in the virtualenv. I wonder if there's something to do with permissions and groups, as per the docs:

Do be aware that the user that Apache runs your code as will need to be able to access the Python virtual environment. On some Linux distributions, the home directory of a user account is not accessible to other users. Rather than change the permissions on your home directory, it might be better to consider locating your WSGI application code and any Python virtual environment outside of your home directory.

But is Ubuntu one of those distributions, and what then, install the virtual environments and project into the / folder?

Grateful for any tips or a solution.

Solution

This now works, mainly thanks to Graham's recipe here and also this answer about loading modules into apache. The proof that it works is that the following variation of the hello world app:

from flask import Flask
import numpy
app = Flask(__name__)

@app.route('/')
def hello_from_np():
  a = numpy.array([4,5,6])
  return str(a)

if __name__ == '__main__':
  app.run()

gives you this in the browser when you go to the IP:

...rather than a bunch of errors in the apache log about numpy. It's all set up for Python 3.6:

AH00489: Apache/2.4.7 (Ubuntu) mod_wsgi/4.5.15 Python/3.6 configured

Wanted to record a few details as a note to self and in case anyone is having trouble and trying to reproduce this, maybe save you from days of barking up the wrong tree..

So I switched from Ubuntu Server 16.04 to 14.04.5 LTS in case that was part of the issue but also because pyenv was performing poorly, crashing half way through the installation of many older Python versions I was testing. Also chose to leave out pyenv-virtualenv as there were no multiple projects planned for this instance and it was one less thing to go wrong.

From the clean install of Ubuntu, in addition to installing apache2, pyenv, libapache2-mod-wsgi, activating it with a2enmod, I deliberately ran the following lines before installing the requirements with pip, which included numpy, so it doesn't complain:

sudo apt-get install gcc
sudo apt-get install g++

This is also needed:

sudo apt-get install apache2-dev

Then inside the environment, after the requirements,

pip install mod_wsgi
mod_wsgi-express module-config

The second line returned:

LoadModule wsgi_module "/home/ubuntu/.pyenv/versions/miniconda3-latest/lib/python3.6/site-packages/mod_wsgi/server/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"
WSGIPythonHome "/home/ubuntu/.pyenv/versions/miniconda3-latest"

Then with loading the module, the docs encourages you to paste into a file httpd.conf which at least in my version of apache and ubuntu does not exist. After a lot of reading, I avoided creating one and also avoided pasting it into apache2.conf which seems to be related. Rather, following the instructions for ubuntu linked to at the top,

edit this file:

sudo vi /etc/apache2/mods-available/wsgi.load

paste this line into it and save:

LoadModule wsgi_module /path/to/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so

then run this and restart the apache server if necessary:

sudo a2enmod wsgi

000-default.conf looks like this:

WSGIPythonHome "/home/ubuntu/.pyenv/versions/miniconda3-latest"

<VirtualHost *:80>
    # The ServerName directive sets the request scheme, hostname and port that
    # the server uses to identify itself. This is used when creating
    # redirection URLs. In the context of virtual hosts, the ServerName
    # specifies what hostname must appear in the request's Host: header to
    # match this virtual host. For the default virtual host (this file) this
    # value is not decisive as it is used as a last resort host regardless.
    # However, you must set it for any further virtual host explicitly.
    #ServerName www.example.com

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html
        WSGIDaemonProcess helloapp threads=5 python-path=/var/www/html/frontend/  
        WSGIScriptAlias / /var/www/html/frontend/helloapp.wsgi

        <Directory flaskapp>
        WSGIProcessGroup helloapp
        WSGIApplicationGroup %{GLOBAL}
        Order allow,deny
        Allow from all
        </Directory>

    # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
    # error, crit, alert, emerg.
    # It is also possible to configure the loglevel for particular
    # modules, e.g.
    #LogLevel info ssl:warn

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    # For most configuration files from conf-available/, which are
    # enabled or disabled at a global level, it is possible to
    # include a line for only one particular virtual host. For example the
    # following line enables the CGI configuration for this host only
    # after it has been globally disabled with "a2disconf".
    #Include conf-available/serve-cgi-bin.conf
</VirtualHost>

Answered By - cardamom

Answer Checked By - Cary Denson (WPSolving Admin)