[SOLVED] Crontab to launch python script on raspberry pi causes import error

Issue

I am running Ubuntu Mate on my Raspberry PI. I wrote a web scraper in Python that I want to run once a day. I figured using sudo crontab -e would be a good way to do way. The problem that I'm having is when the cronjob launches my python script, the python script throws an import error and doesn't run. However, when I execute the python script directly from the command line, it runs without issue.

I was reading that some people use a shell script to launch they python script, so I tried that as well. Again, it works as expected when I execute the shell script directly from the command line, but does not work when executed by the cronjob.

For testing purposes, right now I have the cronjob to execute every minute until I figure out what's going on. Once it works as expected, I'll change the time that it executes to something closer to what I want.

To give you an idea of what my files look like, check it out below. As you can see, all of the files that need to run have execute prilivages.

web_scraper.py

#!/usr/bin/env python

import click
import logging
import os
from datetime import datetime
from bs4 import BeautifulSoup as bs
import re
import urlparse
...

start.sh

#!/bin/bash

cd /home/elmer/

python web_scraper.py

sudo crontab -e

* * * * * sh /home/elmer/start.sh >> /home/elmer/cron.log 2>&1

cron.log

Traceback (most recent call last):
  File "web_scraper.py", line 6, in <module>
    import click

ls -lh

(py27)elmer@elmer-rpi:$ ls -lh
total 56K
-rw-rw-r-- 1 elmer elmer 2.9K Mar 17 20:38 cron.log
-rwxrwxrwx 1 elmer elmer 8.2K Mar 16 09:54 web_scraper.py
-rwxrwxrwx 1 elmer elmer   64 Mar 17 20:02 start.sh

Solution

The problem is that you're running your script as yourself, but then you put it in cron as root. Those two different users have very different environments.

You should just use crontab -e instead of sudo crontab -e to put it in your own account's cron. You can also remove the sh from the start of your cron command line, as that is the default.

In case it still does not work, the solution is likely to echo $PYTHONPATH in your regular shell, then add that setting to cron:

PYTHONPATH=/your/path/here
* * * * * python /home/elmer/web_scraper.py >> /home/elmer/cron.log 2>&1

Another good reason not to use sudo crontab is that if you run your script as root, any bugs it has could ruin your system.

Answered By - John Zwinck