Issue
I need a cron to run a single shell script. This shell script should activate the python virtualenv and then execute python scripts. This is my cron:
0 0 * * 1 /home/ubuntu/crontab_weekly.sh
This is my crontab_weekly.sh file:
cd /home/ubuntu/virtualenvironment/scripts \
&& source /home/ubuntu/venv3.8/bin/activate \
&& python script1.py \
& python script2.py \
& python script3.py \
& python script4.py \
The idea is to enter the directory where the scripts are hosted, then activate the venv, and only then start executing the scripts. The scripts should be executed in parallel. But in my case only script1.py is executed and the following scripts are not executed. Where is my problem?
Solution
Remember that & means to run the entire previous command asynchronously. This includes anything before a &&. Commands that run asynchronously run in separate processes.
To take a simplified example of your problem, let's say we asynchronously change directories, run pwd, and asynchronously run pwd again.
#!/bin/sh
cd / && \
pwd \
& pwd
On my computer, this outputs:
/home/nick
/
The cd / was meant to affect both pwd calls, but it only affected the first one, because the second one runs in a different process. (They also printed out of order in this case, the second one first.)
So, how can you write this script in a more robust fashion?
First, I would turn on strict error handling with -e. This exits as soon as any (non-asynchronous) command returns a non-zero exit code. Second, I would avoid the use of &&, because strict error handling deals with this. Third, I would use wait at the end to make sure the script doesn't exit until all of the sub-scripts have exited.
#!/bin/sh
set -e
cd /
pwd &
pwd &
wait
The general idea is that you turn on strict error handling, do all of your setup in a synchronous fashion, then launch your four scripts asynchronously, and wait for all to finish.
To apply this to your program:
#!/bin/sh
set -e
cd /home/ubuntu/virtualenvironment/scripts
source /home/ubuntu/venv3.8/bin/activate
python script1.py &
python script2.py &
python script3.py &
python script4.py &
wait
Answered By - Nick ODell Answer Checked By - Marilyn (WPSolving Volunteer)