[SOLVED] BashOperator execute python installed on windows

Issue

Please reference:

Pointing bash to a python installed on windows.

Taking that concept and attempting to apply to Apache Airflow. I have Airflow running on a Windows Subsystem for Linux (WSL). I also have a Python installation and .py file installed on Windows 10. I'd like to execute the .py file using Airflow.

If I execute the following in Ubuntu:

/mnt/c/path/to/windows_python/python.exe C:\\path\\to\\my\\python\\file\\my_file.py

my_file.py runs using my Windows 10 installation of Python. However if I make that same call using BashOperator in a DAG I see a "Bash command failed" error. The task in my DAG file looks like:

t1 = BashOperator(
    task_id = 'my_task'
    , bash_command = '/mnt/c/path/to/windows_python/python.exe C:\\path\\to\\my\\python\\file\\my_file.py'
    , dag = dag)

The DAG fails in Airflow with the message in the logs: can't open file C:\path\to\my\python\file\my_file.py. No such file or directory.

The reason I want to call the Python file and executable on the Windows side is because I already have SQL drivers and VPN setup there.

Any assistance is greatly appreciated. Goal is to call .py file from Airflow (running on WSL) by any means necessary.

Solution

Going to answer my own question here in case someone else needs this in the future. I can't explain the behavior I see but as a workaround:

1. Create .sh file containing the bash command
2. Call the .sh file using BashOperator

Example:

command = '/path/to/sh/file/my_file.sh '

t1 = BashOperator(
    task_id = 'my_task'
    , bash_command = command
    , dag = dag)

Answered By - kjmerf