[SOLVED] Airflow on WSL2 Ubuntu: ValueError: Unable to configure handler 'processor'

Issue

I am aware that very similar questions have been asked previously but I have found these have tended to include Docker, which I am not using at this time, nor do I have installed. I am led to believe that WSL2 should be an alternative to Docker regarding running Airflow on Windows.

I am using WSL2 on my Windows 11 laptop and have installed Apache-Airflow from a tutorial from the following link: https://coding-stream-of-consciousness.com/2018/11/06/apache-airflow-windows-10-install-ubuntu/

On WSL2 I have:

• Python version 3.8.10
• Pip version 20.0.2
• Apache-Airflow 2.4.1 (I believe)

I have run the following commands (as per the tutorial) with no issue:

sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get update
sudo apt-get install python-pip
sudo pip install apache-airflow   #I had path issues without 'sudo' command

But when I attempt to use the 'airflow' command in the WSL2 Ubuntu terminal, I am greeted with the following error:

$ airflow
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/airflow/utils/log/file_processor_handler.py", line 45, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/dist-packages/airflow/utils/helpers.py", line 165, in parse_template_string
    import jinja2
  File "/usr/lib/python3/dist-packages/jinja2/__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 15, in <module>
    from jinja2 import nodes
  File "/usr/lib/python3/dist-packages/jinja2/nodes.py", line 23, in <module>
    from jinja2.utils import Markup
  File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 656, in <module>
    from markupsafe import Markup, escape, soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.8/dist-packages/markupsafe/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.8/dist-packages/airflow/__init__.py", line 46, in <module>
    settings.initialize()
  File "/usr/local/lib/python3.8/dist-packages/airflow/settings.py", line 564, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/usr/local/lib/python3.8/dist-packages/airflow/logging_config.py", line 74, in configure_logging
    raise e
  File "/usr/local/lib/python3.8/dist-packages/airflow/logging_config.py", line 69, in configure_logging
    dictConfig(logging_config)
  File "/usr/lib/python3.8/logging/config.py", line 808, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib/python3.8/logging/config.py", line 570, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'

I have googled the ValueError extensively and can't find any clear solutions that don't involve Docker.

Any insight into the error would be much appreciated!

Solution

You should install airflow using constraints.

https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-tools

This is the only way installation of airflow is guaranteed to work.

Answered By - Jarek Potiuk

Answer Checked By - Marilyn (WPSolving Volunteer)