[SOLVED] selenium Webdriver wait until expected conditon not working properly on Amazon EC2 instance

Issue

I made a script to visit a page and log in then get a download link from the page.
The script works fine on my local window machine, but it's not working on Amazon EC2 instance(ubuntu)
The code is as below

from selenium import webdriver  
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

dir_chrome_driver = "c:/selenium/driver/chromedriver.exe"
parser = ConfigParser()
option = webdriver.chrome.options.Options()

url = "https://ams.amazon.com/webpublisher/analytics/requested_downloads"

option.add_argument('--user-agent="Chrome/102.0.5005.115"')
option.add_argument("--headless")
option.add_argument('--no-sandbox')

driver = webdriver.Chrome(executable_path=dir_chrome_driver, options=option)
# driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=option)
driver.get(url)

WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#ap_email')))

driver.find_element(By.ID, "ap_email").send_keys(USER_ID)
driver.find_element(By.ID, "ap_password").send_keys(USER_PASSWORD)
driver.find_element(By.ID, "signInSubmit").click()

WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.download-link')))
download_link = driver.find_element(By.CSS_SELECTOR, ".download-link")

It gives me an error
"File "aps.py", line 46, in <module>
WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.download-link')))
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 90, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException:"
I added fixed time wait between 'click' and WebDriverWait like below.

driver.find_element(By.ID, "signInSubmit").click()

time.sleep(30)

WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.download-link')))

It worked for a while, but it became not work again today.
I tried to change wait time but the driver still in the login page.

Please advise me if there is any possible cause or solution.

Solution

You need a different setup for selenium on ubuntu/debian:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)

Answered By - platipus_on_fire

Answer Checked By - Terry (WPSolving Volunteer)