[SOLVED] Shell script: curl download fails but browser downloads successfully

Issue

I seem to have run into some server side logic that's preventing me from automating the installation of Respondus LockDown Browser for Mac.

When I download the installer from a gui web browser, the installer downloads without issue. However, when I try to download the file from the command line using curl, it fails.

The full installer should weight in at 117.8MB, but curl results in a 5KB file.

Here's the code I'm trying to troubleshoot:

#!/bin/zsh

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Download and install Respondus Lockdown Browser from .pkg
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Define Variables

installerName="RespondusLockdown"
pkgfile="/tmp/$installerName.pkg"
url="https://download.respondus.com/lockdown/download7.php?id=228644488&ostype=2"

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Define functions

checkResult() {
    error="An error occurred"
    if [[ $? = 0 ]]
    then
        printf "$success"
    else
        printf "$error"
    fi
    printf "\n\n"
}

downloadInstaller() {
    local success="Package downloaded successfully"
    curl -Lo "${pkgfile}" "${url}"
    checkResult
}

installPKG() {
    local success="Package installed successfully"
    installer -pkg "$pkgfile" -target /
    checkResult
}

cleanUp() {
    local success="Package removed successfully"
    rm "$pkgfile"
}

main() {
    downloadInstaller
    installPKG
#   cleanUp
}
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# main

main

exit 0

I've tried changing the user agent by adding the following, but to no avail:

userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15"

curl -A "${userAgent}" -L -o "${pkgfile}" ${url}

Solution

The URL you provide is a webpage with an auto download (maybe through JavaScript, I'm not familiar with the front end). You should use the actual file link. If you curl directly on console, you will get this part:

...
<br>
<div style="text-align: left;">Your browser will begin the download momentarily.  If the download does not start, <a href="https://downloads.respondus.com/installs/cmac2.1.1.03/228644488/InstallLDBPackage64c-2-1-1-03.zip">click this link to start the download manually.</a></div>
</td>
...

So the download link should be https://downloads.respondus.com/installs/cmac2.1.1.03/228644488/InstallLDBPackage64c-2-1-1-03.zip

Answered By - Tianshu Wang

Answer Checked By - Pedro (WPSolving Volunteer)