[SOLVED] curl + bash + sed - how to combine mandatory query with optional query in URL? Replacing "?" with "?client=xxx&" adds a "?"

Issue

I have a bash function to repeated curl call, accepting different URLs as function args. The urls are different and one of them has previous query, like "?foo=bar". The API behind these urls are defined by users but they meet the same standard and need a mandatory arg like client=xxx. So I am thinking how to do this in bash scripting.

Now I am doing:

    client_query=client=xxx
    oauth_url_with_client=$(echo $oauth_url | sed 's/?/?'$client_query'&/g') # if found, replace first query with client query
    echo "First step: $oauth_url_with_client"
    # when there is no "?", no replace happens, need to append
    if [[ $oauth_url_with_client != *"$client_query"* ]]; then
        oauth_url_with_client=$oauth_url_with_client?$client_query
    fi
    echo "Final oauth url: $oauth_url_with_client"

But now I see:

First step: https://host/oauth2/token?client=xxx?foo=bar
Final oauth url: https://host/oauth2/token?client=xxx?foo=bar

I see sed is not replacing ? with ?client=xxx&. What is the problem here?

Is there some good way to combine possible/optional query from URL with mandatory query in --data or --form? Not sure where to put these

Solution

Thanks for the suggestions from @tripleee, that I could check with shellcheck.net, which gave me a suggestion to use ${var//search/replace} form instead of sed; then I came up with this which works:

oauth_url_with_client=${oauth_url//"?"/"?$client_query&"}

Here I am not afraid of quoting; in the sed pattern I fear there would be side effects but here I can quote as much as I like.

But I also agree with @user1934428 that actually no expanding should happen in an url; I have considered the possible sequence of not quoting actually. And, quoting does not solve my problem.

And, after that, I am thinking if & should be treated specially and a search of "sed & pattern" leads me to https://unix.stackexchange.com/questions/296705/using-sed-with-ampersand which confirms me that & represents found part in search so it needs escaping:

oauth_url_with_client=$(echo $oauth_url | sed 's/?/?'"$client_query"'\&/g')

which also works.

The last question still stands: if curl could it natively? I will leave it to the future since my problem is solved in two ways already; but I guess this is too much for curl, it does not expect mixing queries from url and from options.

Answered By - WesternGun

Answer Checked By - Cary Denson (WPSolving Admin)