[SOLVED] My script only fetches the response from the last curl request

Issue

I am trying to execute a curl request for each data of an array, but I actually only gets one of the answers (even if all the url are ok and the requests works one by one)

My script is as below:

    tmp="{
       \"data\":{
          \"id\":\"sv-Ps7epZJypQGB4Jb3\",
          \"type\":\"state-versions\",
          \"relationships\":{
             \"outputs\":{
                \"data\":[
                   {
                      \"id\":\"wsout-YDDEzn2h9Y9fJERB\",
                      \"type\":\"state-version-outputs\"
                   },
                   {
                      \"id\":\"wsout-CYzpWK2uMDGFn9Yr\",
                      \"type\":\"state-version-outputs\"
                   },
                   {
                      \"id\":\"wsout-RroeVzLsv6PxJ56A\",
                      \"type\":\"state-version-outputs\"
                   },
                   {
                      \"id\":\"wsout-8ts3sHe7NsE46mN6\",
                      \"type\":\"state-version-outputs\"
                   }
                ]
             }
          }
       }
    }"
    echo "$tmp"
    output_ids=$(echo "${tmp}" | jq -r '.data.relationships.outputs.data[].id')
    echo "iic"
    base_url="https://app.terraform.io/api/v2/state-version-outputs/"

    declare -a tableau

    # Créez un fichier temporaire contenant le texte
    tempfile=$(mktemp)
    echo "$output_ids" > "$tempfile"

    # Parcourez chaque ligne du fichier et ajoutez-la au tableau
    while IFS= read -r ligne; do
      tableau+=("$ligne")
    done < "$tempfile"

    # Supprimez le fichier temporaire

    cat "$tempfile"
    rm "$tempfile"
    declare -p tableau


    # Accédez aux éléments du tableau
    for element in "${tableau[@]}"; do
        echo "Élément du tableau : $element"
        full_url="$base_url$element"
        echo "URL : $full_url"
        output_details=$(curl \
         --header "Authorization: Bearer $TOKEN" \
         --header "Content-Type: application/vnd.api+json" \
         -s "$full_url")
        echo "$output_details"
    done

and the (wrong) result:

{
       "data":{
          "id":"sv-Ps7epZJypQGB4Jb3",
          "type":"state-versions",
          "relationships":{
             "outputs":{
                "data":[
                   {
                      "id":"wsout-YDDEzn2h9Y9fJERB",
                      "type":"state-version-outputs"
                   },
                   {
                      "id":"wsout-CYzpWK2uMDGFn9Yr",
                      "type":"state-version-outputs"
                   },
                   {
                      "id":"wsout-RroeVzLsv6PxJ56A",
                      "type":"state-version-outputs"
                   },
                   {
                      "id":"wsout-8ts3sHe7NsE46mN6",
                      "type":"state-version-outputs"
                   }
                ]
             }
          }
       }
    }
    wsout-YDDEzn2h9Y9fJERB
    wsout-CYzpWK2uMDGFn9Yr
    wsout-RroeVzLsv6PxJ56A
    wsout-8ts3sHe7NsE46mN6
    declare -a tableau=([0]=$'wsout-YDDEzn2h9Y9fJERB\r' [1]=$'wsout-CYzpWK2uMDGFn9Yr\r' [2]=$'wsout-RroeVzLsv6PxJ56A\r' [3]="wsout-8ts3sHe7NsE46mN6")
    Élément du tableau : wsout-YDDEzn2h9Y9fJERB
    URL : https://app.terraform.io/api/v2/state-version-outputs/wsout-YDDEzn2h9Y9fJERB

    Élément du tableau : wsout-CYzpWK2uMDGFn9Yr
    URL : https://app.terraform.io/api/v2/state-version-outputs/wsout-CYzpWK2uMDGFn9Yr

    Élément du tableau : wsout-RroeVzLsv6PxJ56A
    URL : https://app.terraform.io/api/v2/state-version-outputs/wsout-RroeVzLsv6PxJ56A

    Élément du tableau : wsout-8ts3sHe7NsE46mN6
    URL : https://app.terraform.io/api/v2/state-version-outputs/wsout-8ts3sHe7NsE46mN6
    {"data":{"id":"wsout-8ts3sHe7NsE46mN6","type":"state-version-outputs","attributes":{"name":"db_username","sensitive":false,"type":"string","value":"diagngrowdbuser","detailed-type":"string"},"links":{"self":"/api/v2/state-version-outputs/wsout-8ts3sHe7NsE46mN6"}}}

Thank you!

EDITED to provide a minimum reproductible version, with the results I get from execution

Solution

From this line of output in your question:

declare -a tableau=([0]=$'wsout-YDDEzn2h9Y9fJERB\r' [1]=$'wsout-CYzpWK2uMDGFn9Yr\r' [2]=$'wsout-RroeVzLsv6PxJ56A\r' [3]="wsout-8ts3sHe7NsE46mN6")

we can see that all of your IDs except the last one contain Carriage Returns (represented by \r), get rid of those. That's why only the last ID works and why when you hand-coded a list of IDs it also worked.

I can't use your URLs as they require the TOKEN in your code to be populated so here's the difference using example.com.

With a trailing CR:

$ url=$'https://www.example.com/index.html\r'
$ curl -s "$url"

we get no output while without a trailing CR:

$ url=$'https://www.example.com/index.html'
$ curl -s "$url" | head -3
<!doctype html>
<html>
<head>

we get the expected output.

I can't tell where the \r is coming from but I suspect it was added by whatever editor you used to create your code or, more likely, whatever created the file stateVersion.json you mentioned in a previous iteration of your question. See Why does my tool output overwrite itself and how do I fix it? for more information on CRs and how to remove them.

Answered By - Ed Morton

Answer Checked By - Willingham (WPSolving Volunteer)