[SOLVED] cat and curl large JSON value from gitlab ci

Issue

There are many questions on how to post JSON via curl. And I am able to send one word as json but not the whole text.

My requirement is to comment a merge request when one job fail in our GitLab CI.

# From .gitlab-ci.yaml
  after_script:
    - report=$(find services/ -name '*_latest_failure.md')
    - comment="Job failed.\n\n$(cat "$report")"
    - if [[ "${CI_JOB_STATUS}" == "failed" ]] ; then bash -c "source .gitlab-ci/gitlab.sh; gitlab-comment_merge_request "$comment"" ; fi

# From .gitlab-ci/gitlab.sh
function gitlab-comment_merge_request() {
  url="https://$CI_SERVER_HOST/api/v4/projects/$CI_MERGE_REQUEST_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  comment="$1"
  curl -v --location -X POST "$url" \
    --header "PRIVATE-TOKEN: $GITLAB_CI_TOKEN" \
    --header "Content-Type: application/json" \
    --data '{"body": "'"$comment"'"}'
}

Unfortunately, the comment that is created contains only the first word, Job.

I have tried many techniques to build and send the JSON payload. But all give the same result.

So I thought it could be a mistake when creating the $comment variable in my .gitlab-ci.yaml. However, echo $comment in the same file give the right content.

Now, I'm lost. Do you know why the $comment variable that should contain a large text (a markdown text with title, subtitles, and lists) is not sent ?

Thanks

Solution

A simplified version of this would look like:

nl='
'
comment="Job failed.${nl}${nl}$(find services/ -name '*_latest_failure.md' -exec cat -- {} +)"
if [ "${CI_JOB_STATUS}" = "failed" ] ; then
  . .gitlab-ci/gitlab.sh
  gitlab_comment_merge_request "$comment"
fi

Note:

• Because you were quoting the output from find before passing it to cat, cat was looking for a single file with all the filenames concatenated together as its name. Don't do that. Using find ... -exec cat -- {} + correctly handles names with spaces without creating new bugs.
• We've taken out bash-only syntax: Changed == to =, [[ to [.
• We took out the use of bash -c and the exec boundary between parent and child processes it implied. If we hadn't done this we would need to export comment.
• We changed the source keyword to its POSIX-compliant synonym, .
• We took the dash out of the function name (this does require the file being sourced to change).

A POSIX-compliant version of your function might look like:

# note we had to rename this: dashes are not required to work in function names in sh
gitlab_comment_merge_request() {
  url="https://$CI_SERVER_HOST/api/v4/projects/$CI_MERGE_REQUEST_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  body=$(jq -n --var comment "$1" '{"body":$comment}') 
  curl -v --location -X POST "$url" \
    --header "PRIVATE-TOKEN: $GITLAB_CI_TOKEN" \
    --header "Content-Type: application/json" \
    --data "$body"
}

If you don't have jq installed, it's straightforward to write a shell wrapper for Python's json module for the same purpose.

Answered By - Charles Duffy

Answer Checked By - Gilberto Lyons (WPSolving Admin)