Wednesday, October 27, 2021

[SOLVED] shell script parsing issue: unexpected INVALID_CHARACTER (Windows cmd shell quoting issues?)

Issue

What I am doing? I have one JSON file as sonar-report.json. I want to iterate sonar-report.json in shell script, to read values of json. To parse JSON file I am using jq https://stedolan.github.io/jq/

So Following code I was trying to execute in shell script

alias jq=./jq-win64.exe
for key in $(jq '.issues | keys | .[]' sonar-report.json); do
    echo "$key"
    line=$(jq -r ".issues[$key].line" sonar-report.json)
done

Problem When i execute this, console give me error:

jq: error: syntax error, unexpected INVALID_CHARACTER (Windows cmd shell quoting issues?) at <top-level>, line 1:

If I update my above script, and add static index of array then script works fine

alias jq=./jq-win64.exe

for key in $(jq '.issues | keys | .[]' sonar-report.json); do
    echo "$key"
    line0=$(jq -r ".issues[0].line" sonar-report.json)
    line1=$(jq -r ".issues[1].line" sonar-report.json)
done

so at the end what i want : I want to iterate values and print in console like

alias jq=./jq-win64.exe
for key in $(jq '.issues | keys | .[]' sonar-report.json); do
    line=$(jq -r ".issues[$key].line" sonar-report.json)
    echo $line
done

so the output should be 

15

This is my JSON file as sonar-report.json

{
"issues": [
        {
            "key": "016B7970D27939AEBD",
            "component": "bits-and-bytes:src/main/java/com/catalystone/statusreview/handler/StatusReviewDecisionLedHandler.java",
            "line": 15,
            "startLine": 15,
            "startOffset": 12,
            "endLine": 15,
            "endOffset": 14,
            "message": "Use the \"equals\" method if value comparison was intended.",
            "severity": "MAJOR",
            "rule": "squid:S4973",
            "status": "OPEN",
            "isNew": true,
            "creationDate": "2019-06-21T15:19:18+0530"
        },
        {
            "key": "AWtqCc-jtovxS8PJjBiP",
            "component": "bits-and-bytes:src/test/java/com/catalystone/statusreview/service/StatusReviewInitiationSerivceTest.java",
            "message": "Fix failing unit tests on file \"src/test/java/com/catalystone/statusreview/service/StatusReviewInitiationSerivceTest.java\".",
            "severity": "MAJOR",
            "rule": "common-java:FailedUnitTests",
            "status": "OPEN",
            "isNew": false,
            "creationDate": "2019-06-18T15:32:08+0530"
        }
    ]
}

please help me, Thanks in advance


Solution

This looks to me like an instance of Windows/Unix line-ending incompatibility, indicated in jq bugs 92 (for Cygwin) and 1870 (for MSYS2).

Any of the workarounds indicated in those bug reports should work, but once the fix gets into the release binary (presumably v1.7), the simplest solution is to use the new -b command-line option. (The option is available in recent jq preview builds; see the second bug report listed above):

for key in $(jq -b '.issues | keys | .[]' sonar-report.json); do
    line=$(jq -rb ".issues[$key].line" sonar-report.json)
    # I added quotes in the next line, because it's better style.
    echo "$line"
done

Until the next version of jq is available, or if you don't want to upgrade for some reason, a good workaround is to just remove the CRs by piping the output of jq through tr -d '\r':

for key in $(jq -'.issues | keys | .[]' sonar-report.json | tr -d '\r'); do
    line=$(jq -r ".issues[$key].line" sonar-report.json | tr -d '\r')
    echo "$line"
done

However, as pointed out in a comment by Cyrus, you probably don't need to iterate line-by-line in a shell loop, which is incredibly inefficient since it leads to reparsing the entire JSON input many times. You can use jq itself to iterate, with the much simpler:

jq '.issues[].line' solar-response.json

which will parse the JSON file just once, and then produce each .line value in the file. (You probably still want to use the -b command-line option or other workaround, depending on what you intend to do with the output.)



Answered By - rici
at