[SOLVED] Bash losing string escapes when piping

Issue

I have a program a that produces a stream of JSON documents where some of the fields contain strings with escaped double-quotes. This JSON is then piped into a Bash script b which invokes jq to do some filtering basically as follows:

while read j; do
  echo $j | jq 'select(.my_field | match("..."))' | ...
done

jq raises errors (e.g. "Invalid numeric literal") because it cannot parse those strings. This is because the escape \ apparently gets lost between a and the invocation of jq in b: where a sees "\"" (with escape), b sees only """ (without escape). Is there a way to preserve the escapes all the way into jq so that it can do its parsing? My current guess is that the culprit is one of the two pipes (|), but it could also be read.

Solution

By default, read lets you use a backslash as a line-continuation operator:

$ read foo
x \
> y
$ declare -p foo
declare -- foo="x y"

(The documentation describes this as the backslash character escaping the next character, but I'm not aware of any other context where it matters.)

As such, it discards any backslash found in the input. Using -r disables this behavior, preserving any backslashes found in the input.

while read -r j; do
    echo "$j" | jq '...'
done

Answered By - chepner

Answer Checked By - Senaida (WPSolving Volunteer)