Thursday, October 27, 2022

[SOLVED] jq with multiple inputs from different sources

Issue

How can we mix different input sources when using jq ? For a specific usecase, I'd like to add some data from a file into a feed that was pipe in stdout.

$ echo '[{"a": 1}]' > /tmp/a1
$ echo '[{"a": 2}]' > /tmp/a2
$ jq --slurp '.[0] + .[1]' /tmp/a1 /tmp/a2
[
  {
    "a": 1
  },
  {
    "a": 2
  }
]
$ cat /tmp/a1 | jq --slurp '.[0] + .[1]' /tmp/a2  # Expecting the same result
[
  {
    "a": 2
  }
]

As you can see, the last command didn't interpret the piped data.

Right now, I'm forced to save the output from the first operation into a temporary file, so that I can do the jq merging operation, before sending it back to the network. Having a single stream would be much more efficient


Solution

I'd like to add some data from a file into a feed that was pipe in stdout.

There are various ways to do this, depending on the shell and also the version of jq you are using.

Assuming your jq supports the --argfile option, you might find that quite congenial:

cat /tmp/a1 | jq --argfile a2 /tmp/a2 '. + $a2'

Here is another variation that suggests some of the other possibilities:

jq -n --argfile a1 <(cat /tmp/a1) --argfile a2 <(cat /tmp/a2) '$a1 + $a2'

More interestingly:

(cat /tmp/a1 ; cat /tmp/a2) | jq '. + input'

And finally an approach that should work for every version of jq:

jq -s '.[0] + .[1]' <(cat /tmp/a1) /tmp/a2

In general, though, it's best to avoid the -s option.

A note on slurping

If you compare the outputs produced by:

echo '1 2' |
  jq -s --debug-dump-disasm --debug-trace  '.[0], .[1]'

and

echo '1 2' |
  jq --debug-dump-disasm --debug-trace  '., input'

you'll notice the former has to PUSHK_UNDER to store the entire array [1,2], whereas the latter program just reads the two inputs separately.

In the first program, the memory for the array cannot be freed until after all the pointers into it have been processed, whereas in the second program, the memory for . can be freed after the first RET.



Answered By - peak
Answer Checked By - Terry (WPSolving Volunteer)
at