Issue
I'm piping a large file (~5M lines) into xargs that I'm processing in batches using -L to limit to max-lines per command. The command I'm calling with xargs requires not only the arguments (one argument per line in the input file), but additionally a count of the number of arguments I am passing it as the first argument.
e.g. I have a file, numbers.txt containing the English words for the first 17 numbers, one per line.
cat numbers.txt | xargs -n 3 echo prints the following:
one two three
four five six
seven eight nine
ten eleven twelve
thirteen fourteen fifteen
sixteen seventeen
As expected echo is called with 3 arguments, the max-lines limit, on each invocation except the last because we only have two lines remaining.
What I want is to call my target command with a count of the arguments passed followed by the actual args.
Extending the preceding example, cat numbers.txt | xargs -n 3 sh -c 'mycommand "$#" "$@"' almost gives what I want. Strangely (at least to me), the first argument is missing from both $@ and $#.
That is, cat numbers.txt | xargs -n 3 sh -c 'echo "$#" "$@"' prints
2 two three
2 five six
2 eight nine
2 eleven twelve
2 fourteen fifteen
1 seventeen
rather than
3 one two three
3 four five six
3 seven eight nine
3 ten eleven twelve
3 thirteen fourteen fifteen
2 sixteen seventeen
which is what I would expect.
Why is this happening? How can I arrive at the expected output? I'm not married to this approach, if there is something simpler I could do please suggest.
Solution
You need a placeholder to fill in $0 so the first argument becomes $1.
<numbers.txt xargs -n 3 sh -c 'echo "$#" "$@"' sh
Answered By - Charles Duffy Answer Checked By - David Marino (WPSolving Volunteer)