Issue
I'm running into strange behavior with xargs in a bash script where the replacement is not working anymore when placing it in the middle of a variable with special characters. Please see the following:
#!/bin/bash
url=https://myhostname.com/endpoint/
parameters="?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY"
echo 1 2 3 | xargs -n1 -I{} echo $url{}$parameters -o {}
results in:
https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 1
https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 2
https://myhostname.com/endpoint/{}?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANr -o 3
Please note the 1 2 and 3 did not show up but was the literal {} in the endpoint.
I determined it was the parameter because when I removed a few characters, it worked:
#!/bin/bash
url=https://myhostname.com/endpoint/
parameters="?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY"
echo 1 2 3 | xargs -n1 -I{} echo $url{}$parameters -o {}
https://myhostname.com/endpoint/1?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 1
https://myhostname.com/endpoint/2?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 2
https://myhostname.com/endpoint/3?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-WyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 3
Note the 1, 2, and 3 is there.
I am on MacOS. This behavior seems to be specific to the xargs on Mac. This will not even run on linux xargs with the flags passed. Would love some suggestions how to make this portable and work on all systems. Thank you.
Solution
From the man page:
-I replstr Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is specified) arguments to utility with the entire line of input. The resulting arguments, after replacement is done, will not be allowed to grow beyond replsize (or 255 if no -S flag is specified) bytes; this is implemented by concatenating as much of the argument containing replstr as possible, to the constructed arguments to utility, up to replsize bytes. The size limit does not apply to arguments to utility which do not contain replstr, and furthermore, no replacement will be done on utility itself. Implies -x.
Your output string is longer that 255 characters, so it doesn't perform the replacement. Add a -S option that's large than the maximum possible output string.
$ echo 1 2 3 | xargs -n1 -I{} -S 500 echo "$url{}$parameters" -o {}
https://myhostname.com/endpoint/1?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 1
https://myhostname.com/endpoint/2?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 2
https://myhostname.com/endpoint/3?Policy=1&KeyPairId=aaaaa&Signature=aa~UhFXhFy-DalIcQDq~Hfd-b57cm~~kkKfV6~ly1lapp0Elu1N84WKHh7Y4sjR2saA9cx2gJF8jTEUVsimQ21zBbyQE46-pPHyePAB1bV9nVfZ6euU14ovWyZazd3FyjGaN7qVSA5ntf8RUfUCo7Asc0Gilf0FJzzlMtI38hSWj3wGC-4B-7ANrA__&Key-Pair-Id=YYY -o 3
Also remember to quote variables to prevent word splitting and globbing.
Answered By - Barmar Answer Checked By - Robin (WPSolving Admin)