[SOLVED] Automatic Ubuntu terminal commands get cut

Issue

First of all, I'm not an Ubuntu expert and I have barely used its terminal so there might be a better way to do this.

I'm trying to execute this command over several .fits files:

astmkcatalog ${filename} --ids --ra --dec --magnitude --sn --zeropoint=${zeropoints} --brightness --clumpbrightness --brightnessnoriver --riverave --rivernum --clumpscat

The thing is each file has a specific "zeropoint" value. I have these values saved in a .txt file being each value a row so I want to access to each of this rows every time I run the command. I created this .sh file to do so (the .fits files are ordered so that the first one to pass is the one corresponding to the zeropoint in the first row, so no problem there):

I found these 2 methods to access the row values of the .txt files (the awk and sed above), both of them have the same problem. When I run this only the last .fits works properly.

In order to see the problem I added the commented echo lines in the code and when I uncomment them and comment the actual command I want to run (astmkcatalog...) I get this:

As you can see it only passes the zeropoint value for the last one. Also, when I run the code again this happens:

Here it gets crazy. First the problem was understood, it is not passing the zeropoint values for the first commands but now, when I run the exact same .sh file a second time it disorders the command for some reason.

I'm obviously doing something wrong but I don't know what, I guess the awk and sed methods are stopping the process so I tried to save their result in the zeropoints variable, but it doesn't work.

TL;DR: How can I automatically run almost the same command but with a different value for a variable having this value for each command saved as a row in a .txt file?

I tried different methods (awk and sed) to access the values of specific rows in a .txt file.

I tried to save their result in a variable.

Placing the zeropoint="" input of the command at the end doesn't solve the problem.

SOLUTION:

Thanks to tripleee comments I could find the issue. As they mentioned it was a DOS carriage problem.

In this post Are shell scripts sensitive to encoding and line endings? the issue is explained. Just passing in the terminal to my .txt file:

dos2unix zeropointsfile.txt

This removed the invisible ^M at the end of each line and solved the problem,

THANKS!!

Solution

Reading the input file repeatedly to align each line with a line number is rather inelegant and inefficient. See also Counting lines or enumerating line numbers so I can loop over them - why is this an anti-pattern?

Instead, I would refactor the Awk command to output pairs of values, and then simply loop over those.

awk 'NR==FNR { z[NR] = $1; next }
  FNR==1 { print z[++n], FILENAME; nextfile }' zeropoints.txt /path/*.fits |
while read -r zp file; do
    echo \
    astmkcatalog "$file" --ids --ra --dec --magnitude --sn \
      --zeropoint="$zp" --brightness --clumpbrightness \
      --brightnessnoriver --riverave --rivernum --clumpscat
done

(I left in an echo so it doesn't really do anything, just print the command it would execute. If it looks correct, take out the echo line.)

I don't believe this will solve your problem, as such; but at least it should hopefully steer you in the right direction.

If the input file contains DOS line endings, run dos2unix on it, or add sub(/\r/, ""); to the Awk script.

Answered By - tripleee

Answer Checked By - Clifford M. (WPSolving Volunteer)