[SOLVED] Shell script with grep and sed to extract individuals from a pair after comparing the numerical values of a variable

Issue

I want to compare a group of words (individuals) in pairs and extract the one with the lowest numeric variable. My files and scripts are made this way.

Relatedness_3rdDegree.txt (example):

Individual1 Individual2
Individual5 Individual23
Individual50 Individual65

filename.imiss

INDV    N_DATA  N_GENOTYPES_FILTERED    N_MISS  F_MISS
Individual1 375029  0   782 0.00208517
Individual2 375029  0   341 0.000909263
Individual3 375029  0   341 0.000909263

Main script:

numlines=$(wc -l Relatedness_3rdDegree.txt|awk '{print $1}')

for line in `seq 1 $numlines`
do
ind1=$(sed -n "${line}p" Relatedness_3rdDegree.txt|awk '{print $1}')
ind2=$(sed -n "${line}p" Relatedness_3rdDegree.txt|awk '{print $2}')
miss1=$(grep $ind1 filename.imiss|awk '{print $5}')
miss2=$(grep $ind2 filename.imiss|awk '{print $5}')
if echo "$miss1 > $miss2" | bc -l | grep -q 1
then
echo $ind1 >> miss.txt
else
echo $ind2 >> miss.txt
fi
echo "$line / $numlines"
done

This last script will echo a series of line like this :

1 / 208
2 / 208
3 / 208

and so on, until getting to this error:

91 / 208
(standard_in) 1: syntax error
92 / 208
(standard_in) 1: syntax error
93 / 208

If I go to my output (miss.txt), the printed individuals are not correct. It should print the individuals, within the pairs contained in the file "Relatedness_3rdDegree.txt", that have the lowest value of F_MISS (column $5 of the "filename.imiss").

For instance, in the pair "Individual1 Individual2", it should compare their values of F_MISS and print only the individual with the lowest value, which in this example would be Individual 2.

I have manually checked the values and the printed individual, and it looks like it printed random individuals per each pair.

What is wrong in this script?

Solution

Bash version:

#!/bin/bash
declare -A imiss
while read -r ind nd ngf nm fm     # we'll ignore most of these
do 
    imiss[$ind]=$fm
done < filename.imiss

while read -r i1 i2
do
    if (( $(echo "${imiss[$i1]} > ${imiss[$i2]}" | bc -l) ))
    then 
        echo "$i1"
    else
        echo "$i2"
    fi
done < Relatedness_3rdDegree.txt

Run* it like:

bash-imiss

AWK version:

#!/usr/bin/awk -f
NR == FNR {imiss[$1] = $5; next}

{
    if (imiss[$1] > imiss[$2]) {
        print $1
    } else {
        print $2
    }
}

Run* it like:

awk-imiss filename.imiss Relatedness_3rdDegree.txt 

These two scripts do exactly the same thing in exactly the same way using associative arrays.

* This assumes that you have set the script file executable using chmod and that it's in your PATH and that the data files are in your current directory.

Answered By - Dennis Williamson

Answer Checked By - Dawn Plyler (WPSolving Volunteer)