Issue
This is a continuation to my previous question below, just checking whether I will be able to process this model in the same time
Reduce processing time for 'While read' loop
I have a huge csv file, with a varying length field 11, like
"xx","x",x,x,x,xx,xx,"x",x,11,"00000aaaaD00000bbbbD00000abcdD00000dwasD00000dedsD00000ddfgD00000dsdfD00000snfjD00000djffD00000wedfD00000asdfZ"
"xx","x",x,x,x,xx,xx,"x",x,5,"00000aaaaD00000bbbbD00000abcdD00000dwasD00000dedsD"
After splitting the field 11 in size of 10, I need 6-9 characters. then I have to insert it as new rows I need the output as below,
"xx","x",x,x,x,xx,xx,"x",x,11,"aaaa"
"xx","x",x,x,x,xx,xx,"x",x,11,"bbbb"
"xx","x",x,x,x,xx,xx,"x",x,11,"abcd"
.
.
.
"xx","x",x,x,x,xx,xx,"x",x,11,"asdf"
"xx","x",x,x,x,xx,xx,"x",x,5,"djff"
.
.
"xx","x",x,x,x,xx,xx,"x",x,5,"deds"
while read -r line1; do
icount=$[icount+1]
col_11=$( echo $line1 | cut -d',' -f11 )
col_10=$( echo $line1 | cut -d',' -f1,2,3,4,5,7,10)
#echo $col_11
col_11_trim=$(echo "$col_11" | tr -d '"')
#echo $col_11_trim
echo $col_11_trim | fold -w10 > $path/col_11_extract
while read -r line2; do
ocount=$[ocount+1]
strng_cut=$(echo $line2 | cut -c6-9)
echo "$col_10",\""$strng_cut"\" >> $path/final_out
done < $path/col_11_extract
done < $input
Solution
With awk
:
awk 'BEGIN{FS=OFS=","}
{
eleven=$11;
len=length(eleven);
for(i=2; i<len-1; i=i+10){
$11="\"" substr(eleven, i+5, 4) "\"";
print;
}
}' file
for
loop starts at position 2
and ends with len-1
because of quotes in field 11.
Output:
"xx","x",x,x,x,xx,xx,"x",x,11,"aaaa" "xx","x",x,x,x,xx,xx,"x",x,11,"bbbb" "xx","x",x,x,x,xx,xx,"x",x,11,"abcd" "xx","x",x,x,x,xx,xx,"x",x,11,"dwas" "xx","x",x,x,x,xx,xx,"x",x,11,"deds" "xx","x",x,x,x,xx,xx,"x",x,11,"ddfg" "xx","x",x,x,x,xx,xx,"x",x,11,"dsdf" "xx","x",x,x,x,xx,xx,"x",x,11,"snfj" "xx","x",x,x,x,xx,xx,"x",x,11,"djff" "xx","x",x,x,x,xx,xx,"x",x,11,"wedf" "xx","x",x,x,x,xx,xx,"x",x,11,"asdf" "xx","x",x,x,x,xx,xx,"x",x,5,"aaaa" "xx","x",x,x,x,xx,xx,"x",x,5,"bbbb" "xx","x",x,x,x,xx,xx,"x",x,5,"abcd" "xx","x",x,x,x,xx,xx,"x",x,5,"dwas" "xx","x",x,x,x,xx,xx,"x",x,5,"deds"
Answered By - Cyrus