Issue
I recently found a post about how to clone all repos from an org In GitHub. The top answer was the following:
gh repo list myorgname --limit 4000 | while read -r repo _; do
gh repo clone "$repo" "$repo"
done
I understand everything before and after the underscore. What is the meaning of _; in the above command?
Thanks!
Solution
Variable _
is used as a garbage
From man bash
(or man -P'less +"/^ *Shell Variables"' bash
):
Shell Variables The following variables are set by the shell: _ At shell startup, set to the pathname used to invoke the shell or shell script being executed as passed in the environment or argument list. Subsequently, expands to the last argument to the previous simple command executed in the foreground, after expansion. Also set to the full pathname used to invoke each command executed and placed in the environment exported to that command. When checking mail, this parameter holds the name of the mail file currently being checked.
This variable will hold the last argument of previous line.
Little useful case
I ofen use this:
mkdir -p /some/path/somewhere
cd $_
Unusable in other way
If you try to use them as a variable, you may become disapointed
_='Hello world!'
echo "I said: '$_'"
I said: ''
Then (:
in an alias for true
)
: "Not this"
echo "I said: '$_'"
I said: 'Not this'
But if you repeat last command
echo "I said: '$_'"
I said: 'I said: 'Not this''
Used as a trashcan
But in your sample, they are used as a garbage:
So while read -r repo _;...
will parse each line of input by storing first (space separated) word into $repo
and the rest of line into (unusable) $_
.
Sample,showing your GECOS field from your password file:
while IFS=: read -r user _ uid _ gecos ;do
(( UID == uid )) && echo $user ${gecos%%,*}
done < /etc/passwd
About your sample using gh repo list
I prefer to write this in the other way, this will permit to use overall variable in order to create stats and other reusable things:
declare -i cloned=0 total=0
while read -r repo _; do
total+=1
gh repo clone "$repo" "$repo" &&
cloned+=1
done < <(
gh repo list myorgname --limit 4000
)
printf 'There was %d repo successfully cloned, over %d total\n' \
$cloned $total
You may found more sample at How do I set a variable to the output of a command
But as TIMTOWTDI
You could read your gh repo list
into an array before processing elements:
mapfile -t rList < <(gh repo list myorgname --limit 4000)
declare -i cloned=0 total=${#rList[@]}
printf 'There are %d objects to clone.\n' $total
for repo in "${rList[@]/%$'\11'*}"; do
gh repo clone "$repo" "$repo" &&
cloned+=1
done
printf 'There was %d repo successfully cloned, over %d total\n' \
$cloned $total
For sample, you could plan to download simultaneously upto 5 process at a time:
Parallel gh
clone full demo script.
(Copied from Speed up rsync with Simultaneous/Concurrent File Transfers?.)
Save this as gh_parClone.sh
and give him execution right:
#!/bin/bash
maxProc=5
[[ $1 == -v ]] && verbose=1 && shift || verbose=0
orgName="$1"
declare -ai start elap results order
declare -a succeeded failed messages
wait4oneTask() {
printf 'Running task: %d\r' ${#running[@]}
wait -np epid
results[epid]=$?
local _i _line
elap[epid]=" ${EPOCHREALTIME/.} - ${start[epid]} "
if ((results[epid])); then
failed[epid]="${repos[epid]}"
else
succeeded[epid]="${repos[epid]}"
fi
while read -ru "${ghFds[epid]}" -t .02 _line; do
messages[epid]+="$_line"$'\n'
done
exec {ghFds[epid]}<&-
unset "ghFds[epid]"
unset "running[$epid]"
while [[ -v elap[${order[0]}] ]]; do
_i=${order[0]}
printf " - %(%a %d %T)T.%06.0f %-36s %4d %12d\n" "${start[_i]:0:-6}" \
"${start[_i]: -6}" "${repos[_i]}" "${results[_i]}" "${elap[_i]}"
order=("${order[@]:1}")
done
}
mapfile -t rList < <(LANG=C gh repo list "$orgName")
printf 'GH: %d object in \47%s\47. Start clone with max %d instances.\n' \
${#rList[@]} "$orgName" $maxProc
printf " %-22s %-36s %4s %12s\n" Started Repo Rslt 'microseconds'
for repo in "${rList[@]/%$'\11'*}"; do
exec {ghFd}<> <(:)
gh repo clone "$repo"{,} >&$ghFd 2>&1 &
lpid=$!
ghFds[lpid]=$ghFd
repos[lpid]="${repo#$orgName/}"
start[lpid]=${EPOCHREALTIME/.}
running[lpid]=''
order+=("$lpid")
((${#running[@]}>=maxProc)) && wait4oneTask
done
while ((${#running[@]})); do
wait4oneTask
done
printf 'Succeeded: %d.\n%s\n' ${#succeeded[@]} "${succeeded[*]}"
if ((verbose)); then
for pid in "${!succeeded[@]}"; do
printf ' - %s\n' "${succeeded[pid]}"
printf ' %s\n' "${messages[pid]//$'\n'/$'\n' }"
done
fi
printf 'Failed: %d.\n' ${#failed[@]}
if ((${#failed[@]})); then
for pid in "${!failed[@]}"; do
printf ' - %s\n' "${failed[pid]}"
printf ' %s\n' "${messages[pid]//$'\n'/$'\n' }"
done
fi
Sample run:
$ mkdir -p demos/mixxx/foo
$ ./gh_parClone.sh demos
GH: 9 object in 'demos'. Start clone with max 5 instances.
Started Repo Rslt microseconds
- Fri 12 10:41:59.476735 php-logical-filter 0 2100719
- Fri 12 10:41:59.477372 mixxx 1 403229
- Fri 12 10:41:59.477718 Cache 0 2062995
- Fri 12 10:41:59.478078 developer_skins 0 3713021
- Fri 12 10:41:59.478470 Qt-Inspector 0 1949781
- Fri 12 10:41:59.901984 pytaglib 0 2325354
- Fri 12 10:42:01.449372 gmap3 0 2269148
- Fri 12 10:42:01.561883 Motif 0 4105144
- Fri 12 10:42:01.600549 offlineimap 0 4493532
Succeeded: 8.
php-logical-filter Cache developer_skins Qt-Inspector pytaglib gmap3 Motif offlineimap
Failed: 1.
- mixxx
fatal: destination path 'demos/mixxx' already exists and is not an empty directory.
failed to run git: exit status 128
Longer output if run with -v
flag:
$ ./gh_parClone.sh -v demos
Answered By - F. Hauri - Give Up GitHub Answer Checked By - David Marino (WPSolving Volunteer)