[SOLVED] Parallelize curl calls in a bash script to check on API result

Issue

I am trying to measure the time until a particular data piece arrived at all nodes in a p2p network.

This is a bash script. I have several servers (boxes), and on each, some number of nodes is running.

My goal is to measure the time until all nodes have responded through their API that they got the piece of data.

This is in a bash script. I am not aware to what extent I can parallelize calls in a bash script. Because this now is iterating first through each box, then issuing a curl call to each of that box's nodes, and then moving on to the next box. I feel that might skew my results.

Is there a better way to do this?

   found=0
   while :
   do  
     echo "checking data is at node..."
     for i in "${BOXES[@]}"
     do
       for (( j=0; j < $NODE_PER_BOX; j++ )) 
       do
         res=$(curl http://ip:port/api)
         echo $res 
         # HERE I NEED TO CHECK FOR THE CONDITION, NOT DONE YET 
         # I know the exit condition is never met here yet
       done
     done                                               
  done

Solution

You can create processes by placing a command or block of code in the background with & and then wait for them to finish.

Example:

time { # measure the time it takes to finish all lookups in parallel
    for box in "${BOXES[@]}"; do
        for (( j=0; j < NODE_PER_BOX; j++ )); do
        {
            res=$(curl -s http://ip:port/api)
            # HERE I NEED TO CHECK FOR THE CONDITION, NOT DONE YET

            # Write the result of the condition check to a unique file per query:
            echo "$res" > "${box}_${j}.res"
        } &  # & puts the job in a background process
        done
    done

    wait # wait until all jobs are done
}        # end of time measuring block

# read all the result files here

Answered By - Ted Lyngmo

Answer Checked By - Willingham (WPSolving Volunteer)