[SOLVED] How to share a variable between scripts that run in parallel?

Issue

I would like to run two bash scripts simultaneously executed by a parent script that exports a variable with read and write privileges between all these scripts. Let's say I have 3 scripts: parent.sh, children1.sh, children2.sh. From parent.sh I am exporting a SOME_VAR variable and executing ./children1.sh & children2sh simultaneously. Then in children1.sh and children2.sh I am performing some operation. In addition, we can see children2.sh is overriding SOME_VAR variable with value 20. I would like to have this change to be reflected in parent.sh. So, for example, let's say I have such scripts:

parent.sh

#!/bin/bash
SOME_VAR=100
export SOME_VAR
bash ./children1.sh & ./children2.sh
echo "New SOME_VAR value: $SOME_VAR"

children1.sh

#!/bin/bash
for i in {1..3}
do
   echo "### FIRST PROCESS  $i"
done

children2.sh

#!/bin/bash
for i in {1..3}
do
   echo "### SECOND PROCESS  $i"
done
SOME_VAR=20

Now if I execute bash ./parent.sh I get such output:

### SECOND PROCESS  1 
### FIRST PROCESS  1 
### SECOND PROCESS  2 
### FIRST PROCESS  2 
### SECOND PROCESS  3 
### FIRST PROCESS  3 
New SOME_VAR value: 100

Unfortunately, it looks like that children2.sh operation was not reflected back to parent.sh and SOME_VAR variable is still equal to 100 instead of 20. I came up with the idea that each script can create a file and write its output to this file. Later, these files can be read by the parent script after the children's scripts finish their work. This solution sounds a bit hacky to me. I am not a bash expert, so maybe someone can help me with a more sufficient solution.

There is one requirement: You can NOT use any 3rd party libraries/tools like for example, GNU Parallel

Solution

Given what you said in the comments below about not really needing all of your scripts to share a variable but instead just needing children running in the background to be able to each return a value to the parent, something like this might be best:

$ cat parent.sh
#!/usr/bin/env bash

foo=1000
bar=999

run_children() {
    local -A children
    local -a pids
    local var

    children[foo]=./child1.sh
    children[bar]=./child2.sh

    for var in "${!children[@]}"; do
        printf '%s\n%s\0' "$var" "$( "${children[$var]}" )" &
        pids+=( $! )
    done

    wait "${pids[@]}"
}

while read -r -d '' var val; do
    printf -v "$var" '%s' "$val"
done < <( run_children )

printf 'foo=%s\n' "$foo"
printf 'bar=%s\n' "$bar"

$ cat child1.sh
#!/usr/bin/env bash

for i in {1..3}
do
   echo "### FIRST PROCESS  $i" >&2
   sleep 2
done

printf "now is the winter\nof our discontent\n"

$ cat child2.sh
#!/usr/bin/env bash

for i in {1..3}
do
   echo "### SECOND PROCESS  $i" >&2
   sleep 1
done

printf 'wee, sleekit, cowrin\ntimrous beastie\n'

$ ./parent.sh
### FIRST PROCESS  1
### SECOND PROCESS  1
### SECOND PROCESS  2
### FIRST PROCESS  2
### SECOND PROCESS  3
### FIRST PROCESS  3
foo=now is the winter
of our discontent
bar=wee, sleekit, cowrin
timrous beastie

I had the children output multi-line string values just to show the approach works for that case as well as the desired single-number output case.

I gave child1 a longer sleep interval between iterations than child2 to show that the script waits for all processes to end before reading their output and isn't affected by the order they run in.

Note in all of the above that the children scripts know nothing about the parent variables, thereby maintaining loose coupling between them. If any of the children need to use the value of foo or bar as initially set in the parent then, unless there's some reason we don't know about yet, I'd recommend passing the value as an argument to keep that loose coupling rather than exporting it in the parent and then referencing the variable in a child as that would undesirably and unnecessarily create tight coupling.

---

Original Answer:

A child process cannot change the value of a variable in a parent process. export lets the child process see the value of the variable and change it within itself, not modify it's value back in the parent.

You can do:

var=$( child )

to set the value in the parent to a string the child prints, or:

child
var=$?

to set the value in the parent to an integer exit status from the child, or

. child

to make the child run in the same process and so it's variables and the parents variables co-mingle but none of those would let you share a variable between child processes running in the background - for that you need a file:

child1:
    echo 10 > "$TMP"

child2:
    echo 20 > "$TMP"

parent:
    export TMP=$(mktemp)
    echo 100 > "$TMP"
    child1 &
    child2 &
    sleep 5    # or "wait" or similar
    IFS= read -r val < "$TMP"
    echo "$val"

but then you probably should introduce some locking mechanism around the file (see @CharlesDuffy's answer at is-writing-to-a-unix-file-through-shell-script-is-synchronized and the flock man page) in case different processes are trying to read/write it simultaneously if whatever you're doing with it isn't atomic.

flock is available on most systems but it's not POSIX so if it's not available to you due to your requirement that "You can NOT use any 3rd party libraries/tools" then obviously you'd have to come up with some other way to do that part.

Answered By - Ed Morton

Answer Checked By - Clifford M. (WPSolving Volunteer)