[SOLVED] How to start multiple job instance in a shell script to process multiple files in a directory?

Issue

#!/bin/bash

      data_dir=./all
      for file_name in "$data_dir"/*
      do
        echo "$file_name"
        python process.py "$file_name"
      done
   

For example, this script processes the files sequentially in a directory in a 'for' loop. Is it possible to start multiple process.py instances to process files concurrently? I want to do this in a shell script.

Solution

I have another possibility for you, if still needed. It uses the screen command to create a new detached process with the supplied command.

Here is an example:

#!/bin/bash

data_dir=./all
for file_name in "$data_dir"/*
do
  echo "$file_name"
  screen -dm python process.py "$file_name"
done

Answered By - cfgn

Answer Checked By - Marie Seifert (WPSolving Admin)