[SOLVED] gnu parallel single job splitted in multi-threads

Issue

I want to launch a single job, that takes in input two parameters, and split its execution between 128 thread of the CPU with GNUparallel. I explain what I did and how I did it, and ask for help.

I have a bash script "job.sh" where I define two variables and launch a job

#!/bin/bash

first=70
second=10

./executable $((first-1)) $second

echo "$first $second"

now I want to split this intense job in 128 threads of the CPU of the server I'm woring on, using GNU parallel.

After reading the documentation, I tried to substitute the row ./executable $((first-1)) $second with parallel -j+128 ./executable $((first-1)) $second, the I tried to run it as follows:

user@server$ nohup bash job.sh&

the job fails. Anyone knows how to run an executable with GNU parallel?

Solution

I want to launch a single job, that takes in input two parameters, and split its execution between 128 thread of the CPU with GNUparallel.

This cannot be done. GNU Parallel launches jobs. It does not magically make your single threaded job into a multithreaded job.

In general GNU Parallel is useful if you can express your jobs as:

./executable A B
./executable C D
./executable E F
./executable G H
./executable I J
:
./executable Y Z

So if you can explain a way to express your jobs this way, GNU Parallel can help you run these in parallel.

Maybe something like this:

seq 128 | parallel --dry-run ./executable {} '{= $_++ =}'

(Remove --dry-run to actually run it).

I think you have gotten a wrong impression of what GNU Parallel is capable of. Consider spending 20 minutes reading chapter 1+2: https://doi.org/10.5281/zenodo.1146014

Answered By - Ole Tange

Answer Checked By - Clifford M. (WPSolving Volunteer)