Thursday, October 6, 2022

[SOLVED] nvidia-smi monitor only while a specific process is running

Issue

In bash, nvidia-smi command gives you information about the GPU.
We also have option to get this periodically such as nvidia-smi -lms 50

I want to get this info only as long as a particular process is running.

Pseudocode

nvidia-smi -lms 50 & > logfile.txt
(time ./process1) > timelog.txt
while process1 is running:
     keep nvidia-smi running
kill nvidia-smi

How can I do this in bash, cleanly, such that once my bash script exits no process that starts here is left behind for me to clean? A direct nvidia-smi based solution would be preferred to a bash based one, but the latter is also perfectly fine.


Solution

Run both in the background, then wait for the one your job depends on.

nvidia-smi -lms 50 > logfile.txt &
nvpid=$!
time ./process1 > timelog.txt &
prpid=$!
wait "$prpid"
kill "$nvpid"


Answered By - tripleee
Answer Checked By - Gilberto Lyons (WPSolving Admin)
at