[SOLVED] Python subprocess communicate freezes when reading output

Issue

I'm using Gphoto2 to take pictures on a DSLR. As its based on bash commands I tried to use subprocess.communicate but it freezes after the camera takes a picture.

If I try the gphoto2 --capture-image-and-download in the terminal it takes less than 2 seconds. I'm working on a Raspberry Pi.

Code:

import subprocess

class Wrapper(object):

    def __init__(self, subprocess):
        self._subprocess = subprocess

    def call(self,cmd):
        p = self._subprocess.Popen(cmd, shell=True, stdout=self._subprocess.PIPE, stderr=self._subprocess.PIPE)
        out, err = p.communicate()
        return p.returncode, out.rstrip(), err.rstrip()


class Gphoto(Wrapper):
    def __init__(self, subprocess):
        Wrapper.__init__(self,subprocess)
        self._CMD = 'gphoto2'

    def captureImageAndDownload(self):
        code, out, err = self.call(self._CMD + " --capture-image-and-download")
        if code != 0:
            raise Exception(err)
        filename = None
        for line in out.split('\n'):
            if line.startswith('Saving file as '):
                filename = line.split('Saving file as ')[1]
        return filename


def main():
    camera = Gphoto(subprocess)

    filename = camera.captureImageAndDownload()
    print(filname)

if __name__ == "__main__":
    main()

If I exit I get this:

Traceback (most recent call last):
  File "test.py", line 39, in <module>
   main()
  File "test.py", line 35, in main
    filename = camera.captureImageAndDownload()
  File "test.py", line 22, in captureImageAndDownload
    code, out, err = self.call(self._CMD + " --capture-image-and-download")
  File "test.py", line 11, in call
    out, err = p.communicate()
  File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 1409, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

Any ideas?

Solution

Based on the comments above, here's what we came up with. The .communicate() call hung the program, and much to my suspicion it was because the executed command didn't exit properly.

One thing you can use to get around this is by manually polling the process if it's finished and print the output as you go along.
Now the gist above was written on the phone so it didn't utelize this properly, but here's an example code you can use to catch the output as you go and manually poll the command.

import subprocess
from time import time
class Wrapper():
    def call(self, cmd):
        p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        O = ''
        E = ''
        last = time()
        while p.poll() is None:
            if time() - last > 5:
                print('Process is still running')
                last = time()
            tmp = p.stdout.read(1)
            if tmp:
                O += tmp
            tmp = p.stderr.read(1)
            if tmp:
                E += tmp
        ret = p.poll(), O+p.stdout.read(), E+p.stderr.read() # Catch remaining output
        p.stdout.close() # Always close your file handles, or your OS might be pissed
        p.stderr.close()
        return ret

Three important things to notice, using shell=True might be bad, unsafe and tricky.
I personally favor it because I rarely handle user input or "unknown variables" when I execute stuff. But a few words of caution - Never use it!

Second being, if you don't need to separate error and regular output, you can also do:

Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

It will give you one less file handle to worry about.

Finally thing, ALWAYS empty the stdout/stderr buffers, and always close them. These two things are important.
If you don't empty them, they might themselves hang the application because they're full and Popen can't put more data in them, so it will wait for you (in best case scenario) to empty them.
Second being not closing those file handles, that might render your OS to run out of possible file handles to open (there's only a certain ammount of collective file handles a OS can have open at any given time, so not closing them could render your OS useless for a bit).

(note: Depending on if you're using Python2 or 3, p.stdout.read() might give you bytes data back, meaning O = '' should be O = b'' instead etc)

Answered By - Torxed

Answer Checked By - Katrina (WPSolving Volunteer)