Python is such a powerful language that “Python scripts” actually deserve to be called “Python programs” or “Python applications”, but nevertheless, let’s go with this title for todays’s blog posting: “Running a sub process from a Python script”.
Currently I am writing test scripts for one of our storage products and thus apparently I have the need to invoke commands from my Python script, capture the output and look at the return code.
There are various ways in Python how to do this. One way is popen which allows to run an external command in a sub process and return its output in form of a file like stream, so that it can be read and analyzed any further ( see this easy example ). Nevertheless, there are many popens available for Python, from the standard popen provided by the os module to several variations in other modules like popen2 and subprocess, called popen, popen2, popen3, popen4. A bit confusing for a Python newbie, especially since I had come up with some specific requirements:
- I wanted to capture stderr as well – the file stream for error messages – and actually treat it like the standard output stream so that in case of an error the error messages are displayed where usually the output is displayed. And depending on my future test scenarios I probbaly have the need to analyze the error message stream as well.
- I need to capture the return code from the command executed in the sub process.
After working on this for 2 to 3 hours and exploring all the possibilities I got to the point where I thought I either can have 1 or 2, but not both. Until I figured out this solution based on the popen2 module:
1: import popen2
2: # ...
3: cmd = "ping bla" # No, I am not using foo here 😉
4: f = popen2.Popen4(cmd)
5: while True:
6: try: line = f.fromchild.readline()
7: except IOError: break
8: if not line: break
9: # ...
10: rc = f.poll()
Popen4 from the popen2 module by default combines stderr and stdout into one stream, thus no need to handle both streams. This implementation actually seemed to work nicely, except … I got a “depreciation” warning about the popen2 module when importing it saying basically that this module will go away in the future and will be replaced by the subprocess module.
Thus I had to continue my research and find a working solution based on the subprocess module. Here it is:
1: import subprocess
2: # ...
3: cmd = "ping blabla" # Still not using foo, but never mind 😉
4: f = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
5: while True:
6: line = f.stderr.readline()
7: if not line: break
8: m = re.match("shell-init",line) # Ignore shell-init errors
9: if not m:
10: # ...
11: while True:
12: line = f.stdout.readline()
13: if not line: break
14: # ...
15: rc = f.poll()
Here I get two message streams, one for stdout and one for stderr to be handled independently. One hurdle is that I get strange “shell-init” errors always in my stderr stream, wheter or not the command itself finishs successfully. The other observation to make: the return code obtained by using this implementation is different than the one I get from the first implementation. If the command is okay – in my case I ping an existing network address, I get 0 in both cases. If I ping some non-existing network address ( like “foo”, “bla” or “blabla” ) I get a return code 512 with the first code snippet and a return code of 2 with the second alternative.
Even I have a solution now working quiet well I guess there is still more to explore for me.