make python examples using_qiskit* and other examples run as a test

[jaygambetta]

What is the expected enhancement?

Since the using_qiskit are to be use case examples we need to make them part of the tests. We would have found out that job.done was used and I'm hoping user did note make examples based on this when we make this release.

[jaygambetta]

wait until #968 is done.

[mtreinish]

Do you mean the examples in examples/python? Or is using_qiskit something else?

If it is what's in examples/python this felt like it would be fairly simple to execute and a good idea. So I wrote a quick PoC (which assumes it's running in test/python) for this:

import os
import subprocess
import sys

import ddt

from . import common

examples_dir = os.path.abspath(os.path.join(
    os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))),
                 'examples'),
    'python'))
examples = [x for x in os.listdir(examples_dir) if x.endswith('.py')]
#print(examples)

@ddt.ddt
class TestPythonExamples(common.QiskitTestCase):

    @ddt.data(*examples)
    def test_example(self, example):
        """Execute the example pythonf file and pass if it returns 0"""
        print(example)
        example_path = os.path.join(examples_dir, example)
        cmd = [sys.executable, example_path]
        run_example = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                                       stderr=subprocess.PIPE)
        stdout, stderr = run_example.communicate()
        error_string = "Running example %s failed with return code %s\n" % (
            example, run_example.returncode)
        error_string += "\tstdout:%s\n\tstderr: %s" % (stdout, stderr)
        self.assertEqual(run_example.returncode, 0, error_string)

The advantage of testing it this way is that whenever we add (or remove) a new example we don't need to update anything on the testing side it will automatically run.

But after testing this it raised a few questions. First, several of the examples there expect to be able to use the IBMQ provider to talk to the api/backends how did we want to handle that? My first thoughts were to make a blacklist and/or tag those examples somehow (maybe a subdir) and skip those based on the run online tests flag. The other issue it raised is that the visualization example is meant to be run interactively. It calls image show to open up the circuit in a new window which holds the process open, so the test doesn't finish until someone actively closes the window. Should we skip that file or just modify the example to not open a new window.The last question I had was do you think just executing the examples in a subprocess and seeing if they work or not sufficient? Or do you think we should we build targeted tests that run each of the examples from inside the same process as the test runner and probe the internals of the examples?

[jaygambetta]

I think this should not be part of Terra but the mega test. My main point is in the last update things were changing and the examples were not. Since these define how different users use qiskit we need to know when they break. They are more like a full integration test.

I just feel we keep talking about it but I want a full integration to have a green light that says tests pass.

[mtreinish]

Sure, where the test lives doesn't actually matter too much, especially if it's like what I pasted before. That would just need the relative path updated to work from anywhere. Although, if we do have it in an external test suite we will have to fix that examples/ in terra isn't actually in the sdist so it may not be present if it's just pip installed (but we can probably solve that by just adding it to the manifest file I think).

[jaygambetta]

I agree. I am not sure what the best thing is to do. I let you decide. the workflow is more how do we make sure that things dont break for Aer<->Terra<->IBMQ and standard user tasks.

[delapuente]

For the record, @diego-plan9 wrote https://github.com/Qiskit/qiskit-tutorial/blob/master/utils/test/test_tutorials.py to help testing the tutorials. Perhaps we should include it as part of the regular tests.

[jaygambetta]

Maybe once we finish separateing the tutorial from ones that teach quantum using qiskit from those that teach qiskit. This is going to take a long time (your welcome to help there) so I would much rather a solution using these examples.

[jaygambetta]

Also for the record I know it exists but because we don’t make the all tutorials qiskit 0.6 as many are community contributions and we don’t have the bandwidth to keep all updated we decided not to do it.

[ajavadia]

@mtreinish is there any update on this? I think what you pasted above works in the sense that it makes sure no error is raised while executing the examples. Later on we may want to test some correctness aspects as well, but for now just need to make sure they're not broken.