Pyjo.Prove - Run test scripts¶
$ python -m Pyjo.Prove t/Pyjo t/docs
$ python -m Pyjo.Prove -r -j 4 t
$ pyjo-prove -v t/Pyjo/URL.py
Pyjo.Prove runs test scripts which print the
Test Anything Protocol (TAP), such as the ones
written with Pyjo.Test, and reports their results. It is inspired by
Perl’s prove and accepts the options of it that make sense for Python.
A directory runs every script with the extension .py in it, and with
-r in its subdirectories too. Without arguments it runs t.
A test script passes when it exits with status 0, declares a plan, runs as
many tests as planned and none of them fails, except the TODO tests. A
plan of 1..0 # SKIP reason skips the whole script. Bail out! stops the
whole run. Indented lines belong to subtests and are not parsed: the parent
test point stands for the whole subtest.
A script with the inline metadata of
PEP 723 is skipped without running when
the Python that runs it does not satisfy its requires-python, as a script
with the syntax of a newer Python, which an older one cannot compile, so that
the script cannot skip itself.
# /// script
# requires-python = ">=3.14"
# ///
The summary ends with Result: PASS when every test script passed,
Result: FAIL when one did not, and Result: NOTESTS when they passed
without running a test, as when every one of them was skipped. The exit status is 0 when every test script passed and 1 otherwise.
Ctrl-C (SIGINT) stops the run: the test scripts that are running are
killed, with the processes they started, and the harness dies of the signal,
or with --trap prints the summary of the test scripts that have finished.
Options¶
-r,--recurseRun the test scripts in subdirectories too.
-j N,--jobs NRun up to N test scripts at the same time, or with
0as many as the processors that the harness may use.$ python -m Pyjo.Prove -j 0 -r t
provehas no such value.-v,--verbosePrint the TAP output of each test script, and set the environment variable
PYJO_HARNESS_IS_VERBOSEof the test scripts to1, asprovesetsHARNESS_IS_VERBOSE, which tellsPyjo.Test.Pyjoto log everything.-I DIR,--include DIRAdd the directory to
PYTHONPATHof the test scripts. Can be repeated.-l,--libAdd the current directory to
PYTHONPATHof the test scripts, as-I .does, whereproveaddslib.--ext EXTExtension of the test scripts in a directory,
.pyby default.--exec CMDCommand that runs a test script, the current Python interpreter by default. A script whose command cannot start fails with the reason.
--timeout SECONDSThe most time a test script may run, no limit by default. A test script that runs longer is killed, with the processes it started, and fails.
--trapPrint the summary of the test scripts that have finished when Ctrl-C interrupts the run.
--capture-stderrKeep the standard error of each test script and print it after the line of its result, each line after the name of the script, instead of letting it through as the script writes it, where the scripts that run at the same time with
-jmix it.t/Pyjo/URL.py ........ ok t/Pyjo/URL.py: ResourceWarning: unclosed <socket.socket ...>
provehas no such option: its--mergeputs the standard error into the TAP output, which shows it with-vonly.--stderr-lines NWith
--capture-stderr, print at most N lines of the standard error of a test script, no limit by default: the first half of them and the last half, which ends with what the script said last, and between them a line that tells how many were left out.t/Pyjo/URL.py: [922554 lines left out]
--count,--no-count,--nocountShow the X/Y count of the tests of a test script while it runs or not, renewed once a second. By default it is shown when the output goes to a terminal, and never with
-v. With-j, a line shows the number of the tests of all the test scripts, the seconds of the run and the count of each test script that runs.t/Pyjo/URL.py ........ 12/? ===( 158;3 12/? 40/40 3/? )======================================
proveshows it on a terminal only, even with--count.-c,--color,--no-color,--nocolorColor the output or not. By default it is colored when it goes to a terminal, unless
NO_COLORis set orTERMisdumb: the result of a test script that passed and the summary of a run that passed are green, what failed is red, what was skipped and theTODOtests are yellow, and so isNOTESTS. With--capture-stderr, the standard error of a test script is red.--formatter NAME[=FILE]Write the results as the formatter
NAMEto the file, or in the place of the usual output without one, which only one formatter can do. Can be repeated. The formatters are:consoleThe usual output, a line for each test script and the summary, which goes to the standard output unless another formatter writes there.
junitThe JUnit XML of the results, which the reporters of CI systems read, as
junit()makes it.markdownThe summary of the results in Markdown, as the one of a job of GitHub Actions, with a table of what failed and the diagnostics of each test script that did not pass, as
markdown()makes it.
The diagnostics of the failures come from the standard error of the test scripts, which
--capture-stderrcaptures.$ python -m Pyjo.Prove --formatter junit -r t > report.xml $ python -m Pyjo.Prove --capture-stderr --formatter junit=report.xml \\ --formatter "markdown=$GITHUB_STEP_SUMMARY" -r tprovetakes the class of one formatter, which writes in the place of the usual output, asTAP::Formatter::JUnit.
Environment¶
PYJO_PROVE_OPTSOptions that go before the ones of the command line, which override them, split as a shell splits words, as the command of
--execis, where a backslash is kept on Windows, as in its paths. The test scripts do not get it, so that a test script that runsPyjo.Proveitself runs it without them.$ PYJO_PROVE_OPTS='-l -j 4' python -m Pyjo.Prove -r t
provereads its options from.provercinstead.PYJO_HARNESS_ACTIVESet to
1for the test scripts, asprovesetsHARNESS_ACTIVE, which tellsPyjoyment.Commandsnot to list the commands.
Functions¶
- class Pyjo.Prove.Point(number: int, name: str = '', ok: bool = True, directive: str | None = None, reason: str = '', time: float = 0.0, diagnostics: list[str] = <factory>)[source]¶
A test point of the TAP output of a test script, with the lines of its subtest and the diagnostics after it when it fails.
- class Pyjo.Prove.Result(path: str, planned: int | None = None, skip_all: str | None = None, tests: int = 0, failed: list[int] = <factory>, todo_passed: list[int] = <factory>, skipped: int = 0, bail_out: str | None = None, exit_status: int = 0, timed_out: float | None = None, errors: list[str] = <factory>, output: list[str] = <factory>, stderr: list[str] = <factory>, stderr_left_out: int = 0, points: list[Point] = <factory>, time: float = 0.0, pending: list[str] = <factory>, mark: float | None = None)[source]¶
Result of a test script: what its TAP output says and how it exited.
- Pyjo.Prove.find_scripts(paths: Sequence[str], ext: str, recurse: bool) list[str][source]¶
scripts = find_scripts(['t'], '.py', recurse=True)
Expand directories into the test scripts in them, sorted by name. The names of the scripts found in a directory have
/between their levels on every system, asprovegives them.
- Pyjo.Prove.finish(result: Result) None[source]¶
finish(result)
Check the result once the test script has exited: the plan, the number of tests and the exit status. The diagnostics of the failures in the standard error that was captured, as
Pyjo.TestandTest::Morewrite them, go to the test points that failed, in their order, with the diagnostics of the subtest before each of them.
- Pyjo.Prove.junit(results: Sequence[Result]) Pyjo.DOM.object[source]¶
report = junit(results) Path('report.xml').write_text(str(report))
The JUnit XML of the results, as the reporters of CI systems read it, as a
Pyjo.DOMobject: atestsuitefor each test script, with atestcasefor each of its test points, whosefailurehas the diagnostics of a test point that failed, the lines of its subtest and the comments after it, and whoseskippedhas the reason of a test point that was skipped, or is aTODOtest that failed. Atestcasenamed after the script stands for what the script did wrong as a whole, as anerror, as dying, a bad plan or a timeout, or for all its tests that it skipped. A test case has thelineof the firstat FILE line Nof its diagnostics, whichPyjo.Testwrites, where the reporters annotate the failure. The TAP output of a script that did not pass, and the standard error that was captured, are itssystem-outandsystem-err.
- Pyjo.Prove.main(argv: Sequence[str] | None = None) int[source]¶
sys.exit(main(['-r', 't']))
Command line interface: parse the options, after the ones of
PYJO_PROVE_OPTS, run the test scripts and return the exit status.
- Pyjo.Prove.markdown(results: Sequence[Result], elapsed: float, title: str = 'Pyjo.Prove') str[source]¶
summary = markdown(results, 37.2) summary = markdown(results, 37.2, title='Tests of Pyjo')
The summary of the results in Markdown, as GitHub Actions shows it as the summary of a job: the numbers of the files and of the tests that passed, were skipped and failed, as the test cases of
junit()count them, with a test script that skipped all its tests or failed as a whole as one of them, the time, and the result,PASS,FAILorNOTESTS, with a table of what failed, and the diagnostics and the standard error of each test script that did not pass, folded.
- Pyjo.Prove.parse(result: Result, line: str, at: float | None = None) None[source]¶
result = Result('t/test.py') for line in output.splitlines(): parse(result, line) parse(result, line, at=time.monotonic())
Parse a line of TAP output into the result, which came at the time
at, oftime.monotonic(), for the time of its test point.
- async Pyjo.Prove.run(scripts: Sequence[str], command: Sequence[str] | None = None, env: dict[str, str] | None = None, *, jobs: int = 1, verbose: bool = False, timeout: float | None = None, trap: bool = False, capture_stderr: bool = False, stderr_lines: int | None = None, color: bool | None = None, count: bool | None = None, console: bool = True) list[Result][source]¶
results = asyncio.run(run(['t/a.py', 't/b.py'], jobs=2)) results = asyncio.run(run(['t/a.py', 't/b.py'], jobs=0)) results = asyncio.run(run(['t/a.py', 't/b.py'], timeout=60, trap=True)) results = asyncio.run(run(['t/a.py', 't/b.py'], jobs=2, capture_stderr=True)) results = asyncio.run(run(['t/a.py', 't/b.py'], jobs=2, capture_stderr=True, stderr_lines=1000)) results = asyncio.run(run(['t/a.py', 't/b.py'], color=False)) results = asyncio.run(run(['t/a.py', 't/b.py'], count=False)) results = asyncio.run(run(['t/a.py', 't/b.py'], console=False))
Run the test scripts, up to
jobsof them at the same time, or with0as many as the processors that the process may use, print a line for each of them and a summary, and return the results in the order of the scripts. Withtrap, a run that Ctrl-C interrupts prints the summary and returns the results of the test scripts that have finished. The output is colored withcolor, and the tests of the test scripts are counted while they run withcount, both by default when the output goes to a terminal, the count withoutverboseonly. Withoutconsole, nothing goes to the standard output, as when another formatter writes there, and the standard error that was captured still goes to the standard error.
- async Pyjo.Prove.run_script(path: str, command: Sequence[str], env: dict[str, str], *, verbose: bool = False, stream: bool = False, timeout: float | None = None, capture_stderr: bool = False, stderr_lines: int | None = None, color: bool = False, progress: Callable[[Result], object] | None = None, cwd: str | None = None) Result[source]¶
result = await run_script('t/test.py', [sys.executable], dict(os.environ)) result = await run_script('t/test.py', [sys.executable], dict(os.environ), timeout=60) result = await run_script('t/test.py', [sys.executable], dict(os.environ), capture_stderr=True) result = await run_script('t/test.py', [sys.executable], dict(os.environ), capture_stderr=True, stderr_lines=100) result = await run_script('t/test.py', [sys.executable], dict(os.environ), progress=print) result = await run_script('t/test.py', [sys.executable], dict(os.environ), cwd='project')
Run a test script and parse its TAP output. The standard error of the script goes to the standard error of this process, or with
capture_stderrto the lines ofResult.stderr, at moststderr_linesof them, the first half and the last half, with the number of the lines between them inResult.stderr_left_out. Withverboseandstream, the TAP output is printed as it comes, colored withcolor.progressis called with the result when the script starts and after each of its tests. A script that runs longer thantimeoutseconds is killed, with the processes it started. A script whoserequires-pythonthe Python of the command does not satisfy is skipped without running. The script runs in the directorycwd, the current one by default, which its path is relative to.