Pyjo.Prove - Run test scripts

$ python -m Pyjo.Prove t/Pyjo t/docs
$ python -m Pyjo.Prove -r -j 4 t
$ pyjo-prove -v t/Pyjo/URL.py

Pyjo.Prove runs test scripts which print the Test Anything Protocol (TAP), such as the ones written with Pyjo.Test, and reports their results. It is inspired by Perl’s prove and accepts the options of it that make sense for Python.

A directory runs every script with the extension .py in it, and with -r in its subdirectories too. Without arguments it runs t.

A test script passes when it exits with status 0, declares a plan, runs as many tests as planned and none of them fails, except the TODO tests. A plan of 1..0 # SKIP reason skips the whole script. Bail out! stops the whole run. Indented lines belong to subtests and are not parsed: the parent test point stands for the whole subtest.

A script with the inline metadata of PEP 723 is skipped without running when the Python that runs it does not satisfy its requires-python, as a script with the syntax of a newer Python, which an older one cannot compile, so that the script cannot skip itself.

# /// script
# requires-python = ">=3.14"
# ///

The summary ends with Result: PASS when every test script passed, Result: FAIL when one did not, and Result: NOTESTS when they passed without running a test, as when every one of them was skipped. The exit status is 0 when every test script passed and 1 otherwise. Ctrl-C (SIGINT) stops the run: the test scripts that are running are killed, with the processes they started, and the harness dies of the signal, or with --trap prints the summary of the test scripts that have finished.

Options

-r, --recurse

Run the test scripts in subdirectories too.

-j N, --jobs N

Run up to N test scripts at the same time, or with 0 as many as the processors that the harness may use.

$ python -m Pyjo.Prove -j 0 -r t

prove has no such value.

-v, --verbose

Print the TAP output of each test script, and set the environment variable PYJO_HARNESS_IS_VERBOSE of the test scripts to 1, as prove sets HARNESS_IS_VERBOSE, which tells Pyjo.Test.Pyjo to log everything.

-I DIR, --include DIR

Add the directory to PYTHONPATH of the test scripts. Can be repeated.

-l, --lib

Add the current directory to PYTHONPATH of the test scripts, as -I . does, where prove adds lib.

--ext EXT

Extension of the test scripts in a directory, .py by default.

--exec CMD

Command that runs a test script, the current Python interpreter by default. A script whose command cannot start fails with the reason.

--timeout SECONDS

The most time a test script may run, no limit by default. A test script that runs longer is killed, with the processes it started, and fails.

--trap

Print the summary of the test scripts that have finished when Ctrl-C interrupts the run.

--capture-stderr

Keep the standard error of each test script and print it after the line of its result, each line after the name of the script, instead of letting it through as the script writes it, where the scripts that run at the same time with -j mix it.

t/Pyjo/URL.py ........ ok
t/Pyjo/URL.py: ResourceWarning: unclosed <socket.socket ...>

prove has no such option: its --merge puts the standard error into the TAP output, which shows it with -v only.

--stderr-lines N

With --capture-stderr, print at most N lines of the standard error of a test script, no limit by default: the first half of them and the last half, which ends with what the script said last, and between them a line that tells how many were left out.

t/Pyjo/URL.py: [922554 lines left out]
--count, --no-count, --nocount

Show the X/Y count of the tests of a test script while it runs or not, renewed once a second. By default it is shown when the output goes to a terminal, and never with -v. With -j, a line shows the number of the tests of all the test scripts, the seconds of the run and the count of each test script that runs.

t/Pyjo/URL.py ........ 12/?
===(     158;3  12/?  40/40  3/? )======================================

prove shows it on a terminal only, even with --count.

-c, --color, --no-color, --nocolor

Color the output or not. By default it is colored when it goes to a terminal, unless NO_COLOR is set or TERM is dumb: the result of a test script that passed and the summary of a run that passed are green, what failed is red, what was skipped and the TODO tests are yellow, and so is NOTESTS. With --capture-stderr, the standard error of a test script is red.

--formatter NAME[=FILE]

Write the results as the formatter NAME to the file, or in the place of the usual output without one, which only one formatter can do. Can be repeated. The formatters are:

console

The usual output, a line for each test script and the summary, which goes to the standard output unless another formatter writes there.

junit

The JUnit XML of the results, which the reporters of CI systems read, as junit() makes it.

markdown

The summary of the results in Markdown, as the one of a job of GitHub Actions, with a table of what failed and the diagnostics of each test script that did not pass, as markdown() makes it.

The diagnostics of the failures come from the standard error of the test scripts, which --capture-stderr captures.

$ python -m Pyjo.Prove --formatter junit -r t > report.xml
$ python -m Pyjo.Prove --capture-stderr --formatter junit=report.xml \\
    --formatter "markdown=$GITHUB_STEP_SUMMARY" -r t

prove takes the class of one formatter, which writes in the place of the usual output, as TAP::Formatter::JUnit.

Environment

PYJO_PROVE_OPTS

Options that go before the ones of the command line, which override them, split as a shell splits words, as the command of --exec is, where a backslash is kept on Windows, as in its paths. The test scripts do not get it, so that a test script that runs Pyjo.Prove itself runs it without them.

$ PYJO_PROVE_OPTS='-l -j 4' python -m Pyjo.Prove -r t

prove reads its options from .proverc instead.

PYJO_HARNESS_ACTIVE

Set to 1 for the test scripts, as prove sets HARNESS_ACTIVE, which tells Pyjoyment.Commands not to list the commands.

Functions

class Pyjo.Prove.Point(number: int, name: str = '', ok: bool = True, directive: str | None = None, reason: str = '', time: float = 0.0, diagnostics: list[str] = <factory>)[source]

A test point of the TAP output of a test script, with the lines of its subtest and the diagnostics after it when it fails.

property failed: bool[source]
failed = point.failed

True if the test point failed, and is not a TODO test.

property skipped: bool[source]
skipped = point.skipped

True if the test point was skipped, or is a TODO test that failed, as expected.

class Pyjo.Prove.Result(path: str, planned: int | None = None, skip_all: str | None = None, tests: int = 0, failed: list[int] = <factory>, todo_passed: list[int] = <factory>, skipped: int = 0, bail_out: str | None = None, exit_status: int = 0, timed_out: float | None = None, errors: list[str] = <factory>, output: list[str] = <factory>, stderr: list[str] = <factory>, stderr_left_out: int = 0, points: list[Point] = <factory>, time: float = 0.0, pending: list[str] = <factory>, mark: float | None = None)[source]

Result of a test script: what its TAP output says and how it exited.

property passed: bool[source]
ok = result.passed

True if the test script passed.

Pyjo.Prove.find_scripts(paths: Sequence[str], ext: str, recurse: bool) → list[str][source]
scripts = find_scripts(['t'], '.py', recurse=True)

Expand directories into the test scripts in them, sorted by name. The names of the scripts found in a directory have / between their levels on every system, as prove gives them.

Pyjo.Prove.finish(result: Result) → None[source]
finish(result)

Check the result once the test script has exited: the plan, the number of tests and the exit status. The diagnostics of the failures in the standard error that was captured, as Pyjo.Test and Test::More write them, go to the test points that failed, in their order, with the diagnostics of the subtest before each of them.

Pyjo.Prove.junit(results: Sequence[Result]) → Pyjo.DOM.object[source]
report = junit(results)
Path('report.xml').write_text(str(report))

The JUnit XML of the results, as the reporters of CI systems read it, as a Pyjo.DOM object: a testsuite for each test script, with a testcase for each of its test points, whose failure has the diagnostics of a test point that failed, the lines of its subtest and the comments after it, and whose skipped has the reason of a test point that was skipped, or is a TODO test that failed. A testcase named after the script stands for what the script did wrong as a whole, as an error, as dying, a bad plan or a timeout, or for all its tests that it skipped. A test case has the line of the first at FILE line N of its diagnostics, which Pyjo.Test writes, where the reporters annotate the failure. The TAP output of a script that did not pass, and the standard error that was captured, are its system-out and system-err.

Pyjo.Prove.main(argv: Sequence[str] | None = None) → int[source]
sys.exit(main(['-r', 't']))

Command line interface: parse the options, after the ones of PYJO_PROVE_OPTS, run the test scripts and return the exit status.

Pyjo.Prove.markdown(results: Sequence[Result], elapsed: float, title: str = 'Pyjo.Prove') → str[source]
summary = markdown(results, 37.2)
summary = markdown(results, 37.2, title='Tests of Pyjo')

The summary of the results in Markdown, as GitHub Actions shows it as the summary of a job: the numbers of the files and of the tests that passed, were skipped and failed, as the test cases of junit() count them, with a test script that skipped all its tests or failed as a whole as one of them, the time, and the result, PASS, FAIL or NOTESTS, with a table of what failed, and the diagnostics and the standard error of each test script that did not pass, folded.

Pyjo.Prove.parse(result: Result, line: str, at: float | None = None) → None[source]
result = Result('t/test.py')
for line in output.splitlines():
    parse(result, line)
parse(result, line, at=time.monotonic())

Parse a line of TAP output into the result, which came at the time at, of time.monotonic(), for the time of its test point.

async Pyjo.Prove.run(scripts: Sequence[str], command: Sequence[str] | None = None, env: dict[str, str] | None = None, *, jobs: int = 1, verbose: bool = False, timeout: float | None = None, trap: bool = False, capture_stderr: bool = False, stderr_lines: int | None = None, color: bool | None = None, count: bool | None = None, console: bool = True) → list[Result][source]
results = asyncio.run(run(['t/a.py', 't/b.py'], jobs=2))
results = asyncio.run(run(['t/a.py', 't/b.py'], jobs=0))
results = asyncio.run(run(['t/a.py', 't/b.py'], timeout=60, trap=True))
results = asyncio.run(run(['t/a.py', 't/b.py'], jobs=2, capture_stderr=True))
results = asyncio.run(run(['t/a.py', 't/b.py'], jobs=2, capture_stderr=True, stderr_lines=1000))
results = asyncio.run(run(['t/a.py', 't/b.py'], color=False))
results = asyncio.run(run(['t/a.py', 't/b.py'], count=False))
results = asyncio.run(run(['t/a.py', 't/b.py'], console=False))

Run the test scripts, up to jobs of them at the same time, or with 0 as many as the processors that the process may use, print a line for each of them and a summary, and return the results in the order of the scripts. With trap, a run that Ctrl-C interrupts prints the summary and returns the results of the test scripts that have finished. The output is colored with color, and the tests of the test scripts are counted while they run with count, both by default when the output goes to a terminal, the count without verbose only. Without console, nothing goes to the standard output, as when another formatter writes there, and the standard error that was captured still goes to the standard error.

async Pyjo.Prove.run_script(path: str, command: Sequence[str], env: dict[str, str], *, verbose: bool = False, stream: bool = False, timeout: float | None = None, capture_stderr: bool = False, stderr_lines: int | None = None, color: bool = False, progress: Callable[[Result], object] | None = None, cwd: str | None = None) → Result[source]
result = await run_script('t/test.py', [sys.executable], dict(os.environ))
result = await run_script('t/test.py', [sys.executable], dict(os.environ), timeout=60)
result = await run_script('t/test.py', [sys.executable], dict(os.environ), capture_stderr=True)
result = await run_script('t/test.py', [sys.executable], dict(os.environ), capture_stderr=True, stderr_lines=100)
result = await run_script('t/test.py', [sys.executable], dict(os.environ), progress=print)
result = await run_script('t/test.py', [sys.executable], dict(os.environ), cwd='project')

Run a test script and parse its TAP output. The standard error of the script goes to the standard error of this process, or with capture_stderr to the lines of Result.stderr, at most stderr_lines of them, the first half and the last half, with the number of the lines between them in Result.stderr_left_out. With verbose and stream, the TAP output is printed as it comes, colored with color. progress is called with the result when the script starts and after each of its tests. A script that runs longer than timeout seconds is killed, with the processes it started. A script whose requires-python the Python of the command does not satisfy is skipped without running. The script runs in the directory cwd, the current one by default, which its path is relative to.