The new pythonTestAdapter breaks python-subtests

[Winand]

Type: Bug

Behaviour

I cannot use python-subtests when pythonTestAdapter is enabled.

    "python.experiments.optInto": [
        "pythonTestAdapter"
    ],

Exceptions in subtests (except for the first subtest) don't cause the test to be marked as failed.

Steps to reproduce:

Here's an example:

def test(subtests):
    with subtests.test(msg="First subtest"):
        assert 1 == 1
    with subtests.test(msg="Second subtest"):
        assert 2 == 1

Test results:

Running pytest with args: ['-p', 'vscode_pytest', '--rootdir=c:\\Users\\Andrey.makarov\\Documents\\projects\\python-test-adapter-bug', 'c:\\Users\\Andrey.makarov\\Documents\\projects\\python-test-adapter-bug\\test_test.py::test']
============================= test session starts =============================
platform win32 -- Python 3.12.5, pytest-8.3.3, pluggy-1.5.0
rootdir: c:\Users\Andrey.makarov\Documents\projects\python-test-adapter-bug
configfile: pyproject.toml
plugins: subtests-0.13.1
collected 1 item

test_test.py ,u.                                                         [100%]

================================== FAILURES ===================================
____________________________ test [Second subtest] ____________________________

subtests = SubTests(ihook=<_pytest.config.compat.PathAwareHookProxy object at 0x00000206322DCF50>, suspend_capture_ctx=<bound met...'>> _state='started' _in_suspended=False> _capture_fixture=None>>, request=<SubRequest 'subtests' for <Function test>>)

    def test(subtests):
        with subtests.test(msg="First subtest"):
            assert 1 == 1
        with subtests.test(msg="Second subtest"):
>           assert 2 == 1
E           assert 2 == 1

test_test.py:5: AssertionError
------------------------------ Captured log call ------------------------------

=========================== short test summary info ===========================
[Second subtest] SUBFAIL test_test.py::test - assert 2 == 1
=============== 1 failed, 1 passed, 1 subtests passed in 0.13s ================
Finished running tests!

The second subtest has failed but the test is marked as passed on the Testing panel.

Extension version: 2024.20.0
VS Code version: Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813, 2024-11-13T14:50:04.152Z)
OS version: Windows_NT x64 10.0.19045
Modes:

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.5
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Default

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

experiments
• optInto: ["pythonTestAdapter"]

Installed Extensions

Extension Name	Extension Id	Version
Auto Markdown TOC	huntertran.auto-markdown-toc	3.0.13
Dev Containers	ms-vscode-remote.remote-containers	0.388.0
Docker	ms-azuretools.vscode-docker	1.29.3
Even Better TOML	tamasfe.even-better-toml	0.19.2
Excel Viewer	GrapeCity.gc-excelviewer	4.2.62
GitLab Workflow	GitLab.gitlab-workflow	5.18.1
Hex Editor	ms-vscode.hexeditor	1.11.1
isort	ms-python.isort	2023.10.1
JavaScript Debugger	ms-vscode.js-debug	1.95.3
JavaScript Debugger Companion Extension	ms-vscode.js-debug-companion	1.1.3
Jupyter	ms-toolsai.jupyter	2024.10.0
Jupyter Cell Tags	ms-toolsai.vscode-jupyter-cell-tags	0.1.9
Jupyter Keymap	ms-toolsai.jupyter-keymap	1.1.2
Jupyter Notebook Renderers	ms-toolsai.jupyter-renderers	1.0.21
Jupyter Slide Show	ms-toolsai.vscode-jupyter-slideshow	0.1.6
PowerShell	ms-vscode.powershell	2024.4.0
Pylance	ms-python.vscode-pylance	2024.11.2
Pylint	ms-python.pylint	2024.0.0
Python	ms-python.python	2024.20.0
Python Debugger	ms-python.debugpy	2024.12.0
Ruff	charliermarsh.ruff	2024.54.0
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.10
TabOut	albert.TabOut	0.2.2

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-9600K CPU @ 3.70GHz (6 x 3696)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled webnn: disabled_off
Load (avg)	undefined
Memory (System)	31.70GB (8.66GB free)
Process Argv
Screen Reader	no
VM	0%

[eleanorjboyd]

Giving this a try, when I try it from the terminal I get this:

test_subtest.py::test [First subtest] SUBPASS                                                                                                [100%]
test_subtest.py::test [Second subtest] SUBFAIL                                                                                               [100%]
test_subtest.py::test PASSED 

Any idea why test_subtest.py::test is marked as passed if one of its subtests failed?

[eleanorjboyd]

create issue to discuss design in the pytest-subtest repo since another question came up as well. Feel free to comment in either place with your thoughts
pytest-dev/pytest-subtests#184

[Winand]

Any idea why test_subtest.py::test is marked as passed if one of its subtests failed?

I can only confirm that the whole test is marked as PASSED. In the summary section it says that 2 [subtests] failed, 1 [test] passed, 3 subtests passed.

In VS Code test panel it shows the whole "test_x" as failed. And it is more appropriate than just ignoring subtests results as pythonTestAdapter does.

tests/integration/test_api.py::test_x [First subtest] SUBPASS
tests/integration/test_api.py::test_x [Second subtest] SUBFAIL
tests/integration/test_api.py::test_x [Third subtest] SUBFAIL
tests/integration/test_api.py::test_x [Fourth subtest] SUBPASS
tests/integration/test_api.py::test_x [Fifth subtest] SUBPASS
tests/integration/test_api.py::test_x PASSED

===================================================================== FAILURES ===================================================================== 
_____________________________________________________________ test_x [Second subtest] ______________________________________________________________ 

subtests = SubTests(...)

    def test_x(subtests):
        with subtests.test(msg="First subtest"):
            assert 1 == 1
        with subtests.test(msg="Second subtest"):
>           assert 2 == 1
E           assert 2 == 1

tests\integration\test_api.py:185: AssertionError
---------------------------------------------------------------- Captured log call ----------------------------------------------------------------- 

______________________________________________________________ test_x [Third subtest] ______________________________________________________________ 

subtests = SubTests(...)

    def test_x(subtests):
        with subtests.test(msg="First subtest"):
            assert 1 == 1
        with subtests.test(msg="Second subtest"):
            assert 2 == 1
        with subtests.test(msg="Third subtest"):
>           assert 3 == 1
E           assert 3 == 1

tests\integration\test_api.py:187: AssertionError
---------------------------------------------------------------- Captured log call ----------------------------------------------------------------- 

============================================================= short test summary info ==============================================================
[Second subtest] SUBFAIL tests/integration/test_api.py::test_x - assert 2 == 1
[Third subtest] SUBFAIL tests/integration/test_api.py::test_x - assert 3 == 1
========================================== 2 failed, 1 passed, 11 deselected, 3 subtests passed in 3.84s ===========================================