pytest_runtest_call() runs item twice

[joshua-badger]

When I define pytest_runtest_call() and use item.runtest() within it in my conftest.py file, my test runs twice. I expected it to just run once. Are my expectations off, or is there a way to prevent the second test run?

This is bad for two reasons:

1. The database state is not reset between the two runs, so the second run doesn't start from a clean, known state and causes the test to fail when it would otherwise pass
2. We are literally doubling the test run time because it runs twice

A toy example is below:

# root-level conftest.py

@pytest.fixture(autouse=True)
def global_print_check(capsys):
    """Guarantee capsys is available to all tests"""
    yield capsys


def pytest_runtest_call(item):
    """Globally check for some output (toy example, could be validating certain log statements, etc)"""
    item.runtest()
    print_capper = item.funcargs["global_print_check"]  # Grab the capsys object
    out, err = print_capper.readouterr()
    assert out  # Guarantee we printed something

# test_things.py

BAR = 0  # represents the database and its state (won't be reset between test runs)


def test_a_thing():
    global BAR
    add_amounts = [0]
    for _ in range(3):
        add_amounts.append(randint(10, 25))

    for index, add_amount in enumerate(add_amounts, 1):
        print(f"Adding {add_amount} to BAR")
        BAR += add_amount
        expected_amount = sum(add_amounts[:index])
        assert BAR == expected_amount

Output:

 =================================== FAILURES ===================================
 _________________________________ test_a_thing _________________________________
 
     def test_a_thing():
         global BAR
         add_amounts = [0]
         for _ in range(3):
             add_amounts.append(randint(10, 25))
     
         for index, add_amount in enumerate(add_amounts, 1):
             print(f"Adding {add_amount} to BAR")
             BAR += add_amount
             expected_amount = sum(add_amounts[:index])
 >           assert BAR == expected_amount
 E           assert 48 == 0
 
 test_ui_prior_auth_request_list_page.py:126: AssertionError
 ----------------------------- Captured stdout call -----------------------------
 Passed the global check
 Adding 0 to BAR
 =========================== short test summary info ============================
 FAILED test_ui_prior_auth_request_list_page.py::test_a_thing - assert 48 == 0
 ============================== 1 failed in 9.59s ===============================

You'll note that the "Passed the global check" statement happened, which means the test already ran once (and passed) before failing this second time.

Installed packages (developing an internal app using poetry, so here are the contents of our pyproject.toml file:

[tool.poetry]
name = "<redacted>"
version = "0.1.0"
description = ""
authors = ["<redacted>"]

[tool.poetry.dependencies]
python = "^3.8"
django = "^3.2"
celery = "^4.4.0"
redis = "^3.3.11"
gunicorn = "^20.0.4"
psycopg2 = "^2.8.4"
django-getenv = "^1.3.2"
django-redis-cache = "^3.0"
djangorestframework = "^3.11.0"
django-simple-history = "^2.8.0"
django-auth-ldap = "^2.1.0"
django-extensions = "^2.2.6"
gql = "^0.3.0"
dj-database-url = "^0.5.0"
hiredis = "^1.0"
celery-redbeat = "^0.13.0"
httpx = "^0.15.0"
jmespath = "^0.9.5"
pytest-asyncio = "^0.11.0"
django-debug-toolbar = "^2.2"
django-pglocks = "^1.0.4"
django-storages = "^1.9.1"
boto3 = "^1.14.29"
sentry-sdk = "0.16.4"
xifin = "^0.0.33"
django-constance = "^2.7.0"
django-picklefield = "^3.0.1"
django-celery-beat = "^2.0.0"
pillow = "^8.0.1"
datadog = "^0.39.0"
ddtrace = "^0.43.1"
Cython = "^0.29.21"
django-extra-views = "^0.13.0"
django-widget-tweaks = "^1.4.8"
auto-mq = "^1.6"
django-autocomplete-light = "^3.8.2"

[tool.poetry.dev-dependencies]
coverage = "^5.0.3"
ipython = "^7.11.1"
black = "^19.10b0"
flake8 = "^3.7.9"
flake8-tidy-imports = "^4.0.0"
flake8-import-order = "^0.18.1"
flake8-black = "^0.1.1"
mypy = "^0.761"
pytest = "^6.2.4"
pytest-django = "^3.8.0"
pytest-cov = "^2.8.1"
responses = "^0.10.9"
pdbpp = "^0.10.2"
hypothesis = "^5.4"
factory-boy = "^2.12.0"
ipdb = "^0.13.2"
selenium = "^3.141.0"
pytest-html = "^3.1.0"
pytest-xdist = "^2.2.0"
respx = ">=0.16.3"
nplusone = "^1.0.0"

[tool.black]
line-length = 99
exclude = '''(
  manage.py
  | <redacted>
  | <redacted>/migrations
  | <redacted>/.*/migrations
  | \.git/.*
  )
'''

[[tool.poetry.source]]
name = "<redacted>"
url = "https://pypi.<redacted>.com/<redacted>/prod/+simple"
default = true

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

pytest version: 6.2.4 (see pyproject.toml)
OS version: docker container python3.8

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

[bluetech]

When multiple plugins implement a hook, pytest calls all of them (unless the hook is defined as firstresult, which pytest_runtest_call isn't). This means pytest calls your hook implementation in addition to the built-in pytest implementation.

In your example, you should be able to just remove the item.runtest() call as pytest already does that. To make sure your hook implementation runs after all of the others, you can mark it as trylast=True:

@pytest.hookimpl(trylast=True)
def pytest_runtest_call(item):
    ...

Let me know if it's not clear or doesn't work as expected.

[RonnyPfannschmidt]

i strongly recommend the solution by @bluetech - its practically the de-facto standard way to implement your use-case

[joshua-badger]

This is fantastic, I'll give it a whirl right now. :D

[joshua-badger]

Yup, worked exactly as I hoped it would. Thank you so much!

[Formartha]

Superb.
May I ask if there’s a better way to handle that case with registering some global parameter that will provide ability to determine if the plugin ran or not? (Or better yet, a module in the plugin)

[RonnyPfannschmidt]

please open a new discussion with enough context to provide a meaningful answer - hijacking a 3 year old closed discussion with a incomplete question will not get a good answer