Implement event loop factory hook

[tjkuson]

Added the pytest_asyncio_loop_factories hook to parametrize asyncio tests with custom event loop factories. Users can use the loop_factories argument to select a subset of hooks. If omitted, the test will run parametrized with each loop factory item returned by the hook.

import asyncio
from collections.abc import Mapping

import pytest
import uvloop


def pytest_asyncio_loop_factories(
    config: pytest.Config,
    item: pytest.Item,
) -> Mapping[str, Callable[[], AbstractEventLoop]]:
    return {
        "uvloop": uvloop.new_event_loop,
        "stdlib": asyncio.new_event_loop,
    }

import pytest


@pytest.mark.asyncio(loop_factories=["uvloop"])
async def test_with_uvloop_only() -> None:
    assert True

Closes #1101, #1032, #1346.

Relates to #1164 by building on the idea of having a global parametrization for all tests and fixtures instead of a marker. An alternative idea was to expose a configuration option where a user could describe event loop factors and which tests they applied to, but this seemed less ergonomic to me compared to the hook approach (and less powerful than allowing user-defined logic).

Test plan

Added new tests that pass via uvx tox.

Existing tests pass with minimumal changes.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.13%. Comparing base (dbacf7b) to head (74fa976).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1373      +/-   ##
==========================================
+ Coverage   93.64%   95.13%   +1.49%     
==========================================
  Files           2        2              
  Lines         409      473      +64     
  Branches       44       57      +13     
==========================================
+ Hits          383      450      +67     
+ Misses         20       17       -3     
  Partials        6        6              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[seifertm]

Thanks for the great work @tjkuson ! This is much more than a draft. It looks like we finally have a replacement for the policy fixture in pytest-asyncio.

I did a couple of comments, but most of them are very minor. The two largest being the limitation to a single hook implementation and your thought about my idea to use the asyncio marker to limit the loop factory parametrization for a single test.

[seifertm]

Let's say a configured configured pytest_asyncio_loop_factories so that all tests to run with uvloop and SelectorEventLoop. There's one specific test that should only be run with uvloop. This means the user would have to switch to the conftest.py file and find some way to identify the test (e.g. custom marker or item name) to tell pytest-asyncio to use a different set of factories for this test.

Ergonomically, it would be preferable for the user to perform this change directly at the level of the test without having to change the hook implementation. This would also improve visibility that this test is an exception.

Based on the implementation of _get_item_loop_scope my understanding is that we have access to the asyncio marker here. Could we use the information from the marker to modify the test parametrization?

I'm thinking of something like this:

def pytest_asyncio_loop_factories(config, item):
    return {
        "default": asyncio.new_event_loop,
        "uvloop": uvloop.new_event_loop,
    }

async def parametrized_over_all_factories():
    ...

@pytest.mark.asyncio(loop_factories=["uvloop"])
async def  only_runs_with_uvloop():
    ...

@pytest.mark.asyncio(loop_factories=["default"])
async def  only_runs_with_default_asyncio_loop():
    ...

Do you think this is feasible and/or reasonable? I'd love to hear your opinion on this @tjkuson .

[tjkuson]

@seifertm I can see that being easier. The workflow I had in mind was that people would configure their own marks to achieve this (e.g., designing a pytest.mark.uvloop mark), but that can get complex, especially if there are lots of loop factory permutations.

The main downsides I can think of to the alternative approach you're suggesting:

• We now have more than one way to do the same thing, which could be confusing to communicate
• The mapping structure might be misleading (e.g., we don't usually provide mappings where the default behaviour is to collect all of its values), whereas it should be intuitive that all things returned by an ordinary collection are used to parametrize tests
• pytest-asyncio now carries additional complexity that needs to be maintained (namely due to the additional marker check upon test collection) versus just letting it be the user's responsibility

I think the benefit to the developer experience outweighs these negatives, however, so I am in favour of your suggestion. I have a commit working locally that implements this, but I'll tidy it up first before committing!

[tjkuson]

I've updated the PR to use a mapping. The implementation is more a complicated and invasive change, so I'd be interested in your thoughts.

[seifertm]

It looks very good to me.

I suggest we create a pre-release and ask the reporters of related issues to test if their use cases are covered (from the top of my head that's mostly pytest-aiohttp). The pre-release may also uncover regressions, if any, since some projects test against our pre-releases.

Any objections?

[tjkuson]

That sounds good to me! Thank you for taking the time to review. Let me know if there's anything I can do to help out with the pre-release.

I have a couple of minor tweaks in mind that I can push later today, and then I could mark the PR as ready for review and you could take another look?

[tjkuson]

Also, let me know if you'd like me to squash the commits.

[tjkuson]

I changed the hook to using item-scoped dispatch over a global dispatch. When there are multiple hook definitions, instead of one 'winning' and being applied to all tests, using an item-scoped dispatch would mean it leverages conftest locality when resolving the hook. I think this would be less surprising for users than the previous behaviour. What do you think about the change? Apart from the test suite, it's only a one-line change, so it's simple to revert if we want to go back. 85ce9f6