New hook to get the full list of collected items (before any filtering, slicing, etc.) ? #1298
Description
Hi there, thanks once again for this great and reliable plugin !
I use this plugin for parallel execution of large data science benchmarks, in combination with pytest-pilot (to slice in a more intuitive way than the generic '-k' cli) and pytest-cases (to parametrize complex situations without making the test itself heavier).
I face the need to know, inside the test session itself, the full list of tests that pytest "has initially collected" (before slicing was made by '-k', or more generally, before the collection is possibly modified by plugins).
When I do not use pytest-xdist, the solution is straightforward : I use pytest_collection_modifyitems in a wrapper mode to be sure that I have access to the full list of collected items:
@pytest.hookimpl(wrapper=True)
def pytest_collection_modifyitems(items):
# here the `items` list is the full one, before any slicing/filtering/modificationHowever with pytest-xdist this hook is skipped on the master node. We only have access to the pytest_xdist_node_collection_finished hook, that returns the actual tests that will be executed by each node. But then the list might have been modified already.
Could it be worth adding a hook so that each node reports the status of collected items before filtering, too ? Something such as pytest_xdist_node_collection_initial_items(node, items) ?
In the meantime since my nodes are all running locally, I will use a FileLock so that the workers will end their pytest_collection_modifyitems hook by writing the full list to a file. Obviously this will not work when I switch to remote workers (colleagues are considering trying Ray btw - maybe that could result in a new feature idea for xdist)
Thanks once again !