-
Notifications
You must be signed in to change notification settings - Fork 13
Open
Description
There are more than one valid resolution to the dependencies being used, and as seen in #37, resolvers can choose different solutions. To minimize incorrectness in the benchmarks, the resolutions should ideally be identical in the end, e.g., so resolve benchmarks capture how long it takes to get to the same solution, and so install benchmarks are for the same set of files.
It'd be good to write a small script that compares the resolutions across the tools and flags differences. Differences should probably require approval in a configuration file to "allow". I think some small differences are okay, but if the tools have significantly different resolutions they can affect benchmarks in a way that's misleading.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels