We have drafted a profile using the must/ should/ may/ folder structure. However, we now want to "flatten" that structure, and collect related checks in top-level files (e.g. must/1_requesting_agent.ttl and should/1_requesting_agent.ttl get merged into a top-level 1_requesting_agent.ttl file, and so on).
However, when we do this, some of our tests start to fail in the CI. We can't figure out why...
For example, in this PR: eScienceLab#89
we have only merged files with the name 4_sign_off.ttl, but we see failures appear from tests which relate to the 1_root_data_entity_metadata.ttl 2_requesting_agent.ttl files, when neither those files nor the tests were changed.
Even stranger, the failures seem dependent on order in which the tests are run (and/or exactly which command is used to run them and/or the number of cores).
When I run locally:
pytest: 5 failures
pytest -k five-safes-crate (targeting just our profile): no failures
pytest -n 1 -k five-safes-crate (one worker only): no failures
pytest -v -n 1 tests/integration/profiles/five-safes-crate/test_5src_2_requesting_agent.py tests/integration/profiles/five-safes-crate/test_5src_4_signoff_phase.py (two specific test files): 3 failures from test_5src_2_requesting_agent.
pytest -v -n 1 tests/integration/profiles/five-safes-crate/test_5src_4_signoff_phase.py tests/integration/profiles/five-safes-crate/test_5src_2_requesting_agent.py (the same files in opposite order): no failures
and the CI for the PR has only 4 failures for pytest instead of 5.
@kikkomep I know you're not familiar with our profile but any help debugging this would be very welcome - we are completely stumped. If it's at all helpful, I also made a test branch where I "flattened" the workflow-ro-crate profile - https://github.com/eScienceLab/rocrate-validator/tree/upstream_merging_expt - and this also generated some inconsistent test errors (I got different errors with pytest vs pytest -k workflow-ro-crate).
I don't know if it's something to do with the mixed file structure, or something odd in how we have written certain checks or tests. As far as I know there were no such issues with the ISA profile (I helped develop it), which was flattened to begin with.
We have drafted a profile using the
must/should/may/folder structure. However, we now want to "flatten" that structure, and collect related checks in top-level files (e.g.must/1_requesting_agent.ttlandshould/1_requesting_agent.ttlget merged into a top-level1_requesting_agent.ttlfile, and so on).However, when we do this, some of our tests start to fail in the CI. We can't figure out why...
For example, in this PR: eScienceLab#89
we have only merged files with the name
4_sign_off.ttl, but we see failures appear from tests which relate to the1_root_data_entity_metadata.ttl2_requesting_agent.ttlfiles, when neither those files nor the tests were changed.Even stranger, the failures seem dependent on order in which the tests are run (and/or exactly which command is used to run them and/or the number of cores).
When I run locally:
pytest: 5 failurespytest -k five-safes-crate(targeting just our profile): no failurespytest -n 1 -k five-safes-crate(one worker only): no failurespytest -v -n 1 tests/integration/profiles/five-safes-crate/test_5src_2_requesting_agent.py tests/integration/profiles/five-safes-crate/test_5src_4_signoff_phase.py(two specific test files): 3 failures from test_5src_2_requesting_agent.pytest -v -n 1 tests/integration/profiles/five-safes-crate/test_5src_4_signoff_phase.py tests/integration/profiles/five-safes-crate/test_5src_2_requesting_agent.py(the same files in opposite order): no failuresand the CI for the PR has only 4 failures for
pytestinstead of 5.@kikkomep I know you're not familiar with our profile but any help debugging this would be very welcome - we are completely stumped. If it's at all helpful, I also made a test branch where I "flattened" the workflow-ro-crate profile - https://github.com/eScienceLab/rocrate-validator/tree/upstream_merging_expt - and this also generated some inconsistent test errors (I got different errors with
pytestvspytest -k workflow-ro-crate).I don't know if it's something to do with the mixed file structure, or something odd in how we have written certain checks or tests. As far as I know there were no such issues with the ISA profile (I helped develop it), which was flattened to begin with.