New TAP should not care about order

[exodist]

In order to support concurrency, I suggest we drop the requirement that tests appear in order. As long as all test numbers between 1 and the plan are encountered, and only once, that should be considered a pass. It should not matter if 2 comes after 1.

See #2 for details of how much of a difference this will make for streaming subtests.

[jonathanKingston]

👍 from me on this one.

[Leont]

TAP has historically supported unnumbered test, so if you'd really want to go there, that seems like a more obvious route.

That said, I don't think this really solves any problem. Atomicity guarantees on write are fairly limited, for any kind of serious usage you'd want synchronization anyway.

[exodist]

Subtests are really the only place where it is essential, since synchronization is a bigger issue there. One thing that keeps popping up is that streaming is essential (Though I myself do not care about it). Once you have 2 subtests running synchronization means holding back results until the entire subtest completes, otherwise results inside each subtest can intermix which will confuse a lot of things.

The main issue comes down to if the subtest syntax is like this:

ok 1.1
ok 1.2
ok 1

The reason this is an issue is because you need to know the subtest number early, so if you have 2 parallel subtest streaming results, you need to assign them both numbers right away, and if they finish out of order you must wait before sending them, which "breaks streaming".

ok 1.1
ok 1.2
ok 2.1
ok 2.2
ok 2
ok 1.3
ok 1

That is an example, the first of the 2 concurrent subtests takes longer, but starts first, thus it gets the number 1, but number 2 finishes first. It is of course trivial to make it wait and render ok 2 after ok 1, but as we saw in the subtest ticket, people complain that this breaks streaming.

I would argue that requiring the results be in order is silly. What matters is that all results are rendered, being out of order really should not matter. It is an additional restriction that complicates code of both the producer, which must do extra work, and the consumer which must report when the order is wrong. This extra work might be minor, but it is still extra, and I see no tangible gain.

[kinow]

I would argue that requiring the results be in order is silly.

Good point. Though sometimes users might want to keep tests in order too. For instance, when you have tests that share state (yickes, I know). Or for the sake of clarity (I'd probably implement a feature to display tests and subtests in order in Jenkins TAP Plug-in). But TAP implementations can easily handle that through parameterization.

I still don't have a firmly based opinion on subtests and test ordering in TAP. But I'd like to explore the idea of leaving the concurrency issues for implementations.

We could define that the specification does not guarantee that tests will be in order. That for streaming and concurrent tests, tests may be out of order. This way we can have implementations that work with concurrent tests and streaming, and others that don't support. But all adhering to the specification.

[jonathanKingston]

My worry with having two implementations (ordered and unordered) we are making consumers have to cope with both.

[beatgammit]

My worry with having two implementations (ordered and unordered) we are making consumers have to cope with both.

A consumer that can cope with unordered tests can, by definition, cope with ordered tests. I don't see the problem here.

I personally prefer ordered tests, so I think there should be some language in the specification that says that producers SHOULD provide a means of producing ordered test results. Whether this means running tests sequentially or having threads report to another thread that orders the results is up to the implementation (and neither is really that hard).

For unattended tests, it really doesn't matter on the format, so I agree that out-of-order results should also be acceptable.

[kinow]

My worry with having two implementations (ordered and unordered) we are making consumers have to cope with both.

Understandable. As I mentioned, I still don't have a strong opinion on which way would be better, so I'm worried with that as well.

I personally prefer ordered tests, so I think there should be some language in the specification that says that producers SHOULD provide a means of producing ordered test results.

@beatgammit +1

[marcbachmann]

I personally prefer ordered tests, so I think there should be some language in the specification that says that producers SHOULD provide a means of producing ordered test results.

IMO that's the responsibility of the reporter. A reporter knows which test started first and should reorder the events/test status if needed.