Last cycle we started talking about having a test spec to describe what a good interoperability test is and how to make tempest tests better for interoperability purposes. During the time it took to write the spec and get it approved I realised why OpenStack developers were confused about the requirements: it took us almost three months of meaningful discussions to agree on what the recommendations for good interoperability tests for OpenStack are.
Today we have finally reached a consensus and the test spec made it into the DefCore repository.
The DefCore test spec can be found here.
Thanks everyone for the reviews! This document is work in progress but it clarifies a set of rules that were implicit and controversial before, which is a step in the right direction.