Working in a large company with development teams working with all kinds of different development and test tool chains, we are facing the issue that a lot of interfaces within those tool chains are closed off. This forces us to support a diverse set of tools, and frustrate our internal clients with a wide variety of reports - typically structured and visualized very different from each other - that are practically impossible to unify.
As we have explored different options on handling Living Documentation, the Specflow+ LivingDoc CLI has the advantage of being able to use inputs that are very simple to understand and create: feature files are feature files, and the TestExecution.json file is human readable and concise. This inspired us to try and convert the test results from other tool chains into something the Specflow+ LivingDoc CLI tool would understand... And with relatively low effort, we succeeded.
The problem is that since the content in TestExecution.json is assumed to be automatically generated through the Specflow.Plus.LivingDoc plugin, there is no information available on what constitutes as a valid TestExecution.json file, what specific properties inside the file mean, what values of said properties are syntactically / semantically accepted, and what repercussion those values have on the reporting.
I am aware there are conversations ongoing to create a unified interface across all tools, but as is often the case, the lead time on those types of convergent actions means I won't expect a mature implementation that is widely supported by different toolchains any time soon.
Until then, exposing the existing TestExecution.json interface would at least provide us with the option to choose Specflow+ LivingDoc as our test reporting solution, regardless of what toolchain the results are coming from. And seeing that it is the interface of the CLI tool, I'd expect this interface to be already described - just not exposed to the public.
Please sign in to leave a comment.