[SpecFlow+ LivingDoc] Add an option to exclude scenarios without test results from the report
Right now, LivingDoc generates reports with all feature files and Gherkin scenarios. It can generate reports with or without test results. Whenever certain scenarios are not executed, LivingDoc will report them as not executed with "gray" balls (instead of green or red). This behavior is good in general cases because it allows partial results to be reported. It also reveals scenarios that are not executed for transparency.
However, this does not scale well for large SpecFlow projects. For example, my current project has nearly 1400 unique scenarios. Sometimes, I run the entire test suite, in which the current LivingDoc behavior is fine. However, many times, I run a subset of the test suite, such as running a dozen "smoke" tests or a hundred "continuous" tests. I use tags to filter tests at execution time. When I open the generated LivingDoc report for these shorter runs, I still get the entire test suite of 1400 scenarios, and the majority of the results are gray because they were not executed. This presents a "wall of text."
It would be much better to have a report that includes only the scenarios that were executed. That way, when I run my smoke tests, I'll see only those 12 smoke tests in the report.
I know the simple workaround would be to use a tag filter in the report to make it show only the desired tests. However, this does not satisfy my use case. I want to give these reports to people who don't know the tag filters. I want to simply share a LivingDoc report and say, "These were the tests I ran for such-and-such configuration on such-and-such date." I want the report file itself to not include unnecessary tests for those recipients. Furthermore, some of the people with whom I intend to share these reports are my customers, and I don't want them to see scenarios that are inapplicable for their configurations. For example, I don't want Customer A to have any way to see scenarios specific to Customer B.
I know another workaround could be for me to generate a LivingDoc JSON report without test results (FeatureData.json), run my own script to remove inapplicable scenarios, and then generate the LivingDoc HTML with the test results. However, I think it would be much nicer to simply have a command line option that does this for me automatically. The command line option could be a Boolean flag named "--include-executed-scenarios-only" or something similar. (I know that suggestion is a long name.) I presume I'm not the only tester with enormous suites that frequently get filtered.
Thank you for considering this feature request! Much love for SpecFlow, from the Automation Panda. :D
Please sign in to leave a comment.