[SpecFlow+ LivingDoc] Make configuration of HTML result more flexible

Ivan Dolgov

August 09, 2021 21:14

Hi folks,

I've just finished integrating LivingDoc (LivingDoc Generator) into our project (about 4000 UI tests):

and here are some things that I am dissatisfied with:

1. It looks like almost everyone has retries in their test runs, and almost everyone has flaky tests; it will be cool if we could configure projects to see "warnings" or "passed" states for scenarios that passed at least once. For example, use specflow.json?
2. In our project, we use targets (described in .srprofile), but results are displayed as basic folders, and this is quite sad that I don't have the possibility to gather and analyze test run results as parts of these targets; We can also configure this thing in specflow.json
3. As part of targets, we use "stable" and "unstable" tags to define what test results we should take seriously and what tests we should fix/stabilize; for both purposes, we have 2 separated Teamcity configurations: stable runs, unstable runs; But all these tests are part of one test-assembly, and it is ok for "Living Documentation" page (it has filters), but it isn't okay for "Analytics" page since it has summed info for the whole assembly.
4. If one test was run twice or more - only one run displayed, same for pictures and additional info for each step, it could be cool to check results for other runs. The solution you made with outlines would be great(TestExecution.json contains info for other runs).
5. If a test fails in BeforeScenario or BeforeStep for the first step - we can see fail status without any information.

0

• 

  Andreas Willich

  August 11, 2021 07:28

  Proper support for Target and Retries from the SpecFlow+ Runner isn't implemented in SpecFlow+ LivingDoc yet, because we didn't know how many people are using a combination of these features and LivingDoc. And it will be a lot of effort to implement it. It would have delayed the release of the test results features for weeks.

  It's classical, the last 20% of a feature takes 80% time.

  As a workaround at the moment, I would split the stable and unstable scenarios into two different executions, generate two TestExecution.jsons, and generate two LivingDocs.

   

  Point 5 looks like a bug, could you open a ticket for this? Please attach a repro project to make it quicker for us to find the reason for it.

  0
• 

  Karol Czechowski

  October 28, 2021 10:12

  Hi Ivan Dolgov and Andreas Willich
  Regarding point number 1, I didn't want to create a new feature request so I can write here, that this is probably highly demanded feature. In SpecFlow+ Runner it's very useful, that re-run scenario is marked as passed in the end (with orange bar) and in LivingDoc it's still failed and marked red. This requires extra time and investigation to verify that it has actually passed in the end. 
  Summarizing, test results for the same scenario should be overwritten (filtered by timestamp) or merged how it is now - but the final status should be from the last run. It might be parametrized as Ivan mentioned, in specflow.json.
  For now this blocks us for using LD as the first-look report and We stay with SP+ Runner reports. 
  Good job anyway! I'm staying tuned!

  0

Please sign in to leave a comment.