I've just finished integrating LivingDoc (LivingDoc Generator) into our project (about 4000 UI tests):
and here are some things that I am dissatisfied with:
- It looks like almost everyone has retries in their test runs, and almost everyone has flaky tests; it will be cool if we could configure projects to see "warnings" or "passed" states for scenarios that passed at least once. For example, use specflow.json?
- In our project, we use targets (described in .srprofile), but results are displayed as basic folders, and this is quite sad that I don't have the possibility to gather and analyze test run results as parts of these targets; We can also configure this thing in specflow.json
- As part of targets, we use "stable" and "unstable" tags to define what test results we should take seriously and what tests we should fix/stabilize; for both purposes, we have 2 separated Teamcity configurations: stable runs, unstable runs; But all these tests are part of one test-assembly, and it is ok for "Living Documentation" page (it has filters), but it isn't okay for "Analytics" page since it has summed info for the whole assembly.
- If one test was run twice or more - only one run displayed, same for pictures and additional info for each step, it could be cool to check results for other runs. The solution you made with outlines would be great(TestExecution.json contains info for other runs).
- If a test fails in BeforeScenario or BeforeStep for the first step - we can see fail status without any information.
Please sign in to leave a comment.