Skip to main content

Ability to see historical test runs in Specflow + Living Doc

Comments

12 comments

  • Mario Steiner

    Hi Tim, thanks a lot for your request. Do you have an example of how you would imagine the view of the historical test runs? Are you using any other reporting tool for that right now + are you sharing the information with people in your team or outside of your team?

    0
  • Reece, Tim

    Hi Mario

    I currently have a series of function that copies test (Feature/Scenario and Step) details and results into nested dictionaries and then parse these through into json, xml and html files at the end of each test run. The results are copied into a shared folder with a heirarchy of test run/company/browser. We are using Azure Devops Standalone server and the build/release pipelines for the installation and then test run with the Visual Studio Test task. The tests themselves are written in MSTest/Selenium/Specflow

    I then have a python page on our devops server that scrapes the most recent folders and displays the summary results

    This page has a link to the individual results from the html file created within the test runs so they can be viewed individually

    It's very procedural, and very tailored to our environment, but it gives the team and managers an overview of where the tests/system is at in real time. The above shows the release branch only. I did have results shown for the nightly trunk build we do, but, when the last third party reporting tool I was using broke, I began writing my own reports as described above and have not got around to including those builds back into the python file. They used to sit cross the top of the page with a very brief result and link, and only showed the last 5 days worth in one row.

    0
  • Mario Steiner

    very interesting, thanks a lot for this detailed example and explanations. Which reporting tool did you use before? Are the 5 days sufficient for you or do you have the need/requirement to look back way further into the past? e.g last 30 days, 60 days, ... Do you need to report historical test runs and/or are using them for analysis and code quality improvements? 

     

    0
  • Reece, Tim

    Hi Mario

    5 Days history is usually plenty, 1 or two is normal, but I have on occasion gone back a few weeks to track down when things went wrong, in those cases it was when a test was missing a bug and why. It is also a nice way to show to the managers the growth of the system over time, and the expanding coverage and improvements the tests provide. It's a good advertisement for my job :)

    As I said, I am lucky in that we store results in a flat file system, so they are easily accessed.

    I was using Extent Reports, which was a bit of a hassle to begin with, and dragged in a lot of additional packages with NuGet that started to become very unwieldy. The overhead on it became too much though, after some update caused hassles elsewhere in the system, so I abandoned it when I noticed you had Specflow + Living Doc. There was another BDD reporintg tool I'd used earlier which was far superior, but had been abandoned by the developers, which was a shame.

    I guess it's strength was that they both created standalone html files that I could then copy to our results folders along with my own results, which led me to doing it the way I have. Perhaps if you could do something like that with Living Doc I could be persuaded to use it again. People can do with the results as they please then, as I understand everyone's development/test pipeline will differ a lot?

    0
  • Reece, Tim

    Having just looked through other Feature Requests, it appears these two are asking similar things.

    https://support.specflow.org/hc/en-us/community/posts/360014231838--SpecFlow-LivingDoc-Have-Specflow-reports-auto-generate-when-running-test-from-VS-IDE

    https://support.specflow.org/hc/en-us/community/posts/360014202838--LivingDoc-CLI-tool-Generate-report-from-multiple-input-json-files-from-different-executions-e-g-different-browsers-different-tags

    Just he ability to (automatically) download a html file of the test report at the conclusion of the tests will be all most people need I think?

    0
  • Mario Steiner

    can you remember the name of the abandoned BDD reporting tool? Yes you are right, the development/testing pipelines are mostly quite individual. We are currently trying to analyze all the great feedback we received recently and think about how we can incorporate it into our product roadmap. We will keep you updated.

    0
  • Karol Czechowski

    Hi guys, I can share with you my approach for historic test results and how I would like to see it with LivingDoc in the future.
    Currently, each pipeline with tests in Azure DevOps generates HTML report (from SpecRunner) which is copied to Blob Storage and each report has unique name. This is the way I can go through the historic test results. I do the same with LivingDoc, in parallel, however there is no way, afaik, to create custom report name, that is why report is being overwritten each time it generates so I am not able to see the old files. Conclusion is, if there was a way to generate LivingDoc.html with unique name, similar to Specflow+ Runner, that would solve the issue. (e.g. outputName="LivingDoc_{unique_guid}.html")

    0
  • Mario Steiner

    Hi Karol, thanks for your feedback and for sharing your workflow for historic test results with us. It is possible to manually rename your report (name of the LivingDoc) with the following CLI command:

    livingdoc feature-folder C:\Work\MyProject.Specs --output C:\Temp\MyReport.html

    this would at least allow renaming your LivingDoc to e.g. LivingDoc17052021

    Link to the docs: Using the CLI tool — SpecFlow+ LivingDoc documentation

    we are currently evaluating our options on how to provide an option for historic test results in the future. We will keep you updated.

    0
  • Karol Czechowski

    Hi Mario, your hint works perfect for me! Now each run creates a report with suffix:

     LivingDoc_$(Build.BuildNumber).html

    what allows me to have unique reports stored in my Blob Storage.
    I create direct url to such a file and display it in the "Extensions" tab in pipeline summary view. 
    Thanks!
    P.S. Now I wait for the documentation update to implement Test Output in LD reports ;)

    0
  • Andreas Willich

    Ali Mollahosseini is working on them right now. Should be available this week.

    2
  • Raman Tsitou

    Looks like the feature is still not available :(

    0
  • Andreas Willich

    Raman Tsitou I meant that Ali is working on the Documentation. We are not working on this feature. You can find our roadmap at https://docs.specflow.org/en/latest/roadmap.html

    0

Please sign in to leave a comment.

Powered by Zendesk