Specflow + Living Docs reports dont match up?

Antonio Chiong

• January 07, 2022 21:44

Our net5 solution with

<PackageReference Include="SpecFlow.Plus.LivingDocPlugin" Version="3.9.57" />
<PackageReference Include="SpecRun.SpecFlow" Version="3.9.31" />

Configured to run in Azure Devops with Dotnet Test  but the results are not the same:

From the console output of the dotnet Test

Attachments:

C:\windows\ServiceProfiles\NetworkService\AppData\Local\Temp\kvssf3wi.yjm\xxxxx Web API Tests_2022-01-07T161511.log

C:\windows\ServiceProfiles\NetworkService\AppData\Local\Temp\kvssf3wi.yjm\TestResults_2022-01-07T162549.html

Failed! - Failed: 4, Passed: 140, Skipped: 0, Total: 144, Duration: 10 m 30 s - Tests.dll (net5.0)

 

from Living Documents generated 

From the Html report created after run:

 

So Failed tests are not showing in Living Docs as failed but as "others", and some passed tests are showing as "others" as well. Does anyone run into this?

0

• 

  Jordan Western

  • January 12, 2022 12:24

  Hey Antonio Chiong,

  Can you provide some details of the tests that are showing as 'others' in your execution? Anything interesting in the logs? (maybe send them over to info@specflow.org if you can)

  Is the same true when running the livingdoc CLI from your local machine, maybe you can try it? I would be interested to see the testexecution.json file generated by SpecFlow.Plus.LivingDocPlugin. If you try, it would be good for you to send this to me also.

  Thanks, Jordan.

  0
• 

  Antonio Chiong

  • January 13, 2022 15:24

  Hi Jordan,

  I just run it locally and I was watching the testexecution.json and while running it seems to be collecting all the data but almost at the end from let's say from 30k file goes back to like 6k. 

  I email you the results of my local run

  0
• 

  Jordan Western

  • January 14, 2022 08:38

  Hi Antonio Chiong,

  If you are running tests in parallel, then you will need to generate multiple testexecution.json files and then squash them. Otherwise the results will be overwritten as far as I understand it.

  please see this link for generating multiple json files per thread - https://docs.specflow.org/projects/specflow-livingdoc/en/latest/LivingDocGenerator/Setup-the-LivingDocPlugin.html?highlight=testexecution.json 

  and this link for merging the results - https://docs.specflow.org/projects/specflow-livingdoc/en/latest/Guides/Merging-Multiple-test-results.html

  Please let me know if this does no solve your issue.

   

  0
• 

  Antonio Chiong

  • January 18, 2022 21:57

  Yes it did thanks so much

  0

Please sign in to leave a comment.