Merging multiple testExecution.json produce incorrect result

I am using spec flow liv doc in my automation project. We have designed our framework in a way that it reruns the failed test twice to check if they are flaky or not. S ao to generate multiple TestExecution.json files I use livingdocgenerator TestExecution_{ProcessId}_{ThreadId} as mentioned in specflow liv doc.

At first run, I get TestExecution_{1}.json and it has all results pass + fail, say 5 tests - 3 pass and 2 fail

Now on second attempt my 2 failed tests case are executed again, and this time get the same result, again 2 failed, but another TestExecution_{2}.json is generated

Now on third attempt, again my 2 test cases are run and but this time they got passed and got generated TestExecution_{3}.json

In last I merge these files using below command

livingdoc test-assembly Test.dll -t TestExecution_*.json which generates the final livdoc file, but this doc has incorrect results, it says total 5 scenarios out of which 5 passed, 2 failed, and in actually I had total 5 scenarios

Merging multiple testExecution.json produce incorrect result

Comments

Didn't find what you were looking for?