Azure DevOps SpecFlow+LivingDoc publish report from TestExecution.json

Answered

Domenico Del Vecchio

• July 13, 2022 13:37

Hi

 

We are facing an issue generating and publishing the test result report in Azure DevOps.

Our current workflow is to build .net projects and run tests within a docker container. Therefore, we use the LivingDoc CLI tool to generate the TestExecution.json file. This is all good. We also export the file out of the container so that it is available in the Azure DevOps pipeline.

The last step is a SpecFlowPlus@0 task in Azure DevOps. This idea (maybe we got it wrong and it is not possible) is to use the already generated TestExecution.json, together with the feature folder to publish a report to the specflow hub.

Unfortunately, the task ends in error (TargetInvocationException and NullReferenceException). We have tried with different versions of dotnet core on the build agent but same error. The yaml task definition looks like the one below.

Any help would be greatly appreciated.

Thank you

Dom

- task: SpecFlowPlus@0

            inputs:

              generatorSource: 'FeatureFolder'

              projectFilePath: '<path to the source feature folder>'

              projectName: '<our project name>'

              testExecutionJson: '<path of TestExecution.json>'

              projectLanguage: 'en'

0

• 

  Andreas Willich

  • July 14, 2022 09:32

  Can you post the livingdoc cli call that runs in your docker container? Because the TestExecution.json is generated by the LivingDoc SpecFlow plugin and not the cli tool. So I am a little bit unsure what is really happening.

  0
• 

  Domenico Del Vecchio

  • July 14, 2022 09:45

  Hi Andreas.

  These are the steps we run inside the container.

  TestExecution.json is generated (in fact, the html report can be generated in the same way) and then it is copied outside the container with a COPY command.

  We can see the generated TestExecution.json file on the build agent, so it is available for the Azure DevOps pipeline task.

  RUN dotnet add package SpecFlow.Plus.LivingDocPlugin
  
  RUN dotnet build "<our project file>" -c Release -o /app/build
  
  # run tests on docker build
  
  RUN dotnet test
  
  #SpecFlow LivingDoc generation
  
  RUN dotnet tool install --tool-path ~/bin SpecFlow.Plus.LivingDoc.CLI
  
  RUN ~/bin/livingdoc feature-folder "Features" --output /src/livingdoc/TestExecution.json

  0
• 

  Daniel Melao

  • July 28, 2022 09:16

  Hi,

  I'm facing a similar situation. Can you confirm it's possible to generate the TestExecution.json (inside the container where tests run) and then export it so that it's available in the Azure DevOps pipeline?

  This way the json is not being generated by the plugin, but by the living doc cli inside the container. So I'm not sure this will work. At the moment I'm getting the same error described in the initial post (TargetInvocationException and NullReferenceException).

  Thanks,

  Daniel

  0
• 

  Andreas Willich

  • July 28, 2022 14:12

  Domenico Del Vecchio & Daniel Melao

  The TestExecution.json is generated by the SpecFlow.Plus.LivingDocPlugin NuGet package. And only by that.

  It is never generated by the LivingDoc CLI tool.

  The TestExecution.json is always an input for the livingdoc CLI. The parameter for this is "-t". You can't use it as output.

  Also the folder has to be the root folder of your project. Not the folder of your feature files. If this is not correct, we don't can match the test results to the correct scenario.

   

  What you want is to generate a FeatureData.json, that can then be used in the Azure DevOps task.

  The parameter for it is --output-type JSON. This json-file includes everything. You can copy it then out of the container.

  0
• 

  Domenico Del Vecchio

  • August 01, 2022 08:33

  Thanks Andreas. It is working now.

  0
• 

  Andreas Willich

  • August 01, 2022 08:39

  Awesome!

  0

Please sign in to leave a comment.