SpecFlow report after running multiple tasks

I have setup SpecFlow reporting to run after multiple tasks. The issue is that it generates the results for the last task only and all the other tasks output/results are lost. How do we remedy this situation?

Has anyone configured such a project or run into a similar issue? We cannot run them all under the same task as they take too long to run and we want only some to be run parallelly, while others run with different parameters, and hence the different tasks across each with tags to separate the tests and tasks.

The pipeline file is running on Azure DevOps and is provided below for reference:

trigger:
  - master
pool:
  name: Hosted Windows 2019 with VS2019
  vmImage: windows-latest
  demands: vstest
schedules:
  - cron: '00 06 * * MON,TUE,WED,THU,FRI'
    displayName: Weekdays 4PM build
    branches:
      include:
        - master
        - releases/*
      exclude:
        - development
        - feature/*
jobs:
  - job: 'ParallelTesting'
    strategy:
      parallel: 3
    displayName: Run tests in parallel
    timeoutInMinutes: 120
    cancelTimeoutInMinutes: 10
    pool:
      vmImage: windows-latest
    steps:
      - task: PowerShell@2
        displayName: Clone the pipelines repository
        inputs:
          targetType: inline
          script: >
            git clone -b master
            https://azuredevops:$($env:token)@dev.azure.com/proj/aproj/_git/build
            pipeline-repo/
        env:
          token: $(System.AccessToken)
      - task: UseDotNet@2
        displayName: Using SDK Version (3.1.x)
        inputs:
          version: 3.1.x
      - task: DotNetCoreCLI@2
        displayName: Restore packages
        inputs:
         command: restore
         feedsToUse: select
         vstsFeed: project
      - task: DotNetCoreCLI@2
        displayName: Build the solution
        inputs:
          command: build
          projects: '**/*.sln'
      - task: DotNetCoreCLI@2
        displayName: Publish
        inputs:
          command: publish
          publishWebProjects: false
          projects: '**/*.sln'
          arguments: '--configuration Release -o $(build.artifactstagingdirectory)\MyProject.Qa'
          zipAfterPublish: false
          modifyOutputPath: false
      - task: PublishBuildArtifacts@1
        displayName: Publish Artifact
        inputs:
          PathtoPublish: $(build.artifactstagingdirectory)
        condition: succeededOrFailed()
      - task: VSTest@2
        displayName:  Run 1st Test Set
        continueOnError: true
        inputs:
          platform: '$(buildPlatform)'
          configuration: '$(buildConfiguration)'
          testSelector: 'testAssemblies'
          testAssemblyVer2: | 
            **\[M|m]y[P|p]roject.[Q|q]a.[S|s]pecifications.dll
            !**\*TestAdapter.dll
            !**\obj\**
          searchFolder: '$(System.DefaultWorkingDirectory)'
          pathToCustomTestAdapter: '**\*TechTalk.SpecRun.VisualStudio.TestAdapter.dll'
          testFiltercriteria: 'TestCategory=authentication'
          uiTests: true
          vsTestVersion: 'latest'
          testRunTitle: 'SpecFlow BDD Scenarios for Authentication'
          diagnosticsEnabled: false
          rerunFailedTests: false
          rerunType: 'basedOnTestFailurePercentage'
          rerunFailedThreshold: '70'
          rerunFailedTestCasesMaxLimit: '2'
          rerunMaxAttempts: '2'
      - task: VSTest@2
        displayName:  Run 2nd Test Set
        continueOnError: true
        inputs:
          platform: '$(buildPlatform)'
          configuration: '$(buildConfiguration)'
          testSelector: 'testAssemblies'
          testAssemblyVer2: | 
            **\[M|m]y[P|p]roject.[Q|q]a.[S|s]pecifications.dll
            !**\*TestAdapter.dll
            !**\obj\**
          searchFolder: '$(System.DefaultWorkingDirectory)'
          testFiltercriteria: 'TestCategory=dashboard'
          uiTests: true
          vsTestVersion: 'latest'
          testRunTitle: 'SpecFlow BDD Scenarios for Dashboard'
          diagnosticsEnabled: false
          rerunFailedTests: false
          rerunType: 'basedOnTestFailurePercentage'
          rerunFailedThreshold: '70'
          rerunFailedTestCasesMaxLimit: '2'
          rerunMaxAttempts: '2'
      - task: VSTest@2
        displayName: Run 3rd Test Set
        continueOnError: true
        inputs:
          platform: '$(buildPlatform)'
          configuration: '$(buildConfiguration)'
          testSelector: 'testAssemblies'
          testAssemblyVer2: | 
            **\[M|m]y[P|p]roject.[Q|q]a.[S|s]pecifications.dll
            !**\*TestAdapter.dll
            !**\obj\**
          searchFolder: '$(System.DefaultWorkingDirectory)'
          testFiltercriteria: 'TestCategory=other_tests'
          uiTests: true
          vsTestVersion: 'latest'
          testRunTitle: 'SpecFlow BDD Scenarios for Other Pages'
          diagnosticsEnabled: false
          rerunFailedTests: false
          rerunType: 'basedOnTestFailurePercentage'
          rerunFailedThreshold: '70'
          rerunFailedTestCasesMaxLimit: '2'
          rerunMaxAttempts: '2'
      - task: SpecFlowPlus@0
        displayName: Generate SpecFlow+ LivingDoc
        inputs:
          projectFilePath: 'MyProject.Qa.Specifications\FeatureFiles'
          projectName: 'MyProject.Qa.Specifications'
          testExecutionJson: 'MyProject.Qa.Specifications\bin\Debug\netcoreapp3.1\TestExecution.json'
          projectLanguage: 'en'
          workItemPrefix: AUTO
          bindingAssemblies: 'MyProject.Qa.Specifications\bin\Debug\netcoreapp3.1\MyProject.Qa.Specifications.dll'

Andreas Willich

March 09, 2021 07:52

We are going to write some documentation for this use case in the next iteration, but until we published it, here are the steps you need to do:

1. Configure a custom TestExecution.json filename in the specflow.json file - see https://docs.specflow.org/projects/specflow-livingdoc/en/latest/LivingDocGenerator/Setup-the-LivingDocPlugin.html

2. Change the parameter `testExecutionJson` to a glob pattern to catch all your testexecution.json files - see https://docs.specflow.org/projects/specflow-livingdoc/en/latest/Generating/Configuring-the-Build-Step-in-YAML.html#livingdoc-specific-parameters-inputs

So you change it in the YAML to something like this:

testExecutionJson: 'MyProject.Qa.Specifications\bin\Debug\netcoreapp3.1\TestExecution*.json'

Karol Czechowski

April 13, 2021 15:52

Hi Andreas Willich, in response to your comment on jetbrains youtrack thread, I found this topic the most relevant to my question from there. I am having similar case as above, I would like to keep one LivingDoc status from the different Pipelines with tests. I have multiple pipelines and want to see results in one LivingDoc place. On the other hand, I don't mind having more than one LD, preferably accessible straight from Pipeline results(and showing only the tests I run). I think it's a bit general question how to handle documentation when having much more tests, pipelines, environments etc.? :)

April 15, 2021 08:57

Karol Czechowski What exactly do you mean with multiple pipelines? Are they different definitions for the same project in the same repo? Are they for completely different projects?

Or do you simply slice your tests of one project into different pipelines and need to merge it together again?

April 15, 2021 09:25

Andreas Willich I have several pipelines where each of them runs only few test scenarios. All is in one project, one repo. By doing this I want to make pipelines more stable, get a faster feedback and run them selectively by some categories.

April 16, 2021 11:52

This scenario is supported. You need to collect all TestExecution.json from your pipelines and you can specify multiple TestExecution.jsons on the CLI and the Build task.

Like this:

C:\work\BookShop\BookShop.AcceptanceTests\bin\Debug\netcoreapp3.1>livingdoc test-assembly BookShop.AcceptanceTests.dll -t TestExecution_1.json TestExecution_2.json

SpecFlow report after running multiple tasks

Comments

Didn't find what you were looking for?