SpecFlow report after running multiple tasks
I have setup SpecFlow reporting to run after multiple tasks. The issue is that it generates the results for the last task only and all the other tasks output/results are lost. How do we remedy this situation?
Has anyone configured such a project or run into a similar issue? We cannot run them all under the same task as they take too long to run and we want only some to be run parallelly, while others run with different parameters, and hence the different tasks across each with tags to separate the tests and tasks.
The pipeline file is running on Azure DevOps and is provided below for reference:
trigger:
- master
pool:
name: Hosted Windows 2019 with VS2019
vmImage: windows-latest
demands: vstest
schedules:
- cron: '00 06 * * MON,TUE,WED,THU,FRI'
displayName: Weekdays 4PM build
branches:
include:
- master
- releases/*
exclude:
- development
- feature/*
jobs:
- job: 'ParallelTesting'
strategy:
parallel: 3
displayName: Run tests in parallel
timeoutInMinutes: 120
cancelTimeoutInMinutes: 10
pool:
vmImage: windows-latest
steps:
- task: PowerShell@2
displayName: Clone the pipelines repository
inputs:
targetType: inline
script: >
git clone -b master
https://azuredevops:$($env:token)@dev.azure.com/proj/aproj/_git/build
pipeline-repo/
env:
token: $(System.AccessToken)
- task: UseDotNet@2
displayName: Using SDK Version (3.1.x)
inputs:
version: 3.1.x
- task: DotNetCoreCLI@2
displayName: Restore packages
inputs:
command: restore
feedsToUse: select
vstsFeed: project
- task: DotNetCoreCLI@2
displayName: Build the solution
inputs:
command: build
projects: '**/*.sln'
- task: DotNetCoreCLI@2
displayName: Publish
inputs:
command: publish
publishWebProjects: false
projects: '**/*.sln'
arguments: '--configuration Release -o $(build.artifactstagingdirectory)\MyProject.Qa'
zipAfterPublish: false
modifyOutputPath: false
- task: PublishBuildArtifacts@1
displayName: Publish Artifact
inputs:
PathtoPublish: $(build.artifactstagingdirectory)
condition: succeededOrFailed()
- task: VSTest@2
displayName: Run 1st Test Set
continueOnError: true
inputs:
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
testSelector: 'testAssemblies'
testAssemblyVer2: |
**\[M|m]y[P|p]roject.[Q|q]a.[S|s]pecifications.dll
!**\*TestAdapter.dll
!**\obj\**
searchFolder: '$(System.DefaultWorkingDirectory)'
pathToCustomTestAdapter: '**\*TechTalk.SpecRun.VisualStudio.TestAdapter.dll'
testFiltercriteria: 'TestCategory=authentication'
uiTests: true
vsTestVersion: 'latest'
testRunTitle: 'SpecFlow BDD Scenarios for Authentication'
diagnosticsEnabled: false
rerunFailedTests: false
rerunType: 'basedOnTestFailurePercentage'
rerunFailedThreshold: '70'
rerunFailedTestCasesMaxLimit: '2'
rerunMaxAttempts: '2'
- task: VSTest@2
displayName: Run 2nd Test Set
continueOnError: true
inputs:
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
testSelector: 'testAssemblies'
testAssemblyVer2: |
**\[M|m]y[P|p]roject.[Q|q]a.[S|s]pecifications.dll
!**\*TestAdapter.dll
!**\obj\**
searchFolder: '$(System.DefaultWorkingDirectory)'
testFiltercriteria: 'TestCategory=dashboard'
uiTests: true
vsTestVersion: 'latest'
testRunTitle: 'SpecFlow BDD Scenarios for Dashboard'
diagnosticsEnabled: false
rerunFailedTests: false
rerunType: 'basedOnTestFailurePercentage'
rerunFailedThreshold: '70'
rerunFailedTestCasesMaxLimit: '2'
rerunMaxAttempts: '2'
- task: VSTest@2
displayName: Run 3rd Test Set
continueOnError: true
inputs:
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
testSelector: 'testAssemblies'
testAssemblyVer2: |
**\[M|m]y[P|p]roject.[Q|q]a.[S|s]pecifications.dll
!**\*TestAdapter.dll
!**\obj\**
searchFolder: '$(System.DefaultWorkingDirectory)'
testFiltercriteria: 'TestCategory=other_tests'
uiTests: true
vsTestVersion: 'latest'
testRunTitle: 'SpecFlow BDD Scenarios for Other Pages'
diagnosticsEnabled: false
rerunFailedTests: false
rerunType: 'basedOnTestFailurePercentage'
rerunFailedThreshold: '70'
rerunFailedTestCasesMaxLimit: '2'
rerunMaxAttempts: '2'
- task: SpecFlowPlus@0
displayName: Generate SpecFlow+ LivingDoc
inputs:
projectFilePath: 'MyProject.Qa.Specifications\FeatureFiles'
projectName: 'MyProject.Qa.Specifications'
testExecutionJson: 'MyProject.Qa.Specifications\bin\Debug\netcoreapp3.1\TestExecution.json'
projectLanguage: 'en'
workItemPrefix: AUTO
bindingAssemblies: 'MyProject.Qa.Specifications\bin\Debug\netcoreapp3.1\MyProject.Qa.Specifications.dll'
-
This scenario is supported. You need to collect all TestExecution.json from your pipelines and you can specify multiple TestExecution.jsons on the CLI and the Build task.
Like this:
C:\work\BookShop\BookShop.AcceptanceTests\bin\Debug\netcoreapp3.1>livingdoc test-assembly BookShop.AcceptanceTests.dll -t TestExecution_1.json TestExecution_2.json
1 -
We are going to write some documentation for this use case in the next iteration, but until we published it, here are the steps you need to do:
1. Configure a custom TestExecution.json filename in the specflow.json file - see https://docs.specflow.org/projects/specflow-livingdoc/en/latest/LivingDocGenerator/Setup-the-LivingDocPlugin.html
2. Change the parameter `testExecutionJson` to a glob pattern to catch all your testexecution.json files - see https://docs.specflow.org/projects/specflow-livingdoc/en/latest/Generating/Configuring-the-Build-Step-in-YAML.html#livingdoc-specific-parameters-inputs
So you change it in the YAML to something like this:
testExecutionJson: 'MyProject.Qa.Specifications\bin\Debug\netcoreapp3.1\TestExecution*.json'
0 -
Hi Andreas Willich, in response to your comment on jetbrains youtrack thread, I found this topic the most relevant to my question from there. I am having similar case as above, I would like to keep one LivingDoc status from the different Pipelines with tests. I have multiple pipelines and want to see results in one LivingDoc place. On the other hand, I don't mind having more than one LD, preferably accessible straight from Pipeline results(and showing only the tests I run). I think it's a bit general question how to handle documentation when having much more tests, pipelines, environments etc.? :)
0 -
Karol Czechowski What exactly do you mean with multiple pipelines? Are they different definitions for the same project in the same repo? Are they for completely different projects?
Or do you simply slice your tests of one project into different pipelines and need to merge it together again?
0 -
Andreas Willich I have several pipelines where each of them runs only few test scenarios. All is in one project, one repo. By doing this I want to make pipelines more stable, get a faster feedback and run them selectively by some categories.
0
Please sign in to leave a comment.
Comments
5 comments