Specflow+ LivingDoc in Azure DevOps - unable to setup/have test execution results

January 15, 2021 15:00 Edited

I am trying to setup Specflow+ LivingDoc in Azure DevOps to show the test execution results.

I followed instructions in the doc, but I believe some part is missing in my setup. I keep seeing an error that the json file is not found (I believe it's refering to TestExecution.json or FeatureData.json)

I am also confused, I see at some places in the docs that the json file is TestExecutionResult.json, at other places TestExecution.json.

In the v3.4 to v3.5 migration guide it says that FeatureData.json was changed to TestExecution.json but even with the most recent SpecFlow versions (Azure DevOps extension v0.6.724 and nuget packages v3.5.14 and v3.5.286), I see in the pipeline logs that a CLI command is run and is passing an output parameter with "FeatureData.json", not "TestExecution.json". I have no place in my code or setup (that I know of) that sets this output parameter so I think it's defined by SpecFlow or the extension itself.

--output "D:\a\1\s\22\FeatureData.json"

I feel like with this recent feature to include test execution results in SpecFlow+ LivingDoc in Azure DevOps is incomplete, untested, bugus, or misses some helpful documentation.

Any help would be appreciated.

Thank you

Comments

27 comments

Official comment
Viktor Nemes

January 18, 2021 07:38
Hi Martin,

Sorry for the inconvenience!
Could you please open a support request at https://support.specflow.org/hc/en-us/requests/new with the full output of your LivingDoc build task?

Thanks in advance!
Sudhindra Kumar

January 25, 2021 10:28
Hi, I am also facing an issue when running tests in parallel. I have added specflow.json as below but only see a single TestExecution.json file & none of the test results appear correctly:

Is there a working example of Specflow+ LivingDoc in Azure DevOps considering parallel ?
```
{
  "livingDocGenerator": {
    "enabled": true,
    "filePath": "TestExecution_{ProcessId}_{ThreadId}.json"
  }
}
```
0
Viktor Nemes

January 25, 2021 11:19 Edited
Sudhindra Kumar you need to use the latest version of the LivingDoc plugin: 3.5.286 and also make sure that your specflow.json is copied to the output directory with setting the "Copy to Output Directory" property
1
Sudhindra Kumar

February 01, 2021 07:05
Thanks a ton Viktor Nemes. I now see that multiple TestExecution files are generated. How can i update my Test Execution JSON paths in ADO pipeline task so that it can point to multiple TestExecution.json files?
0
Viktor Nemes

February 02, 2021 06:56
You can use wildcards (glob patterns) when specifying the Test Execution JSON path e.g.
```
MyTestProject/bin/Debug/**/TestExecution_*.json
```
0

Sudhindra Kumar

February 03, 2021 05:28 Edited

Viktor Nemes: Its not working for me. I have checked the path & can see json files starting with TestExecution after the test finishes.

Starting: SpecFlow+ build step.

============================================================================== 

Task : SpecFlow+LivingDoc 

Description : SpecFlow+LivingDoc custom build step. 

Version : 0.6.739 


Author : techtalk 

Help : 

============================================================================== 

Error: No files are matching the pattern: D:\agent\_work\19\s\CP.SSR.APIAutomation\bin\Debug\**\TestExecution_*.json 

##[error]Error: Command failed: dotnet D:\agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.739\CLI\LivingDoc.CLI.dll feature-folder "D:\agent\_work\19\s\CP.SSR.APIAutomation" --project-language "en" --output "D:\agent\_work\19\s\65576\FeatureData.json" --output-type JSON --project-name "CP.SSR.APIAutomation" --work-item-prefix "WI:" --test-execution-json "D:\agent\_work\19\s\CP.SSR.APIAutomation\bin\Debug\**\TestExecution_*.json" --binding-assemblies "D:\agent\_work\19\s\CP.SSR.APIAutomation\bin\Debug\netcoreapp3.1\CP.SSR.APIAutomation.dll"

Error: No files are matching the pattern: D:\agent\_work\19\s\CP.SSR.APIAutomation\bin\Debug\**\TestExecution_*.json

My Nuget versions are

    <PackageReference Include="SpecFlow.Plus.LivingDocPlugin" Version="3.5.286" />
    <PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.5.14" />
    <PackageReference Include="SpecRun.SpecFlow" Version="3.5.22" />

Viktor Nemes

February 03, 2021 10:11
The path in the build task should be a relative path
```
CP.SSR.APIAutomation\bin\Debug\**\TestExecution_*.json
```
Are you using the classis UI or YAML to configure your build tasks?
Can you show how do you set the Test Execution JSON path parameter in the build task?
0
Sudhindra Kumar

February 03, 2021 11:12
Viktor Nemes: We are using Classic UI. Please see the screenshot below.

Here $(Parameters.project) is our .csproj & $(BuildConfiguration) is Debug
0
Viktor Nemes

February 03, 2021 11:36
You should try to remove the Build.SourceDirectory reference and only use
```
$(Parameters.project)\bin\$(BuildConfiguration)\**\TestExecution_*.json
```
1
Sudhindra Kumar

February 03, 2021 11:49
This works, Thanks much Viktor Nemes.
0
Karol Czechowski

March 22, 2021 17:11
Hi Viktor Nemes and Sudhindra Kumar!
I am actually facing the same issue. When I point a path to single file, then it works. When I have several files, wildcard is not working.
```
$(System.DefaultWorkingDirectory)/src/SeleniumTestsPipeline/bin/Release/netcoreapp3.1/SpecFlowTestExecution/TestExecution_*.json
```
This code doesn't work but when I provide: TestExecution.json works fine.

In living doc documentation is mentioned that:
To merge multiple JSON files in Azure Devops you have to include them in your build step in separate lines

https://docs.specflow.org/projects/specflow-livingdoc/en/latest/Guides/Merging-Multiple-test-results.html
0
Sudhindra Kumar

March 24, 2021 11:55 Edited
Karol Czechowski: Please check below parameters in your project

Test Execution JSON paths - $(project.name}\bin\$(BuildConfiguration)\**\TestExecution_*.json

Feature folder - $(project.name)

You can go to the location in your system where multiple TestExecution_*.json files are available and provide the path accordingly. Hope this helps.

Note: $(project.name} is my project/solution name
0
Krzysztof Cabaj

August 05, 2021 07:22 Edited
Hey Guys,

Thank you for your work and support!.

I am facing the same problem triggering test report locally:

Two TestExecution.json files are generated (I am using "filePath": "TestExecution_{Now}.json" in my specflow.json) and the are correctly generated in my ..\ApiTests\bin\Debug\netcoreapp3.1 folder so I've got:

TestExecution_20210805073352.json

TestExecution_20210805073825.json

And then when I use this command:

livingdoc feature-folder C:\Repos\Tests.API.BDD\Automation\ApiTests -t C:\Repos\Tests.API.BDD\Automation\ApiTests\bin\Debug\netcoreapp3.1\TestExecution_20210805073352.json TestExecution_20210805073825.json

Then I've got following error message:

Cannot find the specified test-execution-json: C:\Repose\Tests.API.BDD\Automation\TestExecution_20210805073825.json

OR

livingdoc feature-folder C:\Repos\Tests.API.BDD\Automation\ApiTests -t C:\Repos\Tests.API.BDD\Automation\ApiTests\bin\Debug\netcoreapp3.1\TestExecution_*.json

Then I've got following error message:

No files are matching the pattern: C:\Repose\Tests.API.BDD\Automation\ApiTests\bin\Debug\netcoreapp3.1\TestExecution_*.json

If I search for any of these two files separately then it's found.

Any tips?
0
Andreas Willich

August 05, 2021 07:22
Are you using the latest version of the LivingDoc CLI tool installed?
0
Krzysztof Cabaj

August 05, 2021 08:24 Edited
no I am using 3.9.30. When I attempt to update to 3.9.35 it fails:

Is 3.9.35 absolutely needed to make it work?
0
Andreas Willich

August 05, 2021 08:29
Ok, a 3.9.* version should be enough. The wildcards work.

But I see you are using PowerShell. Could you put the path in single quotes?

Like:

livingdoc feature-folder C:\Repos\Tests.API.BDD\Automation\ApiTests -t `C:\Repos\Tests.API.BDD\Automation\ApiTests\bin\Debug\netcoreapp3.1\TestExecution_*.json`

I think to remember that PowerShell is handling these wildcards sometimes a little bit different.
0
Krzysztof Cabaj

August 05, 2021 09:00 Edited
Unfortunately it doesn't work as well with single quotes.

No files are matching the pattern: C:\Repose\Tests.API.BDD\Automation\ApiTests\bin\Debug\netcoreapp3.1\TestExecution_*.json
0
Viktor Nemes

August 05, 2021 10:31 Edited
Wildcards are not supported when you are using absolute paths.

It should work with relative ones: (assuming you are in the folder C:\Repos\Tests.API.BDD\Automation\)
```
livingdoc feature-folder C:\Repos\Tests.API.BDD\Automation\ApiTests -t ApiTests\bin\Debug\netcoreapp3.1\TestExecution_*.json 
```
0

Mateusz Niezgoda (PGS Software)

August 12, 2021 14:08 Edited

Hi.

I'm having the same problem as Krzysztof above. In Azure, SpecFlowPlus@0 task is not able to locate the TestExecution json files when I try using a wildcard. I've checked directly on the server and the files are there:

the yaml configuration:

            - task: SpecFlowPlus@0
              displayName: 'SpecFlow+ LivingDoc'
              condition: succeededOrFailed()
              inputs:
                generatorSource: 'TestAssembly'
                testAssemblyFilePath: '$(Pipeline.Workspace)\BuildPipeline\${{lower(parameters.project)}}\${{parameters.dllName}}'
                projectName: '${{parameters.project}}'
                testExecutionJson: '${{lower(parameters.project)}}\TestExecution*.json'
                projectLanguage: 'en'

and I'm getting the same error as guys before me

Error: No files are matching the pattern: MyProject\TestExecution*.json 
##[error]Error: Command failed: dotnet "C:\Agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.946\CLI\LivingDoc.CLI.dll" test-assembly "C:\Agent\_work\2\BuildPipeline\MyProject\MyProject.Bdd.Tests.dll" --output-type JSON --test-execution-json "MyProject\TestExecution*.json" --include-test-output "All" --output "C:\Agent\_work\2\s\114013\FeatureData.json" --project-name "MyProject" --project-language "en"

As you can see, I've already used the relative path as you suggested in previous comments, it didin't help.

There is no problem when I point to a single TestExecution file without using a wildcard.

I've also tried using FeatureFolder instead of TestAssembly but with the same result:

yaml

            - task: SpecFlowPlus@0
              displayName: 'SpecFlow+ LivingDoc'
              condition: succeededOrFailed()
              inputs:
                generatorSource: 'FeatureFolder'
                projectFilePath: '$(Pipeline.Workspace)\BuildPipeline\${{lower(parameters.project)}}'
                projectName: '${{parameters.project}}'
                testExecutionJson: '\**\TestExecution*.json'
                projectLanguage: 'en'

error

Error: No files are matching the pattern: \**\TestExecution*.json 
##[error]Error: Command failed: dotnet "C:\Agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.946\CLI\LivingDoc.CLI.dll" feature-folder "C:\Agent\_work\2\BuildPipeline\MyProject" --output-type JSON --test-execution-json "\**\TestExecution*.json" --include-test-output "All" --output "C:\Agent\_work\2\s\114017\FeatureData.json" --project-name "MyProject" --project-language "en"

Any idea what am I missing there?

Krzysztof Cabaj

August 25, 2021 07:06 Edited

Hi guys,

It looks like wild cards used for selecting many TestExecution.json files don't work correctly on AzureDevOps for me as well.

For stage setup (yaml):

- stage: Publish_Results
  jobs:
  - job:
    displayName: "Print Test Reports"
    steps:
      - checkout: self

      - task: DownloadPipelineArtifact@2
        displayName: "Download pipeline artifacts"
        inputs:
          buildType: "current"
          targetPath: "$(Pipeline.Workspace)"

      - task: UseDotNet@2
        displayName: "Use DotNet Core version 3.1.x"
        inputs:
          packageType: "sdk"
          version: "3.1.x"

      - task: SpecFlowPlus@0
        condition: succeededOrFailed()
        timeoutInMinutes: 10
        inputs:
          command: "custom"
          custom: "livingdoc"
          arguments: "feature-folder"
          generatorSource: 'FeatureFolder'
          projectFilePath: 'Automation/ApiTests'
          projectName: 'Tests.Api.BDD'
          testExecutionJson: |
            "$(Pipeline.Workspace)/Bdd.TestExecution/client1/TestExecution_*.json" |
            "$(Pipeline.Workspace)/Bdd.TestExecution/client2/TestExecution_*.json" 
          projectLanguage: 'en'

error:

No files are matching the pattern: /home/vsts/work/1/Bdd.TestExecutio/client1/TestExecution_*.json

For single TestExecution.json file and analogous configuration works fine:

feature-folder "/home/vsts/work/1/s/Automation/ApiTests" 
--output-type JSON 
--test-execution-json "/home/vsts/work/1/Bdd.TestExecution/client1/TestExecution.json" 
--include-test-output "All" 
--output "/home/vsts/work/1/s/22593/FeatureData.json" 
--project-name "Tests.Api.BDD" 
--project-language "en"

I also tried to merge two TestExecution.json files with the same name. Each of them belongs to different folder like below:

feature-folder "/home/vsts/work/1/s/Automation/ApiTests" 
--output-type JSON 
--test-execution-json "/home/vsts/work/1/Bdd.TestExecution/client1/TestExecution.json" | "/home/vsts/work/1/Bdd.TestExecution/client2/TestExecution.json" --include-test-output "All" 
--output "/home/vsts/work/1/s/22604/FeatureData.json" 
--project-name "Tests.Api.BDD" 
--project-language "en"

but then I'm getting another message like:

/bin/sh: 1: /home/vsts/work/1/Bdd.TestExecution/client2/TestExecution.json "Permission denied"

any ideas what can be wrong?

Jordan Western

August 25, 2021 14:04
Hi Mateusz Niezgoda (PGS Software) and Krzysztof Cabaj,

Can you try with the relative path to the test TestExecution.json including the root folder (below is the example) -

https://docs.specflow.org/projects/specflow-livingdoc/en/latest/Generating/Configuring-the-Build-Step-in-YAML.html#arguments

Please let me know how you get on.
```
- task: SpecFlowPlus@0
displayName: LivingDoc with TestExecution.json files

inputs:
generatorSource: 'TestAssembly'
testAssemblyFilePath: BookShop.AcceptanceTests\bin\Debug\netcoreapp3.1\BookShop.AcceptanceTests.dll
testExecutionJson: BookShop.AcceptanceTests\**\*TestExecution*.json
```
0

Mateusz Niezgoda (PGS Software)

August 26, 2021 08:49

Hi Jordan

Unfortunately, it didn't help. I've already tried that before and each time, regardless what I put as a root folder, I'm still seeing the same error "No files are matching the pattern:"

Below you can see few logs examples:

1. yaml

testAssemblyFilePath: '$(Pipeline.Workspace)\BuildPipeline\${{lower(parameters.project)}}\${{parameters.dllName}}'
testExecutionJson: '$(Pipeline.Workspace)\**\*TestExecution*.json'

logs

Error: No files are matching the pattern: C:\Agent\_work\2\**\*TestExecution*.json 
##[error]Error: Command failed: dotnet "C:\Agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.946\CLI\LivingDoc.CLI.dll" test-assembly "C:\Agent\_work\2\BuildPipeline\MyProject\MyProject.Bdd.Tests.dll" --output-type JSON --test-execution-json "C:\Agent\_work\2\**\*TestExecution*.json" --include-test-output "All" --output "C:\Agent\_work\2\s\115323\FeatureData.json" --project-name "MyProject" --project-language "en"

and the files are there

2. yaml

testAssemblyFilePath: '$(Pipeline.Workspace)\BuildPipeline\${{lower(parameters.project)}}\${{parameters.dllName}}'
testExecutionJson: '$(Pipeline.Workspace)\BuildPipeline\**\*TestExecution*.json'

logs

Error: No files are matching the pattern: C:\Agent\_work\2\BuildPipeline\**\*TestExecution*.json 
##[error]Error: Command failed: dotnet "C:\Agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.946\CLI\LivingDoc.CLI.dll" test-assembly "C:\Agent\_work\2\BuildPipeline\MyProject\MyProject.Bdd.Tests.dll" --output-type JSON --test-execution-json "C:\Agent\_work\2\BuildPipeline\**\*TestExecution*.json" --include-test-output "All" --output "C:\Agent\_work\2\s\115331\FeatureData.json" --project-name "MyProject" --project-language "en"

3. yaml

testAssemblyFilePath: '$(Pipeline.Workspace)\BuildPipeline\${{lower(parameters.project)}}\${{parameters.dllName}}'
testExecutionJson: '$(Pipeline.Workspace)\BuildPipeline\${{lower(parameters.project)}}\*TestExecution*.json'

logs

Error: No files are matching the pattern: C:\Agent\_work\2\BuildPipeline\MyProject\**\*TestExecution*.json 
##[error]Error: Command failed: dotnet "C:\Agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.946\CLI\LivingDoc.CLI.dll" test-assembly "C:\Agent\_work\2\BuildPipeline\MyProject\MyProject.Bdd.Tests.dll" --output-type JSON --test-execution-json "C:\Agent\_work\2\BuildPipeline\MyProject\*TestExecution*.json" --include-test-output "All" --output "C:\Agent\_work\2\s\115333\FeatureData.json" --project-name "MyProject" --project-language "en"

4. yaml

as assembly and TestExecution.json files are in the same folder (MyProject), I've also tried just this:

testAssemblyFilePath: '$(Pipeline.Workspace)\BuildPipeline\${{lower(parameters.project)}}\${{parameters.dllName}}'
testExecutionJson: '${{lower(parameters.project)}}\*TestExecution*.json'

but it didn't work either

logs

Error: No files are matching the pattern: MyProject\*TestExecution*.json 
##[error]Error: Command failed: dotnet "C:\Agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.946\CLI\LivingDoc.CLI.dll" test-assembly "C:\Agent\_work\2\BuildPipeline\MyProject\MyProject.Bdd.Tests.dll" --output-type JSON --test-execution-json "MyProject\*TestExecution*.json" --include-test-output "All" --output "C:\Agent\_work\2\s\115342\FeatureData.json" --project-name "MyProject" --project-language "en"

Jordan Western

August 27, 2021 12:04
Hi Mateusz Niezgoda (PGS Software),

Can you simply try the following as you path?
```
testExecutionJson: '\**\*TestExecution*.json'
```
0

Mateusz Niezgoda (PGS Software)

August 27, 2021 12:37

Hi Jordan Western

Same results unfortunately.

logs:

Error: No files are matching the pattern: \**\*TestExecution*.json 
##[error]Error: Command failed: dotnet "C:\Agent\_work\_tasks\SpecFlowPlus_32f3fe66-8bfc-476e-8e2c-9b4b59432ffa\0.6.961\CLI\LivingDoc.CLI.dll" test-assembly "C:\Agent\_work\2\BuildPipeline\MyProject\MyProject.Bdd.Tests.dll" --output-type JSON --test-execution-json "\**\*TestExecution*.json" --include-test-output "All" --output "C:\Agent\_work\2\s\115524\FeatureData.json" --project-name "MyProject" --project-language "en"

although the files are in the same folder as the dll file

Guess this might be an issue with the wildcards.

Jordan Western

August 27, 2021 12:43
Hi Mateusz Niezgoda (PGS Software),

At this point I will refer this one back to the devs for clarity.

Will provide an update as soon as I can.
1
Jordan Western

September 01, 2021 12:23
Hi Mateusz Niezgoda (PGS Software),

A note from our devs, please let me know in case of any concern. -

It seems they are using a custom agent with a custom folder structure, which might cause their problems.

The testExecutionJson always needs to be relative path when using wildcards (**). But I think the root path is different in this case.

We are executing our code from the SYSTEM_DEFAULTWORKINGDIRECTORY which is the same as the BUILD_SOURCESDIRECTORY and the testExecutionJson has to be relative to that root.

https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#system-variables-devops-services

In this case it seems the root is defined with: Pipeline.Workspace.

They need to see where their BUILD_SOURCESDIRECTORY points to and update the paths accordingly.
0

Mateusz Niezgoda (PGS Software)

September 02, 2021 10:57

Hi Jordan Western

I've checked your suggestion and it worked. It occurred that the both BUILD_SOURCESDIRECTORY and SYSTEM_DEFAULTWORKINGDIRECTORY is pointing to an empty folder

C:\Agent\_work\2\s

when all the code is in

C:\Agent\_work\2\BuildPipeline\MyProject

Simple, going up to the main folder and then a relative path to the *TestExecution*.json files did the trick. Here is how my yaml file looks like now:

- task: SpecFlowPlus@0
  displayName: 'SpecFlow+ LivingDoc'
  condition: succeededOrFailed()
  inputs:
    generatorSource: 'TestAssembly'
    testAssemblyFilePath: '$(Pipeline.Workspace)\BuildPipeline\${{lower(parameters.project)}}\${{parameters.dllName}}'
    testExecutionJson: '..\BuildPipeline\${{lower(parameters.project)}}\*TestExecution*.json'
    projectName: '${{parameters.project}}'
    projectLanguage: 'en'

Many thanks for the help!

Please sign in to leave a comment.

Specflow+ LivingDoc in Azure DevOps - unable to setup/have test execution results

Comments

Didn't find what you were looking for?