Acceptance tests skipped using 'dotnet test' in Azure Pipeline

February 09, 2021 16:40

Hi,

I want to execute my SpecFlow tests using this task in an Azure pipeline:

- task: DotNetCoreCLI@2
  displayName: Acceptance Tests
  continueOnError: true
  inputs:
    command: test
    testRunTitle: 'Acceptance Tests'
    projects: '**/*Specs.csproj'

When I run 'dotnet test' on my Specs project in the terminal of Visual Studio the tests are executed and the TestExecution.json file is created.

In the log of the azure pipeline for the task above I see that the tests are skipped:

 2021-02-09T16:23:43.7423054Z ##[section]Starting: Acceptance Tests
2021-02-09T16:23:43.7432660Z ==============================================================================
2021-02-09T16:23:43.7433030Z Task : .NET Core
2021-02-09T16:23:43.7433403Z Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command
2021-02-09T16:23:43.7433761Z Version : 2.181.0
2021-02-09T16:23:43.7434029Z Author : Microsoft Corporation
2021-02-09T16:23:43.7434406Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
2021-02-09T16:23:43.7435037Z ==============================================================================
2021-02-09T16:23:44.0977932Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
2021-02-09T16:23:44.3899153Z [command]/opt/hostedtoolcache/dotnet/dotnet test /home/vsts/work/1/s/experimental.MyUntappd.Specs/experimental.MyUntappd.Specs.csproj --logger trx --results-directory /home/vsts/work/_temp
2021-02-09T16:23:45.2781000Z Determining projects to restore...
2021-02-09T16:23:46.2806974Z All projects are up-to-date for restore.
2021-02-09T16:23:46.3819825Z SpecFlowFeatureFiles: Features/BeerRatingsController.feature
2021-02-09T16:23:46.4871220Z -> Using default config
2021-02-09T16:23:46.9742016Z SpecFlowGeneratedFiles: Features/BeerRatingsController.feature.cs
2021-02-09T16:23:47.6168869Z experimental.MyUntappd.Contracts -> /home/vsts/work/1/s/experimental.MyUntappd.Contracts/bin/Debug/net5.0/experimental.MyUntappd.Contracts.dll
2021-02-09T16:23:47.6439227Z experimental.MyUntappd.Data -> /home/vsts/work/1/s/experimental.MyUntappd.Data/bin/Debug/net5.0/experimental.MyUntappd.Data.dll
2021-02-09T16:23:47.9532707Z experimental.MyUntappd.Application -> /home/vsts/work/1/s/experimental.MyUntappd.Application/bin/Debug/net5.0/experimental.MyUntappd.Application.dll
2021-02-09T16:23:48.2196028Z experimental.MyUntappd.Api -> /home/vsts/work/1/s/experimental.MyUntappd.Api/bin/Debug/net5.0/experimental.MyUntappd.Api.dll
2021-02-09T16:23:48.5702286Z experimental.MyUntappd.Specs -> /home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0/experimental.MyUntappd.Specs.dll
2021-02-09T16:23:48.5917586Z Test run for /home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0/experimental.MyUntappd.Specs.dll (.NETCoreApp,Version=v5.0)
2021-02-09T16:23:48.6732908Z Microsoft (R) Test Execution Command Line Tool Version 16.8.3
2021-02-09T16:23:48.6737351Z Copyright (c) Microsoft Corporation. All rights reserved.
2021-02-09T16:23:48.6739330Z 
2021-02-09T16:23:48.8221936Z Starting test execution, please wait...
2021-02-09T16:23:48.9274844Z A total of 1 test files matched the specified pattern.
2021-02-09T16:23:52.9392682Z Starting external program: "mono" "/home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0/SpecFlowPlusRunner/net461/SpecFlow.Plus.Runner.Reporting.exe" "/tmp/tmpmeQZ3w.tmp" in /home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0
2021-02-09T16:23:54.3786797Z Skipped Get empty enumerable of beer ratings when no beer ratings exist [< 1 ms]
2021-02-09T16:23:54.3787917Z Skipped Get all stored beer ratings [< 1 ms]
2021-02-09T16:23:54.3788539Z Skipped Get BAD REQUEST response if beer rating does not exist [< 1 ms]
2021-02-09T16:23:54.3789150Z Skipped Get a single stored beer rating [< 1 ms]
2021-02-09T16:23:55.3607163Z 'mono "/home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0/SpecFlowPlusRunner/net461/SpecFlow.Plus.Runner.Reporting.exe" "/tmp/tmpmeQZ3w.tmp"' took 2420.7365ms
2021-02-09T16:23:55.3612112Z LwBoAG8AbQBlAC8AdgBzAHQAcwAvAHcAbwByAGsALwBfAHQAZQBtAHAALwBVAG4AbgBhAG0AZQBkACAAcAByAG8AagBlAGMAdABfAF8AMgAwADIAMQAtADAAMgAtADAAOQBUADEANgAyADMANQAwAC4AaAB0AG0AbAA=
2021-02-09T16:23:55.3613363Z 
2021-02-09T16:23:55.4672957Z Results File: /home/vsts/work/_temp/_fv-az128-357_2021-02-09_16_23_54.trx
2021-02-09T16:23:55.4673849Z 
2021-02-09T16:23:55.4674417Z Attachments:
2021-02-09T16:23:55.4675500Z /tmp/agupf2ja.00p/Unnamed project__2021-02-09T162350.log
2021-02-09T16:23:55.4676361Z /tmp/agupf2ja.00p/Unnamed project__2021-02-09T162350.html
2021-02-09T16:23:55.4705381Z Skipped! - Failed: 0, Passed: 0, Skipped: 4, Total: 4, Duration: < 1 ms - /home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0/experimental.MyUntappd.Specs.dll (net5.0)
2021-02-09T16:23:55.9305431Z Result Attachments will be stored in LogStore
2021-02-09T16:23:55.9309386Z Run Attachments will be stored in LogStore
2021-02-09T16:23:55.9584906Z Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
2021-02-09T16:23:55.9588926Z ##[section]Async Command Start: Publish test results
2021-02-09T16:23:56.2698503Z Publishing test results to test run '1056004'.
2021-02-09T16:23:56.2701198Z TestResults To Publish 4, Test run id:1056004
2021-02-09T16:23:56.2703909Z Test results publishing 4, remaining: 0. Test run id: 1056004
2021-02-09T16:23:56.8161445Z Published Test Run : https://experimental.visualstudio.com/experimental%20Knowledge%20Boost/_TestManagement/Runs?runId=1056004&_a=runCharts
2021-02-09T16:23:57.0659271Z ##[section]Async Command End: Publish test results
2021-02-09T16:23:57.0684753Z ##[section]Finishing: Acceptance Tests

Do you have any hint what I am doing wrong? Any help is welcome!

Kind regards,

Daniel

Comments

9 comments

Andreas Willich

February 10, 2021 07:23
There should be the log file attached to the test run. In it should be more information, why the tests are skipped.
0

Westhoff, Daniel

February 10, 2021 09:58

Hi Andreas, hi all,

thanks for your response. I found a log file in the "Test Plans > Runs" page for my run. I hope this is the log file you are referring to.

Below you can see an excerpt of this file with an error that states the file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0' could not be found. Unfortunately, I have no idea why this happens and how i can solve it.

Further help or hints how this can be solved would be nice!

Regards, Daniel

2021-02-09T16:23:52.5552826+00:00:2021-02-09T16:23:52.5549417+00:00:[AssemblyCache] Trying to resolve 'Microsoft.AspNetCore.Mvc.Abstractions' to '/home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0/SpecFlowPlusRunner/netcoreapp5.0/Microsoft.AspNetCore.Mvc.Abstractions.exe'
2021-02-09T16:23:52.5558892+00:00:2021-02-09T16:23:52.5556496+00:00:[AssemblyCache] Could not resolve 'Microsoft.AspNetCore.Mvc.Abstractions' in folder '/home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0/SpecFlowPlusRunner/netcoreapp5.0'
2021-02-09T16:23:52.6145047+00:00:2021-02-09T16:23:52.5640817+00:00:Assembly resolving Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
2021-02-09T16:23:52.6672200+00:00:2021-02-09T16:23:52.6203262+00:00:Thread#0:T:-> error: Unable to load one or more of the requested types.
2021-02-09T16:23:52.7109935+00:00:2021-02-09T16:23:52.7103255+00:00:Thread#0:T:Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
2021-02-09T16:23:52.7130537+00:00:2021-02-09T16:23:52.7117909+00:00:Thread#0:T:Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
2021-02-09T16:23:52.7145709+00:00:2021-02-09T16:23:52.7134160+00:00:Thread#0:T: (0.0s)
2021-02-09T16:23:52.7156480+00:00:2021-02-09T16:23:52.7153037+00:00:Thread#0:T:-> error: Type Loader exceptions: (0.0s)
2021-02-09T16:23:52.7188943+00:00:2021-02-09T16:23:52.7171503+00:00:Thread#0:T:-> error: LoaderException: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
2021-02-09T16:23:52.7194684+00:00:2021-02-09T16:23:52.7192277+00:00:Thread#0:T:File name: 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60' (0.0s)
2021-02-09T16:23:52.7363450+00:00:2021-02-09T16:23:52.7356792+00:00:OutOfProcessTestAssemblyExecutor: RunTest error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeModule.GetTypes()
at System.Reflection.Assembly.GetTypes()
at TechTalk.SpecFlow.Bindings.Discovery.RuntimeBindingRegistryBuilder.BuildBindingsFromAssembly(Assembly assembly) in D:\a\1\s\TechTalk.SpecFlow\Bindings\Discovery\RuntimeBindingRegistryBuilder.cs:line 23
at TechTalk.SpecFlow.TestRunnerManager.BuildBindingRegistry(IEnumerable`1 bindingAssemblies) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 94
at TechTalk.SpecFlow.TestRunnerManager.InitializeBindingRegistry(ITestRunner testRunner) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 73
at TechTalk.SpecFlow.TestRunnerManager.CreateTestRunner(Int32 threadId) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 62
at TechTalk.SpecFlow.TestRunnerManager.GetTestRunnerWithoutExceptionHandling(Int32 threadId) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 155
at TechTalk.SpecFlow.TestRunnerManager.GetTestRunner(Int32 threadId) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 136
at TechTalk.SpecFlow.TestRunnerManager.OnTestRunStart(Assembly testAssembly, IContainerBuilder containerBuilder) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 232
at experimental_MyUntappd_Specs_SpecFlowPlusRunnerAssemblyHooks.AssemblyInitialize() in /home/vsts/work/1/s/experimental.MyUntappd.Specs/obj/Debug/net5.0/SpecFlow.Plus.Runner.AssemblyHooks.cs:line 12
at TechTalk.SpecRun.Framework.TaskExecutors.StaticOrInstanceMethodExecutor.ExecuteInternal(ITestThreadExecutionContext testThreadExecutionContext)
at TechTalk.SpecRun.Framework.TaskExecutors.StaticOrInstanceMethodExecutor.Execute(ITestThreadExecutionContext testThreadExecutionContext)
at TechTalk.SpecRun.Framework.TestAssemblyExecutor.RunAssemblyInitializeExecutors(ITestAssemblyExecutorsProvider testAssemblyExecutorsProvider, ITestThreadExecutionContext testThreadExecutionContext)
at TechTalk.SpecRun.Framework.TestAssemblyExecutor.RunTests(ITestAssemblyExecutorState executorState, ITestAssemblyExecutorsProvider testAssemblyExecutorsProvider, ITestThreadExecutionContext testThreadExecutionContext)
at TechTalk.SpecRun.Framework.TestAssemblyExecutor.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.InProcess.InProcAppDomainTestAssemblyExecutor.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.OutOfProcess.OutOfProcessTestAssemblyExecutor.RunTests(TestThreadState testThreadState)
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
2021-02-09T16:23:52.7786047+00:00:2021-02-09T16:23:52.7370190+00:00:OutOfProcessTestAssemblyExecutorReceiver - Error at RunTests - System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeModule.GetTypes()
at System.Reflection.Assembly.GetTypes()
at TechTalk.SpecFlow.Bindings.Discovery.RuntimeBindingRegistryBuilder.BuildBindingsFromAssembly(Assembly assembly) in D:\a\1\s\TechTalk.SpecFlow\Bindings\Discovery\RuntimeBindingRegistryBuilder.cs:line 23
at TechTalk.SpecFlow.TestRunnerManager.BuildBindingRegistry(IEnumerable`1 bindingAssemblies) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 94
at TechTalk.SpecFlow.TestRunnerManager.InitializeBindingRegistry(ITestRunner testRunner) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 73
at TechTalk.SpecFlow.TestRunnerManager.CreateTestRunner(Int32 threadId) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 62
at TechTalk.SpecFlow.TestRunnerManager.GetTestRunnerWithoutExceptionHandling(Int32 threadId) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 155
at TechTalk.SpecFlow.TestRunnerManager.GetTestRunner(Int32 threadId) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 136
at TechTalk.SpecFlow.TestRunnerManager.OnTestRunStart(Assembly testAssembly, IContainerBuilder containerBuilder) in D:\a\1\s\TechTalk.SpecFlow\TestRunnerManager.cs:line 232
at experimental_MyUntappd_Specs_SpecFlowPlusRunnerAssemblyHooks.AssemblyInitialize() in /home/vsts/work/1/s/experimental.MyUntappd.Specs/obj/Debug/net5.0/SpecFlow.Plus.Runner.AssemblyHooks.cs:line 12
at TechTalk.SpecRun.Framework.TaskExecutors.StaticOrInstanceMethodExecutor.ExecuteInternal(ITestThreadExecutionContext testThreadExecutionContext)
at TechTalk.SpecRun.Framework.TaskExecutors.StaticOrInstanceMethodExecutor.Execute(ITestThreadExecutionContext testThreadExecutionContext)
at TechTalk.SpecRun.Framework.TestAssemblyExecutor.RunAssemblyInitializeExecutors(ITestAssemblyExecutorsProvider testAssemblyExecutorsProvider, ITestThreadExecutionContext testThreadExecutionContext)
at TechTalk.SpecRun.Framework.TestAssemblyExecutor.RunTests(ITestAssemblyExecutorState executorState, ITestAssemblyExecutorsProvider testAssemblyExecutorsProvider, ITestThreadExecutionContext testThreadExecutionContext)
at TechTalk.SpecRun.Framework.TestAssemblyExecutor.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.InProcess.InProcAppDomainTestAssemblyExecutor.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.OutOfProcess.OutOfProcessTestAssemblyExecutor.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.OutOfProcess.OutOfProcessTestAssemblyExecutorReceiver.RunTests(TestThreadState testThreadState)
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
2021-02-09T16:23:52.8024448+00:00:test thread #0 error: StreamJsonRpc.RemoteInvocationException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

at StreamJsonRpc.JsonRpc.InvokeCoreAsync[TResult](Nullable`1 id, String targetName, IReadOnlyList`1 arguments, CancellationToken cancellationToken, Boolean isParameterObject)
at TechTalk.SpecFlow.Plus.RPC.TcpRpcClient.Invoke[T](String methodName, Object[] parameters)
at TechTalk.SpecFlow.Plus.RPC.AsyncHelper.RunSync[TResult](Func`1 func)
at TechTalk.SpecRun.Framework.Execution.OutOfProc.StreamJsonRPC.OutOfProcessTestAssemblyExecutorClient.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.OutOfProc.StreamJsonRPC.RemoteExecutor.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.OutOfProc.StreamJsonRPC.OutOfProcessTestAssemblyExecutorHost.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.TestThread.RunNonEmptyTestSuite(CancellationToken cancellationToken, TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.TestThread.Run(ITestExecutionManager executionManagerForRun, CancellationToken cancellationToken)
at TechTalk.SpecRun.Framework.AsyncTestThreadRunner.RunSync(TestExecutionManager executionManager, CancellationToken cancellationToken)
2021-02-09T16:23:52.8034361+00:00:test thread error: TechTalk.SpecRun.Framework.SpecRunException: At least one test thread aborted.
---> StreamJsonRpc.RemoteInvocationException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

at StreamJsonRpc.JsonRpc.InvokeCoreAsync[TResult](Nullable`1 id, String targetName, IReadOnlyList`1 arguments, CancellationToken cancellationToken, Boolean isParameterObject)
at TechTalk.SpecFlow.Plus.RPC.TcpRpcClient.Invoke[T](String methodName, Object[] parameters)
at TechTalk.SpecFlow.Plus.RPC.AsyncHelper.RunSync[TResult](Func`1 func)
at TechTalk.SpecRun.Framework.Execution.OutOfProc.StreamJsonRPC.OutOfProcessTestAssemblyExecutorClient.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.OutOfProc.StreamJsonRPC.RemoteExecutor.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.Execution.OutOfProc.StreamJsonRPC.OutOfProcessTestAssemblyExecutorHost.RunTests(TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.TestThread.RunNonEmptyTestSuite(CancellationToken cancellationToken, TestThreadState testThreadState)
at TechTalk.SpecRun.Framework.TestThread.Run(ITestExecutionManager executionManagerForRun, CancellationToken cancellationToken)
at TechTalk.SpecRun.Framework.AsyncTestThreadRunner.RunSync(TestExecutionManager executionManager, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
2021-02-09T16:23:52.8095073+00:00:Thread#-1: Test #4/0 is started on thread #-1
2021-02-09T16:23:52.8107770+00:00:Thread#-1: Test #4/0 is finished on thread #-1 as Skipped within 0.0005746 seconds

Andreas Willich

February 10, 2021 10:08
Yeah, that is an assembly loading issue in the SpecFlow+ Runner. :-/

As a workaround, you could switch from `Process` test thread isolation to `SharedAppDomain` (https://docs.specflow.org/projects/specflow-runner/en/latest/Profile/Environment.html#testthreadisolation).

Depending on your automation code, you can do this or not.

More information about the different test thread isolation modes can be found here: https://docs.specflow.org/projects/specflow-runner/en/latest/Usage/Parallel-Execution-Features.html

What would us help to fix this issue, is that you create a small repro project and open a support ticket.
0
Westhoff, Daniel

February 10, 2021 10:25
Thanks for the quick response. I will have a look on the links and try to use the info to setup a workaround.

One thing I noticed in the first log I posted was this line:
```
2021-02-09T16:23:52.9392682Z Starting external program: "mono" "/home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0/SpecFlowPlusRunner/net461/SpecFlow.Plus.Runner.Reporting.exe" "/tmp/tmpmeQZ3w.tmp" in /home/vsts/work/1/s/experimental.MyUntappd.Specs/bin/Debug/net5.0
```
I do not get this line at all when I run 'dotnet test' on my development PC. Is it possible that there is something wrong with the frameworks that are used during the Azure pipeline run?
0
Andreas Willich

February 10, 2021 10:29
No, that is fine. The report generation happens in a separate process, which is still implemented only in .NET Framework. If you are on a Linux agent, we are using mono to execute this process.
0
Westhoff, Daniel

February 11, 2021 12:50
The hint about Linux made my clone my project on a macOS machine. On the Mac I could reproduce the problem seen on Azure DevOps when running tests executing 'dotnet test' on the command line.

I then created a simple default.srprofile file in the Specs project of my solution:
```
<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
    <Settings projectName="experimental.MyUntappd"/>
    <Execution retryFor="None" stopAfterFailures="0"/>
    <Environment testThreadIsolation="SharedAppDomain"/>
</TestProfile>
```
After that the acceptance tests were execute during 'dotnet test' on the Mac.

Unfortunately, they are still skipped on Azure DevOps. It looks like the .srprofile file is not taken into account because e.g. the project name is not used in the report file names. There, the reports still start with 'Unnamed project_...'.

In the documentation I saw that the .srprofile file is sometimes written as 'Default.srprofile' and sometimes 'default.srprofile'. I used lowercase. Might this be a case-sensitivity issue on Linux images?
0
Westhoff, Daniel

February 11, 2021 13:22
Ok, it was an issue with case-sensitivity. The filename needs to be Default.srprofile.

On macOS the default is that the file system is case-insensitive. Thus, it works with default as well as Default as filename.

On Azure DevOps using a linux image for the pipline run the filesystem is case-sensitive.

Therefore, my solution is to add a minimal profile file 'Default.srprofile' to the Specs project of my solution:
```
<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
    <Settings projectName="<my_project_name>"/>
    <Environment testThreadIsolation="SharedAppDomain"/>
</TestProfile>
```
0
Ali Mollahosseini

February 11, 2021 15:41
Hi Daniel,

Apologies if the documentation was misleading here. I have now updated it so it correctly states Default.srprofile, to avoid issues on Linux machines.

Cheers
0

Miguel Acosta

June 21, 2021 23:19 Edited

Andreas Willich, after switching to

<Environment testThreadIsolation="SharedAppDomain"/>

test execution in azure pipelines doesn't output an html report

Scenario: Test Modifiers page Values Are Saved, IL in Quote_Modifiers -> Succeeded on thread #*** 
Passed Test Modifiers page Values Are Saved, IL [*** s]

##[error]The active test run was aborted. Reason: Test host process crashed : [16243***816***.197][WARNING]: This version of ChromeDriver has not been tested with Chrome version 91. 

##[error][16243***8163.***73][WARNING]: This version of ChromeDriver has not been tested with Chrome version 91.

##[error]Stack overflow. 

Results File: D:\a\_temp\TestResults\VssAdministrator_fv-az***9-3***3_***21-***6-21_***_42_59.trx 

##[error]Test Run Aborted. 

Total tests: Unknown 

Passed: 62

Failed: 2

Skipped: 3

Total time: 34.***412 Minutes

Vstest.console.exe exited with code 1.

Please sign in to leave a comment.

Acceptance tests skipped using 'dotnet test' in Azure Pipeline

Comments

Didn't find what you were looking for?