stopAfterFailures value defined as 0, it is working fine in my laptop. however in Azure pipeline, stopAfterFailures was not considered, it is running with default value
Having scenario with multiple test data in the parameters example as below. When execute test in local laptop is working as expected. However, when I configure the pipeline in Azure VM. if first test data 7505 is getting failed. Entire scenario is running 10 times for all the test data (examples) defined. this consume more execution time.
Examples:
| company | Country Name |
| 7505 | Poland |
| 7502 | Czech Republic EU1 |
| 7506 | Slovakia EU1 |
| 7599 | Croatia EU1 |
Normally script executed in one or two hours, however this pipeline still running more than 16 hours
Please suggest, why Default.srprofile values are not accepted in Azure pipeline.
-
I am noticing the similar behaviour as well stopAfterFailures may be taking the default value which is 10 when running in pipeline.
0 -
Nashrin Banu Syed Mohamed & thathi Are you using Windows or Linux?
0 -
Andreas Willich I am using windows. Here is the screen shot though all tests are green the test step is reporting as failed. what I have noticed some of the tests are executing more than one at one point of time not sure if it is expected behaviour or not. Hope this give you some context.
I can see this bug has raised in GitHub
0 -
I am using Windows. I managed to fix the issue, in the Test assemblies
Initially I was given
**\TestAutomation.dll
!**\*TestAdapter.dll
!**\obj\**Now I changed
**\TestAutomation\bin\Debug\net5.0\TestAutomation.dll
!**\*TestAdapter.dll
!**\obj\**Because of the general search, Pipeline found multiple DLL. Now I have given the associated path to pick one DLL. Issue is resolved
0 -
Nashrin Banu Syed Mohamed thanks for the update, great that it works!
I think the issue is not related to the stopAfterFailures but probably to the retryCount.
The stopAfterFailures feature stops the execution of the whole tests suite after a certain number of failures detected. The default value is 10, hence after 10 failing scenarios the execution would stop. If you override this to 0 it means that you turn this feature off, hence the execution of the test suite won't be shortcut.
The retryCount specifies how many times the same scenario should be executed. It works together with the retryFor setting. By default the retryCount is 2 and the retryFor is "failing", so the failing scenarios would be executed 2 additional times.
If you see a difference between your local execution and the pipeline execution one explanation could be that the pipeline execution does not pick up your .srprofile settings. But in this case it would fall back to the defaults. If the behavior is different from the defaults then the issue might be something else.
Some questions about the setup that might be relevant:
- What task do you use in the pipeline for the test execution? How is it parametrized? E.g. if you use the Visual Studio Test Task it has also retry capabilities on its own (see the rerunXXX parameters).
- Do you use a "Default.srprofile" or a custom one? How is it configured?
- Do you use a runsettings file? How is it configured?
0 -
Zoltán Tóth Thanks for taking time to look into my query.
Here is the my original query posted here. As you have suggested I was using the following setup in my .srprofile.
<Execution stopAfterFailures="0" testThreadCount="15" testSchedulingMode="Random" retryFor="Failing" retryCount="2" />
<VSTest testRetryResults="Unified" passRateAbsolute="1"/>
- What task do you use in the pipeline for the test execution? How is it parametrized? E.g. if you use the Visual Studio Test Task it has also retry capabilities on its own (see the rerunXXX parameters).
[Thathi] Yes using Visual Studio Test Tasks and not using its retry capabilities
- Do you use a "Default.srprofile" or a custom one? How is it configured?
Setting via .runsettings and I can see number of threads picked up from srprofile so I can see it is picking the right .srprofile
- Do you use a runsettings file? How is it configured?
0
Please sign in to leave a comment.
Comments
6 comments