Parameters from default.srprofile are not used in Azure Pipeline with Linux runner.

March 05, 2021 16:55 Edited

Hi Guys,

It seems the values from default.srprofile are not passed in the pipeline.

1. Environmental variables are not set at the runtime

2. Non of these are set: <Execution retryFor="None" stopAfterFailures="3" testThreadCount="1" testSchedulingMode="Sequential" />

For example, when test fail they are retired 3 times ( retry #1 retry #2), which I do not want. This does not happen when I run locally from TestRunner or cmd.

My setup is:

Local environment: Win 10 with VS19, SpcFlow with SpeFlow Runner

Azure pipeline with Ubuntu 18.04 runner

Tasks ran:

    - task: DotNetCoreCLI@2
      inputs:
        command: 'build'

    - task: DotNetCoreCLI@2
      inputs:
        command: 'test'
        arguments: '--filter TestCategory=auto'

default.srprofile

  <Execution retryFor="None" stopAfterFailures="3" testThreadCount="1" testSchedulingMode="Sequential" />
  <!-- For collecting by a SpecRun server update and enable the following element. For using the 
      collected statistics, set testSchedulingMode="Adaptive" attribute on the <Execution> element.
    <Server serverUrl="http://specrunserver:6365" publishResults="true" />
  -->
  <TestAssemblyPaths>
    <TestAssemblyPath>BDD.dll</TestAssemblyPath>
  </TestAssemblyPaths>
  <DeploymentTransformation>
    <Steps>
      <EnvironmentVariable variable="BUILD_ENVIRONMENT" value="dev" />
      <EnvironmentVariable variable="MACHINE_UUID" value="123" />
      <EnvironmentVariable variable="MACHINE_IP" value="x.x.x.x" />

settings.runsettings

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
    <TargetPlatform>x86</TargetPlatform>
    <TargetFrameworkVersion>Framework40</TargetFrameworkVersion>
  </RunConfiguration>
  <SpecRun>
    <Profile>default.srprofile</Profile>
    <ReportFile>TestReport.html</ReportFile>
    <GenerateSpecRunTrait>false</GenerateSpecRunTrait>
    <GenerateFeatureTrait>false</GenerateFeatureTrait>
  </SpecRun>
</RunSettings>

Comments

15 comments

Andreas Willich

February 19, 2021 14:04
Linux filesystem is case-sensitive. The correct filename is `Default.srProfile. Please rename the file and then it should work on your Linux agent.
0
Victor Sypniewski-Tom

February 22, 2021 10:27 Edited
So I tried that. It didn't work.

1. Environmental variables are not set

I tried with not setting them up at the root of the pipeline

I tried with setting them to empty string at the root of the pipeline (in two different ways)

2. The retry setting is ignored: retry #2 [104 ms]

I tried to `printenv` and variables were empty.

(Just for sanity check I set them up in the root of the pipeline with correct values and it worked)
```
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
    <TargetPlatform>x86</TargetPlatform>
    <TargetFrameworkVersion>Framework40</TargetFrameworkVersion>
  </RunConfiguration>
  <SpecRun>
    <Profile>Default.srProfile</Profile>
    <ReportFile>TestReport.html</ReportFile>
    <GenerateSpecRunTrait>false</GenerateSpecRunTrait>
    <GenerateFeatureTrait>false</GenerateFeatureTrait>
  </SpecRun>
</RunSettings>
```
In fact I do not care that much for env, but I do care for not rerunning the jobs.
0
Victor Sypniewski-Tom

February 22, 2021 15:48 Edited
I also, tried to change the name to `dev.srprofile` in case naming it `Default` causes it not being seen, I still have the same effect.

Edit: When I used the dev.srprofile it did not work locally. Does this need to be passed as CLI parameter?
0
Victor Sypniewski-Tom

February 26, 2021 18:14
Is this an issue for github? I will try to run it locally on Linux dev env and see if it works locally on Linux.
0
Victor Sypniewski-Tom

March 02, 2021 16:09
Hey Guys, I ran it on Ubuntu 18.04 with VSCode and indeed ``dotnet test`` does not observe the *.srprofile.

Is this a feature request or a bug?
0

Victor Sypniewski-Tom

March 05, 2021 12:57

I just found this bit of log, maybe it will help. No matter how I name the .srprofile it is not being used on linux.


 -> -> Loading plugin /home/bin/Debug/netcoreapp3.1/BDD.dll
-> -> Missing [assembly:RuntimePlugin] attribute in BDD, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null. Please check https://go.specflow.org/doc-plugins for details.
-> -> Loading plugin /home/bin/Debug/netcoreapp3.1/SpecRun.Runtime.SpecFlowPlugin.dll
-> -> Loading plugin /home/bin/Debug/netcoreapp3.1/LivingDoc.SpecFlowPlugin.dll
-> -> Using default config

Zoltán Tóth

March 05, 2021 15:40
Hi,

could you please run "dotnet test -v:d" and check the output?

The "Using default config" is printed by the SpecFlow framework telling that it uses the default SpecFlow configuration (there was no specflow.json or app.config configuration found). So this is not about the SpecFlow+Runner profile.

In your .srprofile you can give a name to your profile <Settings projectName="SomeProfileName" />. In the detailed log you should see a snippet as follows:

SpecFlow+Runner execution started
SpecFlow+Runner 3.7.9 in Framework netcore31 in x64 mode execution started
TestAdapter Location:XXX
Current UserName: 'XXX', MachineName: 'XXX'
SpecRun: running tests in XXX
Output folder configured to XXX
Profile: SomeProfileName

...
0
Zoltán Tóth

March 05, 2021 15:57
Some more thoughts:
- if you don't use runsettings then the default file name that is picked up is "Default.srprofile" with capital "D".
- the .srprofile has to be next to the test assembly. Make sure that the file has a "copy if newer" setting in the project.
- if you use runsettings then you specify the srprofile file in the <SpecRun>/<Profile> element.
- if you use runsettings you could specify it explicitly for dotnet test using the "-s|--settings <SETTINGS_FILE>" option
0

Victor Sypniewski-Tom

March 05, 2021 17:18

I use runsettings.
This is set <Settings projectName="SomeProfileName" />
<SpecRun>
<Profile>dev.srprofile</Profile>

The Profile comes back as empty on Linux
The same project on Windows shows Profile: BDD
I added copy if newer.


<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
<Settings projectName="BDD" />
<Execution retryFor="None" stopAfterFailures="1" testThreadCount="1" testSchedulingMode="Sequential" />

A total of 1 test files matched the specified pattern.
SpecFlow+Runner execution started
SpecFlow+Runner 3.7.3 in Framework netcore31 in x64 mode execution started
TestAdapter Location: /home/repos/BDD/bin/Debug/netcoreapp3.1/TechTalk.SpecRun.VisualStudio.TestAdapter.dll
Current UserName: 'xxx', MachineName: 'Virtual-Machine'
SpecRun: running tests in /home/repos/BDD/bin/Debug/netcoreapp3.1/BDD.dll
Output folder configured to "/home/repos/TestResults" (default)
Profile: 
Log file path: /home/repos/BDD/TestResults/Unnamed project__2021-03-05T170340.log
Checking activation
Activated.
Activation check completed
Starting test run
Discovering target: Default

Is it possible that this simply does not work on Linux?

Victor Sypniewski-Tom

March 05, 2021 17:33
Explicitly stating
dotnet test --filter TestCategory=auto -s settings.runsettings

Works! Zoltán Tóth

Andreas Willich is this something worth logging in github for linux? Maybe docs update?
0
Zoltán Tóth

March 08, 2021 08:36
I still don't think that this is related to Linux. If you use the command line you have to specify the runsettings explicitly on all platforms. Only Visual Studio has an "autodetect" feature.

See the documentation of "-s|--settings <SETTINGS_FILE>" at https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test

There are also two further links that explain the usage of settings files in detail.
0
Victor Sypniewski-Tom

March 08, 2021 09:09
Interesting. I would rather not have support in VS (magic always leads to troubles) and expect the same behaviour everywhere.

I ran this from PowerShell without even opening VS19, without -s option, and the profile was picked up.

One or the other way, thanks a million for your help.
0

Sudhindra Kumar

March 15, 2021 10:21

Zoltán Tóth : I am facing this issue in Linux even after following the same process stated in this discussion thread. Any help would be really appeciated.

Packages used with <TargetFramework>netcoreapp3.1</TargetFramework> is as below

 <PackageReference Include="SpecFlow.Plus.LivingDocPlugin" Version="3.7.43" />
<PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.7.13" />
<PackageReference Include="SpecRun.SpecFlow" Version="3.7.3" />

Could not generate test run result reports. An error occurred in the generator.
External program returned exit code '254'

Victor Sypniewski-Tom

March 15, 2021 11:27
Sudhindra Kumar I think this is slightly different symptom to this issue (I did not have problem with report). Could you open new post and give some more details there, please?
0
Sudhindra Kumar

March 15, 2021 12:00
Victor Sypniewski-Tom, Zoltán Tóth : I found out the issue. Since i was using custom test report template, the path needs to be changed to be compatible with Linux. I updated it & it worked fine. Thank you
0

Please sign in to leave a comment.

Parameters from default.srprofile are not used in Azure Pipeline with Linux runner.

Comments

Didn't find what you were looking for?