[Specflow+Livingdoc] How to find a specific TestExecution.json at runtime?
I am using Specflow+NUnit+Livingdoc. When I run tests locally, everything is fine; but now I have to set up automatic runs and report generating, and I have no idea how to connect the run and the report.
I know I have to use two console commands. The first one, $nunit3-console, is to run the test. The second one, $livingdoc, is to create HTML report from TestExecution_<something>.json generated by the first command - and here the problem goes: how do I know which TestExecution file I need to pass to Livingdoc?
https://docs.specflow.org/projects/specflow-livingdoc/en/latest/LivingDocGenerator/Setup-the-LivingDocPlugin.html
The documentation says I can use 3 placeholders to differentiate the TestExecution file: ProcessId, ThreadId and Now. None of these are suitable for identifying the test result programmatically.
The tests are run manually and asyncronously. I know neither the time or the process id.
I cannot use just "TestExecution.json" because it may rewrite the previous result before Livindgoc is generated.
I cannot use some outer ID like environment variable because TestExecution file path can be set only in specflow.json config.
Using command line wildcard and generating a single report from all test results is not an option because all results but a few will be irrelevant.
The best option for me would be a command line parameter like --result in NUnit, but I couldn't find any documentation or discussions about this. Is there any solution or workaround?
-
edit: I've added an alternate solution in a separate comment.
Hello, I feel that I don't fully understand your use case.
I cannot use just "TestExecution.json" because it may rewrite the previous result before Livindgoc is generated.'
The TestExecution.json file is written to disk at the end of the test run. The file will be written to the same directory as the test assembly containing the feature files.
Do you have multiple test runs of the same/different assemblies in the same directory running concurrently? Is that why you state that the TestExecution.json file may be overwritten?
If you intend to run multiple test runs concurrently perhaps you could ensure that you have a separate directory for each run? Assuming you are using the dotnet sdk, one way to achieve this would be to specify an output directory to the dotnet build command: dotnet build <testproject.csproj> --output <path of unique directory>.
I believe MSBuild also has a command line option to specify the output directory (if you are using MSBuild to build the test assembly).
Another approach could be use a script to move/rename the TestExecution.json file immediately after the nunit-console command completes. However this approach will be unreliable due to race conditions if test runs finish at roughly the same time. The script may not time to copy the first json file before the second test run finishes and overwrites it.
0 -
Here's an approach that might fit your needs.
Basically, the `SpecFlow.Plus.LivingDocPlugin` collects test results even if the LivingDocGenerator is disabled via specflow.json, eg:
{
"$schema": "https://specflow.org/specflow-config.json",
"livingDocGenerator": {
"enabled": false
}
}We can use this to our advantage by disabling the LivingDocGenerator and providing our own code that takes the test results to write the TestExecution.json file to disk.
There is a 'TODO' comment in the code. You will need to work out how to provide a value to uniquely identify the TestExecution.json file.
I'm not very familiar with the NUnit Console, but having had a brief look at the documentation it seems that the `--testparam:PARAMETER` may let you specify the file name from the command line.
I haven't tested this but perhaps you could add `--testparam=TestExecutionFileName=<customname>` and then access it in code using `TestContext.Parameters["TestExecutionFileName']?
Code:
using BoDi;
using LivingDoc.SpecFlowPlugin;
using LivingDoc.SpecFlowPlugin.Collectors;
using TechTalk.SpecFlow.Analytics.UserId;
using TechTalk.SpecFlow.Events;
using TechTalk.SpecFlow.Tracing;
namespace SpecFlowTestExecutionPath
{
[Binding]
public class TestExecutionHooks
{
[BeforeTestRun]
public static void Register(ITestThreadExecutionEventPublisher eventPublisher, ResultWriter writer)
{
// Register a handler to invoke our own logic for writing test execution results to disk.
eventPublisher.AddHandler((Action<TestRunFinishedEvent>)(_ => writer.WriteToDisk()));
}
}
/// <summary>
/// An alternative to SpecFlows `LivingDoc.SpecFlowPlugin.FeatureDataGenerator` for writing test execution results to disk.
/// </summary>
public class ResultWriter
{
private readonly GlobalExecutionResultStore _globalExecutionResultStore;
private readonly IReporter _reporter;
private readonly IObjectContainer _container;
private readonly ITestRunnerManager _testRunnerManager;
public ResultWriter(GlobalExecutionResultStore globalExecutionResultStore, IReporter reporter, IObjectContainer container, ITestRunnerManager testRunnerManager)
{
_globalExecutionResultStore = globalExecutionResultStore ?? throw new ArgumentNullException(nameof(globalExecutionResultStore));
_reporter = reporter ?? throw new ArgumentNullException(nameof(reporter));
_container = container ?? throw new ArgumentNullException(nameof(container));
_testRunnerManager = testRunnerManager ?? throw new ArgumentNullException(nameof(testRunnerManager));
}
public void WriteToDisk()
{
try
{
var executionResults = _globalExecutionResultStore.GetResults();
var userUniqueIdStore = _container.Resolve<IUserUniqueIdStore>();
var reportData = _reporter.GetReport(executionResults, DateTime.UtcNow, userUniqueIdStore.GetUserId());
var testAssemblyDir = Path.GetDirectoryName(_testRunnerManager.TestAssembly.Location);
// TODO: add your logic to generate a unique, known, file name.
var fileName = "foobarwhateveryouwant.json";
var filePath = Path.Combine(testAssemblyDir, fileName);
File.WriteAllText(filePath, reportData);
_container.Resolve<ITraceListener>().WriteToolOutput($"Generated Test Execution json file: {filePath}");
} catch (Exception e)
{
_container.Resolve<ITraceListener>().WriteToolOutput("Error generating Test Execution json file: {0}", e);
}
}
}
}The above code was written using the following SpecFlow packages:
<PackageReference Include="SpecFlow.Plus.LivingDocPlugin" Version="3.9.57" />
<PackageReference Include="SpecFlow.NUnit" Version="3.9.40" />0
Please sign in to leave a comment.
Comments
2 comments