Skip to main content

[Specflow+Livingdoc] How to find a specific TestExecution.json at runtime?



  • Sam Shackleton

    edit: I've added an alternate solution in a separate comment.

    Hello, I feel that I don't fully understand your use case.

    I cannot use just "TestExecution.json" because it may rewrite the previous result before Livindgoc is generated.'

    The TestExecution.json file is written to disk at the end of the test run. The file will be written to the same directory as the test assembly containing the feature files.

    Do you have multiple test runs of the same/different assemblies in the same directory running concurrently? Is that why you state that the TestExecution.json file may be overwritten?

    If you intend to run multiple test runs concurrently perhaps you could ensure that you have a separate directory for each run? Assuming you are using the dotnet sdk, one way to achieve this would be to specify an output directory to the dotnet build command: dotnet build <testproject.csproj> --output <path of unique directory>.

    I believe MSBuild also has a command line option to specify the output directory (if you are using MSBuild to build the test assembly).

    Another approach could be use a script to move/rename the TestExecution.json file immediately after the nunit-console command completes. However this approach will be unreliable due to race conditions if test runs finish at roughly the same time. The script may not time to copy the first json file before the second test run finishes and overwrites it.


  • Sam Shackleton

    Here's an approach that might fit your needs.

    Basically, the `SpecFlow.Plus.LivingDocPlugin` collects test results even if the LivingDocGenerator is disabled via specflow.json, eg:

      "$schema": "",
      "livingDocGenerator": {
        "enabled": false

    We can use this to our advantage by disabling the LivingDocGenerator and providing our own code that takes the test results to write the TestExecution.json file to disk.

    There is a 'TODO' comment in the code. You will need to work out how to provide a value to uniquely identify the TestExecution.json file.

    I'm not very familiar with the NUnit Console, but having had a brief look at the documentation it seems that the `--testparam:PARAMETER` may let you specify the file name from the command line.

    I haven't tested this but perhaps you could add `--testparam=TestExecutionFileName=<customname>` and then access it in code using `TestContext.Parameters["TestExecutionFileName']?


    using BoDi;
    using LivingDoc.SpecFlowPlugin;
    using LivingDoc.SpecFlowPlugin.Collectors;
    using TechTalk.SpecFlow.Analytics.UserId;
    using TechTalk.SpecFlow.Events;
    using TechTalk.SpecFlow.Tracing;

    namespace SpecFlowTestExecutionPath
        public class TestExecutionHooks
            public static void Register(ITestThreadExecutionEventPublisher eventPublisher, ResultWriter writer)
                // Register a handler to invoke our own logic for writing test execution results to disk.
                eventPublisher.AddHandler((Action<TestRunFinishedEvent>)(_ => writer.WriteToDisk()));

        /// <summary>
        /// An alternative to SpecFlows `LivingDoc.SpecFlowPlugin.FeatureDataGenerator` for writing test execution results to disk.
        /// </summary>
        public class ResultWriter
            private readonly GlobalExecutionResultStore _globalExecutionResultStore;
            private readonly IReporter _reporter;
            private readonly IObjectContainer _container;
            private readonly ITestRunnerManager _testRunnerManager;

            public ResultWriter(GlobalExecutionResultStore globalExecutionResultStore, IReporter reporter, IObjectContainer container, ITestRunnerManager testRunnerManager)
                _globalExecutionResultStore = globalExecutionResultStore ?? throw new ArgumentNullException(nameof(globalExecutionResultStore));
                _reporter = reporter ?? throw new ArgumentNullException(nameof(reporter));
                _container = container ?? throw new ArgumentNullException(nameof(container));
                _testRunnerManager = testRunnerManager ?? throw new ArgumentNullException(nameof(testRunnerManager));

            public void WriteToDisk()
                    var executionResults = _globalExecutionResultStore.GetResults();
                    var userUniqueIdStore = _container.Resolve<IUserUniqueIdStore>();
                    var reportData = _reporter.GetReport(executionResults, DateTime.UtcNow, userUniqueIdStore.GetUserId());

                    var testAssemblyDir = Path.GetDirectoryName(_testRunnerManager.TestAssembly.Location);
                    // TODO: add your logic to generate a unique, known, file name.
                    var fileName = "foobarwhateveryouwant.json";

                    var filePath = Path.Combine(testAssemblyDir, fileName);
                    File.WriteAllText(filePath, reportData);
                    _container.Resolve<ITraceListener>().WriteToolOutput($"Generated Test Execution json file: {filePath}");
                } catch (Exception e)
                    _container.Resolve<ITraceListener>().WriteToolOutput("Error generating Test Execution json file: {0}", e);

    The above code was written using the following SpecFlow packages:

        <PackageReference Include="SpecFlow.Plus.LivingDocPlugin" Version="3.9.57" />
        <PackageReference Include="SpecFlow.NUnit" Version="3.9.40" />

Please sign in to leave a comment.

Powered by Zendesk