Scaling Specflow Tests Across Multiple Azure Devops Agents: Unexpected Performance Issues
I'm currently trying to enhance my CI pipeline by distributing my Specflow tests across multiple Azure Devops agents using the default batching provided by the VSTest task (see here).
When I run my tests on a single agent, they take 15 minutes to complete.
When running them batched across 10 different agents, they are still taking up to 13 minutes to complete.
I have checked the logs of the parallel test runs to ensure that the tests are being evenly distributed (and that no tests are being run more than once across agents) and I can confirm that they are being distributed evenly.
So I'm struggling to think of why I'm only seeing negligible performance gains when batching across 10 agents.
I'm currently using the “number of tests” batching method (see here). But I've also tried the “based on past running times” method (see here), but that produces even worse results (upwards of 32 minutes for the tests to run on each agent - I killed the pipeline run in the end so I'm not sure how long this would have taken to complete).
One thing that could be causing the issue is if Specflow makes use of
ClassCleanup methods under the hood - as these methods can end up being called multiple times across each agent if tests from the same assembly/class end up getting distributed between different agents.
I just wanted to ask the community if this is likely to be the case? Or if there are other factors at play that are preventing the performance gains.
Please sign in to leave a comment.