[SpecFlow+Runner] VS extension to automatically link test case ID's in Visual Studio test explorer
Not planned
Currently we have an option `Associate to Test Case` in Visual Studio Test Explorer that is compatible with MSTest, Nunit & Xunit. We can have this added for SpecRun to ease integration with VSTS..
Currently we get an error as below when we try to achieve the same with SpecRun
-
do we have any mitigation to associate the Specflow+Runner tests with Azure Devops Test cases
0 -
I would also be interested in understanding if this issue can be resolved as i'd like to be able to execute my SpecFlow+ tests from the Azure test plans feature.
I recently switched from NUnit to SpecFlow+ runner and this is causing me a headache.
0 -
The last time I checked, there weren't any extension points available to make the association between tests from SpecFlow+ Runner and test plans possible.
But this was some time ago. Perhaps something changed in the meantime. If the feature requests get enough upvotes, we will have again a look at it.
At the moment, this is not possible when using the SpecFlow+ Runner.
1 -
I built a tool that associates the test cases using ADO API's (useful to bulk upload and link test cases from the test assembly); it seems the only issue is with the VSTest runner (via the vs test task in the ADO pipeline) understanding which test is required for execution as test can be sent to the runner individually via the test plans feature.
Without knowing how the runner is expecting the test name, and the mechanism by which it locates the tests, then I am at a loss as to what I can try as a workaround.
VSTest has has no issue seeing the tests available to it in the assembly and also no issues when executing the tests from the CLI.
I hope this will be resolved, I think more and more people will start using both of these technologies together!
0 -
I'm looking forward to use SpecFlow+ runner combined with Azure test plans so it would be great to see such integration in place.
1 -
Jordan Western what information would you need? Are we talking about the full qualified test name?
0 -
Andreas Willich I really have no idea as what I thought should have work has failed.
(sorry if you already know about the below, but I will include it just in case)
In the Azure 'test case' work items, there's two parameters that you use to associate a test case, the 'Automated test name' and the 'Automated test storage'.
The 'Automated test storage' of course is the name of the assembly the test exists in, and the 'Automated test name' resolves perfectly fine as the fully qualified name using NUnit, however VSTest was not able to find the tests using the same approach when the project was compiled with the SpecFlow+ Runner.
The problem with all this is that even if we knew what the runner was expecting, I have no idea how the 'test plans' task works or handles this information internally, no idea how this passes the information on to SF+R for execution and so on, so I am at a loss as to what I can try next.
It's probably Microsoft's problem to look into this I guess as it is their pipeline task (although you can get the source code from github for their tasks, but I have never worked with type script so I am not the best person to look)
If there is anything you think I can try, or any information you might need to help then please let me know.
0 -
Jordan Western It could be, that we generate FQNs that VSTest doesn't like via Commandline. That happened already in the past.
Would you have time next week for a call and have a look at what is happening? Perhaps we find together a solution.
Btw, is your tool somewhere available and open-source perhaps, so that other users would also benefit from the work?
0 -
Andreas Willich just as a last resort, can you confirm the format of the FQN?
It would be good to clarify from your side in the case that I have missed something.
0 -
For anyone interested, I spent some time with Andreas to come to a solution for linking and executing SpecFlow+ runner tests to azure dev ops and executing them from test plans.
This can only be achieved by using azure public APIs, I am working on a tool which will achieve this and I will upload it to github, or you can create your own solution.
2 -
All that is required is that the test case work item attribute 'Automated test name' is the fully qualified name of the test. Be aware that the second half of the FQN is url encoded in the case of special chars present in your scenario and or feature name.
0 -
Hi Andreas, Jordan,
If you put together a solution on GitHub, are you willing to share the link to it?
I have a team which could eventually participate in this effort.
I am starting to deploy specflow in my organization alongside, manual and core unit testing. To be able to centralize everything in a test plan is my dream ;-)
Best regards
1 -
Cosials, Benoit absolutely!
It is exactly the same for my team too.
Thanks!
0 -
Any luck in this effort? Our team is also using SpecRun and would like to associate tests with a DevOps Test Case so a group of tests could be triggered from DevOps.
0 -
Please take this code I uploaded to github https://github.com/JordanWestern/ADOTestSync you can take a look at the implementation.
0 -
Jordan, thanks for pointing me to the code. We got it working but are currently thinking that the SpecFlow+ Runner using the Fully Qualified Name (FQN) as the identifier is going to be an issue.
As you know, the DevOps Test Case Name is limited to 255 and this is where the FQN is stored and later used to trigger a test on-demand from DevOps Test Plan. Whatever is building the FQN is using a combination of assembly name, class name, feature name and scenario name, etc. with various separators. Due to this combination and the 255 limitation, we're seeing that our feature and scenario names are going to have to be very short.
Did you find a way around this issue? There seems to be a GUID created for each scenario that you are then using to determine if the scenario exists in DevOps. I am curious why the SpecFlow+ Runner doesn't use that to identify the test instead? We may have to experiment with other test runners to see if we have any better luck. However, we greatly appreciate you sharing your work to allow us to experiment with this option!
0 -
Hi Scott Link,
I never found a solution to the FQN length issue, but it did not pose much of an issue for me to be honest, tests names were mostly short enough. It is an Azure API limitation.
it's VS Test that uses the FQN to identify the tests if I remember rightly when executing via test plans.
NUnit is much less of an issue as you say as the FQN is much shorter and the implementation to extract the methods is very similar using reflection so can be easily extended
0 -
We are stopping the maintenance of the SpecFlow+ Runner, so I am closing this feature request.
For more details, please have a look at our blog post about it: https://specflow.org/using-specflow/the-retirement-of-specflow-runner/
0
Please sign in to leave a comment.
Comments
18 comments