Failed unit test reports in C# give wrong line number windows 10.0 Visual Studio 2017 version 15.2 unit-test Scott Colestock reported Jun 22, 2017 at 02:39 PM. In this video, Robert is joined by Kendra Havens, who shows us some of the excellent unit test tooling in Visual Studio 2017, including testing performance i.
The usual choices for getting code coverage metrics in Visual Studio is to buy the Enterprise version of Visual Studio or buy the dotCover third party tool, both of which are costly, especially if you are developing small applications yourself and want to check the coverage of your unit tests. There is a free NuGet package for generating code coverage metrics called, and together with another free NuGet package, can give you code coverage reports for free. The problem I found, however, was the documentation available to get the tool working was less than adequate for my purposes and it took several hours of faff to get it working for me. To save you time and money I have documented my experience here. The following were all referenced in the creation of this document: I have tested that this works in Visual Studio 2013 Professional and Visual Studio 2015 Community Edition, with both MSTest and NUnit. MSTest unit tests can be run with either mstest.exe or the newer vstest.console.exe. As I primarily use NUnit I have not done anything other than basic testing to ensure OpenCover works with both these test runners so cannot say how either might work on large projects.
I assume you already have a solution with application project that you are testing with a test project, and where you are using NUnit for that testing that you have already included the NUnit NuGet package. You will need to open your solution in Visual Studio and get the following packages using Package Manager Console: REM PM Install-Package OpenCover REM PM Install-Package ReportGenerator If you are using NUnit rather than MSTest you will also need this package: REM PM Install-Package NUnit.ConsoleRunner In the solution root folder you will then need to create a batch file that you will use to produce the code coverage report. I have three batch files shown below, one for use where you are using NUnit, one where you are using MSTest and the mstest.exe test runner, and one for where you are using MSTest and the vstest.console.exe test runner. You will need to modify the batch file to specify the DLL that contains your unit tests.
Unit tests Meson comes with a fully functional unit test system. To use it simply build an executable and then use it in a test. E = executable('prog', 'testprog.c') test('name of test', e) You can add as many tests as you want.
They are run with the command ninja test. Meson captures the output of all tests and writes it in the log file meson-logs/testlog.txt. Test parameters Some tests require the use of command line arguments or environment variables. These are simple to define. Test('command line test', exe, args: 'first', 'second') test('envvar test', exe2, env: 'key1=value1', 'key2=value2') Note how you need to specify multiple values as an array.
Coverage If you enable coverage measurements by giving Meson the command line flag -Dbcoverage=true, you can generate coverage reports. Meson will autodetect what coverage generator tools you have installed and will generate the corresponding targets. These targets are coverage-xml and coverage-text which are both provided by and coverage-html, which requires and or with html support.
The output of these commands is written to the log directory meson-logs in your build directory. Parallelism To reduce test times, Meson will by default run multiple unit tests in parallel.
It is common to have some tests which can not be run in parallel because they require unique hold on some resource such as a file or a D-Bus name. You have to specify these tests with a keyword argument.
Test('unique test', t, isparallel: false) Meson will then make sure that no other unit test is running at the same time. Non-parallel tests take longer to run so it is recommended that you write your unit tests to be parallel executable whenever possible. By default Meson uses as many concurrent processes as there are cores on the test machine. You can override this with the environment variable MESONTESTTHREADS like this.
$ MESONTESTTHREADS=5 ninja test Skipped tests and hard errors Sometimes a test can only determine at runtime that it can not be run. For the default exitcode testing protocol, the GNU standard approach in this case is to exit the program with error code 77. Meson will detect this and report these tests as skipped rather than failed. This behavior was added in version 0.37.0. For TAP-based tests, skipped tests should print a single line starting with 1.0 # SKIP. In addition, sometimes a test fails set up so that it should fail even if it is marked as an expected failure.
The GNU standard approach in this case is to exit the program with error code 99. Again, Meson will detect this and report these tests as ERROR, ignoring the setting of shouldfail. This behavior was added in version 0.50.0. Testing tool The goal of the meson test tool is to provide a simple way to run tests in a variety of different ways. The tool is designed to be run in the build directory. The simplest thing to do is just to run all tests, which is equivalent to running ninja test.
$ meson test You can also run only a single test by giving its name: $ meson test testname Tests belonging to a suite suite can be run as follows $ meson test -suite (sub)projectname:suite Since version 0.46, (sub)projectname can be omitted if it is the top-level project. Sometimes you need to run the tests multiple times, which is done like this: $ meson test -repeat=10 Invoking tests via a helper executable such as Valgrind can be done with the -wrap argument $ meson test -wrap=valgrind testname Arguments to the wrapper binary can be given like this: $ meson test -wrap='valgrind -tool=helgrind' testname Meson also supports running the tests under GDB. Just doing this: $ meson test -gdb testname Meson will launch gdb all set up to run the test. Just type run in the GDB command prompt to start the program. The second use case is a test that segfaults only rarely. In this case you can invoke the following command: $ meson test -gdb -repeat=10000 testname This runs the test up to 10 000 times under GDB automatically.
If the program crashes, GDB will halt and the user can debug the application. Note that testing timeouts are disabled in this case so meson test will not kill gdb while the developer is still debugging it. The downside is that if the test binary freezes, the test runner will wait forever. $ meson test -print-errorlogs Meson will report the output produced by the failing tests along with other useful informations as the environmental variables. This is useful, for example, when you run the tests on Travis-CI, Jenkins and the like. For further information see the command line help of Meson by running meson test -h. NOTE: If meson test does not work for you, you likely have a old version of Meson.
In that case you should call mesontest instead. If mesontest doesn't work either you have a very old version prior to 0.37.0 and should upgrade.