Repository navigation
Subsequent runs produce different results #20
Description
I ran Delphi Code Coverage multiple times on the same code base, and got different results for the coverage even though I didn't change anything about the code in between.
The Command I used to run this was: CodeCoverage -e PBeratungsapp_Tests.exe -m PBeratungsapp_Tests.map -uf test_results/script/dcov_units.lst -spf test_results/script/dcov_paths.lst -od test_results -lt -xml -html -v -ife
The first day I got this result every time:
+-----------+-----------+-----------+
| Lines | Covered | Covered % |
+-----------+-----------+-----------+
| 6991 | 3463 | 49 % |
+-----------+-----------+-----------+
Today, the first time I ran it, I got this result:
+-----------+-----------+-----------+
| Lines | Covered | Covered % |
+-----------+-----------+-----------+
| 6991 | 426 | 6 % |
+-----------+-----------+-----------+
I then rebuilt the test project (even though nothing about the code had changed), and then I got this result:
+-----------+-----------+-----------+
| Lines | Covered | Covered % |
+-----------+-----------+-----------+
| 6991 | 5161 | 73 % |
+-----------+-----------+-----------+
These fluctuations strike me as odd. Any idea as to why this might be happening?
Activity
Laurensvanrun commentedon Mar 17, 2022
This indeed is strange and I have never seens this behaviour. Are you sure all tests ran correctly in your situations and there were no exceptions in the tests? Are you tests clear unit tests, or integration tests (which, if they communicate with datbases/file systems can give different results/behavior in different situations)?
KiraResari commentedon Mar 18, 2022
Some of the tests are integration tests, so maybe that's the issue. However, I have no idea of how to check which tests were executed correctly if I run it with the above command line. Is there another way to do that?
Also, if it were just the integration tests, I would expect a much smaller discrepancy than 6%, 49% and 73%
MHumm commentedon Dec 21, 2022
Which testing framework do you use?
DUnit or DUnitX? If the former you might want to use the GUI runner, which lists all failed tests at the bottom and marks them in red in the tree. If you use the DUnitX one, the console out put should tell you which ones failed and I think I always need to hit enter to close that so it should stay open. If not, create a batch file in which you cann that exe file with the tests and put a pause statement directly beding it. It keeps the console window open until you hit a key so you should see which tests failed.