Improve running tests through ant #10169
Conversation
|
One more disclaimer: I am not quite familiar with ant, so there are probably better ways to approach this. Regardless, it works well as it is now :-) |
matthijskooijman
force-pushed
the
test-run-improvements
branch
from
9bff10c to
565c8e9
Compare
|
I just realized that with this commit, when running all tests, does not produce any detailed output at all. The idea was that you can just look at the HTML report for details, but I just now realized that this is not really convenient when running from CI checks (if a test fails, you cannot easily see what failed). So I just amended the first commit to generate full output for failing tests only (which will probably be easier to read, before this commit it would print full stdout/stderr, but not the failures actually. I also just realized the HTML report could maybe be added as an output asset of the workflow, so let me see if that would work (not sure if you could easily view HTML though). |
matthijskooijman
force-pushed
the
test-run-improvements
branch
from
f283469 to
b68e367
Compare
This allows running an individual test class by specifying -Dsingle-test-class=path.to.Classs and methods within the specified class by specifying -Dsingle-test-methods=testMethod1,testMethod2. Additionally, this improves the error output, but not showing full stderr/stdout output when running the full test suite, and by generating a browsable HTML report with test results (including stdout/stderr output). When single-test-class is used, detailed output (including stdout/stderr) is still printed directly. This also moves the test result XML files into a subdirectory for clarity, which is removed before starting a testrun (so the HTML report does not include older test results).
This ensures that all code is (re)built when running a test. Previously, only the app code was built (because that's where the tests live), requiring a manual full build when changing something in arduino-core. This could lead to confusing situations, where you would changes something but it was not always automatically recompiled.
matthijskooijman
force-pushed
the
test-run-improvements
branch
from
b68e367 to
28f7e68
Compare
This also lets ant print a message pointing to this artifact and explaining the files inside briefly. To make sure this is only printed inside the github action, an extra `-D` option is passed to `ant test` from the workflow file. To allow `if` / `unless` on the echo element, this adds some namespace definitions (this allows them on *any* element, not just the ones that explicitly allow it). See https://ant.apache.org/manual/ifunless.html
matthijskooijman
force-pushed
the
test-run-improvements
branch
from
28f7e68 to
4c6dba9
Compare
|
Ok, that seems to work, I added one more commit that makes the github action publish the HTML report as an asset. It shows up as a zip. I initially tried to add the one-page HTML version, but does not seem to include the full command output for all tests, which I think can sometimes be relevant. So I now included the multi-page version, but also the single-page version since that is a lot easier to open from the zip (you can just open that file, without having to extract all files). I'm not entirely sure it works, since the build seems to fail now, because https://github.com/arduino-libraries/WiFi101-FirmwareUpdater-Plugin/releases/download/v0.10.10/WiFi101-Updater-ArduinoIDE-Plugin-0.10.10.zip is failing to download somehow? Seems to be unrelated, so I'll just leave this for now and maybe retrigger a build tomorrow :-) |
|
@matthijskooijman please push on another branch on your fork if you want to see the gh-actions outcome, otherwise this gets very spammy :-) |
|
Too bad that the frame-less version does not contains the verbose test output (this makes the .zip artifact from github much less handy). |
Yeah, that's indeed annoying. I just doublechecked the XSL source that generates the report ( If we care enough, we could copy the XSL file entirely (setting |
This PR improves running tests through ant, by allowing individual tests to be ran, improving the generated output and automatically (re)building the project before running tests.
I originally also planned to add documentation about this in this PR, but I ended up creating a page on the wiki instead (since the building docs also live there).
That page still has minimal docs (additional docs on running tests from Eclipse or other IDEs welcome, I have no experience with that). When this PR is merged, the CLI-section of that page can be replaced by the following.
Commandline using ant
From the commandline,
antcan be used for running tests, just like building. For this, go into thebuilddirectory and run thetesttarget:This runs the entire testsuite. To run individual tests, you can use e.g.:
This runs all tests from a single test class, or selected test methods (comma separated) from that class respectively.
A class is selected by using its qualified java class name (to find it, look at the
.javafile and join together thepackagename (e.g.processing.app) and theclassname (e.g.CommandLineTest). It is not currently possible to select more than one test class to run.Test results
When running all tests on the commandline, only a minimal test summary is printed for each test. When running just a single test, full test output (failures with backtraces, but also full stdout and stderr output) is shown directly.
In either case, a browsable HTML report is generated. This report contains full details for each test run (including stdout/stderr output, link in the lower right). The path to that report is printed after completing the tests.