Refactor testscript to facilitate running subsets of tests

[mazunki]

This PR refactors the ./test.sh script with three major goals:

• clean up the output
• allow running specific targets manually as ./test.sh net_tests
• make the script more maintainable

Furthermore there's additionally an option to silence the output of the inner tests through the TESTS_STDOUT and TESTS_STDERR variables. I find TESTS_STDOUT=/dev/null ./test.sh particularly useful.

See ./test.sh help for a list of targets. Running ./test.sh with no arguments (or ./test.sh all) runs the same tests it always did.

There's also a custom_tests mock function which just aims to make quick iteration easier.

[alfreb]

Thanks for giving test.sh some love!

I like the idea of allowing the user to run specific categories of tests, but I would prefer if the categories were derived from the folder structure and not introduced as custom functions; this to reduce maintenance. I realize that the folder structure could probably use some improvement (the memory tests, for example, were hard to find), but then let's fix that instead, in a separate PR. So we can keep your outer loop, iterating over parameters, but I prefer that we then match against directory names or full paths, either with exact match or regex, but keep the script as simple as possible.

[mazunki]

I like the idea of allowing the user to run specific categories of tests, but I would prefer if the categories were derived from the folder structure and not introduced as custom functions; this to reduce maintenance.

Interesting, I think I agree. I will work on that after #2316 is realized, since this will already have a bunch of merge conflicts once/if that gets merged.

I realize that the folder structure could probably use some improvement (the memory tests, for example, were hard to find), but then let's fix that instead, in a separate PR.

Yep, that should probably come first then. I'm putting this PR on hold until those are addressed.

[mazunki]

This probably requires a do-over to avoid merge conflicts later down the line. Might be easier to rewrite the whole file too.

I'd like to introduce the following:

• --type= with comma-separated options: --type=unit,integration,stress. Defaults to all of them.
• --exclude to disable specific tests. Repeatable command: --exclude net/slaac --exclude net/router6
• Any depth level should be supported: ./test.sh net would run all tests under the net directory, searching for test.py files.
• A final summary of both passing (currently one has to scroll back to verify all subtests actually passed) and failed tests, with commands for failed tests as we do currently.