Test Framework

[eric-tramel]

Thanks to @sdrangan (and @mborgerding), we now have some base functionality of the vampyre package! And on the Christmas holidays to boot...

I'm curious about the current framework for tests. Currently, there exist a number of testing functions spread across most of the source code. I'm wondering if we shouldn't look at consolidating all of the test code. Additionally, is it necessary that these tests exist as modules within the package, or should they be completely external to the package?

If they remain internal to the package, then perhaps we can write the module vampyre.test (which already has a bit of a scaffold). If the tests exist outside the package, then all of these tests can be written as individual scripts in the top level test directory. Thanks to nose.collector, running all of the tests can be handled automatically; it seems that the test_version() is duplicating some of this functionality.

If agreed upon, I propose to re-write some of this functionality in whichever direction we want to take it.

[sdrangan]

Eric, I think I may have deviated from the preferred path in the test framework. Feel free to re-org the code in any way that you feel is best and I will follow that convention going forward. If you need me to make the changes, let me know. But, I did prefer having the unit tests within each module instead of creating a large number of new files. But, I am open to giving that up too if separating the unit tests is common practice. Thanks, Best, Sundeep

…

On Thu, Jan 26, 2017 at 8:26 AM, Eric W. Tramel ***@***.***> wrote: Thanks to @sdrangan <https://github.com/sdrangan> (and @mborgerding <https://github.com/mborgerding>), we now have some base functionality of the vampyre package! And on the Christmas holidays to boot... I'm curious about the current framework for tests. Currently, there exist a number of testing functions spread across most of the source code. I'm wondering if we shouldn't look at consolidating all of the test code. Additionally, is it necessary that these tests exist as modules within the package, or should they be completely external to the package? If they remain internal to the package, then perhaps we can write the module vampyre.test (which already has a bit of a scaffold). If the tests exist outside the package, then all of these tests can be written as individual scripts in the top level test directory. Thanks to nose.collector, running all of the tests can be handled automatically; it seems that the test_version() is duplicating some of this functionality. If agreed upon, I propose to re-write some of this functionality in whichever direction we want to take it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANO8I58DiP8BDjrTCtQl8cHk6oO1Idpyks5rWJ8igaJpZM4Luonx> .

[eric-tramel]

Hi Sundeep, no worries, just trying to think of sustainable going forward. I looked around at a few places and it seems like a standard convention to keep the development tests segregated from the main code. The reason for this is that the end user isn't very concerned with these kinds of tests, so it doesn't provide much utility to them to build the tests and include them in the outward-facing documentation. Additionally, I saw it pointed out in another place that it might help with readability of the core source to keep the tests in a separate location.

To give an idea of what a set of tests could look like with the nose.collector framework, I made a preliminary branch test-restructure (1757709). Basically I'm going with the following structure:

vampyre/
|- moduleA/
|--submoduleAA.py
|-moduleB/
|--submoduleBA.py
|--submoduleBB.py
|test/
|--test_submoduleAA.py
|--test_submoduleBA.py
|--test_submoduleBB.py

And inside these test files, one just needs to define the functions for each of the unit tests. In the case of test_submoduleAA.py...

from vampyre.moduleA.submoduleAA import functionA, functionB

def test_functionA():
    # do stuff, throw errors maybe

def test_functionB():
    # do stuff, throw errors maybe

There is still some logging/display issues (mostly just some undesired newlines), but these are really minor inconveniences. Let me know what you think.

[ThomasA]

Any plans of using unittest for implementing tests?

[eric-tramel]

At the moment, I've only built the framework for running the test files themselves via nose. I haven't specified how to write the tests themselves within the file. One of the nice features of Nose is that it will run anything that it considers a "test." This includes any unittest classes you define in the test files. So, whether the test is a stand alone function or a unittest, nose should run well.

There are obvious advantages to using gear unitttest classes, e.g. Not duplicating test configuration and cleanup. So this is certainly an approach we should consider.

[sdrangan]

I looked at this unittest framework, and it looks very good. If no one objects, I can start to migrate the tests to this framework.

[eric-tramel]

After speaking with Andre, Sundeep, and Phil at Les Houches, it seemed like the PyTest framework would be a better choice than Nose with respect to both functionality and documentation. So, I've re-written the test framework to use PyTest instead. I'll be submitting a pull request shortly.