Continuous testing and deployment

No user can push changes directly to the unstable branch. Updates to the unstable branch must come either via a pull-request, or via updates of the auto_merge branch. In either case, automated regression tests are run prior to the merge into unstable. The main difference between opening a pull request and pushing to auto_merge is that the former allows code review by other OMFIT developers.

OMFIT automation system is built upon jenkins, and runs on a dedicated workstation named MORTI (Macmini Omfit Regression Test Invigilator). The actual jenkins jobs can be found under OMFIT-source/regression/jenkins.

Writing regression tests

OMFIT regression tests are contained in the OMFIT-source/regression folder. These tests are Python files, each with a filename starting with test_.

They begin with a yaml formatted section with meta-data describing the attributes and requirements of the test. An example of how to construct this section can be found at OMFIT-source/regression/test_small_script.py The primary form of meta-data in the yaml section are labels. Information about each of the possible labels and when to use them can be found here

A tutorial about developing regression tests in OMFIT can be found here (Google docs, PDF)

All OMFIT regression tests should be loaded with the OMFITpythonTest class. Internally, these scripts make use of the OMFITtest class, which builds around Python’s standard unittest.TestCase class to provide all common unittest functionalities as well as interface with GitHub, email, and slack.

Details about the arguments that can be passed to the OMFITtest class can be found here.

Running regression tests

In order to run regression tests a tool called verify_build at OMFIT-source/bin/verify_build.py` should be run. verify_build takes in a set of labels to include and exclude and it selects and runs tests based on these labels appropriately. Additionally, some labels can imply other labels. In addition to the labels specified by the yaml at the top of the test, it is implied that each test has a label containing its own name.

Examples:

verify_build -all

This will run every regression test.

verify_build -i server -e iris

This will run all tests that have labels that imply they need to be run on a server (atlas, sata, etc) with the exception of tests that require the iris server.

verify_build -a -g -n -T 5 -e gui long nersc pppl gacode_local chease_local no_morti test_EFIT++

This is the default invocation of verify_build by MORTI. Broken down:

-a Automatically select tests based off of differences between itself and unstable

-g Post the results of the tests to github.

-n If a newer commit is made to the branch being tested, abort the remaining tests.

-T 5 Abort the test if it is taking longer than 5 times its expected runtime. (Default test runtime is 60 seconds.)

-e gui long nersc pppl gacode_local chease_local no_morti test_EFIT++ Exclude tests that use a gui, are known to be too long, require a nersc account, require a pppl account, require a local installation of gacode, require a local installation of chease, are specifically excluded from running on MORTI, and also don’t run the test with the name test_EFIT++

verify_build -l

List all the available labels and their descriptions.

verify_build -h

Additional help using verify_build