Continuous testing and deployment¶
No user can push changes directly to the
Updates to the
unstable branch must come either via a pull-request, or via updates of the
In either case, automated regression tests are run prior to the merge into
The main difference between opening a pull request and pushing to auto_merge is that the former allows code review by other OMFIT developers.
OMFIT automation system is built upon jenkins, and runs on a dedicated workstation named MORTI (Macmini Omfit Regression Test Invigilator). The actual jenkins jobs can be found under OMFIT-source/regression/jenkins.
Writing regression tests¶
OMFIT regression tests are contained in the
OMFIT-source/regression folder. These tests are Python files, each with a filename starting with test_.
They begin with a yaml formatted section with meta-data describing the attributes and requirements of the test. An example of how to construct this section can be found at
The primary form of meta-data in the yaml section are labels. Information about each of the possible labels and when to use them can be found here
A tutorial about developing regression tests in OMFIT can be found here (Google docs, PDF)
All OMFIT regression tests should be loaded with the
Internally, these scripts make use of the
OMFITtest class, which builds around Python’s standard
unittest.TestCase class to provide
all common unittest functionalities as well as interface with GitHub, email, and slack.
Details about the arguments that can be passed to the OMFITtest class can be found here.
Running regression tests¶
In order to run regression tests a tool called
verify_build at OMFIT-source/bin/verify_build.py` should be run.
verify_build takes in a set of labels to include and exclude and it selects and runs tests based on these labels appropriately.
Additionally, some labels can imply other labels. In addition to the labels specified by the yaml at the top of the test, it is implied that each test has a label containing its own name.
This will run every regression test.
verify_build -i server -e iris
This will run all tests that have labels that imply they need to be run on a server (atlas, cori, sata, etc) with the exception of tests that require the iris server.
verify_build -a -g -n -T 5 -e gui long nersc pppl gacode_local chease_local no_morti test_EFIT++
This is the default invocation of
MORTI. Broken down:
List all the available labels and their descriptions.
Additional help using