User Guide

Logging In

Note

If you have not already joined a server you will need to do this. See Joining a Server for more details.

You can either log in to BrumSoftTest or you can view the options as a guest. To log in go to https://best.sr.bham.ac.uk:8080/login and enter your email for a username and you password. If you have forgotten your password you can ask an admin to change it for you. Alternatively, you can click Log in as guest to view projects that have enabled public viewing.

Joining a Server

To access the datastore, run tests and create projects you will need an account. On the log in page click Register, on the registration page enter your email address, create a password and enter your real name in the first three boxes. These boxes are mandatory.

Optionally, you can add a link to a profile image in the profile image box, this needs to be a URL pointing to a image on a webserver (i.e. https://example.com/me.jpg).

The key is a short passphrase granted by the admin. Email ajones@star.sr.bham.ac.uk for this key.

Running a Test Session

In the left hand side bar, click the project you for which you wish to run a test, a drop down menu should open, click Submit New Test.

Test Submission Form

A complete test submission form.

Once the test submission form loads you will need to choose a test profile. Single tests can be either folders or single test files. Test profiles are groups of these, see Adding a Test Set for more information on setting up and using test profiles.

Click submit to load the test profile and a list of files will be shown. The environment details will list the test environment options, such as which docker container you want to execute the tests in.

Finally select the git branch information. You can test just one commit or run the same test over multiple commits. Enter the commit ID, branch name into the relevant box, or list all the commits you wish to test in the larger box. Alternatively you can select commits from a list.

Once you click submit the your test will be queued. You should be redirected to a status page. Refresh this page to get an updated status.

Viewing Test Session Results

You can find the results of a test session either by searching or you will be redirected to this page after submitting a test session. For information on accessing a test session see Searching Session History and see Running a Test Session for information on submitting a test session.

The first table shows the session information which is common to the to all tests. The second table shows information about the commit which was tested. The third table shows the results of each test.

Session Information

The session information contains information about the environment in which the tests were executed. The session code provides a general indication of the results. At the bottom of this box there are buttons to re-run the test, view data and view the session logs.

Session Status Code

Session status can be one of the following:

  • Passed: The session ran successfully and all tests completed passed all programmed checks.
  • x/y Fails: The session appears to have run successfully and x tests out of y failed.
  • Queued: The session is queued and will execute when there is capacity on the server.
  • Running: The session is currently running, tests results will be updated as they complete.
  • Config Error: The session ran without error but did not execute any tests. Is the profile information correct? Check the the session log and session profile for more information.
  • Session Failed: There was an miscellaneous error running the session. Check the session log to find out why.

Re-running tests

If you are not happy with the results of the test, for example it was killed or there was some other interruption, you can rerun the same test on the same commit by clicking Re-Run Test (Same Commit).

Alternatively, if you have fixed code in a new commit, you may wish to re-run the test on the latest commit for the same branch. For this case click Re-Run Test on latest <branch name>.

Session Data

The session log is the first place to look if the session failed to run correctly. The session log follows the BrumSoftTest logic and should list the following items:

  1. Was a conda environment file found?
  2. Was a requirements.txt file found?
  3. Was a .best_install.sh file found?
  4. Was a .best_env_vars.sh file found?
  5. Expansion of folders into tests.
  6. List of rejected files.

If you are sure that your configuration is correct and your test fails to run correctly please email ajones@star.sr.bham.ac.uk and I will help you fix the problem.

Test Information

The test information is shown in the lower box. The parameters are summarised below.

Column Name Details
Test ID Unique ID for this test.
Status Status of test. If the test was executed without error and returned error code 0 the status will be passed. If the test raised a non-zero error code this will be displayed. If the test was killed the status will be killed. If the test failed in an unexpected way the status be Test Failed Catastrophically. If there was any kind of failure check the test output for details.
Test File Path The name of the test file that was executed. Hover to see the full path.
Duration Duration in seconds of the test.
Project Specific Columns The administrator may define additional columns here. STDOUT will be parsed in order to determine these values. See Project Specific Columns for information on setting up project columns.
Data Files Links to data files generated during the test. Note you will need to be logged in to view these.
Output Links to STDOUT, STDERR and data files.

Visualizing Test Results

The visualize data button builds a matrix with tests on the vertical axis, test sessions on the horizontal axis and return codes in the each of the cells. This helps identify when a bug occurred.

Searching Session History

There are several ways to find interesting test results:
  • Dashboard: A brief summary of your recent tests are available on the dashboard
  • Project Session History: A list of all test sessions is available by clicking the project in the sidebar and then clicking Session History
  • Test History: Search though all tests in a project to find out where an error occurred.

Accessing Cached Data

Note

To access cached data you must be logged in!

See also

See Caching Data for details letting tests cache data