Automation

The final A in RAFIA stands for 'Automation'. We aim to automate as many steps in the software development process as possible, and collect the results of these steps in association with all changes to the software and its supporting documentation. This ensures that these steps are auditable, and ideally repeatable, and enables us to determine the impact of a change in its given context.

Automated builds

The automated construction of the software, and of the tools and environments that are used in its construction and verification, is covered by the TT-CONSTRUCTION tenet.

Automated tests

Wherever possible, verification of the software should be accomplished using automated tests.

System-level and integration tests are the primary focus of RAFIA, but component tests may add another layer of verification before system tests are executed, or be used to verify aspects of system behaviour that it is not feasible to replicate in the available system testing environments.

Test specification and analysis

This diagram shows how the specification of tests interacts with both Test Data Analysis and Risk Analysis.

The TSF Specification is an organised graph of records documenting:

• Expectations related to the software, how it is intended to be used, and how it should be developed, maintained and verified
• Assertions related to these, specifying criteria that need to be satisfied
• Evidence that exists or is generated (e.g. by an automated test) to support these assertions, or to determine the extent to which they are satisfied for a given iteration of the software

The Test Specification forms part of this evidence. It consists of Items in the TSF graph that reference the associated test scenario files, which are stored in a git repository. It documents:

• The Test Scenarios that are used to verify the behaviour of the software, the expected results of executing them, and any test data that needs to be collected for further analysis (see Test Data Analysis below)
• The Test Contexts within which these scenarios are to be executed. These may be actual target systems, virtualised simulations of these, or some other approximation of the environment within which the software under test will be deployed. A context definition may also include specific configurations of the hardware or software with which the software is integrated.

Test Data Analysis examines the results of executing a test, and processes these to produce further evidence.

• Test Results are the direct outputs of an executed test, which may include both the pass/fail outcomes of tests, and test data collected before, during or after the execution of tests.
• Test Metrics are derived from these results, typically by processing this 'raw' data and extracting specific parts of the data or calculating values derived from it. The result data that needs to be collected as an input to this process should be defined in the test scenarios; the desired extracts or inputs and the algorithms for processing them should be specified in, or referenced by, assertions.
• Historical Statistics are derived by analysing metrics recorded for previous executions of the same test(s); for example, examining how often or how frequently a specific test metric fails to satisfy its criteria, or examining trends in a given performance metric over time.

Test design and implementation

This diagram shows how Test Design and Test Implementation both inform and are informed by the specification and analysis.

The Test Design is derived from the Test Specification for execution in a given context. This defines executable sequences of actions that correspond to the scenarios, for use in specific types of test.

Pre-merge Tests are run for every proposed change to the software and its associated test contexts. These tests should all pass (or fail as expected, e.g. in the case of false negative tests). Any issues with these tests must be resolved before a change is permitted to be merged into the software's mainline. These should consist of:

• Happy Path Tests, which execute a scenario under 'normal' conditions, where the software's intended behaviour is expected to be observed.
• Exception Handling Tests, which execute scenarios where a known failure mode or other exceptional condition exists, resulting in an expected deviation from the 'happy path'.
• False Negative Tests, which deliberately create the conditions in which an existing test scenario should fail (using Fault Induction) to verify that the test both detects the condition (i.e. fails) and reports the expected failure (and not a different failure).

Post-merge Tests are periodically executed for integrated sets of changes (e.g. on the mainline, or a release or product branch). These may consist of:

• Soak Tests, which exercise the software in more extended sequences, or over an extended period of time.
• Stress Tests, which extend soak tests to run in a test context that simulates 'abnormal' system conditions, such as constrained memory or storage. capacity, misbehaving system processes, or high volume external input
• Performance Tests, which extend soak or stress tests to collect specific measurements of the software behaviour, or to record observed deviations from expected performance criteria.

A Test Implementation may be required to enable test definitions to be executed in a given test context. This might include:

• Stressor implementations, artificially induced conditions intended to provoke Misbehaviours in a stress test.
• Other Fault Induction implementations for use in false negative or exception handling tests, which may include deliberately misconfigured components, simulated dependencies or client processes, and deliberately broken versions of software components or dependencies.

See Misbehaviours for more information about how different types of test and analysis of test results can help to inform other activities, and how their results can in turn be used to inform test design.

Automated verification

There are many verification actions beyond (as well as connected with) automated tests, which can and should be automated wherever possible.

The results of an automated build or test might be verified as part of an associated CI job, for example, or fed into a separate verification job. Verification actions here might be as simple as checking that the expected result outputs have been produced and successfully stored for each build of test job, or they might involve complex analytical processes to support Test Data Analysis.

The use of static analysis and linting tools to check source code is also a well-established practice. This class of tools can be extended to include spell-checking for documentation, formatting rules for code or commit messages, and other tasks that take the burden of review away from humans.

As part of linting, the status of Statements and their links to the risk analysis help ensure that impact analysis for affected Statements is visible for every change. This is achieved when the example Statement management process is followed, ensuring that link review is applied systematically.

Manual verification

When some aspect of the software cannot be verified by an automated test, this step must be performed by a human reviewer. Examples include reviewing the implementation of a test to verify that it conforms to a test specification, or confirming that a test specification covers all of the criteria specified in an assertion to which it is linked.

Verification actions performed by humans should always be performed on files under change control, or reports generated by an automated process (such as a build or test), or documents produced by an automated documentation generation process.

Automated document generation

Any documents needed for manual verification should be generated by automated processes, using either input files managed under change control (e.g. the content of a document stored as Markdown in a git repository) or the results of some other automated verification actions (e.g. the results of running an automated build or test).

This also allows easy distribution of additional diagrams or figures provided alongside the risk analysis, associating them with Statements as qualifying information. Reviews should ensure these materials are appropriate and understandable (e.g., include clear legends).

Automating and coordinating these processes alongside all of the other automated actions serves two purposes: it minimises the time human reviewers need to spend trying to find and identify the review inputs for a given software iteration, and it can help to ensure that the set of documents required for a review are reproducible and clearly associated with that iteration.

Such tight automatic coupling of documentation ensures that partitioning and prioritisation of work are guided by linked safety analysis outcomes.