Skip to content

Managing Statements for safety-related Software

The Trustable Methodology can be used to track requirements for safety-related software. This document describes our current understanding of how data in a Trustable Graph should be managed to ensure the underlying argument possesses the following desirable safety properties:

  • Completeness
  • Correctness
  • Freedom from Intrinsic Logical Faults
  • Understandability
  • Precision
  • Modularity
  • Fault Detection
  • Defined responsibility

Warning

Completeness is not yet fully addressed by the process in this document. This is because we need weights to track completeness.

Contributors

Managing a Trustable Graph involves three classes of Contributor:

  • The Author(s)
  • The Software Reviewer
  • The Trustable Reviewer

who are collectively responsible for protecting the integrity and quality of the following activities and their outcomes:

The remainder of this document is dedicated to describing the specific responsibilities of Contributors in ensuring the aforementioned properties are achieved and maintained.

Danger

In practice, change requests often reflect the undertaking of several activities in parallel. For instance, often a change to the contents of an item will also include clearing the resulting Suspect links.

Each Contributor should always consider the responsibilities arising from all the activities that are undertaken by a change, whether this is explicit or implied.

Configuration

To apply the Trustable Methodology to a project, the following sets of Contributors must be identified:

Group Definition
Authors All Contributors to XYZ
Software Reviewers A subset of Contributors with proven competence in an area of the project. The collective competence of Software Reviewers should cover all aspects of XYZ.
Trustable Reviewers A subset of Contributors competent in the application of the Trustable Methodology and the use of XYZ.

The assignment of these roles must be documented and enforced. Within the scope of a single change request, separate people should perform each role.

Desirable properties

This procedure contributes to the property of Defined Responsibilities.

General Principles

This section details some general principles that all Contributors should consider when proposing or reviewing a change.

  • Suspect items are indicative of a Fault or Change in Evidence. MRs that introduce a Suspect item should never be accepted, unless they apply to an Evidence Item and are justified by reference to a fault or change in an Artifact.

    Proper uses of the review status

    • Contributor A discovers a bug in a Validator. They open an MR to mark all Evidence items using that Validator as Suspect, preventing inaccurate scores being used until the bug is fixed.
    • Contributor B pushes a change that causes changes to a generated Artifact. This causes the linked Statement to be marked as Suspect.

    Improper uses of the review status

    • Contributor C makes a Statement X and thinks it might support Statement Y. They link it to Statement Y and mark it as Suspect so other Contributors know they are unsure.
    • Contributor D makes a Statement Z. They identify it as Evidence, but being unable to find a suitable Artifact, they mark it as Suspect.

  • Suspect Links are Indicative of a Fault or Change in the argument. Changes that introduce a Suspect link should never be accepted, unless they are justified by significant and unavoidable changes to a Request that must be resolved at multiple levels in the Argument.

    Proper use of a Suspect link

    The Consumer decides they want to pivot immediately to new hardware. Contributor A changes Expectation X to reflect this, but leaves the links to it (from Assertions Y and Z) as Suspect. This then enables Contributors B and C to iteratively rework their argument (in Y and Z) for the new hardware.

    Improper use of a Suspect link

    Contributor D fixes a spelling mistake in an Assertion. They clear the resulting Suspect links to other Statements they own, but leave links to Contributor E's Statements as Suspect, expecting Contributor E to clear them later.

Desirable properties

These principles contribute to the properties of Fault Detection and Freedom from Intrinsic Logical Faults.

Actions

This section sets out the responsibilities of Contributors involved in a change to the Trustable Graph. All changes should be submitted through a merge request. Reviewers indicate that they have completed their responsibilities and are satisfied with the changes by approving the request.

A remark on scope

The scope of this document is restricted to ensuring proper implementation of the Trustable Methodology. This process should complement normal software engineering practice, not replace it. Where a Contributor has no specific responsibilities in the Trustable process, they are still responsible for the overall quality of the changes.

All Actions discussed below can be done either directly in a text editor, or by using trudag.

Adding Items

When a new item is added, Contributors have the following responsibilities.

Author

  • Must submit a complete change that does not require a response from Reviewers.

Software Reviewer

  • Confirms the new item relates to XYZ.
  • Confirms any referenced Artifacts are related to the item's Statement.
  • Confirms the new item is marked as reviewed.

Trustable Reviewer

  • Confirms the new item is a genuine Statement related to XYZ.
  • Confirms the new item is marked as reviewed.

Desirable Properties

This procedure contributes to the properties of Precision, Understandability and Freedom from Intrinsic Logical Faults.

When a new link is added, Contributors have the following responsibilities.

Author

  • Must submit a complete change that does not require a response from Reviewers.

Software Reviewer

  • Confirms the destination item is a necessary, but not sufficient, condition for the source item.
  • Confirms the new link is not marked as Suspect.

Trustable Reviewer

  • Confirms the destination item is a necessary, but not sufficient, condition for the source item.
  • Confirms the link does not adversely affect the modular structure of the Graph without justification. That is, it does not connect two unconnected subgraphs without justification.
  • Confirms the new link is not marked as Suspect.

Desirable properties

This procedure contributes to the properties of Correctness and Modularity.

When a link is removed, Contributors have the following responsibilities.

Author

  • Must submit a complete change that does not require a response from Reviewers.

Software Reviewer & Trustable Reviewer

  • Confirm they agree it is possible for the destination item to be False and the source item to be True.

    OR

  • Confirm the destination item is True if and only if a different combination of existing destination items is True.

Desirable properties

This procedure contributes to the properties of Completeness and Freedom from Intrinsic Logical Faults.

Removing Items

When an item is removed, Contributors have the following responsibilities.

Author

  • Must submit a complete change that does not require a response from Reviewers.

Software Reviewer

  • Confirms the item is not an Expectation for XYZ.

Trustable Reviewer

  • Confirms the item has no links, Suspect or otherwise.

Desirable properties

This procedure contributes to the property of Completeness.

Updating scores

When a score is added or updated, Contributors have the following responsibilities.

Author

  • Must submit a complete change that does not require a response from Reviewers.

Software Reviewer

  • If a Validator is used, confirms the action of that Validator corresponds to the Statement made in the Evidence item.

Trustable Reviewer

  • If a Validator is used, confirms it correctly uses the dotstop plugin system.
  • If a Validator is used, confirms the calculated value can be interpreted as a probability. That is, it is non-dimensional and is confined to the interval \([0,1]\).
  • If an SME score is provided, confirms each of the named SME's is an Author

Desirable properties

This procedure contributes to the properties of Correctness, Precision, and Freedom from Intrinsic Logical Faults.

Editing Items

When an item is edited, Contributors have the following responsibilities.

Author

  • Must submit a complete change that does not require a response from Reviewers.

Software Reviewer & Trustable Reviewer

Desirable properties

This procedure contributes to the properties of Correctness, Completeness and Freedom from intrinsic logical faults.

All Contributors should consider the change as equivalent to adding a new link, then follow the relevant procedure.

Reviewing Items

All Contributors should consider the change as equivalent to adding a new item and (if applicable) adding scores, then follow the relevant procedure.

Modifying SME Scores

Sometimes you may need to re-review an item, but one or more of the original SMEs are unavailable to confirm or adjust their scores. In such cases, it is acceptable to exclude the unavailable SME as an Author and then set their score to zero.

Assigning zero is a conservative approach that prevents inflating the item score beyond what is supported by available SME input. Always include a clear note explaining the reason for the score change.

Alternatively, scores from active maintainers may be updated by new SME Authors. These changes must follow the same review and change-management rules as adding or deleting items, with the new Authors' names and scores replacing those of the original individuals. Any differences in assessment are recorded in the project history and should align better over time as the project matures and calibration improves.

However, you may find it is no longer possible for an existing SME to continue to re-evaluate their score. At this point, it is at the discretion of a Trustable reviewer to determine if it is acceptable to remove the existing SME scores. The Trustable reviewer should evaluate the necessity of the SME score removal, with the aim that existing SME scores are not discarded to circumvent the original contributor's evaluations.

The following provides some guidance for certain scenarios:

  • If an SME is no longer working on the project, but is still available for comment, they should be notified of the action. At which point they should express any concerns about the proposed scoring changes (for example, if in their opinion an SME was scoring too high, this should be discussed, and all SMEs involved should come to some common understanding of the context involved in the statement and the reasons that statement might not be true).

  • If an SME is no longer working on the project, and is not available for comment for any reason, then it is up to the Trustable reviewer for the appropriate action to take. A Trustable reviewer may seek several other SMEs to provide scoring if they feel like this is needed to ensure the score is more accurate.