In regulated industries, it is critical for developers to maintain compliance with recommended safety standards to ensure the safety of the software being developed. Incorrectly implemented code could cause serious harm or loss of life. Industry guidelines and requirements around code coverage mitigate these risks.
Code coverage is a valuable part of the development process that shows the percentage of code executed by automated tests. This process is a source of valuable feedback that helps developers find bugs and errors and determine the parts of the code that require additional testing.
In this list, we’ll look at the automotive, aerospace, rail/freight, medical, and energy industries, noting the regulations that dictate code coverage requirements in order to help developers in those industries stay in compliance.
Automotive
The ISO 26262 standard provides guidelines and requirements for functional safety in automotive vehicles. It aims to mitigate the risks and dangers caused by electrical and electronic automotive systems and applies to all activities during the life-cycle of safety-related systems using electrical, electronic, and software components. By complying with the standard, developers help producers avoid and mitigate systemic failures.
ISO 26262 uses the Automotive Safety Integrity Level (ASIL) to ensure that automotive projects comply with safety measures. ASIL is a risk classification scheme defined within the ISO 26262 standard for the functional safety of road vehicles.
ASIL-A is the lowest degree of automotive hazard and ASIL-D is the highest. ASIL-D applies to systems where failure could result in death, e.g. airbags, anti-lock brakes, and power steering, while ASIL-C applies to features like cruise control, ASIL-B to headlights, and brake lights, and ASIL-A to rear lights.
Automotive software must pass a number of tests to be compliant with the standard. ISO 26262, section 6 provides recommendations for five categories of unit and integration testing. These are:
- Requirement-based testing
- Interface testing
- Fault Injection testing
- Resource Usage testing
- Back-to-back Comparison testing
Each category is assigned an ASIL which indicates whether it is recommended or strongly recommended.
Aerospace
DO-178B was created in 1992, as avionics systems transitioned from analog to digital systems with real-time embedded software. Previous versions included the original DO-178 standard which provided the original guidelines for the certification and approval of software-based equipment and systems and DO-178A, meant to reflect the regulatory and industry experiences gained since DO-178.
DO-178B introduced new certification regulations and guidelines to ensure that software production for aerospace systems was performed at a level that met security and airworthiness requirements.
In 2011, DO-178B was superseded by DO-178C. DO-178C specifies a verification software testing process that calls for increased software testing and documentation as the software level becomes more critical. It also requires that tests be verified against requirements and for an analysis of all code.
The stand-alone document, DO-278, addresses software for communication, navigation, surveillance, and air traffic management systems for ground-based software systems only.
The Verification Software Testing process
DO-178C classifies software into the following five levels of criticality based on whether unexpected software behavior could cause a system to fail.
Software failures can cause billions of dollars in losses for the airline industry, but money isn’t the only concern. Recent software failures on the Boeing 737 MAX planes remind us that software failures can also result in loss of reputation, customer confidence, and unfortunately, loss of life. Code compliance in the airline industry is crucial to ensure that software is well designed, correctly implemented, and thoroughly tested.
Rail / Freight
EN 50128 specifies the process and technical requirements for the development, deployment, and maintenance of safety-related software for programmable electronic control systems and protection applications used in railway control. The software safety route map defined in EN 50128 outlines the following steps:
The following roles and independence requirements are used to verify the competency of those responsible for the development of the software:
- Implement the relevant parts of EN ISO 9001, according to the guidelines in ISO 90003.
- Implement the safety process under the control of an appropriate safety organization that complies with the “Safety Organisation” sub-clause in the “Evidence of Safety Management” clause of EN 50129, except at software safety integrity level zero.
- All persons involved in all the phases of the Software Lifecycle, including management activities, must possess the appropriate level of training, experience, and qualifications.
- An independent assessor for the software shall be appointed.
- The assessor shall be given authority to assess the software.
- The parties involved throughout the Software Lifecycle shall be independent to the extent required by the software safety integrity level
Techniques and Measures Requirements
Four techniques and measures requirements are provided by EN 50128 for assessment, design and code standards, programming languages, and static analysis:
- “M”: the use of a technique is mandatory;
- “HR”: the technique or measure is Highly Recommended for this safety integrity level.
- “R”: the technique or measure is Recommended for this safety integrity level.
- “NR”: the technique or measure is positively Not Recommended for this safety integrity level.
Medical Devices
IEC 62304 is the functional safety standard that guides the safe design and maintenance of medical software or medical devices with embedded software. It provides processes, activities, and tasks that apply to the development and maintenance of medical software to ensure safety.
IEC 62304 defines three software safety levels:
Class A: No injury or damage to health is possible Class B: Non-serious injury is possible Class C: Death or serious injury is possible
One major concern to medical software development is software of unknown provenance (SOUP). SOUP refers to existing software not originally developed for a medical device. This includes off-the-shelf software integrated into the product to reduce development time and costs. Documentation for the development of SOUP is usually not available, making it difficult to verify.
Part 5 of the IEC 62304 standard describes the tasks and activities of the software development process for each medical device class:
- 5.1: Development planning – Creation of a software development plan
- 5.2: Requirements analysis – Specification of the software requirements
- 5.3: Architectural design – Development of a software architecture
- 5.4: Detailed design – development of a detailed design
- 5.5: Unit implementation and verification -implementation of the code according to coding guidelines and verification of code through code reviews and unit tests.
- 5.6: Integration and integration testing – application integration and integration tests
- 5.7: System testing- testing of the complete and fully integrated software product
- 5.8: Release – release of the software
!Software dev process IEC 62304](https://i.imgur.com/pEYo1ct.png “Software dev process IEC 62304”)
Energy
The IEC 61508 standard is an international performance-based standard for electrical, electronic, and programmable electronic safety-related systems. It is closely related to IEC 61511, which provides a similar project model but was designed for process control applications.
IEC 61508 defines requirements for ensuring that systems are designed, implemented, operated, and maintained at the required safety integrity level (SIL). The standard defines four SILs based on the risks associated with the software application starting with SIL 1 (lowest) to SIL 4, the highest level. The higher the SIL level, the higher the associated safety level and the lower the probability that the system will fail to perform as expected.
Unit Testing requirements are found in the IEC 61508 Dynamic Analysis and Testing phase and seek to ensure that all paths and branches of the software are fully tested. Tests must be performed at the unit level and be based on the software and/or code specification.
Techniques and measurements related to unit testing include test case execution from boundary value analysis, error guessing, error seeding, model-based test case generation, performance modeling, equivalence class and input partition testing, and structural coverage for entry points, statements, branches, and MC/DC.
The MC/DC Code Coverage Criterion
The Modified Condition/Decision Coverage (MC/DC) is a criterion used for code coverage that determines the effectiveness of a test suite and is used to ensure that adequate testing has been performed on safety-critical software. If there is a decision, all possible conditions and factors associated with that decision must be tested.
Conclusion
Code coverage is important, especially in industries where a software error can mean the difference between life and death. Some industries have a quagmire of regulations to navigate, which can be confusing and time-consuming.
Codecov makes compliance with industry regulations and standards easier. With Codecov, you can send your code coverage to any storage platform to ensure compliance and review coverage changes to confirm industry requirements.
