DO-178C and DO-178B Software Certification

RTCA/DO-178C, published in 2011 is the de facto guidance document used in the development of airborne software and was published as an improved revision to RTCA/DO-178B. Although not a required standard, the FAA continually refers to DO-178B and DO-178C as the recommended means to address safety in software. Today the use of computer technology in avionics applications is widespread; from the F-35 Joint Strike Fighter to commercial aircraft such as the Boeing 787 and Airbus A380. Applications range from safety-critical flight control functions to on-board passenger entertainment systems. Users in the military community and unmanned systems are also relying on DO-178 as the guidance document for software development.

The level of effort to comply with the objectives of DO-178 will vary based on software criticality (depending on how software can contribute to a failure condition). The level of effort is also proportional to the size of the software under consideration. DO-178 defines five software levels, each related directly to the failure condition that can result from anomalous behavior of the software. The software level definitions given in DO-178 and the number of objectives required to satisfy the requirements of each level are shown below.

DO-178C software levels

DO-178 deliverables

The software life cycle data required by DO-178 includes the following:

Addressing the planning, requirements and verification processes

The planning process begins with the Plan for Aspects of Software Certification (PSAC). The PSAC describes the scope system and software that will be considered for certification. The PSAC also describes the overall software life cycle, the software development plan, the software verification plan, the standards that will be used along with the Software Configuration Management Plan and Software Quality Assurance Plan. These core plans and standards define the framework of how the software will be developed and verified along with the transition criteria for each life cycle phase. The PSAC should also call out any tools that will be used to support development and verification processes and identify whether those tools need to be qualified because they automate a process of the software life cycle.

For Level A, B and C software both high-level and low-level requirements need to be developed and verified for the software. For Level D software, only high-level requirements are developed and verified. The verification activities (for levels A-C) should include reviews of requirements, design, code, test cases, test results and coverage analysis. Verocel’s tools such as VeroTrace can manage and control all life cycle data (including reviews and test procedures and results). Verocel’s VerOCode and VeroSource tools are qualified to DO-330, TQL-5 and help with object code and source code coverage respectively.

Previously developed software and DO-178

It is common for applicants to take an existing set of functional software through the DO-178 certification process rather than develop software in a waterfall model while producing the certification artifacts. DO-178 is written as if a waterfall model of development is used, but any software development model could apply. The same objectives apply to software that is engineered to meet DO-178 as software that is developed to meet DO-178 from the start. As shown above, there are many planning documents produced under DO-178. Planning involves not only a strategy for certification, but also a detailed implementation of how the certification will take place including software quality, configuration management, requirements, design and coding standards, and a detailed plan of how the software will be verified. This planning activity involves the entire engineering and quality team and may likely take a number of months to complete.

Software prototypes

One way to carry out a process is to use a software prototype (assuming the prototype under consideration is functional and operational) and to capture a set of complete requirements that can be used for test and verification purposes. Normally, these requirements would include both high-level and low-level requirements that map to specific functions in the source code. These requirements would need to be reviewed (independently for Level A) and then used as a basis to construct the test cases for the software. However, before the testing process can begin, the detailed design should be either extracted from the existing source code or developed after the high-level requirements are approved.

The design can be extracted from comments in the source code and put into a descriptive textual document (that shows compatibility with high-level requirements). However, this low-level design information is usually not sufficient to reflect the entire software design. There also needs to be a high-level design document that describes how all the software components will work together, their interdependencies and timing relationships, etc. The low-level and high-level designs also need to be reviewed for accuracy and consistency among other objectives. Once the requirements and design are complete and approved, efforts to review the source implementation and production of test cases can begin. It should be noted that these efforts (requirements, design, code and test) can all happen in parallel provided sufficient configuration management of any approved artifact is in place.

Useful metrics

The cost of DO-178 certification varies greatly depending on engineering expertise and code size. One has often heard of the “multi-million dollar” answer when asking the cost of certification of any software to DO-178, regardless of size.

So how to more accurately predict the effort required?

Effort scope

There are a number of ways to scope the level of effort required for DO-178 certification. One useful way to scope the effort is to examine the size of artifacts from previous certification efforts. Consider the example of an operating system certified to Level A: the source code consists of about 12,000 lines of code which resulted in the generation of 1,300 requirements for approximately 700 functions. When determining scope of effort, one useful metric is to plan on 2-4 requirements per function – or 1 requirement for every 5-10 lines of code. Then each requirement will need to be tested, resulting in 1 test procedure for every 2-4 requirements. All requirements, design, code and test artifacts need to be reviewed via checklists. Additionally, the artifacts need to be linked to show traceability between requirements, design, code and tests/results.

What does it cost to certify a line of software to DO-178?

This is a common question asked by software managers. Intuitively, it would seem that it is much costlier to certify software to Level A rather than Level C, given that Level C has only 62 objectives and Level A has 71 objectives to be met. But experience has shown that the difference in cost between Level A and Level C is not that great. This is evidenced by the fact that all the deliverables defined above are required both for Level A and Level C software. In fact, the software planning and software development objectives under DO-178 for Level A and C are identical! The differences between Level A and Level C from a verification perspective impact the requirements, design, testing and analysis processes. Level A verification has the added requirement of independence for some objectives (where the development and verification efforts must be accomplished by different persons). This additional requirement can indeed add engineering labor to the certification effort; however, the added cost is likely to be a small percentage of the total certification effort.

Where Level A certification can get expensive is usually through the certification scrutiny an applicant will face when all verification efforts are complete. Convincing an auditor or host of auditors that have the responsibility to “sign off” on software where loss of life could result from failures can be challenging if the applicant has not adequately addressed the planning, requirements and verification processes. Audits can result in added activities that extend project schedules and increase costs. All Level A certification should be planned with a schedule buffer for planned audits and potential rework.

Assessing cost

Assessing a cost for all this effort requires an estimation of the engineering labor required for each development and review activity. How long does it take an engineer to produce a requirement from an existing implementation? One way to find the answer is to ask the engineers to estimate the effort for each activity (it is presumed that the engineer is very familiar with the software and design). Thus, a combined estimation can be built based on the engineering labor assessments for requirements, design, tests, reviews, etc.

Experience has shown that the cost of certification per line of code can range from $25 to $100 depending on the cost of labor in a given organization. That would mean that certification of 100,000 lines of code could range from $2.5 million to $10 million. This range should not be applied to any software project however; it is only a guideline that must be substantiated and corroborated by the assessment of metrics, a detailed analysis of the source code, engineering judgment, and compensation for risk factors such as certification of object oriented design.

Tools

Documents