Advice from a DER

Patmos Engineering Hardware Development

Advice from a DER from DO-254 newsletter

The following articles were published in the “Advice from a DER” column in the Logicircuit, Inc. DO-254 newsletter. Tammy is the DER who audits Logicircuit’s DO-254 compliant IP.

The Reason Behind DO-254

Importance of “Process Assurance”

Are Checklists Really Necessary

Why Hire an FAA DER?

Don’t Make These Mistake: #1 Creating Plans After the Fact

#2 Mistake: Treating CM as an “End of Process” Activity

#3 Mistake: Improper Retention of Verification Results

#4 Mistake: Not Understanding the Purpose of Traceability

Advice from a DER – The Reason Behind DO-254

Tammy Reeve | Patmos Engineering

Tammy Reeve

(September 2013)
Many times I hear engineering and management teams complain that DO-254 costs them a lot of time and a lot of money.  But when I get to review the work that they have done to comply with DO-254 and the latest guidance from the FAA and/or EASA it really comes down to poor planning and a lack of understanding of why they are applying this design assurance standard in the first place.  Always remember that the reason DO254 is being required is as a means of compliance to the CFRs (or EASA CSs) in order to demonstrate that it “performs its intended function under all foreseeable operating conditions”.  The Design Assurance level (DAL) adjusts the amount of rigor required in order to demonstrate this.  Keeping the primary goal and understanding of the Certification regulation in mind will keep you spending time and effort on what is important, that it functions as intended in the aircraft.

Tammy Reeve
DER/Founder
Patmos Engineering Services, Inc.

top

Advice from a DER – Importance of “Process Assurance”

Tammy Reeve | Patmos Engineering

Tammy Reeve

(November 2013)
Process assurance is a really important role in a DO-254 program. If done well, it can save companies a lot of time and expense by making SOI audits go smoothly and avoiding unnecessary rework. As an example, I was recently holding a SOI-3a audit, which includes checking the ability to rebuild and regenerate the test environment and results. While the company had the necessary documentation as part of their Hardware Verification Plan (HVP) there was still a lot of confusion about which files to pull and where the evidence resided – which boiled down to versioning and control of the files. So I suggested two things. 1) Creating a VCI (verification control index) document to control all the test bench files, models, scripts –basically the collection of items used when testing the FPGA. After all, this repository of data is very similar to the design (hardware) repository, which is controlled by an HCI. 2) Holding a Process Assurance “test readiness review” prior to the formal test for credit audit to run through the process internally prior to the SOI audit. The Job Aid, Section 3-3, provides a really good guide for what to review. Running tests without proper conformity of the test environment and design could result in a full rerun of tests, which can take weeks! So doing these two fairly simple things can save time and frustration in the long run.

In my example above, these small changes made a world of difference the next time around, and the Process Manager referred to these suggestions as one of the most important tips he ever learned from a DER. Try it yourself and see the difference it can make in your processes.

Tammy Reeve
DER/Founder
Patmos Engineering Services, Inc.

top

Advice from a DER – Are Checklists Really Necessary

Tammy Reeve | Patmos Engineering

Tammy Reeve

(January 2014)
No one really likes filling out checklists for reviews. So do you have to have them?  Truthfully, the answer is no. But for practical reasons, I’d like to say “Yes” because they can be a very helpful tool in aiding reviews and showing evidence of compliance to DO-254 objectives.

Many of the verification objectives in DO-254 will be satisfied by reviews and analysis.  When you perform reviews to satisfy objectives in DO-254 you need to provide evidence that they were performed with the correct independence and on a controlled, retrievable version of the data under review.  You also need a way to show that the specific objective was evaluated and any actions were recorded and resolved in a revised, released version of the data.  Checklists provide an excellent means to do this.

Now if you’re doing to take my advice and use checklists, here are a few helpful hints:

  • Annotate and connect checklist items to the development and verification planning standards and activities.
  • Develop checklists of a reasonable size so as to be manageable. The point is to help remind reviewers to check items that are required by the standards and activities.
  • Remember to include the names and roles of the reviewers, the version and title/number of the items under review (including any other items needed to evaluate the data item under review such as trace matrix, standards, etc.)
  • Be sure that actions recorded from the review are tied back to the questions on the checklist. This ensures you can show evidence that all the review evaluations have been considered and actions closed.
  • Show how each action was resolved and in which version of the data item it was resolved.
  • Most importantly know where you keep all these checklists for when the auditor asks for them.  Remember these are verification outputs and should be treated with the same formality as Test results.

Tammy Reeve
DER/Founder
Patmos Engineering Services, Inc.

top

Advice from a DER – Why Hire an FAA DER?

Tammy Reeve | Patmos Engineering

Tammy Reeve

(March 2014)
Nobody likes to wait for things they need.  Waiting can be a waste of time, and time is money.  A key value of having a Designated Engineering Representative (DER) is that they can quickly answer your questions and help you avoid the waiting that will sometimes come when dealing with the FAA alone.

While technically you don’t need a DER in your DO-254 process, the FAA can authorize and typically will encourage your use of FAA approved DERs in your project.  You can stand in long lines (figuratively) and wait for a resource from the FAA. But the FAA has limited resources, and despite their best efforts, sometimes the wait times to get an FAA auditor are quite long.  Even for projects that are not FAA certification programs or are TSOs, if you are being asked to comply with DO-254, working with an FAA authorized Airborne Electronic Hardware DER will definitely save you time, as well as provide you with advice and audit preparation, which is a very valuable facet not offered by FAA auditors. This advice and preparation can save tremendous time, effort and cost in avoided rework.  Also, consider having an FAA DER involved even in the review of proposals you are bidding on.  This can help you avoid costly oversights in what compliance requirements you may not be aware of in the wording and the efforts involved in satisfying these contractual requirements.

DO-254 compliance is complicated and costly. Investing a little more upfront to work with a DER can save significantly in terms of audit scheduling and makes your whole compliance experience run much more smoothly and efficiently.

For more information on how to find a credentialed DER, feel free to email me: tammy@patmos-eng.com and please check the FAA consulting directory as well https://www.faa.gov/other_visit/aviation_industry/designees_delegations/designee_types/media/derdirectory.pdf.

Tammy Reeve
DER/Founder
Patmos Engineering Services, Inc.

top

Advice from a DER – Don’t Make These Mistake: #1 Creating Plans After the Fact

Tammy Reeve | Patmos Engineering

Tammy Reeve

(July 2014)
The next few newsletters are dedicated to discussing the top mistakes that are commonly encountered in the compliance process. The # 1 mistake is…Creating Plans After the Fact

It seems that many folks mistakenly believe that compliance to DO-254 is simply an exercise in filling in the boxes with the documentation required, and that the order or timeline of document creation is not important.  This is a serious misunderstanding of the intent of DO-254 design assurance.

DO-254 is needed because it is nearly impossible to show that today’s complex hardware functions comply to the Certification Authority (FAA/EASA/Etc.) regulation xx.1301 and xx.1309 “Perform intended function under all foreseeable operating conditions.” DO-254 was written as an agreed to industry design assurance strategy that can be used as a “means of compliance” to this regulatory requirement for complex hardware (see AC20-152).

As part of the DO-254 process, Planning is essential because it describes specifically “how” each of DO-254’s general objectives and activities will be met for a particular project. The plans then become the “contract” with the Certification authority for how a company will proceed in all development and testing aspects in order to meet the regulatory rules.  Review and agreement of the plans is important because it shows that there is an understanding of the needed reviews, transitions and analysis throughout the development of the complex hardware in order to ensure that the system is “performing its intended function” and is as free of errors as possible.

The design assurance level (DAL) A-E is a way to communicate the potential level of impact a device failure would have on passengers (with DAL A being very critical). The DAL modulates the objectives of DO-254 such that there is more rigor required when the impact of failure is higher. The planning documents must show this additional rigor and compliance to these additional objectives based on the DAL level.

In addition to these considerations, planning documents need to acknowledge and address certification authority or aircraft specific issue papers or certification review items (CRIs).  One such example is dealing with single event effects (SEE) in hardware, caused by high-speed neutron effects on SRAM based devices.  Another example is certification considerations for using COTS IP, which must adhere to DO-254 standards.  Planning documents should document the developer’s approach on these crucial subjects and applicants should reach agreement with authorities early in the process. Waiting until the end could potentially result in major product redesigns with major cost and schedule implications.

Tammy Reeve
DER/Founder
Patmos Engineering Services, Inc.

top

Advice from a DER – #2 Mistake: Treating CM as an “End of Process” Activity

Tammy Reeve | Patmos Engineering

Tammy Reeve

(November 2014)
This newsletter picks up where we left off last time, identifying the top mistakes that DO-254 applicants make. The # 2 mistake is…Treating CM as an “End of Process” Activity

DO-254 requires Configuration Management (CM) and control of the data not only during the service life but also during development and verification of the item.  Hardware developers commonly misunderstand the intent of DO-254 as it is applied to FPGA/PLD/ASIC development programs and treat CM as an “end of process” activity.  Sometimes this is done to avoid the overhead of the change control (problem reporting) process. This clearly violates the objectives and concepts associated with a DO-254 development assurance process.

The DO-254 process requires change control (and data storage) objectives to be maintained throughout the development life cycle, starting at the Planning Phase.  Data utilized to satisfy a DO-254 objective, and which is then relied upon for downstream development or verification activities, must be controlled formally with proper change management.  This is done to ensure that proper evidence and data control is maintained for these downstream activities and data items (requirements, design, code, tests, review results, etc.).

Data items that are identified in DO-254 as hardware control category 1 (HC1) can then only be changed after a release using a formal change process, which is facilitated through a Problem Report (PR).  This formal process ensures that the impact of any changes to data items, which are crucial in establishing the design and verification of the FPGA/PLD/ASIC, is understood and agreed upon by all affected participants in the development life cycle.

The focus of DO-254 is about controlling the process, not the output. Thus, these “in process” rather than “end of process” configuration management activities are a vital part of the “development assurance” that DO-254 mandates.

Tammy Reeve
DER/Founder
Patmos Engineering Services, Inc.

top

Advice from a DER – #3 Mistake: Improper Retention of Verification Results

Tammy Reeve | Patmos Engineering

Tammy Reeve

(January 2015)
The #3 common mistake: Improper Retention of Verification Results

DO-254 has several objectives related to verification, which can be accomplished through reviews, analysis or test.  The results of these activities must be retained. A common mistake in the DO-254 process centers on retention (or lack thereof) of verification records associated with reviews and analysis as well as retention of test results. This, in addition to review of the final verification results (tests and analysis) is an area that applicants commonly miss or do not give proper emphasis.

Verification records should contain a clear correlation to the pass/fail criteria, which could be a development standard or a test result.  These verification records should contain the author/reviewer, date, and any items used in the including their versions. Any failures or issues found should be correlated to the standard that has been violated. Test results should be clearly linked to their associated tests and requirements, and should be reviewed and summarized in the verification results documents.  Test Results should be reviewed to be sure that the actual and expect results are giving the correct results and that the tests are passing. Review results, which must also be retained, are important because they show compliance to standards during development (and for DAL A and B projects the review participant’s names provide evidence of independence).

Tammy Reeve
DER/Founder
Patmos Engineering Services, Inc.

top

Advice from a DER– #4 Mistake: Not Understanding the Purpose of Traceability

Tammy Reeve | Patmos Engineering

Tammy Reeve

(November 2015)
This newsletter continues with the top mistakes that DO-254 applicants tend to make. The # 4 mistake is…Not Understanding the Purpose of Traceability.

DO-254 is a requirements-based process. As part of this process, requirements must be captured and validated, and then traced (via a Traceability Matrix) into both the corresponding design implementation and verification (test cases, test procedures and results). Traceability is an integral (i.e., Supporting) process verification activity. What traceability provides, if done properly, is assurance that all requirements have been implemented, that all portions of the design tie to requirements, and that the design as implemented behaves as the requirements say it should.  Traceability done throughout the development activity will identify derived requirements that need validation.  But all of this entails integrating traceability into the process as you go. By integrating traceability, it offers insights into design completion and verification coverage, and can even help find design bugs!

Instead what I often see is project teams who think of the Traceability Matrix is an artifact to be generated after the fact, with no thought about it until it needs to be reviewed during SOI audits. If you wait until the last minute to create a traceability matrix, as opposed to incorporating traceability into each phase of design (used as a analysis tool as part of requirements, design, code and test reviews), you miss the point and the benefits that this activity provides. So in your program, try to remember that Requirements Traceability isn’t an output of the program as much as it is a tool to be used throughout the program to help you understand your progress and provide an added measure of verification and visibility for change impact for each phase of the process.

Tammy Reeve
DER/Founder
Patmos Engineering Services, Inc.

top

If you have a topic you’d like Tammy to address, send email to Tammy@Patmos-Eng.com