For almost 20 years of my career, I led assessment teams and assessment organizations for both commercial nuclear and defense nuclear facilities. Along the way I learned to value the power of work observations in assessing safety performance, as well as the important difference between performance and compliance.
Assessing compliance is relatively straightforward. Pretty much all you need to know is the requirement then make a check to see if it was followed – yes, or no. Performance-based assessments (PBA), as discussed in a previous article, is more challenging. Performance-based assessments (PBA) require a deeper look, and analysis, to reach evidence-based conclusions. Those conclusions are derived from observations, document reviews, interviews, and other techniques to determine how effectively desired goals and objectives are being met, regardless of their level of compliance. Non- compliances are not ignored but are not central to assessment conclusions. Perhaps an example will help to better explain the difference.
Several years ago, I was observing routine maintenance on a piece of “safety related” equipment in a defense nuclear facility. The equipment was deemed critical to help ensure safe shutdown of the facility in the event of an accident or unforeseen circumstances. Since the purpose of our performance-based assessments (PBA) was to determine the adequacy and reliability of facility safety systems (e.g., fire protection, radiation detection, ventilation), I picked this particular work activity as one I planned to observe during the assessment. The work was accomplished according to the procedure and the workers were sensitive to the safety requirements (e.g., PPE and lockout/tagout). When they tested the equipment, however, they noted it was not operational. They put the system back in service and retested it to ensure it was working– all in accordance with their procedure. In short, the work I observed was totally compliant with both safety requirements and the procedure. But what about a critical safety system that was not operable at the time of the work and for who knows how long before that?
Clearly the fact that a safety system was discovered out of service was an issue, but effective performance-based assessments (PBA) require more than one data point to develop a conclusion. You need to also determine what led to the problem and to what extent it exists throughout the organization. In PBA parlance this is known as “pulling the string.” Therefore, my next step was to review maintenance documents related to the observed work and interview some of the workers who had performed the work previously. A review of the maintenance records indicated that nine of the 11 previous routine maintenance activities had also found the system out of service prior to initiating the procedure. This disturbing fact had never been escalated to facility management as there was no mechanism to do so. Discussions with workers confirmed that reporting system problems was neither required, nor clearly expected, by facility management. Further review also found that the facility lacked an effective system to track and ensure timely closure of those problems that were identified.
None of the issues identified in this portion of the performance-based assessments (PBA) were, strictly speaking, compliance issues, but they clearly demonstrated that the facility was doing a poor job of insuring that its critical safety systems were reliable, or even operable. That fact, with considerable supporting evidence, was the essence of one of several conclusions cited in the final PBA report. Had the out of service equipment been a one-off event it wouldn’t have made it into our report. By pulling the string, however, significant problem areas were identified and corrected, despite the lack of a compliance driver.
Hopefully the example above will help you better understand the essence of performance-based assessments (PBA) and how it differs from traditional compliance inspections. While there is nothing inherently wrong with compliance inspections, they rarely lead us to the “why” (or relevance) of noncompliance. Performance-based assessments (PBA), on the other hand, are designed to uncover root cause systemic deficiencies. Sustainable improvement depends on our ability to identify and correct such deficiencies and well conducted PBAs are a great tool for reaching this goal.
AUTHOR BIO
Mr. Loud’s over 40 years of safety experience includes 15 years with the Tennessee Valley Authority (TVA) where he served as the supervisor of Safety and Loss Control for a large commercial nuclear facility and later as manager of the corporate nuclear safety oversight body for all three of TVA’s nuclear sites. At Los Alamos National Laboratory he headed the independent assessment organization responsible for safety, health, environmental protection, and security oversight of all laboratory operations.
Mr. Loud is a regular presenter at national and international safety conferences. He is the author of numerous papers and articles. Mr. Loud is a Certified Safety Professional (CSP), and a retired Certified Hazardous Materials Manager (CHMM). He holds a BBA from the University of Memphis, an MS in Environmental Science from the University of Oklahoma and an MPH in Occupational Health and Safety from the University of Tennessee.
