FDA’s Approach to Advancing Quality Surveillance: PQS Effectiveness, Post-Market Quality Surveillance, Quality Metrics and Quality Management Maturity

Nandini Rakala, PhD

Visiting Associate | Data Scientist

U.S. FDA

• Pharmaceutical Quality System (PQS) Effectiveness
• Post-Market Quality Surveillance using Advanced Analytics
• Quality Metrics and Quality Management Maturity
  • Program Overview and Objectives
  • Assessment Framework

In her introduction, Rakala summarized the role of OQS as:

• Monitors quality and manages information about CDER-regulated sites and products
• Uses intelligence collected and analytics to make data-driven decisions that can reduce risk to patients
• Proactively identifies potential quality signals and trends before serious quality problems occur
• Continuously improves surveillance programs by applying innovative methods and novel techniques.

As an example, Rakala presented the integration of advanced analytics to enhance an overall proactive quality surveillance framework. These techniques should assist in predicting future occurrences and should be helpful when applied to FDA’s proposed quality metrics and quality management maturity programs as well as with US Coronavirus Aid, Relief, and Economic Security (CARES) Act (2020) volume reporting requirements and in improving FDA’s site selection model for GMP inspections.

For assessing PQS effectiveness, qualitative and quantitative approaches are being applied.

Assessment of qualitative data and information related to a facility could ensure adequate oversight of product quality utilizing, for example, information from Establishment Inspection Reports, exhibits, product quality defect reports, etc. In a more quantitative manner, assessment of historical data could allow predictions for certain PQS effectiveness metrics related to an establishment (e.g., CAPA effectiveness, investigation times, Human Error, Root cause, Repeat Deviation, and time to initiate recall).

Rakala presented plots of predictions for four PQS metrics derived from machine learning techniques, which used data from Field Alert Reports (FARs) as a surrogate. The plots showed a predicted value for a firm as well as a percentile value indicating where the metric falls relative to a selected benchmark group.

To implement these techniques more formally, there are some more activities required such as addressing gaps in data currently available to complete a full assessment of PQS effectiveness, for example current processes such as inspections and post-marketing reporting do not routinely capture all ICH Q10 elements. This additional information could be requested under the FD&C Act 704 (a)(4) records request.

It is intended that the outputs from these OQS assessments will support agency decision making with respect to assignment of reporting category for Established Conditions (ECs) proposed by a sponsor in an original application or a supplement. Identification and proposals by a sponsor of ECs is extensively discussed in ICH Q12 Guideline, Technical and Regulatory Considerations for Pharmaceutical Product Lifecycle Management.

The Office of Quality Surveillance requires a systematic and data-driven approach to identify potential quality signals in Post-Market Surveillance Reports. FAR regulations are a useful early warning system to protect public health. Information in FARs may assist in identifying changes in the reporting habits with respect to FDA establishment identification numbers (FEI’s), application numbers, and product names leading to prioritization of resources and identification of potential product quality signals.

A function has been created that utilizes statistical process control charts to flag entities with unexpected changes in their quality defect reporting habits. Application of Natural Language Processing algorithms could then discover topic clusters and commonly occurring problems across the network of regulated facilities/products.

Rakala described an overview of the FDA Quality Metrics Program, a program which has been developing since 2013. She indicated that Quality is cost effective since it reduces deviations, reworks, investigation times, adverse events, drug shortages, and product recalls. It is a positive attribute for purchasing decisions. Quality metrics are an objective means of measuring the product and process lifecycle to proactively identify and mitigate risks to quality thereby providing higher levels of performance. They are used extensively throughout the drug and biologicals product industry to monitor quality systems and drive continual improvement opportunities in quality and manufacturing.

Metrics can also be useful to FDA to:

• Help develop compliance and inspection policies and practices, such as risk-based inspection scheduling of drug manufacturers
• Improve the Agency’s ability to predict, and therefore, possibly mitigate, future drug shortages
• Facilitate a data-driven surveillance program
• Encourage pharmaceutical industry to implement state-of-the-art, innovative quality management systems for pharmaceutical manufacturing

Rakala stressed that the Quality Metrics program is intended to be mandatory. CDER’s vision is that the Quality Metrics Program is a quality surveillance tool and not an enforcement tool.

She summarized feedback from two recent learning opportunities for FDA staff in a Site Visit Program (14 sites visited) completed in 2019 and a Feedback Program (13 sites) completed in 2021. This feedback as well as a brief history of the program are also described in a recently issued Federal Register Notice, Request for Comments, Docket Number, FDA-2022-N-0075, March 9, 2022. The main learning points were:

• Different industry sectors prefer different metrics due to their individual operations and business dynamics needs.
• Any metric chosen to be reported should be meaningful
• The majority of participants prefer to report data at an establishment level and context is important
• The effectiveness of the quality system is a critical component of a QM Reporting Program as evidenced by numerous establishments collecting data around their PQS. Examples include metrics related to the effectiveness of CAPA programs, repeat deviations, maintenance programs, and timeliness.
• Certain metric calculations from the 2016 draft guidance can result in mathematical discrepancies and some metrics were not discerning.

This request for comments was open until June 7, 2022 and requested feedback on the scope of the proposed program as well as what to report. It provided four Practice Areas, each Area having suggested metrics from which to choose as most appropriate for an establishment’s operations and business dynamics. The Practice Areas with proposed metrics are:

• Manufacturing Process Performance
  • Process Capability/Performance Indices (Cpk/Ppk)
  • Lot Acceptance Rate
  • Right-First-Time Rate
  • Lot Release Cycle Time:
• PQS Effectiveness
  • CAPA Effectiveness
  • Repeat Deviation Rate
  • Change Control Effectiveness
  • Overall Equipment Effectiveness
  • Unplanned Maintenance
• Laboratory Performance
  • Adherence to Lead Time
  • Right-First-Time Rate
  • Invalidated Out of Specification Rate
  • Calibration Timeliness
• Supply Chain Robustness
  • On-Time In-Full (OTIF)
  • Fill Rate
  • Disposition On-Time
  • Days of Inventory On-Hand

ISPE has submitted comments in response to the Federal Register Notice.

A key takeaway highlighted was that quality metrics are a critical component of an optimized Quality Management Maturity program.

Rakala ended her presentation with a discussion of FDA’s latest thinking on the topic of Quality Management Maturity. She gave the definition as:

“Quality Management Maturity (or QMM) is a state attained by having consistent, reliable, and robust business processes to achieve quality objectives and promote continual improvement.”

She went on to explain that meeting Current Good Manufacturing Practices (CGMPs) is the minimum standard for legally marketing drug products in the U.S. CGMPs assure proper design, monitoring, and controls for manufacturing processes and facilities. Fully realizing the pharmaceutical quality vision for the 21st century, however, requires movement toward richer quality management systems. QMM is not a discrete concept, it is a holistic concept that includes Pharmaceutical Quality System effectiveness and many related aspects of business processes that enable manufacturers to proactively monitor quality related events.

QMM was introduced in the FDA 2019 Report, Drug Shortages: Root Causes and Potential Solutions where one of three identified major root causes of drug shortages was “Market Does Not Recognize and Reward Manufacturers for Mature Quality Management Systems”. The report recommends enduring solutions, including “Developing a rating system to incentivize drug manufacturers to invest in QMM for their facilities.”

FDA has initiated development of a program aimed at recognizing and rewarding manufacturers for “mature quality systems” that achieve sustainable compliance and focus on continual improvement, business continuity plans, and early detection of supply chain issues. The program is intended to be voluntary.

FDA is learning from other related programs and research such as that conducted by the University of St.Gallen, the FDA Center for Devices and Radiological Health (CDRH) Case for Quality Program, a Dun and Bradstreet Quality Benchmarking Study and PDA’s Quality Culture Initiative. Reference was also made to ISPE’s Advancing Pharmaceutical Quality (APQ) Program. ISPE’s APQ Program is an industry-for-industry program which uses as much company data and site procedures as possible and includes an option to conduct a benchmarking exercise using St Gallen criteria. Three APQ Guides have been issued covering Change Management, CAPA System, and Management Responsibilities and Review. The fourth Guide, Process Performance and Product Quality Monitoring will be issued before the end of 2022.

FDA has executed two QMM pilot programs developed in cooperation with third party appraisers. One pilot was for finished dosage forms and was conducted by Pacific Force with seven US domestic sites, ending in September 2021. This self-assessment process examined six program areas e.g., leadership and governance, stakeholder engagement, knowledge management. The second pilot was of eight foreign API manufacturing sites and ended in March 2022. Facilitated assessment by Shabas Solutions examined four QMM pillars of sustainability, risk management, compliance, and quality culture.

Rakala concluded the QMM section of her presentation by providing links to recent updates such as:

• FDA White paper: Quality Management Maturity: Essential for Stable U.S. Supply Chains of Quality Pharmaceuticals
• FDA Quality Management Maturity webpage
• FDA Quality Metrics for Drug Manufacturing Quality Metrics webpage