Patient safety indicators (PSIs) are designed to identify potentially preventable adverse events occurring in the acute inpatient setting. They were developed by the Agency for Healthcare Research and Quality (AHRQ) almost 20 years ago as a marker of the quality of care provided at a specific hospital.
PSIs are captured through claims data, obviating the need for expensive and time consuming data abstraction. Each individual PSI score is reported as a ratio between the observed rates of patient safety events to the risk-adjusted, expected rates of events. The CMS Patient Safety and Adverse Events Composite (PSI-90) combines these individual measures following a weighted approach.
Each individual PSI weight is based on how likely it is to cause patient harm and how often it happens. The number of PSIs included in the PSI-90 score has changed over the years.(3) Table 1 shows the 10 current PSIs incorporated in the PSI-90 score and their respective weights.
CMS uses PSI-90 scores to generate Hospital Compare ratings. They are also part of the Value-Based Purchasing Program and the Hospital-Acquired Condition Reduction Program. Elevated PSI-90 scores may trigger significant financial penalties and have a detrimental effect on people’s perception of the quality of care provided at the hospital.
Despite federal programs’ use to assess quality of care, there is no consensus on the ability of PSI-90 scores to accurately monitor potentially preventable complications. There is no clear correlation between PSI-90 scores and mortality or readmission rates for a specific hospital.(1,2,4)
Positive predictive value (PPV) is the percentage of events flagged as PSIs that were verified as true PSIs after a chart review. Some studies have set the PPV of some PSIs as low as 28% (postoperative hip fracture) while the PSI with the highest PPV is wound dehiscence with a PPV of 87%.(5) Conditions present on admission, miscoding, and software limitations all limit PSI PPV.
Methods
To evaluate to what degree PSI-90 scores reported to the institution in 2015 indicated opportunities for process improvement, the Department of Quality Improvement and Patient Safety at Samaritan Medical Center completed a retrospective medical record review of every patient encounter flagged as a PSI-90. Each record was independently reviewed by two registered nurses experienced in chart abstraction using standardized chart abstraction tools. They were asked to identify the occurrence of any patient safety event. When the abstractors did not agree, the medical record was referred to the director of quality improvement and patient safety and the chief medical officer for resolution.
Results
The review identified six indicators for which the institution performed at a lower level than most hospitals in the country:
Postoperative respiratory failure
Perioperative PE/DVT
Postoperative sepsis
Accidental puncture or laceration
Birth trauma injury to neonate
Trauma secondary to vaginal delivery without instruments
Lapses in clinical care existed for patients with postoperative respiratory failure and perioperative PE or DVT. Coding inaccuracies played a substantial role in encounters coded as accidental puncture or laceration, birth trauma injury to neonate, trauma secondary to vaginal delivery without instruments, and postoperative sepsis. In response to these findings, we formed a workgroup to address postoperative respiratory failure. Simultaneously, a second group focused on perioperative PE/DVT.
Lapses in Clinical Care
In the case of respiratory failure, we found opportunities in medication management. The choice of opioids and the sequencing of pain control modalities was not always optimal. Some records contained no sedation assessment at the time narcotics were given. In addition, once an overdose was identified, patient rescue sometimes took longer than necessary.
Within weeks, the group developed standardized pain order sets that provided clinical decision support on medication selection. The order sets also defined two levels of patient monitoring, depending on the patient’s clinical condition. The sets, which were incorporated into the electronic medical record, included individual orders that nursing staff could follow to address common opioid side effects. The group also recommended the use of an evidence-based sedation scale built into the electronic medication administration record.
The Medical Executive Committee (MEC) approved the pain order sets and the sedation scale in May 2015. In addition, the MEC requested that a designated pharmacist review each new opioid order. These interventions became effective in June 2015. Standardized educational sessions were offered to the staff. The rate of postoperative respiratory failure fell from 12.61 per 1,000 surgical discharges in 2014 to 4.55 in 2015, and 1.88 in 2016 (Figure 1).
Figure 1. Rate of Postoperative Respiratory Failure (per 1,000)
The group working on perioperative PE/DVT identified lack of compliance with DVT prophylaxis as the root cause for the elevated rates. The group developed an order set for DVT prophylaxis that was added to every admission order set. Members also created an electronic screening tool for nurses to assess patients at risk for developing PE/DVT. An elevated score would prompt a call to providers for orders for pharmacological or mechanical prophylaxis.
Management Information Systems (MIS) and the Department of Quality Improvement and Patient Safety built a status board within the EMR for concurrent monitoring. The status board made it possible to efficiently review multiple patients and units. Weeks after implementing the status board, DVT prophylaxis compliance reached 95%. In 2014 the rate of PE/DVT was 4.15/1,000. The rate decreased to 3.49 in 2015 and 2.99 in 2016 (Figure 2).
Figure 2. Rate of Postoperative PE/DVT (per 1,000)
Coding Inaccuracy
Conditions present on admission were the most common coding inaccuracy. Ambiguity in ICD-9-CM coding guidelines, and the codes’ inability to differentiate between new events versus those that were present on admission allowed for imprecise coding. Multiple patients had been erroneously coded as one of the following PSIs:
Postoperative sepsis
Accidental puncture or laceration
Birth trauma injury to neonate
Trauma secondary to vaginal delivery without instruments
Overall Improvement
To eliminate potential false positive events, the Department of Quality Improvement began to review every patient encounter that could be flagged as PSI. Since then, the rates of postoperative sepsis, accidental puncture or laceration, neonatal birth trauma, and trauma secondary to delivery without instruments have decreased well below national averages (Figure 3).
Figure 3. Rate of Coding Opportunities (per 1,000)
Like any observed over-expected metric, PSI-90 values above 1 indicate a worse-than-average performance, while a PSI 90 ratio below 1 suggests a hospital performs better than the national average across these indicators.
In 2018, the PSI-90 score for Samaritan Medical Center was 1.23. This score was based on discharges from July 1, 2014, to June 30, 2016. The quality-improvement initiatives described above were all fully implemented by July 30, 2015. In 2019, the PSI-90 score was significantly better at 0.94. The following year showed additional improvement with a score of 0.73. PSI-90 scores for 2021 and 2022 were 0.64 and 0.56, respectively (Figure 4).
Figure 4. PSI-90 Scores for Samaritan Medical Center, 2018–2022
Conclusion
In our experience, PSI-90 scores are a useful tool that quality improvement professionals can use to identify potential lapses in the quality of care provided at the hospital level.
Due to the poor PPV of some PSIs, potential areas of concern identified through PSIs must be validated via chart reviews. For that reason, the use of unvalidated PSI-90 scores to assess hospital performance may yield inaccurate results.
Through PSI-90-targeted interventions, SMC was able to decrease the number of patients with postoperative respiratory failure and perioperative PA/DVT. In addition to its potential impact on patient care, a programmatic approach to improving PSI-90 scores may also result in enhanced financial outcomes and better Hospital Compare ratings.
Limitation of the Study: We did not have the resources needed to thoroughly blind the record review process. To limit information bias, each record was independently reviewed by two trained abstractors and inter-rater reliability was monitored throughout the study.
References
Kaafarani HM, Borzecki AM, Itani KM, et al. Validity of Selected Patient Safety Indicators: Opportunities and Concerns. J Am Coll Surg. 2011;212:924–934
Rajaram R, Barnard C, Bilimoria KY. Concerns About Using the Patient Safety Indicator-90 Composite in Pay-for-Performance Programs. JAMA. 2015;313:897–898
Patient Safety and Adverse Events Composite (modified version PSI 90) for ICD-9 CM/PCS, v6.0 (FY2016). Agency for Healthcare Research and Quality; https://qualityindicators.ahrq.gov/News/PSI90_Factsheet_FAQ_v1.pdf . Accessed April 13, 2022.
Nguyen MC, Moffatt-Bruce SD, Strosberg DS, et al. Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicator for Postoperative Respiratory Failure (PSI 11) Does Not Identify Accurately Patients Who Received Unsafe Care. Surgery. 2016; 160: 858–868.
Rosen AK, Itani KM, Cevasco M, et al. Validating the Patient Safety Indicators in the Veterans Health Administration. Med Care. 2012;50:74–85