Understanding Hospital Ratings (and how hospitals can game those ratings)

Consumers use ratings in everything that they buy. Amazon rates merchandise, Consumer Reports rates cars and appliances, Trip Advisor rates hotels, Yelp rates restaurants, and Google rates about everything. There are also rating systems for hospitals. But are these rating systems a reflection of the actual quality of care in the hospital or a reflection of the patient population served by the hospital? And if the latter, do hospitals attempt to choose their patients in order to improve their ratings?

The main four hospital rating systems are published by U.S. News & World Report, CMS’s Hospital Compare, Healthgrades, and Leapfrog. In an 2019 article in NEJM Catalyst, the authors rated the rating systems and found that all of the rating systems had flaws. In the article, each rating system was given a letter grade and none received an “A”.

• U.S. News & World Report: B
• CMS Hospital Quality Star Ratings: C
• Leapfrog Safety Score: C-
• Healthgrades: D+

Each of these rating systems uses different measures to determine hospital ratings. In order to understand what the rating really means, you need to first understand how the ratings were made.

U.S. News & World Report

The highest rated system was U.S. News & World Report’s annual hospital ranking. Each year, U.S. News & World Report updates their methodology in order to continually improve on its accuracy and relevancy. This process of annual methodology revision has resulted in the U.S. News & World Report ratings becoming more accurate each year. This year, the rankings are determined by four elements that are each weighted differently in determining the overall hospital rank:

1. Patient outcomes (37.5%) – measured by the number of Medicare patients who died within 30 days of being admitted to the hospital in 2018, 2019, & 2020. The measure was adjusted to factor in each patient’s age & gender, the type of care received, co-morbidities, and whether patients were on Medicaid as well as Medicare. The Medicaid adjustment is important because it is a marker of low-income Medicare patients. A criticism of this measure is that the survival data is 3-5 years old and may not reflect the hospital’s current survival data.
2. Patient experience (5%) – measured by the HCAHPS patient satisfaction surveys
3. Other care-related indicators (30%) – measured by nurse staffing, patient volume, certain clinically proven technologies and professional & specialty-specific recognition
4. Expert opinion (27.5%) – measured by an annual survey to board-certified physicians about which 5 hospitals they consider to be the best for their particular specialty.

CMS Hospital Quality Star Ratings

The current (2022) year CMS Hospital Quality Star Rating is based on five elements:

1. Mortality (22%) – measured by the number of Medicare patients with specific diseases who died within 30 days of admission in 2017, 2018, and 2019. The diseases include: myocardial infarction, coronary artery bypass & graft surgery, COPD, heart failure, pneumonia, stroke, and surgical patients.
2. Patient safety (22%) – measured by eight different safety measures obtained from 2017 to 2020. Some of the measures include central line infections, C. difficile infections, and surgical site infections.
3. Readmission (22%) – measured by the number of patients with 11 specific diagnoses who required readmission within 30 days of discharge from the hospital. Data are from 2017, 2018, and 2019.
4. Patient experience (22%) – measured by the HCAHPS patient satisfaction surveys from 2019.
5. Timely and effective care (12%) – measured by 13 different elements from 2019 and 2020. Some of the measurements include percent of staff vaccinated for influenza, ER length of stay times, percent of ER patients who left without being seen, appropriate follow-up interval for routine screening colonoscopy, and number of patients with low back pain who got an MRI without first getting physical therapy.

Leapfrog Safety Score

The Leapfrog Group was founded in 2001 by a group of large employers who wanted to have objective quality metrics in order to change the way that these companies purchased healthcare for their employees. They give hospitals a grade of A through F based exclusively on patient safety. Leapfrog uses data from CMS and from their own survey. A criticism of Leapfrog is that only about half of U.S. hospitals return the Leapfrog survey. In addition, there is no audit process in place to validate the self-reported survey responses, creating opportunities for falsifying survey responses in order to appear more favorable. The Leapfrog grade is based on two elements:

1. Process measures (50%) – these are 12 different measures of how healthcare is delivered by the hospital such as whether there is computer order entry, how the ICU is staffed by physicians, and how frequently staff practice hand hygiene. The measures are assessed on data from 2020 to 2021.
2. Outcome measures (50%) – these are 10 different measures such as: frequency of MRSA infection, frequency of patient falls, and frequency of air embolism. The measures are assessed on data from 2018 to 2021.

Half of the Leapfrog Safety Score is based on process measures that only indicate whether the hospital has those specific processes in place and does not indicate whether those processes actually improve patient care in that particular hospital. Another criticism of the Leapfrog Safety Score is that it does not incorporate mortality, which can be argued is one of the more important measures of the effectiveness of hospital care.

Healthgrades

Healthgrades bases its rating on outcomes for 33 medical conditions and procedures. The data for hospitals in 34 states is derived only from Medicare reported data. In the other 16 states, data is derived from both Medicare reports and all-payer reports. Outcomes are adjusted for a large number of co-morbidities. However, there are a large number of exclusions, for example, any Medicare patient under age 65. The elements of the rating are:

1. Mortality cohorts – these measures are based on the number of patients with 17 different diagnoses who die within 30 days of admission. The diagnoses include conditions such as bowel obstruction, cranial neurosurgery, and pancreatitis.
2. In-hospital complication cohorts – these measures are based on whether complications occurred in patients undergoing 15 different surgical procedures such as appendectomy, prostatectomy, and hip replacement. In addition, complications occurring during one medical condition (diabetic emergencies) is included.

The problems with the rating systems

All four of the hospital rating systems have flaws. One of the most important flaws is that they rely heavily on Medicare data. This data is quite robust for assessing the outcomes of Medicare patients. However, the only patients in the Medicare database are those who either are over age 65, are receiving dialysis for kidney failure, or are disabled. Currently, only 18.4% of the U.S. population is on Medicare so there is no hospital outcome data for the majority of Americans. Because hospital ratings drive hospital quality improvement processes, American hospitals have been more strongly motivated to improve care to patients older than age 65 with less attention given to improving care to younger patients.

A second flaw is that most of the outcome data is based on information that is several years old. In the case of U.S. News & World Report, the data is up to 4 years old and in the case of CMS, the data is up to 5 years old. Hospitals are constantly improving their patient care practices and most hospitals have made changes in those practices over the past 5 years. As a result, the data from which the ratings are derived can be significantly out of date and not reflective of current hospital practices.

A third flaw is that the rating systems rely on surveys. The HCAHPS survey data is used by the U.S. News & World Report rating and by the CMS Hospital Quality Star rating. Nationwide, on average only 26.7% of patients respond to the HCAHPS survey and there is wide variation with many hospitals having significantly lower response rates. A 2019 study in the Patient Experience Journal found that the higher a hospital’s HCAHPS survey response rate, the higher that hospital’s average HCAHPS score was from those surveys. The implication is that patients are more likely to fill out a survey if they were unhappy with their care so that the sample of patients responding to the survey is not representative of the total hospital patient population. Those hospitals that can convince more patients to fill out surveys will thus have higher HCAHPS scores.

Leapfrog sends surveys to hospitals to fill out but only about half of U.S. hospitals respond to their survey. Those Leapfrog surveys are typically filled out the the hospital’s quality staff who can have a conflict of interest in their survey responses since those staff generally also have their job performance evaluations based on the reported quality outcomes. As a result, the Leapfrog surveys can portray the hospital as performing better than it actually is. For hospitals that do not respond to their survey, Leapfrog obtains surrogate data from other sources. It is not clear if data from those other sources is equivalent to the survey data so it is uncertain if valid comparisons can be made between those hospitals that do fill out Leapfrog surveys and those hospitals that do not return surveys.

A fourth flaw in the ratings is that only a limited number of medical conditions are evaluated. U.S. News & World Report’s rating is based on overall mortality with the result that not much is known about the quality of care for patients who do not die. The CMS Hospital Quality Star rating also uses mortality but only for 7 specific conditions. CMS readmission data is limited to patients with 11 specific diagnoses and its patient safety data is based on only 8 complication diagnoses. The Leapfrog rating does not incorporate mortality data and only incorporates a very limited number of complication diagnoses. The Healthgrades rating is disproportionately based on surgical outcomes and incorporates very little outcome data on non-surgical patients.

The problem of healthcare gerrymandering

In politics, gerrymandering is when politicians set district boundaries in order to choose their voters to win elections. In medicine, the equivalent of elections are annual hospital ratings. Medical gerrymandering is when hospitals choose their patients in order to improve their ratings. After years of having to explain low hospital ratings to hospital CEOs, Deans, and hospital board members, I’ve come to realize that it easy for hospitals to game the rating systems. Here are some of the specific ways that hospitals can improve how they look on the various surveys. Some are legitimate but others are quite nefarious.

Diagnosis selection. It might seem like a patient’s diagnosis is pretty straightforward but this is not always the case. For example, pneumonia is typically defined as a respiratory infection accompanied by an infiltrate on a chest x-ray. But what if the x-ray is normal and the infiltrate is only seen on a chest CT scan? Or what if the x-ray is normal but the doctor believes that the patient has pneumonia based on physical exam? These pneumonia patients tend to be less sick and therefore less likely to either die or be readmitted. Consequently, by being liberal with diagnosis definitions in less ill patients, the hospital can reduce the death rate and readmission rate by including more patients who have mild illness. Often, the diagnosis that is submitted to CMS or other rating organizations is based on the DRG diagnosis that is selected for a given patient’s admission. The DRG diagnosis is usually chosen by the hospital’s coding department staff and if there are 2 possible diagnoses that they can chose from, they will usually chose the DRG diagnosis that pays more. For over a year, I reviewed the charts of all patients who died in our hospital and found that in some, the DRG diagnosis did not really match the patient’s actually clinical diagnosis. By changing the DRG diagnosis, the patients were sometimes re-classified with a diagnosis that was not included in the rating data.

Co-morbidity selection. As a general rule, the more co-morbidities that a patient has, the more likely they are to die or be readmitted to the hospital. Once again, these co-morbidities are generally selected by the coding staff. When I reviewed the inpatient charts of those patients who died in the hospital, I was often able to find co=morbidities that the coding staff overlooked. Because the U.S. News & World Report rating system takes into account these co-morbidities, the more you can list, the better your overall rating will be.

Classify dying patients as being in observation status. Hospitals usually lose money on those patients who are kept in observation status. Observation patients are considered to be outpatients so they have more co-pays and the hospital cannot charge the insurance company for a lucrative inpatient DRG. As a general rule, patients who are anticipated at the time of admission to require a hospital stay of “less than 2 midnights” are considered to be in observation and are not considered to be inpatient admissions. This turns out to be very important in hospital rankings because the mortality rates are only based on those patients who have an inpatient admission. In other words, patients in observation status who die are not included in the mortality calculations for hospital rankings. It is very common to have patients admitted to the intensive care unit after an out-of-hospital cardiac arrest or some other catastrophic medical event and those patients die in the ICU a few hours later. The hospital finance department will want those patients to be classified as inpatients (since they have a medical condition that would have required a stay of greater than 2 midnights if they had lived). However, I trained our admitting physicians to put those patients (who were anticipated to die within 24 hours) in observation status when first admitted to the ICU. If they died within the first day of their ICU stay, they would die in observation status and not be included in our inpatient mortality data. If they survived for more than a day in the ICU, the physician would change their admission level of care order from observation to inpatient admission so that the hospital got paid for the admission. Because inpatient mortality is based on Medicare patients, some hospitals further game the system by only classifying Medicare patients with impending death as being in observation status and leaving commercially-insured patient who are not on Medicare as being inpatient status.

Keep “frequent flyers” in observation status. Similar to mortality rates, only patients with an inpatient admission are included in readmission rate calculations. There are some patients who you know are likely to return to the hospital within 30 days. If you keep them in observation status rather than admit them as inpatients, they won’t count against the hospital’s 30-day readmission rate.

Enroll dying patients in hospice. Medicare does not include patients who are enrolled in hospice in mortality data. However, patients must either already be enrolled in hospice prior to an inpatient admission or become enrolled in hospice during their first hospital day. Identifying those patients who have ultimately terminal diseases and getting them enrolled in hospice early not only helps serve the patients’ palliative care needs but also eliminates those patients from counting toward the hospital’s mortality rate if they die within 30 days of an inpatient admission.

HCAHPS survey response rates. As described in the last section, the patients who do not fill out HCAHPS surveys tend to be those who were more happy with their care. Hospitals that have tactics in place to get more patients to fill out their HCAHPS surveys will get a higher average score on their surveys. Therefore, hospitals that put resources into getting as many patients as possible to respond to the HCAHPS survey will have higher ratings than hospitals with a low survey response rate.

Flood U.S. News and World Report with expert opinion surveys from your own physicians. Americans in 2022 have survey fatigue. We are constantly receiving phone surveys, mail surveys, and email surveys. There are just too many surveys so we don’t bother to fill most of them out. Physicians are no different and many (or most) physicians who receive a U.S. News & World Report expert opinion survey just toss it in the trash. But if a hospital can convince all of its doctors to respond to the expert opinion survey (and rank the hospital in their top 5), then it can move up in the overall ratings. With expert opinion accounting for 27.5% of the overall U.S. News & World Report rating, this strategy is low-hanging fruit for hospitals, especially for hospitals with a large medical staff.

Choosing your patients. The best way to improve the hospital’s rating (and most immoral) is for the hospital to select the patients that it admits. Whether a patient dies within 30 days of being admitted and whether a patient is readmitted within 30 days of discharge is only partially dependent on the medical care delivered while that patient was admitted to the hospital. Socioeconomic factors that the hospital cannot control are at least as important. Age, income level, employment status, housing status, health insurance status, access to transportation, level of education, smoking status, primary language spoken, marital status, alcohol use, drug use, psychiatric co-morbidity, and race can also have a profound impact on disease outcomes, particularly after discharge. Hospitals that care for a larger percentage of older, low-income, unemployed, homeless, uninsured, smoking, low education level, foreign born, or racial minority patients will inevitably have worse mortality and readmission rates than hospitals that mainly care for patients coming from a high socioeconomic group. There are several ways that a hospital can alter their inpatient population in order to improve their overall outcomes and thus their ratings.

1. • Location, location location. By building a hospital (or a satellite hospital facility) in an affluent suburban area, that hospital will naturally attract a more affluent patient population. For hospitals that own primary care practices, by locating those physician offices in affluent suburban areas, it can ensure more affluent patients being admitted to the hospital.
   • Nurture referring physician relations. One of the most important reasons that patients choose to go to a particular hospital is whether their physician recommends that particular hospital. If the hospital fosters relations with private practice primary care physicians who are located in affluent neighborhoods, it can improve the average socioeconomic status of that hospital’s inpatients and by doing so, improve the hospital’s rating.
   • Nurture referral hospital relations. For those hospitals that receive a relatively large number of hospital transfers, by fostering referrals from smaller hospitals that are located in affluent communities, the hospital can skew its inpatient population to a patient group that is more likely to have better readmission and mortality rates.
   • Discourage unfavorable patients. In large cities, patients usually have a choice of emergency departments and hospitals to utilize. When patients have a bad experience at one hospital, they will tend to go to another hospital in the future. Cab vouchers and free meals in the emergency department can encourage low income patients to come to the hospital. On the other hand, liberal use of collection agencies for unpaid bills can discourage those patients. Hospitals have ways to tacitly discourage low income patients, minorities, smokers, and foreign-born patients from coming back. If the outpatient physicians affiliated with the hospital do not accept Medicaid or require up-front full payment from uninsured patients, then those patients will migrate to other health systems. This is the ugly side of American healthcare but unfortunately, ugly exists in every state and every large city.
   • Encourage favorable patients. Hospitals cannot get away with giving something tangible to one group of patients and withholding that something from another group. At the worst, it may be against the law and at best, it results in bad publicity. However, there are subtle ways to attract patients who are more likely to have better mortality and readmission outcomes. Since these outcomes are based on Medicare data, the trick is to attract “favorable” patients over age 65. Adding extra free wellness programs as part of commercial insurance contracts makes the hospital attractive to those seniors who can afford to purchase secondary health insurance. Similarly, free hospital-sponsored wellness programs in affluent neighborhoods can attract more affluent seniors. Hospital advertising campaigns that feature physically fit seniors hiking, swimming, and going on vacations to foreign countries will appeal to healthy, affluent Medicare enrollees.

I spent most of my career practicing in an urban hospital that served a patient population that would be considered “unfavorable” from a socioeconomic standpoint… and if I had to do it all over again, I would not change a thing. Idealism was one of the reasons I went into medicine in the first place. But for every hospital that is motivated by idealism, there is at least one hospital that is motivated by profit and fame. Unfortunately, our hospital rating systems reward the latter and not the former.

September 16, 2022