Last month, the fall 2024 Leapfrog hospital safety grades were released, giving nearly 3,000 U. S. hospitals safety grades of A through F. I was very happy to see that my former hospital, Ohio State East Hospital, received an “A”, but I was surprised to see that another hospital just 2 miles away, that I know also provides great care, received an “D” grade. This is the 23rd year of the Leapfrog reports and it has evolved considerably over those years. Leapfrog’s goal is to answer the question “How safe is my hospital?“. With its first report in 2002, I had a lot of concerns about the validity of the scoring system used by Leapfrog. But has Leapfrog overcome those concerns?
Who rates hospitals?
Hospitals are graded by many different organizations. The big 3 are The Joint Commission, individual state departments of health, and CMS. In addition, there are a myriad of organizations that provide accreditation or certification for individual hospital services, for example, AACVPR certification for cardiac rehabilitation programs and the College of American Pathologists accreditation for hospital clinical labs.
The Joint Commission is a non-governmental organization that accredits U.S. healthcare organizations and programs. Participation is voluntary but many insurance companies use Joint Commission accreditation as a basis for determining in-network hospitals and in negotiating reimbursement rates. The Joint Commission accreditation hinges on unannounced site surveys that occur approximately every 3 years. Although there are alternatives organizations that accredit hospitals, The Joint Commission has the advantage of being approved by Medicare. Consequently, 70% of U.S. hospitals participate with the Joint Commission at a cost of an average of $46,000 per year.
State departments of health license hospitals to operate. These departments generally perform periodic surveys and inspections for hospitals to maintain licensure. However, in most cases, there is not a state department of health grade associated with licensure, only a yes/no designation of whether or not the hospital has a license to operate.
The Centers for Medicare and Medicaid Services (CMS) operates a website that grades hospitals on a one to five star scale based on data derived from Medicare bills as well as hospital self-reported data for metrics such as ER wait times and the percentage of healthcare workers who are up to date on COVID and influenza vaccines. The star rating system was first implemented in 2007. CMS also provides a separate star grade based on satisfaction surveys sent to patients after hospitalization.
The Leapfrog Group
In 1999, the Institute of Medicine issued a report titled “To Err is Human” that claimed that 98,000 Americans died each year due to medical errors occurring in hospitals. The implication was that 1 out of 25 people die from a medical error in a hospital. In response to this report, the Leapfrog Group was started by a group of business leaders in 2001. At the time, the CMS hospital star ratings did not exist. Large companies wanted to be sure that they were getting good value for the healthcare insurance premiums that they were paying to insurance companies and did not want to pay insurance companies to send employees to hospitals where those employees had a high risk of complications that could delay or prevent their return to work. At issue was that insurance companies were primarily motivated to provide the lowest cost of healthcare whereas the employers were additionally motivated by the desire to keep employees healthy enough to be able to stay in the workforce.
The first Leapfrog survey was very simple and looked at 3 things: (1) whether hospitals had computerized physician order entry, (2) whether ICUs were staffed appropriately with intensivists, and (3) whether hospitals had high surgical volumes for certain high-risk procedures. Beginning in 2012, Leapfrog began assigning letter grades based on its annual surveys. In the first years, the Leapfrog Group surveys were heavily criticized as being well-intentioned but inaccurate and misleading. Since that time, the survey has expanded to cover more items and require more documentation in an effort to improve its relevance. The current survey is 330 pages long and covers 22 different measures of patient safety. Each measure requires answering a number of questions. Each question additionally requires hospitals to submit documentation to back up the answers to those questions.
Leapfrog does not survey all hospitals. For example, pediatric hospitals, critical access hospitals, VA hospitals, and specialty hospitals are not included. Of the approximately 3,000 hospitals that Leapfrog invited to submit surveys, 2,200 actually responded this year. In the previous report (spring of 2024), 30% of hospitals received an “A”, 26% received a “B”, 37% received a “C”, 7% received a “D”, and <1% received an “F”.
The problems with the Leapfrog Survey
- Completing the survey is onerous. Realistically, it will take one person a couple of weeks working full-time to complete the survey. For example, for the single question of whether the physicians covering the ICU are board certified in critical care, the hospital has to provide the physician call schedule for the ICU for every day for the last 3 months and provide a copy of each physician’s current board certification in critical care. For a large hospital, with multiple different ICUs, this can involve dozens of physicians who have variably obtained critical care board certification from either the American Board of Internal Medicine, Anesthesiology, Pediatrics, or Surgery. Ask any administrative staff member who has been responsible for compiling Leapfrog survey materials and he or she will tell you that it is a confusing, complicated, time-consuming, and all-around unpleasant process.
- It penalizes hospitals that do not submit surveys. The Leapfrog Group will give your hospital a letter grade whether or not you respond to a survey. If you do not respond, Leapfrog obtains as much information as possible from public sources, such as the CMS website. Leapfrog states: “If a hospital performs well on certain measures on the Leapfrog Hospital Survey, they have the opportunity to earn more points in their Hospital Safety Grade for those certain measures than they would if they did not report to the Leapfrog Hospital Survey.” Currently, about three-quarters of eligible hospitals submit surveys. For some hospitals, this can feel like extortion – by not filling out the survey, you risk getting a lower publicly reported grade than you would have if you filled out the survey.
- It focuses on hospital processes and not on outcomes. The Leapfrog survey centers around workforce staffing, specific hospital policies, use of electronic medical records, and volume of surgical procedures performed. These are variables that in the literature have been shown to be associated with better patient outcomes. However, the Leapfrog survey does not tell you much about actual patient outcomes at specific hospitals.
- It is heavily weighted on surgery and ICU care. Since the beginning, Leapfrog has been all about ICU staffing and surgical volume. These are two areas that were of particular concern to the companies that founded Leapfrog because the ICU and the operating room were targeted in the Institute of Medicine’s 1999 report “To Err is Human”. These companies wanted to be sure that their employees who underwent elective surgeries or had catastrophic injuries or accidents resulting in ICU care were able to return to the job after their operation or ICU stay. The Leapfrog grade tells you relatively less about other types of care including non-ICU medical conditions, primary care, preventive medicine, etc.
- There is a poor correlation between the Leapfrog grade and other rating systems. A 2017 study found that there was there was only a relatively weak association between the Leapfrog safety grade and compulsory Medicare outcomes and penalties. Since 2012, twelve hospitals have received “A” grades each year but only half of these received a 5/5 star rating from CMS. This is primarily because Leapfrog and CMS use different metrics and weight metrics differently when assigning a hospital its rating.
- Leapfrog makes its money from the ratings. According to the most recent Leapfrog Annual Report, it had revenues of $6,974,962. Of this amount, 78% ($5,434,948) came from the sale of reports and licensing fees. Hospitals that want to use their annual Leapfrog grade in advertising must purchase a license agreement from the Leapfrog Group. For a 500-bed hospital, this costs $29,900.
So, should you care about your hospital’s Leapfrog grade?
At its inception, the Leapfrog rating provided ground-breaking data about individual hospital safety practices. Since that time, other organizations have created competing rating systems. There is still value in the Leapfrog Safety Grade, however. Other organizations, such as The Joint Commission, only give a yes/no accreditation. This is like getting a pass/fail grade for a class rather than a letter grade and it is impossible to know which is the best hospital among all of the hospitals accredited by The Joint Commission. In this regard, the Leapfrog letter grade is akin to the CMS star rating, allowing consumers to compare one hospital to another when making a choice of which hospital to utilize. However, the Leapfrog Safety Grade and the CMS star rating measure different things, so they should be viewed as complimentary rather than competitors.
A problem with all rating systems that give hospitals one single grade is that each hospital has different outcomes for different medical conditions. A single overall hospital grade can result in over-generalization of the care of different conditions. For example, assume two hospitals have identical Leapfrog grades but one hospital may have great orthopedic surgery care and the other has terrible orthopedic care but a great obstetric unit. A pregnant patient would be well-advised to choose a hospital based on its obstetric care rather than its hip replacement surgery outcomes.
In the ideal world…
I spent far too much of my life compiling various accreditation and certification surveys, preparing for site surveys by The Joint Commission, and generating quality reports for an endless number of organizations and agencies that rate, review, license, accredit, certify, or otherwise judge hospital care. The Leapfrog Safety Grade survey has grown from something that I could do myself in just a matter of minutes to a behemoth that now takes a whole department of administrative staff days or weeks to complete.
Hospitals are straining at the time and expense to complete all of the various surveys for organizations that rate them. Given the similarities between the CMS star rating and the Leapfrog safety grade, it would be ideal to combine the two. Those who advocate for less government might argue to outsource a single rating system to the private sector (i.e., Leapfrog). However, a considerable component of the Leapfrog grade is based on Medicare data derived from protected personal information on individual patients – I personally do not want my entire medical history, social security number, address, etc. being exported from Medicare to a private company. It would be preferable for CMS to adopt the good things about Leapfrog and incorporate them into the CMS star rating. Having two hospital rating systems with so much overlap creates too much hospital administrative costs that then get passed on to consumers. It’s time to improve hospital administrative efficiency.
December 12, 2024
