Given that hospital rankings may present limited information and fail to adequately assess relevant factors (such as social determinants of health, health disparities, and health inequities), the following offers some context about how they are calculated, as well as their transparency, comprehensiveness, utility, and limits.  Also included is a list of kidney disease resources for patients and family members.

Detailed methodology for the US News & World Report’s annual hospital rankings 2020–2021 rankings, which introduced the Kidney Failure rating in the Procedures & Conditions portfolio that replaces adult nephrology rankings, were not available at the time this information was prepared.

What are hospital rankings? 
Currently, there are four national hospital quality assessment,¹ including US News & World Report’s annual hospital rankings,² the Centers for Medicare & Medicaid Services (CMS) Hospital Compare,⁸ Healthgrades Top Hospitals,⁹ and Leapfrog Top Hospitals.¹⁰  Although hospital ranking organizations aim to provide patients and their caregivers with the best information to assess their care options, this information may be limited by the underlying data used to compile the rankings. Note that the scope of hospital rankings differs by rating organization. For example, US News & World Report’s hospital rankings only assess complex areas of care and procedures, and are not focused on non-acute, outpatient care. 

How are hospital rankings calculated? 
All hospital rankings rely in part on CMS data, with most using 30-day mortality rates in their calculations.^1,7–10  However, mortality rates may not accurately quantify patient care delivered.⁶  For example, although detailed methodology for the US News & World Report’s 2020–2021 rankings was not available at the time this information was prepared, previously the bulk of a hospital’s overall score (67%) in their rankings came from ratings of 12 “data-driven” specialties (including nephrology) comprising components for patient experience (patient surveys, 5%), discharge to home metric (7.5%), reputation (27.5%), structure (capturing staffing and patient services, 30%), and 30-day mortality rate (30%). The reputation component—contributing 27.5% of each specialty’s score—was calculated using a survey that asks physicians their opinion on which five US hospitals “provide the best care …for patients who have the most challenging conditions and/or surgical procedures.”⁷ The latter is a concern as a recent assessment of hospital ratings systems found “… a considerable disconnect between the top hospitals identified by the rating systems and those thought by clinicians to be major referral centers. Thus, the potential misclassification of hospital performance is a major concern in need of evaluation.”¹

How open are hospital rankings? 
Although some data used to calculate the ratings are publicly available (for example CMS claims data), the level of transparency varies between rating organizations.¹ In the case of U.S. News & World Report’s rankings neither the physician survey data—source of the reputation component accounting for 27.5% of a “data-driven” specialty’s (including nephrology) rating—nor the weightings of some variables (from publicly available data) used in the logistic regression modeling calculating a hospital’s score have been made public.⁶ Given that some hospital ratings are calculated using proprietary methods, they have not undergone the rigors of peer-review, the gold standard in ensuring the highest quality of scientific and medical research. 

How comprehensive are hospital rankings? 
Hospital ratings rely in large part on limited data—cause of death. Mortality cases can often be misattributed—where deaths can be assigned to a specialty based solely on the coding in the health record instead of the actual care a patient received or didn’t receive from a specialist. More importantly, “… hospital mortality rates have been shown to be lacking in predicting the quality of care delivered….”⁶ Other data used to compile rankings could be subject to biases, such as the “reputation” data in US News & World Report’s ranking.⁷

How are hospital rankings used? 
While the ratings methodology and any external organizations involved in compiling hospital rankings (e.g., for U.S. News & World Report, RTI International, https://www.rti.org/) are independent from commercial influence, patients and their families should be aware that US News & World Report, ³ Leapfrog,¹¹ and Healthgrades¹²monetize rankings through licensing their respective marks and offering marketing and advertising services associated with their rankings to rated hospitals.¹ Often, the cost to license these marks in advertising is not publicly disclosed.¹

What are their limitations? 
While there is no gold standard for assessing hospital ratings,¹ there are important limitations that patients and their caregivers should be aware of. The Medicare claims data used may not accurately represent all populations that hospitals serve.⁴ Furthermore, the mortality data may not accurately reflect a specialist physician’s involvement with a patient, with deaths attributed to subspecialists based on hospital codes and not a physician’s care.⁶ While the potential for bias in the Hospital Ratings may be less than for US News & World Report’s College Ranking (where reciprocity bias can skew the expert opinion portion of the rating) it still is a concern.⁷ And hospital rankings do not undergo peer-review, the gold standard in the scientific and medical publishing that helps ensure the accuracy and validity of the underlying science and conclusions. A recent assessment of hospital ratings systems found they “… frequently publish conflicting ratings: Hospitals rated highly on one publicly reported hospital quality system are often rated poorly on another. This provides conflicting information for patients seeking care and for hospitals attempting to use the data to identify real targets for improvement.”¹ 

How should patients and caregivers use hospital ratings? 
Caveat emptor. Be aware of the limitations in the underlying data, the lack of transparency, and the potential biases involved in calculating the ratings.^4,5 A recent assessment of all four current hospital rating systems “found that [they] should be used cautiously as they likely often misclassify hospital performance and mislead.”¹ 

What kidney disease resources are available for patients and family members? 

AMERICAN ASSOCIATION OF KIDNEY PATIENTS

• General resources, information on clinical trials and peer mentors through the AAKP Center for Patient Research and Education 

AMERICAN KIDNEY FUND

• Fact sheets on kidney disease
• General information and resources about kidney health

NATIONAL INSTITUTE OF DIGESTIVE, DIABETES AND KIDNEY DISEASES 

• General information about kidney disease

NATIONAL LIBRARY OF MEDICINE (Some of the scientific articles include patient-focused information.)

NATIONAL KIDNEY FOUNDATION 

• NKF Cares: nkfcares@kidney.org; 855.NKF.CARES (855.653.2273)
• NKF Ask the Doctor  
• NKF Peer Mentoring  

RENAL SUPPORT NETWORK 

• RSN Kidney Health Library
• RSN online support groups

References* 

1.    Bilimoria KY, Birkmeyer JD, Burstin H, Dimick JB, Joynt Maddox KE, Dahlke AR, et al. Rating the Raters: An Evaluation of Publicly Reported Hospital Quality Rating Systems. NEJM Catalyst. April 14, 2019. Available at https://catalyst.nejm.org/evaluation-hospital-quality-rating-systems/. 

2.    U.S. News & World Report. FAQ: How and Why We Rank and Rate Hospitals. Available at https://health.usnews.com/health-care/best-hospitals/articles/faq-how-and-why-we-rank-and-rate-hospitals.

3.    U.S. News & World Report. Best Hospitals: Hospital Marketing Opportunities. Available at https://health.usnews.com/best-hospitals/marketing-opportunities. 

4.    Harder B, Comarow A. Hospital Quality Reporting by US News & World Report: Why, How, and What's Ahead. JAMA 313: 1903–1904, 2015. Available at https://jamanetwork.com/journals/jama/article-abstract/2293304.

5.    Lydgate C. The Ominous Cracks in the US News College Ranking System. Reed Magazine. Available at https://www.reed.edu/reed-magazine/articles/2021/rankings-gladwell-usnews.html. 

6.    Mendu ML, Kachalia A, Eappen S. Revisiting US News & World Report’s hospital rankings—moving beyond mortality to metrics that improve care. J Gen Intern Med 36: 209–210, 2021. Available at https://link.springer.com/article/10.1007%2Fs11606-020-06002-x. 

7.    Olmsted MG, Powell R, Murphy J, et al. US News & World Report Best Hospitals 2020-21. US News & World Report. July 15, 2020. Available at https://health.usnews.com/media/best-hospitals/BH_Methodology_2020-21. 

8.    Centers for Medicare and Medicaid Services’ (CMS) Star Ratings. Available at https://www.medicare.gov/care-compare/?providerType=Hospital&redirect=true. 

9.    Healthgrades. Mortality and Complications Outcomes 2021 Methodology. Available at https://www.healthgrades.com/quality/mortality-and-complications-outcomes-2021-methodology. 

10.    Leapfrog Hospital Safety Grade Scoring Methodology. Spring 2021. Available at https://www.hospitalsafetygrade.org/media/file/Safety-Grade-Scoring-Methodology-Spring-2021_2.pdf. 

11. Leapfrog Licensure & Permissions. Available at https://www.hospitalsafetygrade.org/Licensure. 

12. Healthgrades. Advertising Guidelines. Available at https://www.healthgrades.com/quality/healthgrades-advertising-guidelines. 

*NB: While some of these resources are publicly available, others may require paid access. Check your local library to find your viewing options.