The initial pass rate for the nephrology certification exam dropped by 9% in 2019, setting off concerned discussions on Twitter and a search for explanations by many in the nephrology community.
The decline in the pass rate to 74% represents a precipitous drop from the 90% rate of 2016. The rates in both 2017 and 2018 were 83%. For the 15 subspecialties tested under the aegis of the American Board of Internal Medicine (ABIM) in 2019, the average pass rate was 90.5%.
Although the size of the decline raised questions about a problem with the test, ABIM is steadfast in defending the reliability of its psychometric methods for holding the test difficulty steady from year to year. Given that, other proposed explanations include a change in the quality of the test-taking pool, a failing in the quality of education trainees are receiving, and a mismatch between the test material and the clinical experience of fellows.
By the numbers
Although scores temporarily recovered from a similar drop in the past, it’s unlikely that this drop is a single-year aberration. The scores dropped 7 percentage points to 80% in 2014, only to bounce back to 89% in 2015 and 90% in 2016. But that recovery was followed by another decline to 83% in both 2017 and 2018. The nephrology pass rates reflect a long-term decline: in the five-year period from 2006 to 2010, the pass rates averaged 94%; from 2010 to 2014, they averaged 88%; and from 2015 to 2019, they averaged 84% (and for the past three years, 80%).
During that recent five-year period, the only other ABIM-tested subspecialties with pass rates averaging under 90% were geriatric medicine at 86% (and 88% for the past three years) and endocrinology, diabetes, and metabolism at 89% (89% for the past three years).
A sampling of other 5-year averages include: rheumatology, 92%; gastroenterology, 97%; infectious disease, 97%; and cardiovascular disease, 93%.
Although several observers raised the statistical possibility that nephrology, with fewer exam takers, would be subject to larger swings than some larger subspecialties, subspecialties such as rheumatology and infectious disease have fewer examinees than nephrology and have not experienced swings greater than 3 or 4%.
A drop in candidate numbers
Perhaps the leading explanation for the drop in scores is a change in the quality of the trainee pool. “It is public knowledge that over the last several years, nephrology programs in the US have had difficulty filling their positions,” said Gary Singer, MD, a senior partner at Midwest Nephrology Associates in St. Peters, Mo., and a member of the ABIM nephrology board. Ten to 15 years ago, nephrology training programs saw 1.5 candidates applying for every position. Currently, only six of every 10 positions are filled directly through the National Resident Matching Program match. And nephrology is not the first choice of a significant number of fellows matching into it.
Nephrology also has a higher proportion of international medical graduates now than in the past—and also more than many other subspecialties—and international graduates do not perform as well as US graduates on the nephrology certification exam.
Evidence of a change in the candidate pool comes from their performance on the internal medicine certification exam taken before they enter into nephrology training. “Candidates who completed the nephrology certification exam had the lowest scores on the internal medicine certification exam compared to other subspecialties,” said Bradley Brossman, PhD, vice president of psychometrics at ABIM. “Ten years ago, candidates who completed the nephrology certification exam had among the highest scores on the internal medicine certification exam compared to other subspecialties.”
Scott Gilbert, MD, of Tufts Medical Center and chair of the ASN Workforce and Training Committee, said: “Given the declining interest in nephrology, it raises the concern we are accepting trainees into our program through the match and the subsequent scramble who we might not have considered in earlier years in order to fill our complement. This highlights the need to maintain our standards even if it means not filling all of our positions.”
Another factor could be that some fellows are not pursuing nephrology certification because their ultimate goal is the higher-paying field of being a hospitalist—perhaps with special expertise in nephrology—according to ASN Executive Vice President Tod Ibrahim. The number of fellows taking the test for the first time has declined in the past two years, from a rough average of around 420 in the prior decade to 365 in 2018 and 375 in 2019.
Further indirect evidence for the candidate-pool argument is that the other subspecialty with the lowest pass rates is geriatrics—another field that has had trouble attracting a sufficient number of applicants.
Need for better education?
Regardless of any change in the candidate pool, the drop in the pass rate is “tough to see,” said Matthew Sparks, MD, assistant professor of medicine at Duke University and associate director of its fellowship program. “The most controversial aspect of this is whether we are letting individuals into our field that have more deficits to fill. But all these people passed the internal medicine boards, and we should be able to get them to pass the nephrology boards on the first attempt. We should be able to identify those that need more help and utilize resources to help them. The individuals taking the test represent more than just themselves. They represent the program in which they trained, the educational opportunities they have, and the emphasis of their education.”
Sparks said that the low pass rate “is a hard pill to swallow when the nephrology community has put a lot more effort into education recently by including educational sessions at ASN Kidney Week and by more grassroots efforts to start online educational websites.”
Training program challenges?
As long ago as 2014, Christina Yuan of Walter Reed National Military Medical Center and two co-authors wrote an editorial in the American Journal of Kidney Diseases wondering whether “training programs are not providing adequate education,” and concluding that, based on calculations from the general pass rates, many nephrology training programs “are perilously close to or have fallen below” the minimum pass-rate threshold required by the Accreditation Council for Graduate Medical Education (ACGME) to remain accredited. Despite repeated requests to confirm or deny this conclusion, ACGME declined to respond, instead referring questions to ABIM, which is not involved in program accreditation.
Suspicion of the test
Based on his conversations with ABIM’s experts in psychometrics, Jeffrey Berns, MD, of the University of Pennsylvania and chair of the ABIM nephrology board, is confident that the difficulty of the test hasn’t changed: “ABIM does everything within the power of its psychometricians to make sure that the exam difficulty doesn’t vary from year to year.”
But given the current climate in which the unpopularity of the maintenance of certification process is spilling into a general suspicion of ABIM, some critics question whether there could be a mismatch when it comes to training, current clinical practice, and what the test covers.
In their editorial, Yuan et al. said that the board exam committee “is composed of distinguished members of the nephrology community; not surprisingly, the committee overwhelmingly includes older academic nephrologists. Increasing the number of expert voices in developing the certifying exam, in particular including younger nephrologists and those who practice outside of academia, would be valuable.”
The exam committee has become younger and more diverse in recent years, according to Berns, who believes there is evidence the exam remains relevant: “We know that performance on the ASN In-Training Examination is predictive of the ABIM nephrology board pass rate. The good news is that people can re-take the test if they don’t pass it the first time, and historically, the ultimate pass rate has been in the high 90s.” According to ABIM, the ultimate pass rate is 97% across all disciplines.
“One of the ways of interpreting it is that people who don’t pass get out their books or their Internet resources, and they study and learn the material,” Berns told Kidney News.
The number re-taking the exam can be significant. In 2019, 120 candidates who had previously failed the nephrology certification exam took it again.
“I personally think we need to focus on our trainees and how/what we are teaching them. Not on the exam,” Berns said. “We are continuing to get outstanding applicants who become fellows in our training programs, and this is true among both the US medical graduates and international medical graduates.”
Regardless of the reasons for the drop in the pass rate, Sparks argues: “We have to focus on the things we can control ourselves, which is our programs, our educational content, and our curriculum, and identifying trainees that might need additional resources.”
