What can Law Schools Learn about Bar Passage from Medical Schools’ Approach to Studying Students Who Struggle with Licensing Exams?

It’s not unusual for a provost or a colleague or a relative at Thanksgiving to ask a legal academic why law students have so much trouble passing the bar exam when the pass rates for medical students are usually in the high 90th percent.  The short answer to that question is that the two processes are completely different—and there’s no obvious trick, technique, or intervention that could convert our bar passage rates into their licensure passage rates.   For one thing, it’s the wrong question.  “Passing” the medical licensing exams is certainly important, but unlike the “all or nothing” process of passing the bar exam, the score achieved on Step 1 affects medical students’ entire career path.  But there is a lot to learn about the methods that medical schools use in studying the very few students who have trouble as well as how they evaluate the effect of changes to their curriculums on scores on the licensing exams.

Quick recap on professional licensing—future doctors take a series of three exams over the first six years of their undergraduate medical education and the start of their residency.  (more links in a post I wrote earlier this year here).  The exams are almost entirely national although the actual process of being licensed is conducted on a state by state basis.   Law students take a licensing exam in the state where they intend to practice upon graduation.  For purposes of this post, the closest analogy to the bar exam is the more academic Step One students take during their second year of medical school.  Like  our NCBE, the National Board of Medical Examiners which produces United States Medical Licensing Examination works with medical licensing boards and depends on their confidence.  It issues annual reports.

The focus of this post is on the methods that medical schools use to study the small number of their students who do have trouble passing the licensing the exams as well as the factors that can affect the scores students achieve.  I’ve tried to focus on articles outside of paywalls, and would certainly encourage you to conduct your own searches in the various data bases to which you have access.  There are several journals devoted directly to studying medical education—although these articles can pop up anywhere.

Medical educators use a wide range of research techniques to learn more about students who struggle with licensure exams.  Like us, medical schools would prefer students pass the first time and many articles like this one look for characteristics who fail the first time but eventually pass.  Others look for characteristics of students at risk for failure here and here  or even  what students think of the exam.    Another area for inquiry involves the role stress plays in the score students achieve.   In partnership with social scientists at our schools or in our communities, we too could be conducting studies to help us learn more about students who face difficulty passing the bar exam.  These studies can be part of graduate student work or may even be funded by groups like Access which is making money available to study bar passage.

 

The actual reason the medical school pass rates are so high, though, may not be all that helpful.

It’s not just because they are able to limit admission to students who have already demonstrated an ability to score very highly on the MCAT.  A test that is much more similar to step 1 than the bar exam is to the LSAT.  Indeed, medical schools have direct input in both the MCAT and the Licensing Exams—so when one changes, the other can too. And it’s not clear that anything in the curriculum makes a difference at all—the industry offering study aids and licensure prep courses dwarfs the bar prep and study aid market to a point where students often start studying for the licensing exams before the first day of medical school.

But if it is the curriculum, it’s important to remember the vast difference in time scale between medical and legal education.  We have students for three years post B.A. Medical schools in the U.S. plan their curriculum based on  8 plus years of increasingly specialized medical education.  They are therefore comfortable holding off on the direct teaching of practice skills for the first two years while they are aligning their curriculum with the content of the Step 1 exam.

Even Step 1, though, is far more focused on practice than on knowledge accumulation or deliberately confusing question formulations that characterize the bar exam. Step 2,  the second round of licensing exams prior to graduation medical school,  go past paper and pencil in that they actually test students’ ability to conduct exams and exercise medical judgement.  Another reason for the high pass rate is that most medical schools have stopped developing their own tests and instead use assessment instruments (shelf exams) provided by the same company that produces the exam.   Sure, there is grumbling and criticism about content & timing of the licensing exams, but medical schools work hard to make sure that their curriculums are aligned with the content of the exams.  Finally, medical education is extremely self-reflecting–they are constantly aware of the risks that come from confusing correlation and causation.  How do you know that a change in one part of the curriculum is the cause of a change in test scores?  You run Pearson correlations followed by stepwise linear regressions.  Seeing is not believing when comes to identifying factors that affect performance on licensure exams.   Look here, here, here, and here for studies evaluating curriculum changes.  They take nothing for granted—does attendance make a difference, does flipping classrooms really work? Does reducing the number of hours spend in the anatomy lab reduce USMLE scores?

Another standard practice in medical schools is curriculum mapping— an essential first step for any school that wants to understand what they are teaching—let alone make changes.   Like all maps, curriculum maps are DESCRIPTIVE, not PROSCRIPTIVE.  Here is   Harvard’s curriculum map, but you can find examples on the home page of just about every U.S. Medical School.This is a an article walking through how to map a curriculum.

So what’s helpful to us isn’t so much what medical schools are doing, but how they are evaluating themselves. 

In recap, neither I nor anyone else who has ever practiced law thinks it would be a good idea to emulate medical schools by fully aligning our curriculum with the bar exam so as to turn the three years of law school into one extended bar prep course.  Among other reasons, the material tested on the bar is quite static and doesn’t reflect the realities of today’s law practice.   It also wouldn’t make much sense for schools whose students take the bar exam in many different jurisdictions.   Also, the bar exam is just not equivalent to the three rounds of USMLE exams in actually testing both the knowledge and application of knowledge needed to be a successful lawyer.  If it was, we wouldn’t hear so many complaints about how students who have passed bar are never-the-less not “practice ready.”

Tomorrow—where can we get the help we need to find out this information, and who is going to pay for it?  Spoiler--Access Lex has a program.

Advertisements

7 Responses

  1. No, it’s mostly because they limit admission to competent people who belong in medical schools. Law schools would do well to do the same. ABA is a failed governing organization and watchdog in this respect.

  2. Happy to talk with you about this–what’s hard to capture in these comparisons is that “graduates of US Medical Schools” is actually only a portion of “practicing physicians in the US” (maybe not even half) while almost all lawyers practicing in the US are graduates of US Law Schools. So you’re right that the range of admission scores among US medical school graduates is quite low–which is why they focus more on “score” than “passage” Feel free to PM me.

  3. I meant to say “narrow range”

    • Sure. I appreciate your point. I think it’s hard to dispute that med schools have a much higher criteria, though, and that the main culprit in the drop of bar passage rates is relaxation of admissions criteria in law schools.

  4. This is not rocket science and needs little study. LSAT and GPA is HIGHLY predictive of bar passage. And when standards relaxed as applications drop, passage rates shrunk. As long as law schools continue to admit many students with low LSAT scores and low GPAs (from poor and mediocre undergraduate institutions), bar passage will remain low. And law schools shouldn’t NEED to “teach” to the test. A couple months of Barbri or whatever is the popular test prep these days is all that should be needed.

  5. Good article that is relevant to this post.

    https://www.thefacultylounge.org/law_school_news/

    “A number of states have released bar results from the July 2018 examination over the last several weeks, including New York, Texas and Florida. Overall, performance on the bar was very poor, although there have been a few bright spots. The national average on the MBE was down 2.2 points to 139.5, a 34 year low. To those who have followed law school admission trends closely, this should not have come as a surprise. The 2014 and 2015 entering law school classes were arguably the weakest ever in terms of entrance credentials, and these cohorts (part-time students who started in 2014 and full-time students who started in 2015) made up the majority of the first-time test takers in July. There were also a lot of repeat takers from 2017 and February 2018 who also had very low credentials.

    The purpose of this post is to remind everyone, especially law school faculty and administrators at lower tier schools, of the perils of admitting very high risk (LSAT of 145-146) and extremely high risk (144 and below) students. As law schools start to experience an uptick in applications, the focus, especially at low-performing schools, should be on raising the quality of admitted students, not growing the class. A few examples should suffice to demonstrate this point.”

    ————————————————
    “These five examples demonstrate (once again) that when law schools admit high percentages of very high and extremely high risk students, these classes are virtually certain to have very low bar passage rates. Put another way, no law school has figured out a magic formula to take students with very limited aptitude for the study of law, as demonstrated by their LSAT scores and college grades, and turn them into bar passers. All the law schools that have closed in the last couple of years (or are in the process of or in peril of closing), including Whittier, Indiana Tech, Charlotte, Arizona Summit, Valparaiso and Thomas Jefferson, amply demonstrate this point. Although the overall decline in demand for legal education played some role in the downfall of these schools, it was their decision to lower admission standards to an unethically low level that started the vicious cycle of dropping bar pass rates, declining reputation, decreasing applications and increasing transfers that ultimately doomed them.”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: