Unmasking Assumptions about Employment Outcomes and Legal Education

In an upcoming Wisconsin Law Review article, Robert Kuehn, Associate Dean for Clinical Education and Professor of Law at the Washington University Law School, presents a cogent, well-supported and thoughtful article describing the limitations of and lessons we can learn from the existing empirical analysis correlating student enrollment in clinical education and employment outcomes.  Kuehn’s article, entitled Measuring Legal Education’s Employment Outcomes is particularly powerful because it provides a thorough empirical rejection of the claim that clinical coursework might actually harm employment outcomes, as asserted by Professor Jason Yackee and which attracted some sound-bite attention earlier this year. In what is, perhaps,  an unexpected twist, Kuehn demonstrates that using Yackee’s statistical assumptions and methodology also would produce negative correlations for those students who participate on law journals or in moot court competitions.  Kuehn argues that one can’t draw any reliable conclusion from Yackee’s 2013 model, and perhaps not from any nationwide statistical model – as opposed to a particularized analysis of one school –  on the likely effect of clinical courses (or other activities like law journal or moot court) on employment, and surely not the negative effect Yackee posits. Kuehn points out that as to clinical coursework, the available evidence (through surveys) indicates that such experiences do aid some students in securing employment.

If you, like me, still become a bit nervous about how much you actually remember from undergraduate statistics courses, do not be alarmed by this post!  You will find Kuehn’s article accessible and a quick good read, even when he is using words like “regression analysis,” “granular data” and “variable choices.”   Here are the points made in Measuring Legal Education’s Employment Outcomes which I found most helpful:

  1. Kuehn’s reminder that when one confuses correlationwith causation one is bound to come up with a “misdiagnosis.” One problem with Yackee’s analysis is the lack of granular data to calculate the true employment rate for those who took a clinic (or who did not).  In fact, the data is so poor that “the results never account for more than half of the variability in employment across schools.”
  2. Kuehn’s explanation of the “confounding effect of prestige” and bar passage on employment outcomes.
  3. The problems of validity and reliability raised by analyses which employ information from ABA questionnaires, particularly those self-reports submitted prior to 2014.
  4. The fact that “13% of law schools” provide 80% of the school-funded jobs to law graduates. Not surprisingly, Kuehn found this factor biases many results if you examine nationwide statistics. And when Kuehn removes those jobs from the statistical analysis, Yackee’s correlation with clinical education falls apart even using his own assumptions and methodology.
  5. Yackee’s model yields completely different results if one uses the US News Lawyers/judges data versus academic peer data to control for the possible influence of perceived prestige.
  6. Application of Yackee’s model to “Law Journals” and “Skills Competition” and S. Newssub-groups also show no relationship to employment outcomes!
  7. In Yackee’s model, a better ranking is “strongly associated with improved employment outcomes.” However, Kuehn points out that a “closer examination of the relationship between rank and employment indicates that this positive association, although statistically significant when applied across the entire range of top 100 schools, does not hold true for schools ranked 51 through 100 (emphasis added).” 
  8. Kuehn’s documentation of employers who require, “strongly prefer” or identify law clinic experience as a positive factor in hiring such as The U.S. Department of Homeland, legal services and  legal aid offices, district attorney, public defender, fellowships and private law firms.
  9. Kuehn’s description of National Association of Law Placement (NALP) existing information: such as the  2011 survey of lawyers with non-profit and government offices;  the NALP survey of lawyers in firms of predominantly more than 100 attorneys; the NALP survey of public interest legal employers;  and the NALP 2013 presentation on the employment market reporting that ” law firms say they want new graduates to have ‘more experiential learning, client-based and simulation.”
  10. Kuehn provision of good information on other employer information such as the Lexis-Nexis WHITE PAPER: HIRING PARTNERS REVEAL NEW ATTORNEY READINESS FOR REAL WORLD PRACTICEProfessor Neil Hamilton’s employer survey to determine the relative importance of twenty-one different competencies in employer hiring decisions, and Professor Susan Wawrose’s legal employer focus groups which found employers prefer new hires with ” well developed professional or ‘soft skills” along with “strong fundamental practice skills.”

Professor Kuehn concludes by recommending that studies could best be done on a school-by-school basis by “surveying likely employers to find out what educational experiences of students are most valued.”  Professor Kuehn also recommends that schools could also “retrospectively look at various employment outcomes for graduates and any relationship” to students’ experiences while in school.

I agree with Professor Kuehn and am happy to report that  Albany Law School,  through its faculty Assessment committee and Admissions office,  is currently engaged in conducting employer focus groups and analyzing what best helps our students obtain employment in their desired career paths.  Until good data and information suggests otherwise, Professor Neil  Hamilton’s advice to law students,which Professor Kuehn quotes in his “must read” article, bears repeating:

In this challenging market for employment, a law student can differentiate herself from other graduates by demonstrating to legal employers that the student both understands the core competencies that legal employers and clients want and is implementing a plan to develop these competencies, including an ability to demonstrate that the student has experience with these competencies.

Building on Best Practices now available as eBook

Are you trying to:

  • Develop a meaningful law school mission statement?
  • Understand new accreditation requirements, learning goals, and outcomes assessment?
  •  Expand your experiential offerings?  Decide whether to use modules or courses?  An on-site clinic, an externship, or community partnership?
  •  Teach ALL of your students in the most effective ways, using a full range of teaching methods?
  • Add to your curriculum more of the professional identity, leadership, intercultural, inter-professional and other knowledge, skills, and values sought by 21st century legal employers?
  • Lead thoughtfully in the face of the challenges facing legal education today?

These and other topics are addressed in Building on Best Practices:  Transforming Legal Education in a Changing World,  now available in ebook format from LexisNexis at no charge.

The print version is not yet out.  LEXIS-NEXIS is taking advance orders for $50, plus shipping.  BUT we understand that they will make one copy available to every US legal educator for free upon on request.  Details on this and international availability still to come.

Thanks, and congratulations, to book project sponsor Clinical Legal Education Association (CLEA), the more than fifty legal educators who participated as authors, and the countless others who assisted as readers and in numerous other ways.

And, a huge shout-out to my wonderful and talented co-editors, Lisa Radke Bliss, Carrie Wilkes Kaas, and Antoinette Sedillo Lopez.

Building on Best Practices for Legal Education Manuscript Submitted to Publisher

Four editors,  59 authors, 92 readers, three copy editors, librarians from two schools, a secretary, miscellaneous consultants, three student assistants for bluebooking, and one for setting up perrmacc links.*

Many people, occasionally in multiple roles, were needed to produce the manuscript sent to Lexis last Monday for the forthcoming book Deborah Maranville, Lisa Radtke Bliss, Carolyn Wilkes Kaas, and Antoinette Sedillo López (eds.),  Building on Best Practices:  Transforming Legal Education in a Changing World. (Lexis 2015).  A monster project — but, as I assured a friend, no, not a manuscript about monsters and not monstrously unpleasant to produce – just big, ambitious, and sometimes exhausting for the editors and authors.  A big thank you to all who participated!

The book is a follow up to CLEA’s Best Practices for Legal Education, the 2007 volume by Roy Stuckey and others that inspired this blog.  Like Best Practices, this book will be distributed for free to legal educators.  Lexis has promised to make it available in electronic format through their e-book library and to provide print copies on request.  Look for it in four to six months — if all goes smoothly perhaps in time for the AALS Clinical Legal Education Conference in early May.

The coverage of Building on Best Practices is wide-ranging.  To quote from the Introduction, “[t]his volume builds on the call to link mission and outcomes; emphasizing the themes of integrating theory, doctrine and practice, developing the broader spectrum of skills needed by lawyers in the twenty-first century, and taking up the question how best to shift law school cultures to facilitate change.”

Advance praise for the book has included:

  • “[M]ilestone in legal education . . . that legal educators will rely on as much as . . . on the first Best Practices book.”  (Patty Roberts, William & Mary)
  • “Educational for folks who don’t know much about experiential education and insightful for those who do. . . .Really something to be proud of . . . an invaluable resource to schools as they go to work on implementing the ABA’s new requirements for learning outcomes and assessment. . .The perfect product coming out at the perfect time.” (Kate Kruse, Hamline)

Once again, CLEA deserves kudos for its support of an important scholarly project on legal education.  And the Georgia State University, University of New Mexico, Quinnipiac University, and University of Washington Law Schools deserve a big round of thanks for supporting the co-editors in this project.

https://perma.cc/ provides an archive for those annoying website links that quickly become outdated.

Congratulations UNM and Editors of the proposed new Best Practices Book!

This weekend, the University of New Mexico hosted a workshop BEST PRACTICES IN LEGAL EDUCATION: The Walls Are Coming Down” in which draft chapters of a new “Best Practices” book were reviewed and discussed.  The proposal to create a second book focused on best practices in legal education is the brainchild of Professor Antoinette Sedillo Lopez, ably assisted by Professors Deborah Maranville, , Carolyn Kaas and Lisa Bliss. The symposium workshop brought together law professors from throughout the country interested in how legal education and the world of law schools has changed since the publication of the 2007 book Best Practices in Legal Education. Facilitated by Professors Beryl Blaustone and Alex Scherr, the conference explored how many law professors fluidly move from former silos of clinical, legal writing, lawyering, librarian, doctrinal, theory, or skills concentrations to pioneer a new kind of curriculum, better prepare students for the profession, explore the limits and usefulness of technology, and deepen the understanding and learning of law students through self-improving assessment processes.

Fully cognizant of the pressures on legal educators, the fact that not all in legal education welcome the need to change, and the moral imperative to address the concerns of debt-ridden unemployed law students, the authors, editors, advisory board members and readers reviewed challenges, cross-cutting themes and areas of promise. They engaged in innovative thinking about how to move legal education forward for the good of the profession, society and the students who desire to be lawyers of tomorrow. The keynote speaker for the Friday night dinner and author of the first book, Professor Roy Stuckey, directed the participants’ attention to what legal education should look like in 2027. At the same time, he reminded us that those seeking to improve legal education today stand on the shoulders of folks such as the honorable Rosalie Wahl and former ABA president Bob MacCrate who paved the way for the changes we have seen in the last 40 years. He recalled their joint mission to prepare “agents for justice in our communities.”

Every law graduate needs to understand fully that civic professional role of the lawyer. And every admittee to the bar has a sworn duty to improve our system of and access to justice. Returning to those principles can help prioritize our cost-cutting and can position us to move forward in the best interests of our students, our institutions and the society our profession is pledged to serve.

Best Practices Goes International

It is only a few days before Christmas–and, as I write this, the first night of Channukah–but my thoughts turn not  to the holidays but rather to Best Practices.  As David Segal continues to write his maddening articles in the NYT on legal education–the most recent, this past Sunday, on how ABA accreditation causes law schools to be overly costly and cookie-cutter in form and content–it is well to recall the basic underpinnings of the CLEA Best Practices book, and its focus on thoughtful curricular planning and learning outcomes.  Excessive student debt, less-than-full transparency in law school statistics and practices,  and a failure to focus on preparing students to practice law are valid criticisms, and Segal is right to make them, but we all know there is more to legal education than the narrow consumerist focus that Segal (who also writes the consumer complaint-oriented The Haggler column in the Times’s Sunday Business Section ) brings to bear.

I will have more to say about those specific questions in a subsequent post, but for now I wanted to mention that Best Practices has spread its wings and reached China.  There is a now an authorized Chinese version of the book (alas, I can’t attest from personal knowledge to the accuracy of the translation, as my Mandarin consists of only a few words) that has been produced in conjunction with some ongoing workshops on clinical education in China in which US law professors from University of Pacific-McGeorge School of Law and American University Washington College of Law have been involved for the last 4-5 years.  In its current focus, Brian Landsberg and Dorothy Landsberg (director of clinical programs) at McGeorge are spearheading an effort by Chinese legal educators to develop a version of Best Practices that will work in the Chinese legal (and legal educational) context.  I went with Brian and Dorothy to Beijing in September to assist the group with organizing the project, and Brian and Dorothy will be returning there in January to provide feedback on the draft chapters. (Frank Bloch has been working with the group on a related project, and my colleague Elliott Milstein has been deeply involved with the China project from the beginning; other McGeorge, WCL and other faculty have been involved as well.)

As others who have done international clinical work can attest, the translation, literally and figuratively, of clinical concepts  (including those in Best Practices) to other legal systems can be quite challenging.  During the first workshop in 2006, for example, several of our Chinese colleagues resisted our efforts to articulate a client-centered approach to lawyering, stating that such an approach could not work in China.  When we pressed our colleagues on this point, they said that in China if the lawyer deferred to the client (as they understood client-centeredness to require) clients would importune their lawyers to engage in illegal actions.  We had not emphasized in our discussion of the concept of client-centeredness that it assumed that the client’s means and ends were legal and legitimate within the system. Absent that qualification, our colleagues heard our discussion of the concept as promoting lawlessnesss (ironic in a program sponsored by the US AID Rule of Law Project).  With the clarification, our colleagues thought that such an approach could work, though, as we all agreed, the nature of the counseling relationship and the choices actually available to clients and lawyers might well look different from how they would look in the U.S.

We had other moments of language and cultural miscommunication.  For example, it took us some time to realize that our use of the term “advocacy”–meant to convey trial-related skills of direct and cross-examination, among other things–was confusing to our Chinese colleagues since the term was translated as “criminal defense.”  Because we came to know our colleagues well–the first three years of the project consisted of three-week workshops in China, and some of the first group of attendees became co-teachers in our subsequent trainings–we were able to figure out which concepts traveled well and which did not.  The result, we hope, will be the development of a distinctively Chinese form of clinical legal education whose value, we hope, will stand the test of time.

In the workshop this fall, we did come upon one other language/conceptual roadblock, which is relevant to Best Practices. The Chinese group was concerned about use of the term “Best Practices.”  Our Chinese colleagues thought that if they entitled their book “Best Practices” it would create problems for them, because the implication would be that if anyone did not adopt their approach they would by definition be doing something that was less than the best.  The implied criticism of others would be too sharp, in their eyes, and would inevitably lead to hurt feelings and perhaps more.


Note that this criticism was not the same as that which my AU colleague Ira Robbins had leveled against CLEA’s Best Practices project (to which Roy Stuckey responded in a colloquy published in the Clinical Law Review)–that we were wrong to call our book Best Practices because we could not prove the practices we advocated were the best, as opposed to, say, good or recommended practices.  Indeed, the Chinese critique presumed that use of Best Practices did in fact mean that they would be saying that they were proposing what was best, and that others would react negatively to this assertion.

Indeed, before we arrived in Beijing in September we had thought there might be some concern with the term Best Practices, and had proposed the term “Effective Practices”  as one that might be less controversial.  That approach seemed to resonate with our Chinese colleagues in September, though it remains to be seen what title the group will adopt for the final product.   It could be argued that “Effective Practices” implies that those who do not adopt the suggested approach are promulgating Ineffective Practices, which would be just as much of a an apparent dismissal as not using what some thought was the best approach. But I suppose that since the book is not entitled “The Most Effective” practices, the term “Effective Practices” may seem to suggest a wider range of potentially acceptable approaches.


Of course, the term Best Practices as we use it in the CLEA book, and as many in the US have used it in different contexts (especially in social sciences or in regulatory environments) doesn’t really mean “best” in the sense of provably better than everything else.  (I disagree with Prof. Robbins’s assumption to the contrary.) Rather, the term is meant to suggest something like “here is our best thinking, at this time, about the particular matter.”  It is more than what might be minimally required, and, indeed, it is not required at all, but rather exists as a source of information and guidance that people may find useful and therefore may choose to adopt for their own purposes.

In the final analysis, the linguistic vagaries of the term Best Practices are less interesting than the recognition that thoughtful legal educators in China, and perhaps elsewhere around the world, may come to realize the value of the concepts articulated in the CLEA Best Practices book.  If the book generates dialogue and self-conscious examination of our pedagogical goals, it will have gone a long way toward achieving the goals that many of us have had for this project.


–Bob Dinerstein

December 20, 2011



Building on Best Practices: Call for Ideas and Authors

The Clinical Legal Association, Best Practices Implementation Committee is planning a follow-up publication to Best Practices for Legal Education by Roy Stuckey and others.     The vision of the book is to build on ideas for implementing best practices, and to develop new theories and ideas on Best Practices for Legal Education.   If you would like to author a section in the book please let us know as soon as possible.   Then by December 1, 2011 send either of us a 3-5 page abstract identifying the knowledge, skills and values as well as the learning objectives and methodology of your innovative teaching idea.   The Editorial Board will meet at the AALS meeting in January to select pieces for inclusion in the book.


If you have any questions or thoughts about the project please feel free to contact either of us.


Looking forward to drawing  on the expertise of the legal academy to build on Best Practices for Legal Education!


Antoinette Sedillo Lopez ,Chair, Publication Committee

Deborah Maranville,  co-editor


Sharing Scholarship, Building Teachers Conference

Albany Law School will be hosting the Sharing Scholarship, Building Teachers conference on February 3-4, 2012.

This workshop is intended for law faculty who do not have tenure and who seek an opportunity to develop their scholarship and discuss their teaching with other, similarly situated law faculty. This program will provide a safe and comfortable forum for untenured faculty to present works in progress, solicit feedback from peers ahead of the February-March Law Review submission season, and network with other untenured faculty in the region about teaching practices and related issues.

There is no fee to attend the program (whether you are presenting a paper or not), but you must register before the deadline: November 15, 2011. Albany Law School will provide all meals and drinks during the workshop at no charge to attendees. 

I hope to see you all there,

 Sarah Rogerson
Assistant Clinical Professor of Law
Director, Family Violence Litigation Clinic


Get every new post delivered to your Inbox.

Join 993 other followers

%d bloggers like this: