Saw this update from NCBE's Beth Donahue:

Changes in Multiple-Choice Examination

Mirroring the accelerating changes of the information age in which we live, NCBE’s multiple-choice examination programs—those for the Multistate Professional Responsibility Exami- nation (MPRE) and the Multistate Bar Examination (MBE)—have undergone significant change in the past five years. While most of the changes have been reported and discussed in past issues of the BAR exAMineR, this article gives an overview of the two programs as a whole and highlights their recent evolutions.

Underlying Focus

As with all NCBE exams, our touchstone question in developing the MPRE and the MBE is What does a minimally competent new lawyer need to know and how can we test for that knowledge? Although our drafters are experts in their fields and are capable of drafting highly complex problems, they focus on fundamental aspects of the beginning practice of law when they write for our exams, and we have maintained the goal of testing for minimal competence in every recent development.

Retesting

For psychometric enthusiasts (they’re everywhere, aren’t they?), the addition of pretesting—using items on an examination that are being “tried out” but do not count in the examinees’ scores—to the MPRE (in 2005) and the MBE (in 2007) has been an enormous milestone. The pretesting process established 10
unscored question slots on both the MPRE, which moved from 50 questions to 60 questions to accommodate the pretest additions, and the MBE, which
remained at 200 questions overall by having the number of scored questions reduced from 200 to 190.

Because each of these examinations is administered with eight different forms each time, multiplying the 10 available pretest question slots on each form by the eight forms results in 80 questions being tested on a trial basis during every administration. The significance of pretesting is hard to overstate. It allows NCBE’s drafting committees to rework questions that do not perform as expected, making them harder or easier, or eliminating subtle ambiguities that may have gone unnoticed during the drafting and editing processes. When questions that have been pretested are placed on exams as scored questions,there is greater assurance that they will function as intended (by discriminating between highly knowledgeable applicants and less knowledgeable applicants). With the limited amount of testing time, each question on the examination should shoulder some of this discriminatory burden, and the pretesting process has gone a long way toward ensuring that result.

Re-editing

All multiple-choice questions that appear on NCBE exams are drafted by individual experts and then reviewed multiple times by committees of content experts. Beginning with the MPRE drafters in 2005, multiple-choice-question drafters have been adopting NCBE’s pre-editing process in which newly drafted questions are edited and proofed at NCBE, and then reviewed and approved again by the drafters before being reviewed for the first time by the full committees.

Through this process, which has received high praise from the drafters, questions are more focused at their first committee review, making the reviewing
process more efficient. From a program-wide perspective, questions have also become more consistent in style and format, making for a cleaner end product. Specifically, multiple-choice questions—which are divided into the fact pattern setting up the problem, or the “stem,” the question at the end of the stem, known as the “lead-in,” and the choices available to the examinee for answering the question, or the “options”—are regularly reframed in the pre-editing process to adhere to the following conventions (supported by psychometric research):
  • Questions should be concise.
  • Fact patterns and options should be reduced in length wherever practicable, without decreasing the difficulty or complexity of the legal issues being tested.
  • Questions should never be presented in the “K-type” format (using Roman numerals in complex options—e.g., I is true, but II and III are not true).
  • Common nouns should be used in lieu of proper nouns wherever practicable.
  • All of the facts necessary to answer the question should appear in the fact pattern.
  • The appearance of hypothetical facts in the options of multiple-choice questions (typically introduced by the words “if” or “unless”) has been significantly reduced since 2005.
  • The use of formatting that places different facts in each option, asking the applicant to choose the set of facts that meets a particular standard, has also declined.
  • Quotes should be avoided unless necessary to the legal issue being tested—e.g., quoting a statute may be required in an interpretation question, or quoting language might be necessary in a contract question.
  • Language in the options should be parallel.
  • To the extent possible, options should demand that the applicant compare and choose between similar elements—e.g., the four options might lay out four different causes of action, four different defense theories, etc.
  • No option should be the equivalent of “None of the above” or “All of the above.” Questions should be answered adequately by a single option without speculation.
  • Examination questions should cover a variety of tasks. Drafters are encouraged to write questions covering a range of skills and tasks—e.g., the ability to gather information, the ability to spot dispositive issues, the ability to synthesize the law with the facts, etc.
We believe that the pre-editing process has made the committee reviews more substantive and efficient for the experts participating, and more importantly, the pre-editing process produces a more integrated and professional examination.onors from Chicago-Kent College of Law.
Does anyone know when this goes into effect?