Monday, January 16, 2017

VIRGINIA BAR EXAM: LACK OF TRANSPARENCY

Update 36.  Last month I provided the deans of Virginia's law schools with the twenty-page expert report in relation to my case and all candidates who experience similar technical problems saving their bar exam essays using the Exam4 software.  Also included were my sworn statement as to what I experienced at the test site (and observations of others) and print-offs of the policies of neighboring state board of bar examiners who, unlike Virginia, recognize the due process rights of the applicants and allow access to the graded essays following the exam.  See, for example, Maryland's Rule 19-207, available at http://www.courts.state.md.us/ble/pdfs/baradmissionrules.pdf

Last year, when the Courts Committee held a hearing on this, the Board told them that it has decided to change its unwritten policy of not allowing anyone access to their essays to allowing a review of the essays after second and subsequent failures, but still not after the first exam.  This is unacceptable and poses no remedy to applicants like Eric Zeni in New York, who was told his essays were properly received only to discover that was in fact not the case once he actually obtained his essays as of right. Mr. Zeni also provided a well written letter to the Courts Committee in Richmond explaining how critical the right to obtain his essays in New York was to identifying and correcting a grading mistake as a result of a technical failure of the board's software (his symptoms were very similar to what we are experiencing in Virginia). 

With the help of friends, an analyst for a federal intelligence agency and a lawyer for the Federal Reserve, I have prepared the following letter presenting the issues to the deans of the law schools of Virginia.  It is time for Virginia to join the rest of the States in ensuring transparency in the system and guaranteeing the due process rights of the applicants.


December 3, 2016

Re:  Transparency of the Virginia Bar Exam

1.  Virginia does not allow an unsuccessful candidate to obtain their essay answers or review them until a second failure, even in situations where there is evidence of a system software malfunction or other technical failure in the grading.  This undercuts the spirit and purpose behind Virginia Code Section 54.1-3929, which requires the Board to preserve the essay answers for one year.

2.  This is fundamentally unfair, makes little sense, and is contrary to the recommendations of professional associations, the approach taken by 43 other states, and the analysis of outside experts.

3.  This policy seriously adversely impacted myself and others, and will continue to do so.  It reflects poorly on the Commonwealth, calls into question unnecessarily the credibility of the exam results, and can create the impression that Virginia law schools are lagging in successfully preparing their students to pass the bar exam.

4.  It is high time for a change in policy now that the bar exam has made the transition to being administered as a computer-based test.


Dear [Dean]:

As Justice Brandeis once said, "Sunlight is the best disinfectant."  I am writing to you and the other Virginia law schools in reference to a matter that directly affects the integrity of the bar admissions process in Virginia, a problem that directly impacts law school rankings and has gone largely ignored by the courts and the legislature after it was presented to them.  And that is the existence of an unwritten rule that no applicant can access their bar exam essays after their first bar exam.  This rule was applied to me after I experienced a system software malfunction and reboot on Exam4 software provided by the Board of Bar Examiners (see my sworn affidavit enclosed and, separately, article published in the Computer Law & Security Review by myself and a leading computer forensics expert).  As you will see when you read the article, these software problems are occurring nationwide and are directly tied to misgraded bar exams (fn.  for example, see this story of a New York bar applicant who experienced a software problem and how he was able to protect his interests by obtaining his essays as of right:  http://www.newyorkpersonalinjuryattorneyblog.com/2008/02/i-passed-the-new-york-bar-exam.html).  I have no doubt that my essays were impacted, as the essay score was nowhere near how I had actually performed.  Shortly after the exam I was able to reconstruct my essays by memory and was able to compare them to the approved solution.  The relatively recent national transition to computer-based testing is therefore a significant reason for a policy change at this time for the Board to allow applicants to obtain their essays upon request, especially if something went wrong at the test site (fn.  this would be an inexpensive proposition, as the other states are simply charging a nominal fee to cover the costs of making the copies.  Florida, for example, charges $50; North Carolina $20; New York $40).   43 of the states have made this transition already (see enclosed chart) given the inherent risks of computer-based testing, and Virginia would do well to follow suit in order to protect the due process rights of the applicant.

I brought a constitutional challenge to the policy of withholding the essays from review in federal district court in Richmond and filed a 20-page expert report in support (see report enclosed).  This report was filed pro bono by Dr. Stephen Castell, a leading computer forensics expert with over forty years of experience as an IT consultant in some of the highest profile cases in the world.  For example, in 2012 he led a team of three experts in  Australia's T-Card highway transportation system failure (ITSL & ERG v. PTTC, a $200M claim against the government for contract termination was ultimately settled on the eve of trial largely due to Dr. Castell's involvement and findings).  See http://www.smh.com.au/it-pro/government-it/tcard-battle-settled-out-of-court-20120217-1tegm.html.  He also testifies in high dollar patent cases in American federal courts for law firms such as Jones Day and Morrison & Foerster.  Virginia is certainly very fortunate to have him looking at this issue (cv attached).  Dr. Castell describes how my case was a "textbook case" of a system software failure and how there is no way to properly determine completeness checking without including the applicant in the process.  In Par. 20 of the expert report he states as follows:  "In no way does this pose a remedy to, or even a sensible investigation of, any potential corruption problem caused by the Exam4 software itself . . . for which Mr. Bolls could not have been responsible."  I have consulted two additional experts who have arrived at the same conclusion as Dr. Castell, which may be one reason why the Board has never brought forward an expert witness of its own.  In the end, the case was dismissed for lack of subject matter jurisdiction, but one good aspect that came out of it was this:  in their Answer, the Board admits to having made a uniform and unwritten rule that no applicants may have access to their essays and that rule was applied to the circumstances of my case.  I have enclosed a copy of my sworn affidavit on what took place at the test site for your review as well, which clearly affected many others as well as myself.  To this day, my essays remain cloaked in secrecy, encrypted on my laptop, and I have never seen them to verify completeness (fn.  it is also interesting to note that Virginia Code Section 54.1-3929 requires the Board to preserve an applicant's essays for one year.  The Board, however, has interpreted discretion accorded to it in another code section as license to make a uniform rule that no applicant can access their essays.  This, of course, is not a proper use of discretion). 

Meanwhile I took my MBE (multiple choice) score, which was high enough to waive into other jurisdictions, and waived that portion of the test into Washington, D.C.,  taking the remainder of the essay exam only.  I successfully passed in D.C. on my first attempt, in a test where over half of the applicants failed.  I then took the Maryland Bar Exam and passed on the first attempt as well, scoring highly.  I have been working as a full-time magistrate in Fairfax, VA.  I also served as president of the Washington Inn of Court in D.C. last year and I teach trial advocacy at a local prep school.  So fortunately things here worked out for me despite the actions of the Virginia Board of Bar Examiners (fn.  a member of the admissions committee of the Maryland Board of Law Examiners actually apologized to me for what the Virginia Board did and said they hope that I will have a better experience with the legal community in Maryland).  Nevertheless, I am cognizant of the fact that the facts of my case militate so strongly in favor of a change in policy at this time to allow applicants access to their essays in a computer-based test (which are preserved for a year by Va. Code Section 54.1-3929) that it would be unfair to future applicants for me to look the other way on this issue.  As previously stated, I was one of many applicants who experienced the software system malfunction and presumably all of them were treated the same way I was if they had later asked to see or obtain their essay answers.

To borrow the words of the computer forensics expert:

"Virginia, as I understand it, does not allow applicants to obtain their essays.  In my view and experience, this is an unusual way to design a system and immediately gives rise to concerns over validity and completeness checking, and difficulties in identifying and assessing any data corruptions or omissions potentially caused by system malfunction or otherwise.  Clearly, to apply such a policy, even to the circumstances of Mr. Bolls' case, is to overlook the possibilities of systemic error and to have no objective and evenhanded methodology of investigation in regard to ruling out such possibilities.  Mr. Bolls knows what he wrote and, with his input, it may be discovered that software malfunction was the cause of the discrepancy.  Without an allowance for applicants such as Mr. Bolls to obtain their essays, in my opinion the board may very well have a serious problem on its hands and not even know it."   Par. 30 of expert report.

The Board saw all of this information, including my sworn statement as well as the expert report; nevertheless, they still refused to release the test essays, even though they had the discretion to do so.  When asked by senior Circuit Court Judge Alden in Fairfax why not, they told the judge in open court that they used their discretion to make a blanket rule that no applicant can access their essays.  This is why when it was ultimately determined that the Board was exempt from the Administrative Process Act, in her dismissal the judge made it clear in her written opinion that she is preserving the issue of whether the Board abused its discretion in making a blanket rule that no one can review their essays.  As it stands, I was not able to view my essays following the system reboot at the test site and I have not been allowed to see them ever since.  Furthermore, due to my blog which tells this story (and has nearly 11,000 hits from the legal community), I have been informed by applicants as recent as last year that these software problem persist at the test site for many people.   Presumably, all of them are being offered the same sham remedy that I was (see below, an offer I did not accept as this would lead to a misleading result).  The Board's lack of cooperation is likely a result of not wanting to employ the National Conference of Bar Examiners' alternative grading methodology for when a software malfunction occurs.  See Par. 5 of http://web.archive.org/web/20071123015445/www.nybarexam.org/press.htm.  This would necessarily include all applicants affected, which they consider an inconvenience to their usual way of doing things prior to the advent of computer-based testing.

SHAM REMEDY OFFERED BY THE BOARD

The sham remedy being offered to applicants who experience a software system crash while the essays are being saved is as follows:  compare the encrypted file on their Exam4 software (saved on the applicant's computer) to what was graded.  But herein lies the problem:  the file saved on the Exam4 program on the applicant's computer itself is likely corrupted to begin with.  Therefore, what the graders graded was necessarily a duplicate of such potentially corrupted data.

As Dr. Castell states in his report:

"It is my opinion that the Board's offer to compare Mr. Bolls' encrypted file against the one graded does not in any way address the matter at hand.  That is to say, an exact match between the two does not inform the board or the Supreme Court of Virginia that there was no data corruption in this instance.  It merely confirms that the work is that of Mr. Bolls, without providing any illumination as to whether or not the work itself had been corrupted.  The only way that I can see this can be reasonably resolved is for there to be a policy in place for applicants to obtain their essays."
Par. 36 of the Expert Report

"Like Mr. Zeni, applicants such as Mr. Bolls would and in my view should then have the opportunity to identify places where portions of the essays went missing or were corrupted in some way.  If, as in my view, there were in Virginia to be the reasonable, and technically sound, policy in place for applicants to obtain and inspect their essays, then, if such portions were thereby identified, it seems to me that there would then be an obvious and sensible remedy, viz to employ the alternative grading methodology developed by the National Conference of Bar Examiners." 
Par. 38 of the Expert Report

What ends up happening is most applicants, generally unfamiliar with computer data storage methods, will take the Board up on their offer to do this comparison exercise, and be naively satisfied with it.  In reality, had they actually seen their essays, it is likely they would be seeing what the applicants in New York observed in 2007- missing or altered data that impacted the grading.  This is what Mr. Zeni experienced after facing a similar reboot of his software in the middle of the bar exam in New York.  See the enclosed letter from Mr. Zeni to the Courts Committee in Richmond.

Because of my experiences with the Board, I no longer have confidence in the integrity of bar admissions in Virginia, so rather than retaking the exam I have decided to go to other jurisdictions to become barred.  However, I have recorded every stage of my legal case as well as a hearing before the Courts Committee on my blog.  Included in the packet for your review are the proposal and other materials set before the committee.  Ultimately, the committee concluded that my case, while badly handled by the Board, was not enough to take action that would ensure applicants rights to review the essays.  However, it is important to remember that I was certainly not the only one affected (and many more continue to be affected), and our test results remain unsolved mysteries in that first exam sitting.

Since the litigation of this case, the Board has changed its policy to allow applicants to review their essays on the second or subsequent failure; however, the first bar exam is the most important.  This is the exam where many applicants (like myself) have jobs on the line; law school rankings are also impacted by the first exam results.  It is essential that the first bar exam be graded accurately and any software failures include a satisfactory review which according to my expert (and all other experts that I have consulted) necessarily includes the applicant in that process.  It is no longer proper for a computer-based test to have a rule that no applicant can review their essays afterwards, and it is high time for a change in policy.

The American Bar Association, the National Conference of Bar Examiners, and the American Association of American Law Schools have drafted a "Comprehensive Guide to Bar Admission Requirements 2016."  See http://ncbex.org/publications/bar-admissions-guide (page X).  In Section 25 it has this to say:

"25.  Rights of Failing Applicants . . . [a]n applicant who fails a bar examination should have the right, within a reasonable period of time after the announcement of the results of the examination, to see the applicant's answers to the essay questions and the grades assigned thereto, and to compare each of these answers with an approved answer." 

I have included in this packet the printed rules of surrounding jurisdictions Maryland, District of Columbia, North Carolina, and New York that adhere to this best practice.  These are but a small sample of the vast majority of the States that have the same transparency protections in place for the applicants (see chart, enclosed). 

If you are interested in this subject and in using your influence to help Virginia join the 43 other states who have already protected the due process rights of the applicants, I would be happy to discuss or answer any questions you have of me.  What needs to be done to fix this problem is what most of the other states area already doing- to allow applicants to obtain, not just review, their essays.  And this must be after every examination, not just second or subsequent failures (see legislative proposal, enclosed).  We live in a time where there is such a high degree of reliance on computer systems, which has many benefits, but we must ensure that our policies are appropriately tailored to the changed circumstances to ensure that rights are protected while the Board of Bar Examiners performs its functions.  Indeed, that was the clear intent of the General Assembly when they enacted Va. Code Section 54.1-3929, which requires the Board to preserve essay answers for one year.

Thank you for your attention to this matter and I look forward to hearing from you.


Regards,

Jonathan Bolls
Magistrate, 19th Judicial District
Fairfax, VA

Cc:  the law schools of Virginia