Update (32): The following was just published in this quarter's edition of Computer Law & Security Review (Volume 29, Issue 4 (2013), p. 446-449, Computer Based Testing, available at www.sciencedirect.com :
"Computer based testing - Bolls v. W. Scott Street, Sec'y of Va. Bd. of Bar Exmnrs., August 11, 2009, case no. 090915"
Jonathan Bolls and Stephen Castell PhD
The most significant change to law licensing in the United States is the recent reliance on computer-based testing for the bar exam in all of the fifty states. Virginia, for example, has been providing the computer-based test on a wide scale since 2005. New Jersey was the very first, in 2001. Since then, almost all of the states are now administering a computer-based test. The following comment piece is meant to inform the computer and technology law community how the use of testing software must necessarily be accompanied by additional procedural protections and corrective processes that may not have existed previously. Chief among these is the availability of discovery of the item in question, namely in this case the essays in a computer-based bar exam, which constitutes evidence that is essential in vindicating one's rights when a software failure occurs.
In July of 2008 I (Jonathan Bolls) took the essay portion of the Virginia Bar Exam on special testing software provided by the bar examiners. At the conclusion of the morning session, the software froze while in the process of saving the essays. There was a team of technicians on standby and the candidates were instructed to seek their assistance if anything went wrong with the software. At the conclusion of the test, when the proctor asked if there were any problems, a significant number of hands went up including my own, visibly overwhelming the full team of technicians on standby. It took a full 15 min before one could come around to assist me. He worked on my computer hands-on and, when that didn't work, he instructed me to reboot and try saving the answers again. So I did. But after the reboot I was not able to see any of the content of what supposedly was being saved. Then I had no choice but to transfer the file to a USB drive which was then handed in for grading. That afternoon, prior to the start of the second session, the candidates were informed by loudspeaker that over the lunch hour they discovered that 24 candidates had essays which were "misplaced" in the USB drives. These candidates did not know who they were and would receive further instructions in their afternoon test booklets on how to rearrange their essays properly. After the afternoon session, I again experienced the same software malfunction that happened in the morning requiring hands-on technical assistance, which ultimately failed, and had to reboot. Again, I was by far not the only one.
Months later I received my results and became aware of a major scoring discrepancy in my essay score. With a career position hanging in the balance, I immediately contacted the Board of Bar Examiners and requested that copies of my essays be sent to me. My request was denied at once by the secretary of the board simply because, as he stated, there is no review or appeal process available and the results are final. Furthermore, there exists an unwritten rule that no applicant can obtain their essays.
I began to do some research and it was not long before I discovered that software system malfunctions are happening throughout the different state bar exams of the country and, in fact, data has been found missing, misplaced, or otherwise altered in some way. Indeed, it has been proven that misgrading happened as a result. Most interestingly, perhaps, is the fact that these affected applicants had been previously reassured (just as I was) that their answers were intact. It was not until they received their essays as of right (which the vast majority of the states allow) where applicants became so outraged that they forced a corrective process not previously in existence.
With a career position at a district attorney's office on the line, I had no choice but to bring a mandamus action (judicial command order) in Fairfax, Virginia to force the state agency to release the essays. At the hearing, the local judge asked the attorney representing the state what process is available to aggrieved bar applicants, to which the answer was "that is an interesting question, Your Honor." This set the case on a constitutional detour that I would never have expected to be on so soon out of law school.
The case was taken under advisement, and while I was informed by a judicial law clerk that the judge was persuaded that the essays had to be released, it was ultimately determined that since the Board is an arm of the Virginia Supreme Court, only that court could issue such an order.
Before taking the case to the Virginia Supreme Court, I conducted some research of my own, and what I discovered was interesting and troubling. The year before (2007), applicants who sat for the New York bar exam complained of software glitches and system reboots in the process of saving the test essays, basically the same symptoms that I and numerous others in Virginia experienced. See New York Law Journal, available at http://www.abajournal.com/news/ny_bar_exam_marred_by_glitch/print/ One key difference between New York and Virginia, however, is that applicants in New York have a right to obtain their essays.
This right proved crucial for 47 applicants who discovered portions of their essays went missing, were duplicated, or otherwise altered in some way by the software system failure. Initially, it was the testing board's position that there was no cause for concern, that all the essays were received in their original and complete form. Even after the board released a press release about the 47 (New York Bd. of Law Exmnrs, available at (5th Paragraph Down) http://web.archive.org/web/20071123015445/www.nybarexam.org/press.htm ), we know of at least one applicant in particular (Eric Zeni) who demonstrated that the 47 was not all that were affected (New York Personal Injury Law Blog, available at http://www.newyorkpersonalinjuryattorneyblog.com/2008/02/i-passed-the-new-york-bar-exam.html ).
Eric Zeni, for example, demonstrated that two essays, not just one, were cut short by the software. He has told me privately that the board actually tried to place the blame on him, as if it was somehow his fault their software wasn't working properly. In the end, he was able to prove two of his essays were impacted by the software failure and should not have been graded because they were not reflective of his performance on the test. After a proper review, his failing grade was therefore changed to a passing grade. New York is therefore an example of how transparency upheld the integrity of the system- despite some level of inconvenience to the bar examiners themselves.
Since 2007, the National Conference of Bar Examiners has created a remedy in response to what happened in New York, which includes identifying the essay portions that were affected and then grading the balance of the examination. See New York Board of Law Examiners' press release, Par. 5, available at http://web.archive.org/web/20071123015445/www.nybarexam.org/press.htm
Prior to bringing the case to the Virginia Supreme Court, I called every state board of bar examiners' office and inquired whether applicants have rights to the essays following an exam. 42 of the 50 states said they do (with Colorado most recently changing its policy to allow this right, increasing the number to 43). Virginia stood as one of a small minority to operate a computer-based test with an unwritten rule that no applicant can obtain their essays, even in the event of a software system malfunction. The other states that are still doing this include Arkansas, Georgia, Hawaii, Missouri, South Carolina, South Dakota, all of which are currently administering computer-based tests.
One right that all applicants in each of the States have in common is they may present their grievance to their state supreme court, which has inherent authority over bar admissions. As in the case when any legal claim is brought, evidence is needed to show injury. In this case, only the release of the essays could do that. The Board merely cited to an unwritten rule instituting in the early 1970's that no applicant can obtain their essays.
I therefore had no choice but to bring a mandamus action, seeking a command order of the state supreme court to compel a public official to do his duty under an abuse of discretion claim. I have confirmed that in my case no guidelines for discretion were followed. No investigation of the software failure took place whatsoever. It simply came down to a blanket rule that no applicant can obtain their essays. With my sworn affidavit, Eric Zeni's testimony for what happened in New York just the year prior, the existence of a remedy developed by the National Conference of Bar Examiners, and the presence of a witness who can confirm my observations at the test site, the court still refused to hear the case or order the release of the test essays. That being the highest court in the state, I was left without recourse and without being able to have my day in court.
Because of the failure of Virginia's judicial system to provide any forum for handling software claims of bar examinees, I decided to bring a constitutional challenge in federal court to the Virginia state board's policy preventing applicants from obtaining their essays even following a system software failure. Such a rule runs directly afoul of the due process protections of the applicants, particularly given the recent changed circumstances of the computer-based test. The relief I was requesting was simply the release of test essays. No request for damages or monetary compensation was included in state or federal court as I was seeking merely to vindicate my own rights and fix the obvious problem for all future examinees.
First, I needed a computer forensics expert to look at this issue and provide the science behind the common sense as to why this rule is no longer appropriate in the environment of computer-based testing. I had the great fortune to come across Dr. Stephen Castell, an IT Consultant of the Year Professional Award Medalist of the British Computer Society (http://www.CastellConsulting.com). Over the course of his thirty-five year career, Dr. Castell has become a leading international expert in his field and has testified in the English High Court, the Sydney Commercial Court, and American Federal Courts, for both plaintiff and defense, and for many major clients worldwide. He has acted in both the longest (GEC-Marconi v LFCDA, 1991-1992) and largest (AirTours v EDS, 2001; total claims $250M pounds sterling) computer software development contract lawsuits heard in the English High Court. After hearing about this case, he was kind enough to submit a 20-page report pro bono in support of the case.
The federal court had all the evidence that the Supreme Court of Virginia had with the added benefit of Dr. Castell's report in which he was able to form certain preliminary conclusions and opinions based on what happened at the test site.
First, he was able to conclude that there was no way for a proper review to take place without including the applicant, the key creator of the data. That is because, as he states, the need for a reboot under these circumstances is "a very real cause for concern," and "the nature of computer-based testing systems is that there will inevitably be instances where cooperation with the applicant, the key creator of the critical data involved, is necessary, if not vital . . . [i]t is as important also to remember that Mr. Bolls was not the only one affected, and his case may prove helpful to improving the accuracy, integrity, and reliability of systems if, like Eric Zeni, corruption of the essay responses is discovered."
Secondly, he was able to conclude that the Board's offer to compare what was saved on the USB drive for hand-in to the graders with what was saved in encrypted format on my computer was just a sham remedy because the software system failure took place before the data was even transferred to the USB drive. Therefore, as he states, "[T]his can only merely confirm that the file was written by Mr. Bolls using the Exam4 software. In no way does this pose a remedy to, or even a sensible investigation of, any potential corruption problem caused by the Exam4 software itself, or through other (temporary or permanent) system fault, for which Mr. Bolls could not have been responsible."
Third, he affirmed the usefulness of the remedy as developed by the National Conference of Bar Examiners, which is entirely foreclosed to applicants in Virginia until there exists a means for obtaining the essays: "If, as in my view, there were in Virginia to be the reasonable, and technically sound, policy in place for applicants to obtain and inspect the essays, then, if such portions were thereby identified, it seems to me that there would then be an obvious and sensible remedy, viz, to employ the alternative grading methodology developed by the National Conference of Bar Examiners." It is important to note that no expert was brought forward by the Board of Bar Examiners; in fact, I had found two other experts who came to the same conclusions that Dr. Castell did.
The federal court nevertheless dismissed the case without a hearing or determination of the merits on the grounds that I was challenging the agency's rule as applied to my own exam and not as a rule applied to all. This actually wasn't true as I made it very clear in my complaint to the court that besides relief for myself (in the form of releasing the test essays) I was also challenging the rule on its face and sought in my relief to have the rule struck down for future applicants who will have to undergo a computer-based test ("That this Court take the necessary steps to protect the rights of all future applicants to the Virginia Bar by declaring Policy of Nondisclosure and Policy of Finality unconstitutional"). The appellate court affirmed without explanation.
While preparing for the next step in the case, I decided to take the bar exam in another jurisdiction, and was able to successfully pass the Washington, D.C. bar on my first attempt (handwritten not typed). I have taken a job as a magistrate judge in Fairfax, Virginia, a position that is appointed by the Supreme Court of Virginia.
It was becoming clear something was very wrong with how the judicial system in Virginia was approaching software disputes with the Board of Bar Examiners. Either they did not want to take the time to understand the science and hear the expert or they were simply hostile to the issue. There is no doubt that with the evidence presented, this case should never have been dismissed without a proper hearing.
Taking the cue from the last decision, I brought the case again in federal court, this time not asking for relief for me at all, but rather a declaratory judgment that the rule that no applicant can obtain their essays and the corresponding state law preventing Freedom of Information Act requests for test essays was unconstitutional on its face in light of the recent transition to computer-based testing in Virginia. But that judge too went out of his way to deny jurisdiction, this time extending just the opposite reasoning to deny a hearing on its merits. This time, the court said that I lacked standing. In other words, it was the court's opinion that I was not applying the case enough to my own injury and so could not bring it on behalf of all the others. It was a true Catch-22. Again, this was based on a false premise but nevertheless the only way for the court to ensure the evidence and witnesses were not heard in open court.
Because of the high importance of the integrity of bar admissions and the novel issues at play with technology, due process, and movement towards more transparent policies nationwide to protect the applicants, I brought a petition before the U.S. Supreme Court to correct the antequated policies of the 7 hold-out states, including Virginia, where applicants have no recourse when a software malfunction affects their test essays. Certiorari was not granted for that. Unfortunately, the Court can only hear a mere 2% of its cases, though it is widely known that is only a small fraction of the many cases that have actual merit.
So we see the Judiciary clearly has its limitations when it comes to software related disputes. I have no doubt that something went wrong with my essays, and the Board's having fought tooth and nail to keep me from seeing them, for years, says something in and of itself. To this day, the essays remain saved on an encrypted file held on my laptop, the key to which remains in the hands of the Board. Had my test been administered in Alaska, for example, the essays would have been sent to me upon request for a charge of $10; in Florida it would have been $50.
As Dr. Castell put it well, under these circumstances where a dispute presents itself, the essays themselves represent sine qua non evidence i.e. no chance of bringing a claim that could withstand a motion to dismiss for lack of evidence. A healthy judicial system must be willing to address technology and people's rights, even if it is not a high dollar case, because it goes to the core of individual liberty. Short of this, rights will be trampled by those who take advantage of the courts' limited understanding of technology and unwillingness to deal with the obvious implications for citizens' rights. This will inevitably lead to an unjust result for the individual, which in this case impacts not only the candidates but the public at large.
There are many benefits to technology. But as technology increases, transparency must increase in like measure even if it means additional procedural safeguards are put in place. That is the only way to protect the rights of the individual and our freedom in today's Information Age (*footnote).
*Footnote: Particularly when evidence derived from Information Technology may arguably be 'ontologically unreliable', as noted earlier in this Journal by Dr. Castell - see for example 'A computer of the simplest kind...', Letter, page 158,  10 CLSR, May - June 1994, and its appended Footnotes.