Maura R. Grossman and Gordon V. Cormack

Selected Court Citations

The goal is for the review method to result in higher recall and higher precision than another review method, at a cost proportionate to the "value" of the case. See, e.g., Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Rich. J.L.& Tech., Spring 2011, at 8-9, available at http://jolt.richmond.edu/v17i3/article11.pdf.

Given the tedious nature of the assignment of examining every single document and the difficult and subjective nature of deciding what is and is not responsive, it would have been wise of the FBI to run a few verification tests using sophisticated search techniques to ensure that the manual review was actually capturing the universe of responsive documents. Such tests would have given the Court significantly more confidence regarding the adequacy of these manual reviews. See Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech. 11 (2011). See generally the Text Retrieval Conference (TREC) Legal Track at http://trec-legal.umiacs.umd.edu/.

For an explanation of the technology and a glossary of the terms used in it, see Maura R. Grossman & Gordon V. Cormack, The Grossman-Cormack Glossary of Technology-Assisted-Review, with a Foreword by John M. Facciola, U.S. Magistrate Judge, 2013 Fed. Cts. L.Rev. 7 (2013).

Predictive coding has emerged as a far more accurate means of producing responsive ESI in discovery. Studies show it is far more accurate than human review or keyword searches which have their own limitations. See, e.g., Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can be More Effective and More Efficient Than Exhaustive Manual Review, Richmond J. L. & Tech. Vo. XVII, No. 3, Article 11 (2011)

The literature that the Court reviewed at that time indicated that predictive coding had a better track record in the production of responsive documents than human review, but that both processes fell well short of identifying for production all of the documents the parties in litigation might wish to see. See Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech. 1, 37 (2011), attached as Ex. C to July 20, 2012 Letter of JPMorgan Chase, 11 Civ. 6188, Dkt. No. 120 at 49 (reporting that manual reviewers identified between 25% and 80% of relevant documents, while technology-assisted review returned between 67% and 86%).

If the TAR methodology uses "continuous active learning" (CAL) (as opposed to simple passive learning (SPL) or simple active learning (SAL)), the contents of the seed set is much less significant. See generally Gordon V. Cormack & Maura R. Grossman, Evaluation of Machine Learning Protocols for Technology-Assisted Review in Electronic Discovery, in Proceedings of the 37th Int'l ACM SIGIR Conf. on Research & Dev. in Info. Retrieval (SIGIR '14), at 153-62 (ACM New York, N.Y. 2014), http://dx.doi.org/10.1145/2600428.2609601; Maura R. Grossman & Gordon V. Cormack, Comments On "The Implications of Rule 26(g) on the Use of Technology-Assisted Review," 7 Fed. Cts. L. Rev. 285, 298 (2014) ("Disclosure of the seed or training set offers false comfort to the requesting party. . .").

Dr. Khadvezic cited a study by Maura R Grossman and Gordon V Cormack entitled "Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review" (Richmond Journal of Law and Technology). As the title indicates the study concluded that technology assisted processes can yield superior results as measured by recall, precision as well as the f-measure. The Grossman/Cormack study is cited in many of the US cases dealing with the issue.

[quoting Irish Bank Resolution v. Quinn] "66. The evidence establishes, that in discovery of large data sets, technology assisted review using predictive coding is at least as accurate as, and, probably more accurate than, the manual or linear method in identifying relevant documents. Furthermore, the plaintiff's expert, Mr. Crowley exhibits a number of studies which have examined the effectiveness of a purely manual review of documents compared to using TAR and predictive coding. One such study, by Grossman and Cormack, highlighted that manual review results in less relevant documents being identified. The level of recall in this study was found to range between 20% and 83%. A further study, as part of the 2009 Text Retrieval Conference, found the average recall and precision to be 59.3% and 31.7% respectively using manual review, compared to 76.7% and 84.7% when using TAR. What is clear, and accepted by Mr. Crowley, is that no method of identification is guaranteed to return all relevant documents. [ . . . ]"