Selected Court Citations
Moore v. Publicis Groupe & MSL Group, Dist. Court, SD New York 2012. Opinion and order of Andrew J. Peck, Magistrate Judge:
The goal is for the review method to result in higher recall and higher precision than another review method, at a cost proportionate to the "value" of the case. See, e.g., Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Rich. J.L.& Tech., Spring 2011, at 8-9, available at http://jolt.richmond.edu/v17i3/article11.pdf.
National Day Laborer Organizing Network v. US Immigration and Customs Enforcement Agency, 877 F. Supp. 2d 87 - Dist. Court, SD New York 2012. Opinion and Order of Shira A. Scheindlin, District Judge:
Given the tedious nature of the assignment of examining every single document and the difficult and subjective nature of deciding what is and is not responsive, it would have been wise of the FBI to run a few verification tests using sophisticated search techniques to ensure that the manual review was actually capturing the universe of responsive documents. Such tests would have given the Court significantly more confidence regarding the adequacy of these manual reviews. See Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech. 11 (2011). See generally the Text Retrieval Conference (TREC) Legal Track at http://trec-legal.umiacs.umd.edu/.
In the Matter of the Search of Information Associated with the Facebook Account Identfied by the Username AARON.ALEXIS that is Stored at Premises Controlled by Facebook, Inc. Memorandum Opinion of John M. Facciola, United States Magistrate Judge:
For an explanation of the technology and a glossary of the terms used in it, see Maura R. Grossman & Gordon V. Cormack, The Grossman-Cormack Glossary of Technology-Assisted-Review, with a Foreword by John M. Facciola, U.S. Magistrate Judge, 2013 Fed. Cts. L.Rev. 7 (2013).
Progressive Casualty Insurance Company v. Delaney, Dist. Court, D. Nevada 2014. Order (Motion to Compel, Docket #89) of Peggy A Leen, Magistrate Judge:
Predictive coding has emerged as a far more accurate means of producing responsive ESI in discovery. Studies show it is far more accurate than human review or keyword searches which have their own limitations. See, e.g., Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can be More Effective and More Efficient Than Exhaustive Manual Review, Richmond J. L. & Tech. Vo. XVII, No. 3, Article 11 (2011)
Federal Housing Finance Agency v. HSBC North America Holdings Inc., Dist. Court, SD New York 2014. Memorandum Opinion and Order of Denise Cote, District Judge:
The literature that the Court reviewed at that time indicated that predictive coding had a better track record in the production of responsive documents than human review, but that both processes fell well short of identifying for production all of the documents the parties in litigation might wish to see. See Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech. 1, 37 (2011), attached as Ex. C to July 20, 2012 Letter of JPMorgan Chase, 11 Civ. 6188, Dkt. No. 120 at 49 (reporting that manual reviewers identified between 25% and 80% of relevant documents, while technology-assisted review returned between 67% and 86%).
Rio Tinto PLC v. Vale SA, Dist. Court, SD New York 2015. Opinion and order of Andrew J. Peck, Magistrate Judge:
If the TAR methodology uses "continuous active learning" (CAL) (as opposed to simple passive learning (SPL) or simple active learning (SAL)), the contents of the seed set is much less significant. See generally Gordon V. Cormack & Maura R. Grossman, Evaluation of Machine Learning Protocols for Technology-Assisted Review in Electronic Discovery, in Proceedings of the 37th Int'l ACM SIGIR Conf. on Research & Dev. in Info. Retrieval (SIGIR '14), at 153-62 (ACM New York, N.Y. 2014), http://dx.doi.org/10.1145/2600428.2609601; Maura R. Grossman & Gordon V. Cormack, Comments On "The Implications of Rule 26(g) on the Use of Technology-Assisted Review," 7 Fed. Cts. L. Rev. 285, 298 (2014) ("Disclosure of the seed or training set offers false comfort to the requesting party. . .").
Irish Bank Resolution Corporation Limited v. Sean Quinn, High Court, Ireland. Judgment of Mr. Justice Fullam:
Dr. Khadvezic cited a study by Maura R Grossman and Gordon V Cormack entitled "Technology-Assisted Review in E-Discovery
Can Be More Effective and More Efficient Than Exhaustive Manual Review" (Richmond Journal of Law and Technology). As the title
indicates the study concluded that technology assisted processes can yield superior results as measured by recall, precision as well as
the f-measure. The Grossman/Cormack study is cited in many of the US cases dealing with the issue.
Pyrrho Investments Ltd v. MWB Property Ltd, High Court of Justice Chancery Division, U.K. Approved Judgment of Master Matthews:
[quoting Irish Bank Resolution v. Quinn] "66. The evidence establishes, that in discovery of large data sets, technology
assisted review using predictive coding is at least as accurate as, and, probably
more accurate than, the manual or linear method in identifying relevant
documents. Furthermore, the plaintiff's expert, Mr. Crowley exhibits a number
of studies which have examined the effectiveness of a purely manual review of
documents compared to using TAR and predictive coding. One such study, by
Grossman and Cormack, highlighted that manual review results in less
relevant documents being identified. The level of recall in this study was
found to range between 20% and 83%. A further study, as part of the 2009
Text Retrieval Conference, found the average recall and precision to be 59.3%
and 31.7% respectively using manual review, compared to 76.7% and 84.7%
when using TAR. What is clear, and accepted by Mr. Crowley, is that no
method of identification is guaranteed to return all relevant documents. [ . . . ]"
McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd & Ors (No 1)  VSC 734 (2 December 2016) [quoting Irish Bank Resolution v. Quinn].