Getting More for Less...In Document Review

It is no surprise to anyone who has performed a  linear review of documents that there is a better way.  By better, we mean cheaper and more accurate.  However, moving clients from the comfort of knowing that a lawyer has looked at every document to accepting a process that only puts eyes on a portion of the documents is, at times, difficult.   Why?  Because they trust lawyers and are skeptical about trusting the computer’s “black box”.
 
A recent comparative case study “Iterative Legal Analysis & Sampling (“ILAS”) vs. Linear Document Review” by Anne Kershaw, Esq. at the Columbia University School of Professional Studies,  will help clients to see the light. Ms. Kershaw and her team answer the question "Can lawyers conducting iterative data analysis and sampling,  identify relevant documents more efficiently than legal teams undertaking linear review?” The answer is yes: by using software and highly trained legal and technical resources, document review is more efficient.
 
The study is interesting for a few reasons.  First, it is not a lawyer vs. machine evaluation. Rather, it is a comparison of skilled lawyers working with strong technical support and using litigation support software effectively vs. lawyers just going through documents in a linear fashion. The study shows that by combining  legal insight with an appropriate use of technology, review can be cheaper. Second, the study only evaluated the assessment of relevance - no issue coding or privilege identification was conducted. One might think that a linear review would be at least as accurate in a binary assessment, however, the linear review missed 11.95% of the relevant documents, whereas the iterative analysis review only missed 4.28%.  Pause and reflect on this - linear review missed almost 12% of relevant documents! Third, and to give humans some credit, the linear review had a much lower false positive rate - only 23 documents marked relevant were incorrectly coded. Contrast this to the 974 false positives coded by the iterative analysis team. From a risk management perspective, many clients will be much more comfortable with an error rate that includes too much, rather than misses too much. Further, these false positives can be purged in later steps in the disclosure process. Finally, the total number of hours invested in linear review was 98, compared with only 14 by the iterative analysis team. The differential more than justifies the higher hourly rate for the iterative analysis team.
 
Overall, this study provides a ringing endorsement for the use of multi-disciplinary teams with experience on the use of litigation support software.  And, rather than suggesting that computers will replace lawyers in document review, it provides a strong rationale for paying higher hourly rates for skilled and trained lawyers.  If your document review outsourcing decisions are based upon getting the lowest hourly rate, this study shows that you may not be getting good value.  With a skilled team using the right processes, the result will be both better and cheaper.