Is Your Culling Method Reasonable?

A study conducted as part of the 2009 Legal TREC found that conducting enterprise wide automated content analysis returned significantly more relevant documents than employing traditional culling-based methodologies.

To conduct its research, ZL Technologies had two teams – one using content analysis and the other employing manual culling, search through nearly 3 million emails from more than 100 mailboxes of Enron employees.

The results found that culling by first selecting custodians and then applying keyword filters led to incomplete information upon which to base their important legal decisions. The team failed to find 84% of relevant documents and only found four employees storing relevant information. The content analysis team, however, successfully found all relevant documents and identified all 77 relevant employees.

Of course, the above results are based in part on the skill, or lack thereof, of the people who made up the culling group in developing good search terms. This was highlighted recently in Victor Stanley, Inc. v. Creative Pipe Inc. In his May 29 opinion, U.S. Chief Magistrate Judge Paul Grimm faulted the defendant’s keyword search methodology. “[W]hile it is universally acknowledged that keyword searches are useful tools for search and retrieval of [electronically stored information], all keyword searches are not created equal; and there is a growing body of literature that highlights the risks associated with conducting an unreliable or inadequate keyword search or relying exclusively on such searches for privilege review”.

While a less than 100% accurate production is obviously a concern, the test should not be perfection. Instead, the litigator should be confident that the method used in deciding how to cull documents is reasonable. While that means that some records could potentially fall through the cracks, holding one to a standard of perfection is contrary to the concept of proportionality.

​Essentially, if you spend too much time being nervous about having a perfect collection and culling process, your results will not be cost-effective or practical. The process must be balanced – err on the side of caution, but with a reasonable approach.