Probability: The Mathematics of Document Review and How to Dramatically Reduce Costs

by Andy Kraftsow

When is litigation like Las Vegas? The more cynical among us might answer “Always,” arguing that the outcome of any trial is a crapshoot. But for the rest of us, “always” is not the right answer. Most believe there is a relationship between the outcome of a trial and the hard work that went into preparing for it. Most believe that our actions influence the outcome. But there is an aspect of litigation that is almost exactly like Las Vegas, at least from a mathematical perspective, and that aspect is first pass document review.

Las Vegas is at once the land of probability and a haven for the hopeful denial of probability. I am playing blackjack, and I am winning. I have fourteen points in my hand, and the dealer is showing a ten. Do I hold or ask for another card? The odds tell me one thing, but my gut tells me another. Or maybe I am playing craps and wondering if I should double down since the person rolling the dice seems to be hot. Can we really beat the odds? What is the probability that the roulette wheel will make my number a winner? If I close my eyes and cross my toes, will that improve my chances?

All of these questions deal with the probable occurrence of random events, events upon which my actions have no impact. Surprisingly, the same mathematics that determines winners and losers in Vegas impacts first pass review. Consider an example: I have collected 1,000,000 documents, and I suspect that about 15% of them are relevant to the case issues. I want to read all of the relevant documents and none of the nonrelevant ones. Query: How many documents will I have to examine before I find all of the relevant ones? Or stated as an event threshold, how likely is it that the next document I look at will be responsive?

The answer to this question is of great economic significance because first pass review has become the most expensive part of e-discovery. We are drowning in a deluge of documents! It costs about 20 times as much to review a document using a contract attorney ($1) as it does to collect and process the document ($.05). So the question of how to avoid reading nonrelevant documents is very timely. In the example above, I would literally waste $850,000 if I were forced to read all of the nonrelevant documents in order to find all the relevant ones.

From a mathematical perspective, there are two attributes of first pass review that make it expensive. First, the relevant documents that I am seeking are relatively rare (15%) within the set collected. Second, I have no idea how they are distributed. These factors compound my retrieval problems – I do not know where in the collection to look and since the ones I want are rare, I may look for a long time without seeing any relevant documents at all. Anyone who has participated in a large document review has experienced these issues. A reviewer can work for hours without seeing a single relevant document, especially if the documents are ordered for review chronologically.

The mathematical concept that describes the occurrence of random events is called the geometric random variable, and it is at the heart of every Las Vegas triumph and disaster. It is also the factor that predicts how long it will take a reviewer to see the next relevant document. Understanding the geometric random variable is worthwhile because knowledge of its consequences will allow us to seriously reduce first pass review costs by 80% or more. The math is not hard, even for those who are not fond of math.

To get a sense of the geometric random variable, let’s consider an example:

How many times must you roll a single die before you are 95% certain of rolling a 2?

Examining the problem, one notes that there are six numbers on a die so the probability that any particular roll will show a 2 is one-in-six. But we have all had the experience of rolling the same number many times in a row; we know that can happen. If I want to be 95% certain that at least one roll is a 2, the geometric random variable tells me that I have to roll the die 17 times. The formula looks like this, where k is the number of rolls:

k > = ln(.05) / ln(5/6) = 16.431[1]

What stands out is that even though the probability of a 2 showing on any one roll is one-in-six, to be 95% certain that at least one roll is a 2, I need to roll the die 17 times. To be very certain takes a lot of rolls.

Now let’s think about document review. Suppose I have collected 1 million documents, and 15% of them are relevant. How many documents do I have to examine before I am 95% certain to see one relevant document? It is the dice problem translated into document review. Once again, the geometric random variable answers the question:

k > = ln(.05) / ln(850,000/1,000,000) = 18.433

I have to look at 19 documents to be 95% certain of finding the first relevant one. That is a lot of documents to go through just to find one that is relevant – especially since I have to find 149,999 more – but at least it is predictable. At least I know my worst case. The predictability comes from randomness. Rolling dice creates random outcomes.

The 19th-century French mathematician Siméon Denis Poisson discovered that using randomness is also the best way to find needles in a haystack; and what is document review if not looking for needles in a haystack? So we need to introduce randomness into the document review process. We do this by deploying a random “next document” algorithm. When the reviewer asks for the next document, the system picks one at random. It is that simple.

Random examination overcomes the “bunchiness” that one sees in a review organized by custodian or date. It makes sense to review known hot custodians and hot date ranges first, but once those areas are mined, switching to a random algorithm pinpoints where you are and how much more there is to be done.

The geometric random variable tells us that we should expect to see about 18 nonrelevant documents for every relevant one we come across. After a document is examined, it is removed from the review queue. Thus we are removing more nonrelevant documents from the set than relevant ones. In fact, we are removing them at a ratio of 18 to 1. As a result, the relevant documents are getting “denser” in the collection, which means that they will begin to appear more frequently: first once in every seventeen documents, then once in every sixteen and so on. If you run out the sequence, the geometric random variable predicts that about 90% of the collection (900,000 documents) must be read before one can be 95% certain of having found every relevant document. Reviewing 750,000 nonrelevant documents is better than reading 850,000 nonrelevant documents, but it is still an expensive and wasteful proposition. What else can be done to cut the costs?

One answer is to increase the percentage (density) of relevant documents in the collection by removing documents that cannot possibly be relevant. If we could double the density of the set so that at the beginning of review 30% of the documents were relevant, then reviewers would only have to read 45% of the documents to be 95% certain of finding all the relevant ones. That is a big improvement, and there are technologies available to accomplish just this task.

Increasing the density of relevant documents is the goal of predictive coding and related technologies. The idea is to use machine algorithms, or in the case of RenewData’s Language-Based Analytics human language comprehension, to identify and remove documents that are deemed not possibly relevant. Generally these techniques identify about 50% of the collection as not worth reviewing.

By reviewing randomly and using linguistic analysis to increase the density of the set, we have decreased the cost of first pass review by about 50%. Good but not great. In our example we are still wasting $265,000 and almost a thousand man-days reviewing nonrelevant documents. Can we do better?

The geometric random variable says “yes.” There is one more very simple strategy we can deploy, and it has perhaps the greatest impact of all on the cost of first pass review.

The strategy relies on leveraging the redundancy of language, specifically the redundancy of relevant language. Anyone who has participated in a document review knows that the language in a collection is terribly redundant. People say the same things to each other over and over again. Our experience is that every “string” of relevant language identified by a reviewer appears in at least 24 other documents.[2]

Bulk-tagging every document that contains the same identified relevant language string reduces the number of documents that must be read by 50%. That is an astounding result from introducing nothing more than a highlighter and a Boolean bulk-tagger. (RenewData has built a highlight-driven bulk-tagger that sits on top of Relativity). Grabbing every document that contains an identified relevant language string is definitely the way to go.

To summarize: The geometric random variable suggests three strategies to reduce first pass review costs:

1.    Use a random next document algorithm to overcome uneven distribution.
2.    Increase the density of the set by removing documents that could not possibly be relevant.
3.    Use a bulk-tagger to grab every document that contains the relevant language highlighted by the reviewer.

By combining all three strategies, one can avoid reviewing 80% or more of the collection and still be 95% confident of finding every relevant document.

About the Author: RenewData’s Chief Scientist Andy Kraftsow leads the company’s efforts to develop groundbreaking technologies. He joined the company from Digital Mandate via acquisition in 2009. While at Digital Mandate, he spearheaded the company’s efforts to build the Vestigate legal review solution, which remains an integral piece of RDC Analytics’ content analytics portfolio.

Over the past 25 years, Kraftsow has founded and been CEO of three software companies, each of which was ultimately sold to a public company. In 1999, he founded DolphinSearch Inc to bring the power of advanced neural networks mathematics to the legal profession. Trained as a mathematician, his specialty has been in the field of information analysis using various aspects of applied mathematics.

Responsibilities will include:

Familiarisation with Law firm IT architecture.
Liaise with 3rd party providers to ensure that Our Law Firm Client bespoke applications are installed correctly.
Supervise the transfer of accounting system, Linux-based intranet and website to a new managed platform. These applications need to communicate with one another and Exchange.
Manage the transfer of 130 users to a secure document back-up system.
Organise the IT and phone set up of a new 150 sqm office.
Prepare infrastructure for SharePoint and Dynamics CRM.
Oversee the IT support of all users.
Identify areas for performance improvement.

Skills and Experience required:

Experience of Microsoft environments featuring Exchange, SharePoint, Dynamics CRM, remote and mobile working including Blackberry, Apple Macs, VPN and Citrix.
Familiar with SQL databases, and Linux environments.
You as an individual will come from a Law Firm IT background or Professional Services and have g ood organisational skills with experience of managing projects.

This role is initially a 3 month contract position but may become a permanent role for the right individual if they are interested. The position is based in Central London although some travel maybe required as a large proportion of users are remotely based.

To apply contact Mark Lennard at Lewis Paige on 0207 871 9909 or email your CV to mark@lewispaige.com or visit www.lewispaige.com

George Katrivesis, of event organisers Chilli IQ, says the event will feature presentations on technology, human resources, marketing, leadership and law firm management. It is intended as a complementary event to the annual Managing Partners Forum which focuses on larger firms and remains a particularly valuable networking forum that is a proven source of important referral work for smaller firms.

Peter Magee, a partner in Armstrong Legal, said that he regularly attended the Managing Partners Forum, but generally for the networking opportunity rather than the content. While news about globalisation and international law firm mergers was interesting it was not of direct relevance for Armstrong Legal with its family law, criminal law and contested estate focus and staff of 70. He said that the HR, marketing, technology, or even business strategy issues challenging large firms were distinctly different to those that smaller firms were grappling with. “We are swimming in a pool rather than the ocean.”

The areas of interest for boutique and small law firms included issues such as:
·         Staff and partner recruitment, remuneration, retention;
·         Online marketing and social media;
·         Structures for equity, tax, risk management, growth and succession;
·         Practical applications of technology;
·         Practical ideas for business planning and budgeting; and
·         Effective staff supervision approaches to maintain quality control.

Speakers lined up for the conference include:
·         Tim Aspinall, managing partner, DMH Stallard (UK)
·         Peter Magee, managing partner, Armstrong Legal
·         Dan Simmonds, managing principal, Harwood Andrews & Sladen Legal
·         Henriette Rothschild, managing director, Hay Group and
·         Joel Barolsky, managing director, Barolsky Advisors
·         John Kain, managing partner, Kain C+C Lawyers
·         Michael Bradley, managing partner, Marque Lawyers

George Katrivesis said the event would offer delegates insights about how to effectively deploy technology, develop strategies to streamline operations, attract and retain key staff, and receive fresh insights about emerging business and leadership models. Delegates attending the event will also be invited to participate in a specially designed survey intended to provide fresh clarity about the challenges currently faced by Australia’s boutique and small law firms.

www.chilliiq.com.au

Experience Required:

Extensive proven and recent experience (at least 3+ years) in a similar background, preferably within the Professional Services Market. This needs to include a proven track record of annual sales achievement (against target)
Ability to demonstrate success and experience of managing major customer accounts – particularly resulting in sales development.
Excellent client facing skills.

Responsibilities:

Generate and maintain a credible forecast and pipeline for new business sales and
provide this information of a regular basis to the Senior Regional Sales Account Manager
Be able to present a good and current working knowledge of the company’s portfolio of products
Serve as the primary customer contact for the commercial relationship for those customer accounts assigned
Proactively cultivate relationships with all business unit heads of departments.
Set a positive and professional example to all colleagues, and foster positive communication with all departments.
Extensive experience of client relationships, along with the confidence and presence to deal with the world’s leading law firms.
Communicate effectively with clients to identify needs and evaluate alternative solutions.
Continually defines ways to increase customer satisfaction and deepen client relationships.

The Package:

This role is offering up to £45,000 base salary depending on experience plus and OTE of £35,000. The position is based out of the clients London offices.

To apply contact Mark Lennard at Lewis Paige on 0207 871 9909 or email your CV to mark@lewispaige.com

“As a CEO experienced in leading high-growth technology companies, Greg brings significant expertise to Microsystems and further strengthens the company’s ability to deliver market-leading solutions to document-intensive businesses,” said Tom O’Sullivan.

Greg previously oversaw rapid company growth as CEO of CVM Solutions and Capital H Group. He also held several leadership roles at Leo Burnett Worldwide, where he developed a rich foundation in business strategy, sales/marketing, corporate development and global operations. In his new role, Tom O’Sullivan will focus on organic growth opportunities, strategic M&A and other key initiatives relating to customer care.

According to Ken Hooten, Partner at Concentric Equity Partners, “Greg’s expertise in building business strategies, innovating new products and expanding into new markets will enhance the value we bring to our clients and lead to exciting growth for our business. His hiring is one of many pivotal actions and investments planned to drive the company’s long term growth plan.”

There was awaiting list for exhibition slots for this year’s event (which took place in April) and here on the Insider we’ve been fielding a steady stream enquiries over the past week from legal IT vendors who had previously not exhibited at LawTech but were now interested in becoming involved. You can find the full details here http://lawtechfutures.com/event-overview

As rekoop celebrates the third anniversary of the launch of its desktop offering back in 2010, the company says it is intent on marching onwards and upwards.

According to rekoop… for so long dominated by Carpe Diem, to the point where it effectively became synonymous with time recording, the market today looks very different: rekoop now has five of the top 15 international law firm brands, more than any other, and is the only vendor left in this space solely dedicated to time capture. It is also the only supplier based* in the UK, selling and supporting its UK and European clients directly.

For rekoop chief executive Phil Wedgwood, these are all factors that he hopes will propel rekoop to the same levels of synonymity as Carpe enjoyed in the 90s. “You can still talk to finance directors and managing partners today who will quote Carpe at you, such was its dominance in its day. But the time recording market is much changed, particularly in the light of the recent acquisition of DTE Axiom by IntApp. Of the 75% of leading firms who have purchased a dedicated time capture solution, it is rekoop that has the largest share. That translates into a strong geographical spread too, from global giant DLA Piper and UK top ten firm Norton Rose to the biggest firms in Ireland and Spain.

“For all our big hitters though, we mustn’t forget that this is an everyman solution that is used across the entire legal market. Our recent partnering agreement with Advanced Legal (previously IRIS) and our current list of mid-tier clients, including a growing base in both South Africa and Australia, should help propel us down the same path so successfully taken by the likes of BigHand – one that sees our software being used globally by firms of all sizes.

“As regards the technology, we see ourselves as the only vendor wholly focused on time capture, which means 100% of our R&D effort is going into make our solutions the most advanced, powerful, intuitive and agile as they can be. It’s this single-mindedness that has allowed us to catch-up and overtake our competitors in a relatively short timeframe.

“In this period we’ve developed a system that allows for both passive and active time entry; we’ve introduced seamless mobile solutions for both Blackberry and Apple devices; we’ve set the standard for native iPhone and iPad applications, fully leveraging the functionality of Apple iOS as opposed to just mimicking PMS web views; and we’re shortly set to launch the latest version of our software which, for the first time anywhere, will automatically complete a fee earner’s timesheet without any user input required.

“It’s the speed and quality of our technical development, coupled with our proximity and responsiveness to our users, that will hopefully see rekoop take the place of Carpe as the perceived default choice for time capture. But there’s still plenty of work for us to do to ensure that law firms are maximising their efficiency and in the best position to rekoop more time.”

Comment: Strictly speaking Carpe Diem is now a UK-based product thanks to its acquisition by Tikit.

The Insider Guide to Probate, Wills & Trust Accounts Systems Suppliers is available as a FREE two-part download now. This is a new and revised edition of the guide that has been appearing online and in the Probate Practitioner’s Handbook for the past 15 years and comprises a brief inductory PDF + an XLSX spreadsheet containing product and contact information on a total of 24 products from 22 suppliers. The spreadsheet covers case management, estate accounts, will drafting & assembly, and IHT forms completion systems, including software to handle Powers of Attorney and Court of Protection work, while the PDF includes articles and buying and selecting probate systems from Pauline Freegard of Ochresoft Technologies and Gregory van Dyk Watson from Isokon Limited. The spreadsheet will be updated on an ongoing basis.

http://cloudbox.legaltechnology.com/ProbateGuide.pdf

http://cloudbox.legaltechnology.com/NewProbateGuide.xlsx