Moneyball for Law – Analyzing the big firms
The Brad Pitt film, Moneyball, portrays Billy Beane, baseball team manager of the Oakland A’s, struggling against accepted dogma in player selection. Beane’s insight was to use already available player data to pick athletes. This flew in the face of accepted conventional wisdom in the game. Talent scouts prided themselves on their “eye” for talent, choosing drafts by gut feel, pitching and batting style, even factors such as their demeanor off the field and the attractiveness of their wives and girl- friends. Despite a rocky start, strongly opposed by the A’s talent scouts, Beane put together a team that went down in baseball history. Derided, The Oakland A’s put together a roster of players believed to be lacking in talent, past their prime or unusable. Not only did the team prove itself by results, they were assembled with one of the lowest budgets in the sport. Win rates were not correlated with salary rates.
Lawyer selection is much like the bad old days of baseball player selection, con- tends Premonition L.L.C., a Florida based Artificial Intelligence company targeting the legal industry. Premonition claims to know which Attorneys usually win before which Judges. People pick attorneys based on recommendations from friends, online reviews, because they’re friends, friends of friends, went to a particular law school, have nice offices, work for a well known firm, saw an advertisement, their name was first in the phone book, etc. Many publications run “Top Lawyer” lists, people who are recognized by their peers as being “the best”. Premonition analyzed the Win Rates of these Attorneys and it turned out most were average. The only way that they stood out was a disproportionate number of appealed and re-opened cases, i.e. they were good at dragging out litigation. They discovered that even the law firms themselves were poor at picking Litigators. In a study of the United Kingdom Court of Appeals, it found a slight negative correlation of -0.1 between Win Rates and re-hiring rates, i.e. a Barrister 20% better than their peers was actually 2% less likely to be re-hired!
If generally accepted methods for choosing a Lawyer don’t work, what does? “Win Rate”, says Toby Unwin, inventor of the Premonition system. “The only item that affects the likely outcome of a case is the Attorney’s prior Win Rate, preferably for that case type before that Judge.” Wins and losses have traditionally not been tracked in law, perhaps being seen by the profession as a bit tawdry. “Attorneys themselves don’t know their own statistics, nor do the firms they work for. The only thing that gets tracked in law is hours and fees”, Unwin explains. Premonition was formed in March 2014 and expected to find a fertile market for their services amongst the big law firms. They found little appetite and much opposition. “In retrospect it was a stupid idea”, confides Premonition C.E.O and co-Founder, Guy Kurlandski. “It was like interviewing cows, asking them what they thought of McDonalds. You don’t ask the candlemakers what they think about lightbulbs.”
Despite being public record, court data is surprisingly inaccessible in bulk, nor is there a unified system to access it, outside of the Federal Courts. Clerks of Court refused Premonition requests for case data. Resolved to go about it the hard way, Unwin, who was suspended from school for computer hacking at the tender age of 13, wrote a web crawler to mine Courthouse web sites for the data, read it, then analyze it in a database. Unwin founded NetSearch, one of the most profitable Internet businesses in Europe in 1998, selling it a year later at a $160M valuation. Additional courts would be painstakingly re-coded by hand, each new source taking a week or more. “It was the wrong approach. We had thousands of courts to cover and it just wouldn’t scale. We scrapped it and I developed an Artificial Intelligence based system that could actually learn how to crawl a courthouse. Now each new one takes us an hour”, he says.
Win Rates
Having downloaded hundreds of thou- sands of cases, Unwin set about the painstaking task of teaching the Artificial Intelligence system to read the cases. Like a child, the system learned slowly at first, but rapidly gained in speed and accuracy. The system downloads new cases in real time, as they are filed. It analyzes them, then can create complex tables, mining overall Win Rates, differences from Judge to Judge looking for specialists in varying case types. Most of all, it looks for outliers, Litigators with long strings of unbroken wins before particular Judges. “Every Judge has their favorites”, confides Kurlandski with a twinkle in his eye.
Premonition in action
Kurlandski recounted with glee the story of a paralysis case on the verge of trial for a very nervous insurance client. Premonition downloaded 15,000 cases, three years’ worth of civil actions in the Reno, Nevada Courthouse. The system found an Attorney with 22 straight wins before the Judge – the next person down was seven. A bit of checking revealed the Lawyer was actually a criminal defence specialist who operated out of a strip mall and devoted most of his Web site to his “advocacy for the lesbian, gay, bi-sexual and trans- gender community”, the firm discovered. The Lawyer had little reason to be before a civil Judge, let alone so often and with such a stratospheric Win Rate. Premonition advised he be hired as co-Counsel. The case settled within weeks.
The firm claims such outliers are far from rare. Their web site, www.Premoni- tion-Analytics.com, shows an example of an Attorney with 32 straight wins before a Judge in Orange County, Florida. The Attorney, an Associate at a major national law firm, has a Win Rate outstripping any of the Senior Partners at his office.
Fees
“In law you don’t always get what you pay for”, Kurlandski claims. The firm found no correlation between Win Rate and billing rate. While there are undoubtedly highly paid top performers amongst the big firms, the majority of the most successful Counsel in each Court are found amongst the small firms and solo practitioners that Premonition refers to as “Strip mall super- stars.” “The legal marketplace is broken. It has significant opportunities for ‘Perception Reality Arbitrage’ and Premonition is the only company to know each Litigator’s true worth”, he continues. Law has long been seen as a Geffen Good, an economics term whereby the more a consumer pays for something, the better it is perceived to be.
Case duration
Win Rates aren’t the only metric Premonition tracks. “Many Attorneys are notorious clock runners. They’ve got away with it for a long time, because there’s been no transparency”, Unwin says. “Clients will often short-sightedly focus on an Attorney’s hourly rate. You need to look at your total spend: Hourly rate x length of case + losses.” Over a portfolio of cases, these savings can be significant. Premonition can fine-tune individual litigators cherry picked for each case, or pick a better panel for the firms not willing to take the plunge into selecting new litigators every time. The firm claims Attorney selection has a 30.70% effect on case success. “There’s the facts, the law and the people”, says Kurlandski.
Analyzing the big firms
Premonition recently completed a study of the major law firms. Targeting the Miami 11th Circuit Court, one of the busiest in the nation, they looked at data from 2010- October 2014, over 100,000 civil cases. Restricting the survey to just the 20 largest law firms by Attorney count in Florida they were able to extrapolate the first ever ranking of Big Law by their results. The insights were quite startling:
Firm choice matters
Firm success rates ranged from 93.51% for a specialist insurance defence firm to 0% for a firm reputed to be “legal surgeons”, they had only four cases in the system for that court during the period analyzed; only two had been concluded and they lost them both. A couple of Statewide firms posted respectable 64.67% and 64.38% Win Rates, but three National firms scraped in at 37.93%, 33.93% and 27.78%. Why do General Counsel hire White Shoe firms that lose three out of four times they go to court? “They don’t know. No one keeps score”, explains Kurlandski.
Individual Attorney choice matters more
Success rates are not consistent within firms; they vary from office to office. Unwin showed a Statewide firm that did a lot of litigation in Miami with a good 64.67% win rate but did relatively little in Orlando at a win rate that averaged 50%. Premonition showed the stats for the busiest big firm in Miami, with a 58.31% average Win Rate for that office. The bulk of the data came from just one Litigator who was responsible for 35.42% of their cases, handling 243 for that period. His Win Rate was 62.14%, and it strongly affected the average for the firm. There were two 100% Attorneys with four and five straight wins, but worryingly also one with 0% and 17 straight losses whose chart mark nestled next to a 3.45% Lawyer with 28 losses out of 29 appearances. “It’s often a lottery who you get assigned to in these firms”, opines Kurlandski. “If a General Counsel feels they have such a strong relationship with their existing firm, they could never leave, they should, at least, pick a results-based panel from within that firm, not just accept the first person offered, or insist blindly on a Senior Partner in the hope they are hiring quality.”
There are few Top 20 performers among the Big Firms
There is usually one representative from the top firms before each Judge, occasionally two. However many Judges have no Big Law top performers before them. It became very clear that the majority of top performers were from small firms or solo practitioners.
Should companies just hire from the small firms? “No. You see that just because there are more solo and small firm Attorneys than those employed at the big firms. Win Rate isn’t correlated with firm size”, Unwin says.
C.Y.A.
Why do General Counsel like Big Law firms so much? “It’s C.Y.A.”, Kurlandski believes. “There hasn’t been any reliable data before to track what you’re actually getting. So you hire a big firm and you hope. There’s nothing wrong with hiring a big firm, as long as you have the data to back up your Attorney choice within it. If the 33rd case is a loss after 32 wins in a row before that Judge, clearly you picked the very best person possible. Having data to back up your hires is the ultimate C.Y.A. At the end of the day, the numbers don’t lie and the Attorneys that win more, win more.” Is Premonition at the vanguard of a new era of accountability in law? “Every General Counsel in America will eventually be our client”, Unwin boasts, “The only question is when.” [IA]
