The situation
A law firm in a Supreme Court lending dispute had issued more than twenty subpoenas: to banks, an accountant, real estate agents and others. The returns came to about 2,000 files. Several hundred were emails with documents attached, and a few hundred were scanned pages with no text layer. A litigation support provider was assembling the court book, still in draft, with about 1,000 tabs. The book was due with the judge in a week. The firm needed an independent answer to one question: what had been produced under subpoena that was not in the book. This is the work we now offer as the court book completeness check.
All NDF case studies are anonymised and may combine several matters with identifying details changed. The methods and outcomes are accurate.
Why the files could not simply be compared
Every document in the court book had been re-made during assembly, with tab stamps and new page numbers, so no file in the book matched any produced file. Comparing files found 0 of about 1,000 tabs; comparing page text found 26. Filenames were no help either, because both sets number their files from one, and 129 of the produced statements were duplicate copies. The two sides could only be matched on what is printed on each document: for a bank statement, the bank, the account number and the statement period.
How we did it
Software did most of the work. Scripts opened every file, including attachments inside emails and scans run through OCR. Specialist recognition AI, built by us and run in an isolated environment, read each page and identified what the document is and the details printed on it. Scripts then matched the two sides on those details and built the registers, so every match can be traced to what is printed on the page. The examiner wrote the rules, reviewed every case the software could not decide, and signed the report. Every figure in the pack comes from a saved script that anyone can run again, and the report says on page one what the AI did, what the scripts did and what the examiner did. All electronic documents were stored securely on servers in Australia, from receipt to delivery.
How we know the error rate
Repeatable is not the same as right, so the result was checked by a person reading pages. Every row a finding rested on was read in full, with the analysis hidden. The rest of the pack was sampled at random, with the sample size and the pass mark fixed before the draw and the seed written down so anyone can redraw it. A second person read every sampled page by hand and wrote down what was printed; the answer key was opened last.
On this matter the sample found no error in any account or period read. In an appropriate number of samples it found three disagreements on whether a page was a type of financial document relevant to the case. The report states that rate. It does not claim there were no errors. Our standard now is fewer than 1 in 100 rows wrong, at 95% confidence, with the sample sized so a clean result supports that claim.
The outcome
The firm went into the settling of the next court book version with a register it could hand across: every produced statement by account and period, whether the book held it, and the unread and locked items that bounded the answer. Every row is keyed to the document rather than the tab, so the pack can be refreshed against the final book in 2 business days.
Lessons for similar matters
- A court book is a re-made copy of the production, so comparing files will report almost everything as missing.
- Productions carry duplicate copies, so count statement periods, not files.
- Locked attachments whose passwords went by SMS to a party can only be opened through the solicitor, so ask for them on day one.
- Check the book against its own index first. An index typed from covering pages carries disagreements a script will find in minutes.
- State the measured error rate in the report, so it cannot be raised later as a surprise.