According to a recent LinkedIn post from Reducto, the company is introducing Pairwise AI Ranking (PAIR), an open-source framework for evaluating document parsers using organizations’ own data. The post suggests the tool is designed to identify errors that alter meaning, rather than penalize benign formatting differences.
PAIR uses an LLM judge to compare two parser outputs against the original document, while masking parser identities and varying output order to limit bias. Reducto’s reported testing across 300 comparisons found 84% agreement with human reviewers when both selected a winner, increasing to 91.6% when human judgments were unanimous.
The framework’s open-source availability could help businesses assess parsing tools against the documents used in their workflows without building separate grading rules for each format. For Reducto, the initiative may strengthen its position in document-processing infrastructure by promoting a measurement approach tied to information fidelity, although the reported results do not by themselves establish commercial adoption or financial impact.