The first-phase results of the FineBooks project show that open optical character recognition (OCR) models have become capable, in most use cases, of extracting text from historical books with sufficient accuracy to make reprocessing massive collections of public-domain books practical. At the same time, however, the results reveal important limitations involving the preservation of historical textual features, the compatibility of outputs with library systems, and the handling of document types and scripts not included in the evaluation.
The project is a collaboration between Hugging Face and EleutherAI, with two goals: to test whether current open OCR models are accurate, fast, and inexpensive enough to process historical books at scale, and then to reprocess collections of public-domain books and publish the improved texts as open datasets. In its first phase, the project selected the Biodiversity Heritage Library (BHL), a global collection containing more than 300,000 historical natural-history documents, comprising more than 64 million pages of scientific knowledge.
An Evaluation Based on 2,165 Verified Pages
FineBooks released the FineBooks BHL OCR Leaderboard, along with the finebooks/bhl-impact-gt reference dataset and the finebooks/bhl-ocr-eval evaluation tool. The leaderboard compares 14 open-weight OCR models licensed under licenses that permit reuse, allowing them to run on private hardware without API keys or per-page fees.
The evaluation was based on expert-corrected transcriptions produced within the IMPACT and BHL-Europe projects between 2011 and 2012. The material includes 2,165 pages from six volumes in English, French, German, and Latin, with an accuracy level of approximately one error per 2,000 characters. These transcriptions were rebuilt into a modern dataset linking each scanned image to its corresponding text, taking advantage of the availability of the entire BHL collection for bulk download through AWS Open Data.
What Does the Leaderboard Measure?
The primary metric is the character error rate (CER), which calculates substitutions, deletions, and insertions compared with the correct text. For ease of presentation, the rate was converted into an accuracy percentage: when the CER is 0.024, this means that 97.6% of the characters were recognized accurately.
The leaderboard does not rely on this metric alone. It also displays two versions of CER, one for reading and another diplomatic version that preserves historical distinctions such as the long s character “ſ”. It also measures recall of words actually present on the page, the proportion of extra text that does not appear on the page, and the repetition rate that occurs when a model continues reproducing text until it reaches the token limit. Pages that enter a repetition loop are excluded from some other results, so the repetition rate should be read alongside the accuracy scores.
Accurate Models with Low Operating Costs
According to the published sample of results, the dots.mocr model ranked first with reading accuracy of 97.6%, followed by OvisOCR2 at 96.9%, and then PaddleOCR-VL-1.6 at 96.1%. olmOCR-2 scored 95.7%, LightOnOCR-2 95.1%, and Qwen3.5-9B 94.9%, while DeepSeek-OCR achieved 93.8%.
- Processing 1,000 pages using Hugging Face Jobs cost approximately $1.94 for the dots.mocr model.
- The cost was $0.46 for OvisOCR2, $0.34 for PaddleOCR-VL-1.6, and $0.45 for olmOCR-2.
- Qwen3.5-9B recorded a cost of $0.89 per 1,000 pages.
Hugging Face Jobs runs the task through a single command that starts a graphics processing unit, runs the model over the dataset, and then shuts down the environment. Each operation also pins the model revision, container image, and script version, enabling results to be reproduced and new models to be added to the leaderboard with a single task.
Are the Models Sufficient for Real-World Use?
For datasets used to train language models, the project’s authors believe the answer is generally yes. Public-domain texts represent a large training source, but their value depends on their quality. The article notes that the Talkie project found that a language model trained on OCR-extracted texts learned with 30% of the efficiency of a model trained on human transcriptions of the same books. Common Pile, launched by EleutherAI and its collaborators, also includes approximately 300,000 public-domain books whose text was extracted through older OCR processing pipelines, making reprocessing with better models a significant improvement for open training datasets.
Replacing legacy OCR pipelines in libraries, however, depends on the required technical infrastructure. Traditional systems usually produce ALTO XML files that provide word-level coordinates, whereas newer models produce Markdown or plain text and sometimes coordinates for text regions, but not word coordinates. Libraries therefore cannot necessarily insert the new text directly into the existing infrastructure.
The models also do not yet appear sufficient for precise academic transcription. The main problem is not always that the page is misread, but that the model silently modernizes the long s or typographic ligatures and other historical features. The article’s authors believe that specialized fine-tuning may address these limitations.
Evaluation Limitations and Next Steps
The results cover only four languages and six volumes written in scripts from the antiqua family. They therefore cannot be generalized to German Fraktur, non-Latin writing systems, non-European languages, or handwritten manuscripts. The project also deliberately focuses on books, which are visually more regular than newspapers, magazines, and archival materials, and the leaderboard does not test multi-column pages or complex elements.
FineBooks plans to continue adding new models to the leaderboard and then reprocess the entire public BHL collection. Of approximately 300,000 items in the collection, nearly 200,000 carry data indicating that they are in the public domain. The project will use one of the advanced models to process these materials and publish the resulting text as the first FineBooks dataset, while openly sharing the datasets, processing pipelines, and supporting models that will be developed.