Skip to content

awesome_ocr.md

Latest commit

 

History

History
121 lines (118 loc) · 6.83 KB

awesome_ocr.md

File metadata and controls

121 lines (118 loc) · 6.83 KB
Raw

VLMs

  • Wed, 18 Sep 2024 Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

  • Fri, 27 Sep 2024 MinerU: An Open-Source Solution for Precise Document Content Extraction

    • opendatalab/MinerU2.0-2505-0.9B
    • Built on Qwen2 + siglip-so400m-patch14-384

  • Wed, 19 Feb 2025 Qwen2.5-VL Technical Report

  • Tue, 25 Feb 2025 olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models

    • Built on Qwen2-VL-7B-Instruct
    • We use single node with 8 x NVIDIA H100 (80GB) GPUs. A single training run took 16 node hours, with all training experiments totaling 365 node hours

  • Tue, 20 May 2025 Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting

    • Swin Transformer + mBart

  • Sun, 1 Jun 2025 Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing

    • Built on Qwen/Qwen2.5-VL-8B-Instruct
    • optimized via Group Relative Policy Optimization (GRPO)
      • Edit Distance Reward (Rdist)
      • Count Reward (Rcount)
      • Order Reward (Rorder)
    • We fine-tune the Qwen2.5-VL-7B model using GRPO within a distributed training setup based on Verl [45, 66], utilizing 8 A100 GPUs (80GB).

  • Thu, 5 Jun 2025 MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm

    • Structure Detection, Content Recognition, Relation Prediction
    • Built on Qwen/Qwen2.5-VL-3B-Instruct
    • Our 3B model was trained for 53 hours on 32 A800 GPUs.

  • Thu, 12 Jun 2025 OCRFlux-3B

    • Built on Qwen/Qwen2.5-VL-3B-Instruct

  • Wed, 30 Jul 2025 dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model

    • Built on Qwen/Qwen2.5-VL-3B-Instruct

  • Fri, 1 Aug 2025 DocTron-Formula: Generalized Formula Recognition in Complex and Structured Scenarios

    • Built on Qwen/Qwen2.5-VL-8B-Instruct

  • Mon, 18 Aug 2025 DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model

    • Built on Qwen/Qwen2.5-VL-7B-Instruct
    • on a single node with 8 NVIDIA A100 GPUs
    • four-stage process
      • think: it starts with its own OCR read
      • tool : then consults specialized external tools for a second opinion
      • rethink : re-examines the image with that extra context
      • answer : finally delivers its response

  • Fri, 26 Sep 2025 MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing:

    • Language Model. For the decoder, we employ a 0.5B-parameter Qwen2-Instruct model [42], as document parsing tasks typically exhibit relatively low dependency on large-scale language models. To better accommodate diverse resolutions and aspect ratios in cropped image parsing, we replace the original 1D-RoPE [39] with M-RoPE [48], thus enhancing the model’s generalization ability across varying resolutions.
    • Vision Encoder. Inspired by Qwen2-VL, MinerU2.5 incorporates a native-resolution encoding mechanism. Although the Qwen2.5-VL series [3] adopts window attention to improve efficiency, this design causes performance degradation in document parsing tasks. Therefore, we employ a 675M-parameter NaViT [10] initialized from Qwen2-VL. This vision encoder supports dynamic image resolutions and employs 2D-RoPE for positional encoding, enabling it to flexibly handle inputs of various resolutions and aspect ratios.
    • Patch Merger. To balance efficiency and performance, the architecture uses pixel-unshuffle [38] on adjacent 2 × 2 vision tokens, preprocessing the aggregated vision tokens before passing them into the large language model. This design effectively achieves a trade-off between computational efficiency and task performance.
    • Two-Stage Parsing Strategy
      • Stage I: Layout Analysis. 1036 × 1036 pixels
      • Stage II: Content Recognition. Cropped regions are fed at native resolution with an upper bound of 2048 × 28 × 28 pixels
    • Training Recipe
      • Stage 0-Modality Alignment
        • Language-Image Alignment. Image Caption 2048 × 28 × 28
        • Visual Instruction Tuning. VQA 4096 × 28 × 28
      • Stage 1-Document Parsing Pre-training
        • Layout&OCR 2048 × 28 × 28
      • Stage 2-Document Parsing Fine-tuning
        • Layout&OCR 2048 × 28 × 28

  • Thu, 16 Oct 2025 PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model

    • 两阶段 Layout Analysis + PaddleOCR-VL-0.9B
      • PP-DocLayoutV2 RT-DETR-based
      • PaddleOCR-VL-0.9B
        • Architecture
          • txt: ERNIE-4.5-0.3B
          • img: we employed a NaViT-style [15] encoder initialized from Keye-VL’s
          • projector: 2-layer MLP projector
    • Training Recipe
      • Layout Analysis
      • Element-level Recognition
        • Stage 1 pre-training
        • Stage 2 instruction fine-tuning
          • OCR
          • Table Recognition
          • Formula Recognition
          • Chart Recognition
    • Dataset
    • Evaluation
      • OmniDocBench v1.5
      • OmniDocBench v1.0
      • olmOCR-Bench

  • Tue, 21 Oct 2025 DeepSeek-OCR: Contexts Optical Compression

    • Architecture
      • txt: DeepSeek3B-MoE-A570M
      • img: 80M SAM-base + 300M CLIP-large
      • projector:
        • we borrow from Vary [36] and use a 2-layer convolutional module to perform 16× downsampling of vision tokens
        • the DeepEncoder will segment it into 1024/16×1024/16=4096 patch tokens.

  • 27 Jan 2026 DeepSeek OCR2

    • MinerU2.5 90.67
    • PaddleOCR-VL 92.86
    • DeepSeek-OCR 87.36
    • DeepSeek-OCR2 91.09

  • 2 Mar 2026 FireRed_OCR

    • Architecture Qwen3-VL
    • Our approach is founded on two pillars: a high precision data engine and a progressive training strategy.
      • “Geometry + Semantics” Data Factory
      • Three-Stage Progressive Training pipeline
        • Multi-task Pre-alignment
        • Specialized SFT
        • Format-Constrained GRPO
    • Results
      • Pipeline OCR Systems 依然很能打
      • Pipeline OCR Systems
        • MinerU2.5 90.67
        • PaddleOCR-VL 92.86
        • PaddleOCR-VL-1.5 94.50
      • General VLMs
        • Qwen3.5-397B-A17B 90.80
      • End-to-End OCR Models
        • DeepSeek-OCR2 91.09
        • FireRed-OCR-2B 92.94

Benchmark