NoStoneUnturned

Automatic Index Builder

Contents

• Overview
• Requirements
  • Optional 3rd Party Accessories
  • Python 3.10
• Workflow (Linux)
  • 3rd Party Installs
  • Python Environment Setup
  • Index Building
    • Decrypt PDFs
    • Extract text from images
    • Convert to TXT
    • Build index
    • Merge indices
• Workflow (Windows)
  • 3rd Party Installs
  • Python Environment Setup
  • Index Building
    • Decrypt PDFs
    • Extract text from images
    • Convert to TXT
    • Build index
    • Merge indices

Overview

Build indexes from PDF or TXT files.

Casts a wide net but it gets everything. Could be a good option for any open-book with index test.

---

Requirements

Optional 3rd Party Accessories

• qpdf
• pdftotext

Python 3.10

• PyMuPDF
• Pillow
• numpy
• python-doctr
• torch
• torchvision
• pdfminer.six
• pypdf
• wordfreq
• nltk
• tqdm

---

Workflow (Linux)

3rd Party Installs

sudo apt update
sudo apt install qpdf poppler-utils

Python Environment Setup

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt

Index Building

1. Decrypt PDFs (as required)

qpdf --password='PASSWORD' --decrypt book_n.pdf decrypted_n.pdf

2. Extract text from images (as required)

.venv/bin/python extract_img_text.py decrypted_n.pdf processed_n.pdf

This is a time-consuming step that is not required for most PDFs. Even some PDF that look like they contain images are actually composed of text elements only.

3. Convert to TXT (optional)

pdftotext processed_n.pdf text_n.txt

pdftotext generally gives slightly better results than extracting from the raw PDF. However, PDF layout is lost during conversion to text and multi-column elements may be reordered incorrectly.

4. Build index

.venv/bin/python build_index.py -v -o 2 -l 2 -L 50 -F 10 -z 4.0 -r '[a-zA-Z0-9 :.&_-]+' text_n.txt index_n.txt

Recommended Settings:

• -v
  Verbose output (does not affect index content)
• -o 2
  PDF page numbering starts on the second page
• -l 2
  Minimum token length for items added to the index
• -L 50
  Maximum token length for items added to the index
• -F 10
  Exclude words appearing on more than 10 pages
• -z 4.0
  Exclude words with a Zipf score > 4.0
• -r
  Regex filter for accepted tokens

5. Merge indices

.venv/bin/python index_merge.py -o index.txt -F 10 index_1.txt index_2.txt index_n.txt

Recommended Settings:

• -F 10
  Exclude words appearing on more than 10 pages total

---

Workflow (Windows)

3rd Party Installs

Install the following tools and ensure they are available on your PATH:

• qpdf
• Poppler for Windows (provides pdftotext)

Using Winget:

winget install QPDF.QPDF
winget install oschwartz10612.Poppler

Or download manually:

• QPDF: https://qpdf.sourceforge.io/
• Poppler: https://github.com/oschwartz10612/poppler-windows/releases

Python Environment Setup

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

Index Building

1. Decrypt PDFs (as required)

qpdf --password="PASSWORD" --decrypt book_n.pdf decrypted_n.pdf

2. Extract text from images (as required)

.\.venv\Scripts\python.exe extract_img_text.py decrypted_n.pdf processed_n.pdf

This is a time-consuming step that is not required for most PDFs. Even some PDFs that appear to contain images are actually composed entirely of searchable text.

3. Convert to TXT (optional)

pdftotext processed_n.pdf text_n.txt

pdftotext generally produces slightly better results than extracting text directly from the PDF. However, PDF layout is lost during conversion and multi-column content may be reordered incorrectly.

4. Build index

.\.venv\Scripts\python.exe build_index.py -v -o 2 -l 2 -L 50 -F 10 -z 4.0 -r "[a-zA-Z0-9 :.&_-]+" text_n.txt index_n.txt

Recommended Settings:

• -v
  Verbose output (does not affect index content)

• -o 2
  PDF page numbering starts on the second page

• -l 2
  Minimum token length for items added to the index

• -L 50
  Maximum token length for items added to the index

• -F 10
  Exclude words appearing on more than 10 pages

• -z 4.0
  Exclude words with a Zipf score greater than 4.0

• -r
  Regex filter for accepted tokens

5. Merge indices

.\.venv\Scripts\python.exe index_merge.py -o index.txt -F 10 index_1.txt index_2.txt index_n.txt

Recommended Settings:

• -F 10
  Exclude words appearing on more than 10 pages across all input indexes.