[Doc Review 25.09] Huy - Video Concepts + Text Advanced

[huvunvidia]

📄 Pages to Review

• Key Abstractions - https://docs.nvidia.com/nemo/curator/25.09/about/concepts/video/abstractions.html
• Architecture - https://docs.nvidia.com/nemo/curator/25.09/about/concepts/video/architecture.html
• Data Flow - https://docs.nvidia.com/nemo/curator/25.09/about/concepts/video/data-flow.html
• Video Curation Concepts - https://docs.nvidia.com/nemo/curator/25.09/about/concepts/video/index.html
• Exact Duplicate Removal - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/deduplication/exact.html
• Fuzzy Duplicate Removal - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/deduplication/fuzzy.html
• Deduplication - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/deduplication/index.html
• Semantic Deduplication - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/deduplication/semdedup.html
• Language Identification - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/language-management/language.html
• Stop Words in Text Processing - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/language-management/stopwords.html
• Classifier-Based Filtering - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/quality-assessment/classifier.html
• Distributed Data Classification - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/quality-assessment/distributed-classifier.html
• Heuristic Filtering - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/quality-assessment/heuristic.html
• Quality Assessment & Filtering - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/quality-assessment/index.html
• Code Filtering - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/specialized-processing/code.html
• Specialized Processing - https://docs.nvidia.com/nemo/curator/25.09/curate-text/process-data/specialized-processing/index.html
• Split and Remove Duplicates Workflow - https://docs.nvidia.com/nemo/curator/25.09/curate-video/tutorials/split-dedup.html

Description

Usage

# Add snippet demonstrating usage

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[greptile-apps]

Greptile Overview

Greptile Summary

This PR implements documentation improvements across video concepts and text processing sections, enhancing clarity, consistency, and completeness. Most changes are minor editorial improvements including wording clarifications, grammar fixes, markdown link formatting, and added explanatory content for parameters and workflows.

Key improvements:

• Enhanced video concept documentation with clearer explanations of streaming mode and data flow
• Added valuable guidance on embedding model selection (avoiding decoder-only LLMs)
• Improved code filtering documentation with additional language support and parameter clarifications
• Converted raw URLs to proper markdown links for better documentation formatting
• Fixed frontmatter formatting and whitespace issues

Critical issue:

• Two debugging comments (## NEED FIX:) were left in production documentation files and must be removed before merge

Confidence Score: 3/5

• PR cannot be merged with debugging comments present in production docs
• Score reflects high-quality editorial improvements throughout, but two critical syntax issues block merging: debugging comments with "NEED FIX" markers were left in code.md and index.md files that must be removed
• Pay close attention to docs/curate-text/process-data/specialized-processing/code.md and docs/curate-text/process-data/specialized-processing/index.md - both contain debugging comments that must be removed

Important Files Changed

File Analysis

Filename	Score	Overview
docs/about/concepts/video/data-flow.md	5/5	Enhanced explanations of streaming mode, writer outputs, and data flow
docs/curate-text/process-data/deduplication/semdedup.md	5/5	Added embedding model guidance, improved output descriptions, fixed code block syntax
docs/curate-text/process-data/specialized-processing/code.md	2/5	Added language support, parameter clarity, imports - but contains debug comments that must be removed
docs/curate-text/process-data/specialized-processing/index.md	2/5	Fixed heading capitalization but contains debug comment that must be removed

Sequence Diagram

sequenceDiagram
    participant Author as Documentation Author
    participant PR as Pull Request
    participant Review as Review Process
    participant Docs as Documentation System
    
    Author->>PR: Submit doc improvements
    Note over PR: 14 files changed<br/>Video concepts + Text processing
    
    PR->>Review: Trigger review
    Review->>Docs: Check video/architecture.md
    Docs-->>Review: ✓ Wording improvements
    
    Review->>Docs: Check video/data-flow.md
    Docs-->>Review: ✓ Enhanced explanations
    
    Review->>Docs: Check deduplication docs
    Docs-->>Review: ✓ Frontmatter fix, clarifications
    
    Review->>Docs: Check quality-assessment docs
    Docs-->>Review: ✓ URL formatting, improvements
    
    Review->>Docs: Check specialized-processing/code.md
    Docs-->>Review: ⚠️ Debug comment found (line 48)
    
    Review->>Docs: Check specialized-processing/index.md
    Docs-->>Review: ⚠️ Debug comment found (line 79)
    
    Review->>PR: Report findings
    Note over PR: 2 critical issues<br/>Must remove debug comments
    
    PR-->>Author: Request changes

Loading

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

_{9 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{14 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{15 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{15 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{15 files reviewed, 6 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{14 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[huvunvidia]

/ok to test 93ad6f6

[greptile-apps]

_{14 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{14 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

syntax: debugging comment left in production docs

Suggested change

## NEED FIX: TypeError: ScoreFilter.__init__() got an unexpected keyword argument 'score_fn'

[greptile-apps]

syntax: debugging comment left in production docs

Suggested change

## NEED FIX: NameError: name 'code_dataset' is not defined

[sarahyurick]

Thanks! It seems like there is a lot of unnecessary repetitions in the deduplication pages, maybe they can be cut down?

[sarahyurick]

I personally haven't tested TextDuplicatesRemovalWorkflow with exact deduplication. Were you able to verify it?

[sarahyurick]

This page still needs more updates imo.

[sarahyurick]

Have we confirmed cloud storage works with the deduplication modules?

[huvunvidia]

I haven't tested with cloud storage due to it's require more complicated settings.
Although I have discussed to Lawrence Lane, due to the cloud storage is not Dedup-exclusive, but for text-curator in general, it is not needed in Dedup docs. This would help the Dedup docs more concise and to the point.