Skip to content

Yodas2 dataset processing: config and processors #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 92 commits into
base: main
Choose a base branch
from
Open

Yodas2 dataset processing: config and processors #102

wants to merge 92 commits into from

Conversation

ssh-meister
Copy link
Collaborator

Added:

  • Faster Whisper Inference processor
  • Lambda Expressions processor
  • Extract Tar

@ssh-meister ssh-meister self-assigned this Apr 1, 2025
ssh-meister added 29 commits May 5, 2025 05:49
@ssh-meister
yodas2 config is added
4d287d5 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
ListToEntries is added
dc61432 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
ListToEntries is added
6921035 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
ListRepoFiles and SnapshotDownload are added
ea5f874 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
ListRepoFiles and SnapshotDownload docs are added
3fc4895 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Merge branch 'yodas2_pr' of https://github.com/NVIDIA/NeMo-speech-dat…
8fe7d92 
…a-processor into yodas2_pr
@ssh-meister
removed build
0c093d6 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
init updated
29daaa3 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
ListYodas2Data
4eb64ef 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
ListYodas2Data upd
8938345 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
LambdaExpression
59a2d3f 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
DownloadYodas2Data
c632344 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
init for yodas2 processors
cba6b06 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Fixed docs
613a486 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Fixed docs
8220deb 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
ExtractTar
9e71140 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
CreateInitialManifestYodas2
813f670 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Audio converstion processors moved to convert_audio.py
8ea0094 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
RemoveFiles
24f8cc0 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
ASR inference refactoring
e73bd83 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
FasterWhisperInference
5a293c6 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Doc fix
5ab8afc 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Fix typo
312ec51 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
DropSpecifiedFields
c725a7c 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
WhisperHallucinationFeatures
2327a18 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
vLLMInference
29cf370 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
CleanQwenGeneration
71ebb33 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Updated SubRegex
1525c0c 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
CountNumWords updated
994a5f2 
Signed-off-by: Sasha Meister <[email protected]>
ssh-meister added 16 commits May 7, 2025 10:17
@ssh-meister
Prevented requirements collection in subfolders
5ae3072 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Prevent no space left during building
dc0391e 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Lightweight granary requirements
8b80117 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Prevent no space left during building
f3210fb 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
E2E tests check
f12f81f 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Branch update
2c41c13 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
requirements mofidifcation
c9e8d43 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Fix device setup for FasterWhisperInference
8f6e1cf 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
added prepare_yodas2_data.py
45fdc8a 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
added text preprocessing in FastTextLangIdClassifier
214a1be 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
prepare_yodas2_data.py updated
98ab125 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
added wget to add tests/prepare_test_data/prepare_yodas2_data.py
6153185 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Removed already imported module
223bd5d 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Added missed param in FasterWhisperInference
54a0af5 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Added HfHubDownloadYodas2Data, HfHubDownload, GetGranarysYodas2
c9e72fd 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Simplified prepare_yodas2_data.py
7a65dbe 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister ssh-meister marked this pull request as ready for review May 19, 2025 11:54
@ssh-meister ssh-meister requested a review from karpnv May 19, 2025 11:54
ssh-meister and others added 10 commits May 20, 2025 02:55
@ssh-meister
New structure of files
33ef48a 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Fix typo in ListYodas2Data
71d84cb 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Added JoinManifests
0bda056 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
mkdir in SnapshotDownload process
674f1fd 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Yodas from Granary
3e12532 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Granary cfg moved
b7ee0f9 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
license
7421bb0 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Removed extra line
ef4515b 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Added constant fields
3b8aff7 
Signed-off-by: Sasha Meister <[email protected]>
@ssh-meister
Update README.md
3dc4971
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AO696 AO696 AO696 approved these changes

@karpnv karpnv Awaiting requested review from karpnv

@Jorjeous Jorjeous Awaiting requested review from Jorjeous

@lilithgrigoryan lilithgrigoryan Awaiting requested review from lilithgrigoryan

At least 1 approving review is required to merge this pull request.

Assignees

@ssh-meister ssh-meister

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ssh-meister @AO696