Add nucleotide tokenizer for llama3 model to recipes

[savitha-eng]

Description

This PR adds a nucleotide tokenizer for the llama3 model, following HuggingFace's PreTrainedTokenizerFast pattern with Rust backend for performance. It follow's NeMo special token conventions (i.e., for EOS, BOS, PAD etc.).

Key features:

• ASCII-based tokenizer that maps nucleotides (A, T, C, G, N, R, Y) to their ASCII values
• NeMo-style special tokens: EOS=0, PAD=1, BOS=2, UNK=3
• Pre-built tokenizer files in nucleotide_fast_tokenizer/ directory
• create_tokenizer.py script for reproducibility
• Comprehensive unit tests covering tokenization, padding, and attention masks

This tokenizer will be used in genomic data loading pipelines for training llama3 models on nucleotide sequences.

Usage

from transformers import AutoTokenizer

# Load the nucleotide tokenizer
tokenizer = AutoTokenizer.from_pretrained("bionemo-recipes/models/llama3/nucleotide_fast_tokenizer")

# Tokenize a nucleotide sequence
sequence = "ATCGATCG"
encoded = tokenizer.encode(sequence, add_special_tokens=True)
# Returns: [2, 65, 84, 67, 71, 65, 84, 67, 71, 0]
# [BOS, A, T, C, G, A, T, C, G, EOS]

# Batch tokenization with padding
batch = tokenizer(["AAAA", "TTTTTTTT"], padding=True, return_tensors="pt")
# Returns: {'input_ids': tensor([[2, 65, 65, 65, 65, 0, 1, 1, 1, 1],
#                                  [2, 84, 84, 84, 84, 84, 84, 84, 84, 0]]),
#           'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0],
#                                      [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}

Type of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactor
• Documentation update
• Other (please describe):

CI Pipeline Configuration

Configure CI behavior by applying the relevant labels. By default, only basic unit tests are run.

• ciflow:skip - Skip all CI tests for this PR
• ciflow:notebooks - Run Jupyter notebooks execution tests for bionemo2
• ciflow:slow - Run slow single GPU integration tests marked as @pytest.mark.slow for bionemo2
• ciflow:all - Run all tests (unit tests, slow tests, and notebooks) for bionemo2. This label can be used to enforce running tests for all bionemo2.
• ciflow:all-recipes - Run tests for all recipes (under bionemo-recipes). This label can be used to enforce running tests for all recipes.

Unit tests marked as @pytest.mark.multi_gpu or @pytest.mark.distributed are not run in the PR pipeline.

For more details, see CONTRIBUTING

Note

By default, only basic unit tests are run. Add appropriate labels to enable an additional test coverage.

Authorizing CI Runs

We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.

• If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
  automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
• If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
  /ok to test comment on the pull request to trigger CI. This will need to be done for each new commit.

Pre-submit Checklist

• I have tested these changes locally
• I have updated the documentation accordingly
• I have added/updated tests as needed
• All existing tests pass successfully

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[pstjohn]

/ok to test bc7100f