Skip to content

[BUG] Context parallelism not enabled in SFT #189

Description

@slimfrkha

Describe the bug
get_batch doesnt support cp broadcasting. Running with cp > 1 on current code yields wrong loss

To Reproduce
any same SFT experiment with cp=1 then cp=2

Expected behavior
cp=1 or cp=2 should yield very similar lm loss

Screenshots
Screenshot 2024-12-24 at 3 43 11 PM
Screenshot 2024-12-24 at 3 43 26 PM

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions