Skip to content

Improve the algorithm for single-token generation #14

Description

@Francesco215

As of now, the code of model.generate() doesn't cache the value of each node of the model, therefore it has to re-calcutate everyting when generating each new token

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions