Skip to content

Tokenizers: handle special_tokens #31

Open
@pcuenca

Description

@pcuenca

They are now being parsed and added to a list, but ignored.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions