Add more datasets, support for huggingface datasets, etc

[vfdev-5]

Let's add more datasets that user can select from the CLI:

• generic HF datasets option
• built-in HF dataset wrapper for python programming, python-edu, https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus
• built-in HF dataset wrappers for:
  • https://huggingface.co/datasets/Salesforce/wikitext
  • https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k

[aaishwarymishra]

ok what should be the scope of these? I get the built in wrappers, but what about the generic datasets how should we handle it?

As we have a TransformDataset class which basically takes in datasets and applies the transform if provided. When user is using the hugging face dataset should he provide the download hugging face url? or the local dataset path and it's, her own transform?

[atharva7-g]

I'd be interested in contributing to this.

ok what should be the scope of these? I get the built in wrappers, but what about the generic datasets how should we handle it?

Usually when pulling datasets from HF, datasets are id'd as: owner/name. So URLs are probably not ideal. And I think we can support local files as well. Parsing should be durable for inferring HF or local. It would also be better if we can add HF as an optional dep here, then we can use the HF API directly for loading datasets.

[aaishwarymishra]

Hi @atharva7-g contributions are welcome,

whats in my mind is this,

• For generic wrapper we can take an __target__ in the the dataset config and have an generic transform which user has to play with a little with inputs to get it working.
• For the case of the specific datasets as listed in the original issue, we will have pre-built transforms for them.

Well I am not sure how code needs to be changed as it's first time we are integrating the external dataset support.

Regarding "HF API directly for loading datasets" I am not sure how to handle it exactly there are multiple ways to load the HF dataset, the easiest way I know of is using the HF datasets package but it loads dataset as there dataset object that then needs to be converted to pytorch dataset which feels a little redundant in my opinion. Also for private datasets requires authentication which requires some HF package I don't remember exactly. Also I am not sure if the HF package should be required dependency or we can keep it optional for projects that require it @vfdev-5 what do you think? I think we can keep it required if its small enough for all the projects.

well I am open for ideas if you want to work on this :)

[TahaZahid05]

This sounds good to me. For specific HF datasets, it is doable since we only need to write transforms for them to make them work with Trainite.

For generic HF datasets, we need to design it such that the generic transform is configurable by the user so that the transform is compatible with the dataset. For instance, a configurable inside yaml like text_column.

[atharva7-g]

Regarding "HF API directly for loading datasets" I am not sure how to handle it exactly there are multiple ways to load the HF dataset, the easiest way I know of is using the HF datasets package but it loads dataset as there dataset object that then needs to be converted to pytorch dataset which feels a little redundant in my opinion.

torch.utils.data.DataLoader can handle HF Datasets as well, so that itself doesn't seem to be an issue. HF's datasets is also small enough that adding it as a required dep can be considered. Though if we wanted to load it from a parquet for example, we can do that as well AFAIK.

And yeah for private datasets HF Hub does seem to be required.

Apart from that, the plan to add a text_column like parameter sounds good. Though there are cases where text itself spans multiple columns, that will have to be taken into account as well.

[aaishwarymishra]

ok so I have made an draft pr for the generic datasets possibly going to iterate over it and it can be blueprint for other specific datasets for the issue.

Few things I am not sure about

1. currently we tie tokenizer to the dataset itself, my question will be in this case what should be the default behaviour? should we tie an default tokenizer which will be generated when the generic hf dataset is selected? or should we let user choose tokenizer thus maybe decoupling it from dataset and making it independent maybe.
2. What should be the default configuration of the dataset config as we have 2 different configs for data:

• we have a single dataset which we split into train, test, val
• we can have separate configs for train, test , val dataset itself