Add pre-tokenized Delta to MDS conversion script

[mattyding]

This PR

Adds conversion script for pre-tokenized data in a Delta table.

Testing

MCLI IFT and CPT runs trained successfully.

[dakinggg]

why does this assume single turn?

[mattyding]

Forge reduces/joins multi-turn CHAT into a single turn. I thought this was intended but after looking at the code, I don't think it is / PySpark might coerce the data into that format 🙃 . Will do some more debugging.

IFT regression tests might be necessary after all.

[dakinggg]

just following up that we confirmed multi turn is preserved, and so this script should support it too

[mattyding]

Last few commits address comments, add support for multi-turn CHAT data. Provided testing plan + YAMLs in Slack thread. Looking for guidance on to test with PEFT enabled

[dakinggg]

I don't think this is right? see

llm-foundry/llmfoundry/command_utils/data_prep/convert_finetuning_dataset.py

Lines 244 to 250 in 1e997df

	sample_to_write = {'turns': []}
	for turn in sample['turns']:
	turn_to_write = {}
	for key in ['input_ids', 'labels']:
	turn_to_write[key] = list(turn[key])
	sample_to_write['turns'].append(turn_to_write)
	out.write(sample_to_write)

for another example of writing turns data to MDS