EU data set

[jorgtied]

Training data for a multilingual model covering translations between official EU languages.

• English-centric data (initial model)
• non-English language pairs (further iterations)

[jorgtied]

• Raul sends existing scripts for fetching and preprocessing for official EU languages to Anssi
• use Tatoeiba challenge data from https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/data/README-v2023-09-26.md
• Anssi develops some workflow to keep data on allas

[amikael]

Here is a short update to help tracking my progress.

• This does not seem to be a challenging objective after the toolchain is working.
• I have spent only 2% of time to data curation and mainly worked on the toolchain (not this objective)
• I am perfectly capable of doing more elaboration and development when necessary and timed

Here is my account of the minimal (current) account of the objective:

1. I downloaded Tatoeba as a whole to Puhti, and looked arount its structure, and the categorization of the available language pairs. It seemed elaborate and complex variation in overall. The data amount is immensive and one can only get an overal grip of the things.
2. I did not study or test the quality of the data files at all in this point. I understood from Chao that there is a need for filtering the Tatoeba data in various ways. Furthermore, from the Europarl preprocessing script I got that this kind of data sets, including Tatoeba, could need stripping, uniquisation, empty removal and other kinds of trivial preprocessing. Uniquisation makes sense when data files are combined from different sources with union, but I am astonished why empty or orphaned examples and other kind of normalisations like stripping has not been taken care of by Tatoeba curators themselves. Perhaps this is in the documentation, but I did not allocated time to read all what is said about Tatoeba.
3. For weeks, I thought I would come back to Tatoeba data curating in a large scale and left it to Puhti for "next day". Regretting this, I am going to remove Tatoeba from my puhti dirs to reduce storage use.
4. As my milestone is approaching, I have come to the conclusion that I will not have time to study Tatoeba carefully to some substantial improvements to its curation. Instead, I just plan to download eng-eng (and eng-* for some other languages) to get started with Mammoth runs. I also found our some other datasets that are not the largest and would relate languages that I am reasonably familiar with. Here is my pick (excludes eng-* for now):

• https://object.pouta.csc.fi/Tatoeba-Challenge-v2023-09-26/eng-fin.tar
• https://object.pouta.csc.fi/Tatoeba-Challenge-v2023-09-26/eng-eng.tar
• https://object.pouta.csc.fi/Tatoeba-Challenge-v2023-09-26/fin-krl.tar
• https://object.pouta.csc.fi/Tatoeba-Challenge-v2023-09-26/est-fin.tar
• https://object.pouta.csc.fi/Tatoeba-Challenge-v2023-09-26/est-fin.tar

6. There was a task/requirement that "Anssi could develop some Allas workflow". I kept wonder what was the goal. Weeks back, I had an incorrect though that .sp would have been embedded, but it is just filtered and tokenized version of the data. I am still tinking what was being asked and ment by the task. Allas or Lumi's storage server do not seem to be relevant at the current point, but I can imagine that if we want to randomize sentences in Tatoeba and freeze the order, then it is certainly useful to have this done in advance. I decided to postpone the task until it is fully specified and designed because I do not have time for unspecified development.
7. Since I have used most of my time in trying to get the software workflow adequately installed and set up to LUMI and still finalizing that part, I update the goal as follows:

This week (1-2 working days that break my holiday):

• I hope to get Mammoth quickly running on LUMI 23.xx/24.03 and Python 3.10 or 3.12 since there are optimized pytorch containers for these software stacks. For now the latter has been a struggle since pyonmttok compilation procedure has produced the executables ((de)tokenizer), but could not install its python libraries due to some CXX toolchain incompatibilities. The remedy plan Hf integration #1 is to use Python 3.10 that installs cleanly, and Migration of various features from the v2 repo #2 to fall back running tests on Puhti - which would not remove the forthcoming challenge to get Mammoth installed to a recent stack on LUMI.
• I hope that I can follow the quick setup pattern for one GPU / one language pair and the running eng-eng autoencoding task to get mammoth runnability (libraries etc.) basically tested without much more hassle. At this point I would not bother about Singshot 11 or other settings, as Raul proposes.
• 18th July, is an updated timing for my goal of being ready by 14th Mon. This is due to my changed travel plans that reallocated 12-14 July for travelling and settling.

(While in Wienna):

• After the toolchain is validated, I plan to modify the sbatch script by adding the suport for Slingshot and retest to validate it is working. I am not expecting any training performance improvement at this point (since 1 GPU), but I want to know that the toolchain still works.
• Once the toolchain is proven working, I plan to add enough languages and run such training that involves interconnected processes and the use of Slingshot etc. This task gives me an opportunity to benchmark the difference between six settings: (1GPU vs 8 GPUs vs 2 nodes) x (no Slingshot vs full Slingshot). Although this is not a project objective, correct configuration will serve the project's efficiency and be sharable accross other project members, ultimately paying back in may ways in savings of the billing units (up to 3x times faster).

In August,

• I am looking forward to add more languages and to make more experiments. This will require increasingly elaborate sets of language pairs, discussions about the GPU setups and mammoth layers. I am definitely not yet at this stage of my experience. However, I do have much more internalized picture of challenges in distributed training and a better advice of how to set up the distributed jobs so that all communication channels between the processes are optimized.