We can show first that training a transformer to learn arithmetics does not generalize to OOD examples.
Next, we should provide a "solution" from existing papers showing how to make it work.
Here is a list of papers I found on the topic, but we can check and use others. It can be also helpful to check the recent research papers of these authors if they continued working on the problem and found something interesting.
Abacus embedding and kind of recurrent Transformer can generalize to larger lengths: trained for additions on numbers with 20 digits and tested on numbers with 100 digits. https://github.com/mcleish7/arithmetic
- Models can't generalize to unseen lengths
- Addition with reverse order helps
- Scratch pad
https://github.com/HanseulJo/position-coupling
Trained models on easy task can iteratively retrained on self generated data to be able to inference on harder tasks
Importance of filtering for the generated pseudo-labeling data: length size, majority votes on preds with multiple models
https://github.com/JackCai1206/arithmetic-self-improve
Transformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalization arises. In this paper, we investigate length generalization—the ability to extrapolate from shorter to longer inputs—through the lens of \textit{task transfer}. We find that length generalization can be \textit{transferred} across related tasks. That is, training a model with a longer and related auxiliary task can lead the model to generalize to unseen and longer inputs from some other target task. We demonstrate this length generalization transfer across a diverse suite of algorithmic tasks, including arithmetic operations, string transformations, and maze navigation. Our results show that transformer models can inherit generalization capabilities from similar tasks when trained jointly. Moreover, we observe similar transfer effects in pretrained language models, suggesting that pretraining equips models with reusable computational scaffolding that facilitates extrapolation in downstream settings. Finally, we provide initial mechanistic evidence that length generalization transfer correlates with the re-use of the same attention heads between the tasks. Together, our findings deepen our understanding of how transformers generalize to out-of-distribution inputs and highlight the compositional reuse of inductive structure across tasks.
This is an old note: https://openreview.net/forum?id=aLUAzLDIOc / Code can be found in attached materials
https://github.com/lacoco-lab/length_generalization
The outcome of the experiment can be a blog post showing the usage of trainite and it's usefulness in a small research project.
We can show first that training a transformer to learn arithmetics does not generalize to OOD examples.
Next, we should provide a "solution" from existing papers showing how to make it work.
Here is a list of papers I found on the topic, but we can check and use others. It can be also helpful to check the recent research papers of these authors if they continued working on the problem and found something interesting.
The outcome of the experiment can be a blog post showing the usage of trainite and it's usefulness in a small research project.