Error Analysis & Metrics

[pvomelveny]

Fill this in

[kai-110]

https://docs.google.com/document/d/1HdvwU7ANihkTeyAxpsbarw7CiXNP3cUcNla3N-uxESs/edit?usp=sharing

[ArielFuXinyue]

https://github.com/uw-math-ai/LLMsLean/tree/error-classification

[ArielFuXinyue]

Using the multiturn + error feedback does increase the accuracy:

• Overall Success Rate: 61/100 (61.00%)
• First-Turn Success Rate: 43/100 (43.00%)
  Only generate 100 examples which took 7+ hours [7:57:14<00:00, 286.35s/it]

Here was the command:
python code/data_pipeline.py
--input code/data/data.json
--output code/data/final_multi_turn_results_testl.json
--num_samples 100