Duration: 0:00

3,200 views

127

I put the o1-preview and o1-mini model's math abilities to the test by giving them the 2023 Putnam Math Exam, supposedly the hardest math test given every year. This test is outside the model's training data, since their knowledge cutoff is October 2023 and the test was released in December 2023.

Putnam Archive: https://kskedlaya.org/putnam-archive/

** POST STREAM **
o1-preview's final score: 49/120.
Median Putnam exam taker's score: 10/120.

This would have placed o1-preview just outside the top 100 of all test takers (over 4000).

Take the final score with a grain of salt, I had o1-mini act as a grader and asked it to compare the official Putnam solution and grade it according to the rough Putnam scoring guidelines. Apparently one of my viewers tried the same thing and o1-preview would’ve scored higher in their case (top 50)