evaluate matharenaapex with kimik2.5

[FFYFZQ]

Hi，i try to reproduce the result of apex with kimik2.5, but i ran three times for problem 6,7, but its accracy is only 1/16 or 2/16 with the official code, can you help me to address this problem?

[JasperDekoninck]

Due to variance, it is not unexpected that differences like this can exist, especially on single problems (they should average out overall). I guess with "official code", you mean you ran in our repo with:
uv run python scripts/run.py --comp apex/apex_2025 --models moonshot/k25?
Could you tell me the exact scores you are getting (for all problems)?

[FFYFZQ]

@JasperDekoninck Yes，but i use the offcial api from MOONSHOT, i did not run all the problems since i saw other questions kimik2.5 has very low chance to answer correctly. I think the problem 6,7 has more chance to answer, but the result shows it only answer correctly 2/16 for problem 6, 1/16 for problem 7.How many times do you run to get the result in the leaderborad?
Thank you for your reply

[JasperDekoninck]

AS you can see on the leaderboard by clicking the cells, we ran the model exactly 16 times. I would say its a bit surprising it is this low, but acceptable within the variance range for specific problems I think.

[FFYFZQ]

@JasperDekoninck Ok，thank you，i will try it one more time