Skip to content

Conversation

@kleinercubs
Copy link

The evaluation results must be saved in the order of their keys (problem_id). Otherwise, when some eval results are missing, the speedup computed with geometric_mean_speed_ratio_correct_only would use mismatched pairs and produce incorrect results.

@simonguozirui
Copy link
Collaborator

Thanks for catching that @kleinercubs.

@pythonomar22 let's look into this as we are revamping metrics calculations and summary statistics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants