(WIP) Tie-breaking extensibility #110
Conversation
kge/config-default.yaml
Outdated
| # ['a': score 10, 'b': score 10, 'c': score 10, 'd': score 11]. Correct query | ||
| # answer: 'a'. Possible options are: | ||
| # - rounded_mean_rank: Calculate the rounded mean rank. The final rank is calculated | ||
| # as 1 + #scores_higher + #answers_with_same_score // 2, |
There was a problem hiding this comment.
write as round_up(...), use /2. Easier to read this way.
There was a problem hiding this comment.
I have addressed the review. Only here I used round_down() as this matches X//Y.
kge/config-default.yaml
Outdated
| # Correct query answer: 'a'. Possible options are: | ||
| # - rounded_mean_rank: Calculate the rounded mean rank. The final rank is | ||
| # calculated as 1 + #scores_higher + | ||
| # round_down(#answers_with_same_score / 2). In the example, |
There was a problem hiding this comment.
We always round up to err on the worst_rank side.
There was a problem hiding this comment.
Ahh, I read this as "other_answers_with_same_score". Prehaps clarifiy: rounded (up) mean rank,
There was a problem hiding this comment.
I'm just describing what the code does. And in ranks + num_ties // 2 , "//" rounds the rank down?
There was a problem hiding this comment.
See, you are confused as well. We round the number of ties (>=1) down, which corresponds to rounding the mean rank up. E.g., if there is one other answer with he same score than the correct one, then num_ties=2. We round down num_ties/2, but this corresponds to rounding up the mean rank. Perhaps its best to not show this computation here, it's only conusing. Mean rank rounded up is clear enough.
kge/config-default.yaml
Outdated
| # 1 + 1 + round_down(3 / 2) = 3. | ||
| # - best_rank: Assign the lowest rank for the correct answer. The final | ||
| # rank for answer (s, p, 'a') is 2. We do not recommend to | ||
| # use this option, it leads to misleading results. |
There was a problem hiding this comment.
To weak. Write DO NOT USE THIS OPTION. It's the reason for a number of problematic results/papers
kge/job/entity_ranking.py
Outdated
| @@ -13,6 +13,11 @@ class EntityRankingJob(EvaluationJob): | |||
|
|
|||
| def __init__(self, config: Config, dataset: Dataset, parent_job, model): | |||
| super().__init__(config, dataset, parent_job, model) | |||
| self.config.check( | |||
| "eval.entity_ranking.tie_handling", | |||
There was a problem hiding this comment.
Why not: eval.tie_handling. It makes sense for all ranking tasks, even the ones we still plan to implement
There was a problem hiding this comment.
Yes that also makes sense. So dropping the key 'entity_ranking' completely for now.
| # misleading evaluation results. | ||
| # - rounded_mean_rank: Average between worst and best rank, rounded up | ||
| # (rounded fractional ranking). In example: 3. | ||
| tie_handling: rounded_mean_rank |
|
Thanks! |
Tie-breaking options/documentation, see #96