milia.evaluate.ranking#
Quality of a ranking against the true, graded relevance of the items it orders.
A ranking orders items by a score, such as the predicted risk of each transformer, and is judged against the outcome it is meant to order, such as the transformer’s simulated violation hours. The functions here follow the information-retrieval definitions: discounted cumulative gain and its normalised form [Järvelin and Kekäläinen, 2002], precision at a cut-off, the reciprocal rank of the most relevant item, and Kendall’s rank correlation. Ties in the score are broken by the original order of the items, so every result is deterministic.
Each metric takes the true relevance, true, which must not be negative, and the ranking score, score,
higher first. With groups, each group is a separate ranked list (a query, a network, a penetration level)
and the metric is averaged over the groups.
Functions
|
Discounted cumulative gain of gains listed in ranked order. |
|
Kendall's rank correlation between the true relevance and the score. |
|
Normalised discounted cumulative gain: DCG of the ranking over the DCG of the ideal ranking. |
|
Share of the |
|
Reciprocal of the position at which the score ranks the most relevant item. |
- milia.evaluate.ranking.dcg(gains, k=None)#
Discounted cumulative gain of gains listed in ranked order.
\[\mathrm{DCG}@k = \sum_{i=1}^{k} \frac{g_i}{\log_2(i + 1)}\]- Parameters:
- Return type:
- Returns:
The discounted cumulative gain.
Examples
>>> round(dcg([3, 2, 0]), 3) 4.262
- milia.evaluate.ranking.ndcg(true, score, k=None, groups=None)#
Normalised discounted cumulative gain: DCG of the ranking over the DCG of the ideal ranking.
\[\mathrm{NDCG}@k = \frac{\mathrm{DCG}@k(\text{order by score})}{\mathrm{DCG}@k(\text{order by truth})}\]One for a ranking that orders the items as their relevance does; a list without relevant items scores one.
- Parameters:
- Return type:
- Returns:
NDCG at
k, averaged over the lists.
Examples
>>> ndcg([3, 2, 0, 1], [0.9, 0.8, 0.1, 0.2]) 1.0 >>> round(ndcg([3, 2, 0, 1], [0.1, 0.2, 0.9, 0.8]), 3) 0.614
- milia.evaluate.ranking.precision(true, score, k, groups=None)#
Share of the
kitems ranked first by the score that are among thekmost relevant.- Parameters:
- Return type:
- Returns:
Precision at
k, averaged over the lists.
Examples
>>> precision([5, 4, 3, 2, 1], [5, 1, 4, 2, 3], k=2) 0.5
- milia.evaluate.ranking.reciprocal(true, score, groups=None)#
Reciprocal of the position at which the score ranks the most relevant item.
Averaged over lists, this is the mean reciprocal rank: one when the most relevant item is always ranked first, one half when it is ranked second.
- Parameters:
- Return type:
- Returns:
The (mean) reciprocal rank.
Examples
>>> reciprocal([1, 9, 2], [0.5, 0.4, 0.9]) 0.3333333333333333
- milia.evaluate.ranking.kendall(true, score, groups=None)#
Kendall’s rank correlation between the true relevance and the score.
- Parameters:
- Return type:
- Returns:
Kendall’s tau-b, averaged over the lists; zero for a list whose score or truth is constant.
Examples
>>> kendall([1, 2, 3, 4], [10, 20, 30, 40]) 1.0 >>> kendall([1, 2, 3, 4], [40, 30, 20, 10]) -1.0