milia.evaluate.ranking#

Quality of a ranking against the true, graded relevance of the items it orders.

A ranking orders items by a score, such as the predicted risk of each transformer, and is judged against the outcome it is meant to order, such as the transformer’s simulated violation hours. The functions here follow the information-retrieval definitions: discounted cumulative gain and its normalised form [Järvelin and Kekäläinen, 2002], precision at a cut-off, the reciprocal rank of the most relevant item, and Kendall’s rank correlation. Ties in the score are broken by the original order of the items, so every result is deterministic.

Each metric takes the true relevance, true, which must not be negative, and the ranking score, score, higher first. With groups, each group is a separate ranked list (a query, a network, a penetration level) and the metric is averaged over the groups.

Functions

dcg(gains[, k])

Discounted cumulative gain of gains listed in ranked order.

kendall(true, score[, groups])

Kendall's rank correlation between the true relevance and the score.

ndcg(true, score[, k, groups])

Normalised discounted cumulative gain: DCG of the ranking over the DCG of the ideal ranking.

precision(true, score, k[, groups])

Share of the k items ranked first by the score that are among the k most relevant.

reciprocal(true, score[, groups])

Reciprocal of the position at which the score ranks the most relevant item.

milia.evaluate.ranking.dcg(gains, k=None)#

Discounted cumulative gain of gains listed in ranked order.

\[\mathrm{DCG}@k = \sum_{i=1}^{k} \frac{g_i}{\log_2(i + 1)}\]
Parameters:
  • gains (ArrayLike) – Relevance of the items, in the order they are ranked.

  • k (int | None) – Cut-off; every item when omitted.

Return type:

float

Returns:

The discounted cumulative gain.

Examples

>>> round(dcg([3, 2, 0]), 3)
4.262
milia.evaluate.ranking.ndcg(true, score, k=None, groups=None)#

Normalised discounted cumulative gain: DCG of the ranking over the DCG of the ideal ranking.

\[\mathrm{NDCG}@k = \frac{\mathrm{DCG}@k(\text{order by score})}{\mathrm{DCG}@k(\text{order by truth})}\]

One for a ranking that orders the items as their relevance does; a list without relevant items scores one.

Parameters:
  • true (ArrayLike) – True relevance of each item, not negative.

  • score (ArrayLike) – Ranking score of each item, higher first.

  • k (int | None) – Cut-off; every item when omitted.

  • groups (ArrayLike | None) – Ranked list of each item; one list when omitted.

Return type:

float

Returns:

NDCG at k, averaged over the lists.

Examples

>>> ndcg([3, 2, 0, 1], [0.9, 0.8, 0.1, 0.2])
1.0
>>> round(ndcg([3, 2, 0, 1], [0.1, 0.2, 0.9, 0.8]), 3)
0.614
milia.evaluate.ranking.precision(true, score, k, groups=None)#

Share of the k items ranked first by the score that are among the k most relevant.

Parameters:
  • true (ArrayLike) – True relevance of each item.

  • score (ArrayLike) – Ranking score of each item, higher first.

  • k (int) – Cut-off.

  • groups (ArrayLike | None) – Ranked list of each item; one list when omitted.

Return type:

float

Returns:

Precision at k, averaged over the lists.

Examples

>>> precision([5, 4, 3, 2, 1], [5, 1, 4, 2, 3], k=2)
0.5
milia.evaluate.ranking.reciprocal(true, score, groups=None)#

Reciprocal of the position at which the score ranks the most relevant item.

Averaged over lists, this is the mean reciprocal rank: one when the most relevant item is always ranked first, one half when it is ranked second.

Parameters:
  • true (ArrayLike) – True relevance of each item.

  • score (ArrayLike) – Ranking score of each item, higher first.

  • groups (ArrayLike | None) – Ranked list of each item; one list when omitted.

Return type:

float

Returns:

The (mean) reciprocal rank.

Examples

>>> reciprocal([1, 9, 2], [0.5, 0.4, 0.9])
0.3333333333333333
milia.evaluate.ranking.kendall(true, score, groups=None)#

Kendall’s rank correlation between the true relevance and the score.

Parameters:
  • true (ArrayLike) – True relevance of each item.

  • score (ArrayLike) – Ranking score of each item.

  • groups (ArrayLike | None) – Ranked list of each item; one list when omitted.

Return type:

float

Returns:

Kendall’s tau-b, averaged over the lists; zero for a list whose score or truth is constant.

Examples

>>> kendall([1, 2, 3, 4], [10, 20, 30, 40])
1.0
>>> kendall([1, 2, 3, 4], [40, 30, 20, 10])
-1.0