Evaluation - GlaS Challenge

Detection¶

The ground truth for each segmented object is the object in the manual annotation that has maximum overlap with that segmented object.

A segmented glandular object that intersects with at least 50% of its ground truth will be considered as true positive, otherwise, it will be considered as false positive. A ground truth glandular object that has no corresponding segmented object or has less than 50% of its area overlapped by its corresponding segmented object will be considered as a false negative.

Let

$TP$ be the number of true positives,
$FP$ be the number of false positives,
$FN$ be the number of false negatives.

A metric for gland detection is the F1-score, defined by

$\mathrm{F1score}~=~\frac{2\cdot\mathrm{Precision}\cdot\mathrm{Recall}}{\mathrm{Precision}~+~\mathrm{Recall}}$

where

$~\mathrm{Precision}~=~TP/(TP~+~FP)$ , $~\mathrm{Recall}~=~TP/(TP~+~FN)$

Segmentation¶

Given $G$ a set of pixels annotated as a ground truth object and $S$ a set of pixels segmented as a glandular object, the Dice index is defined as follows

$\mathrm{Dice}(G,S)~=~\frac{2|G\cap{S}|}{|G|+|S|}$

Further, let

$\mathcal{G}_a$ denote a set of ground truth objects in image $a$ .
$\mathcal{S}_a$ denote a set of segmented objects in image $a$ .
$S_i~\in~\mathcal{S}_a$ denote the $i$ th segmented object in image $a$ .
$G_i~\in~\mathcal{G}_a$ denote a ground truth object that maximally overlaps $~S_i~$ in image $a$ .
$\tilde{G}_i~\in~\mathcal{G}_a$ denote the $i$ th ground truth object in image $a$ .
$\tilde{S}_i~\in~\mathcal{S}_a$ denote a segmented object that maximally overlaps $~\tilde{G}_i~$ in image $a$ .
$\mathcal{G}~=~\cup_a~\mathcal{G}_a~$ a set of all ground truth objects.
$\mathcal{S}~=~\cup_a~\mathcal{S}_a~$ a set of all segmented objects.
$n_S$ denote the total number of segmented objects in $~\mathcal{S}~$ .
$n_G$ denote the total number of ground truth objects in $~\mathcal{G}~$ .

We define the object-level Dice index as

$\mathrm{Dice}_\text{object}(G,S)=\frac{1}{2}\left[\sum_{i=1}^{n_S}\omega_i\mathrm{Dice}(G_i,S_i)+\sum_{i=1}^{n_G}\tilde{\omega}_i\mathrm{Dice}(\tilde{G}_i,\tilde{S}_i)\right],$

where

$\omega_i=|S_i|/\sum_{j=1}^{n_S}|S_j|,$ $\tilde{\omega}_i=|\tilde{G}_i|/\sum_{j=1}^{n_G}|\tilde{G}_j|.~$

The object-level Dice index will be used to evaluate the performance of segmentation.

Shape Similarity¶

Let $G$ denote a set of pixels annotated as ground truth and $S$ denote a set of pixels segmented as glandular objects. A Hausdorff distance between $G$ and $S$ is defined as

$\mathrm{H}(G,S)~=~\max\{\sup_{x\in{G}}\inf_{y\in{S}}\|x-y\|,\sup_{y\in{S}}\inf_{x\in{G}}\|x-y\|\}$ .

Now, let

$\mathcal{G}_a$ denote a set of ground truth objects in image $a$ .
$\mathcal{S}_a$ denote a set of segmented objects in image $a$ .
$S_i~\in~\mathcal{S}_a$ denote the $i$ th segmented object in image $a$ .
$G_i~\in~\mathcal{G}_a$ denote a ground truth object that maximally overlaps $~S_i~$ in image $a$ . If there is no ground truth object overlapping $~S_i~$ , $~G_i~$ is defined as ground truth object $~G~\in~\mathcal{G}_a$ that has the minimum Hausdorff distance from $~S_i~$ .
$\tilde{G}_i~\in~\mathcal{G}_a$ denote the $i$ th ground truth object in image $a$ .
$\tilde{S}_i~\in~\mathcal{S}_a$ denote a segmented object that maximally overlaps $~\tilde{G}_i~$ in image $a$ . If there is no segmented object overlapping $~\tilde{G}_i~$ , $~\tilde{S}_i~$ is defined as segmented object $~\tilde{S}~\in~\mathcal{S}_a$ that has the minimum Hausdorff distance from $~\tilde{G}_i~$ .
$\mathcal{G}~=~\cup_a~\mathcal{G}_a~$ a set of all ground truth objects.
$\mathcal{S}~=~\cup_a~\mathcal{S}_a~$ a set of all segmented objects.
$n_S$ denote the total number of segmented objects in $~\mathcal{S}~$ .
$n_G$ denote the total number of ground truth objects in $~\mathcal{G}~$ .

We measure the shape similarity between all segmented objects in $\mathcal{S}$ and all ground truth objects in $\mathcal{G}$ using the object-level Hausdorff distance

$\mathrm{H}_\text{object}(\mathcal{S},\mathcal{G})~=~\frac{1}{2}~\left[~\sum_{i~=~1}^{n_s}~\omega_i~H(G_i,S_i)~+~\sum_{i=1}^{n_G}~\tilde{\omega}_i~H(\tilde{G}_i,\tilde{S}_i)\right],$

where

$\omega_i=|S_i|/\sum_{j=1}^{n_S}|S_j|,$ $\tilde{\omega}_i=|\tilde{G}_i|/\sum_{j=1}^{n_G}|\tilde{G}_j|.~$