Evaluation metrics - Object detection

[csudre]

Is your feature request related to a problem? Please describe.
Need for the following metrics when dealing with object detection problems

Intersection over Union (IoU) measurements can either be made at the level of the bounding boxes of the objects or of the segmentation themselves.

At the element level

• Dice per element
• Centroid distance

At the level of the image considering independent elements:

• F1 score with IoU thresholds ranging from 0.5 to 0.95
• Average precision with IoU thresholds ranging from 0.5 to 0.95 https://cocodataset.org/#detection-eval
• Average recall with IoU thresholds ranging from 0.5 to 0.95 https://cocodataset.org/#detection-eval
• Correlation of detected volumes
• Number difference
• F1 with minimum overlap of 1 voxel
• Outline error https://link.springer.com/article/10.1186/1471-2342-12-17
• Detection error https://link.springer.com/article/10.1186/1471-2342-12-17

Additional context
This list of metrics is the output of the initial brainstorming of the metrics task force

[mibaumgartner]

Amazing list of detection metrics, thanks for all the effort of the metrics task force to collect them!

I am curious about the F1 with minimum overlap of 1 voxel metric, could someone add some papers which use that metric? (I would like to read them ^^)

Also a small addition:
The Free-Response Receiver Operating Characteristic (FROC) might also be a metric which should be added (used by LUNA , DeepLesion and RibFrac)

[iliathesmirnov]

Hello everyone,

I am new here and would like to contribute to implementing some (or all) of the evaluation metrics on @csudre's list. Did anyone start working on them? If not, should I create a pull request (labelled [WIP]) and get going?

Thank you,
Ilia

[wyli]

Hello everyone,

I am new here and would like to contribute to implementing some (or all) of the evaluation metrics on @csudre's list. Did anyone start working on them? If not, should I create a pull request (labelled [WIP]) and get going?

Thank you,
Ilia

welcome and please go ahead! (I don't think anyone is currently working on them)

[wyli]

today we discussed the requirement of FROC curves for lesion based evaluation

[Can-Zhao]

Please check https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/detection/metrics/coco.py

[TheHuntsman4]

Hey everyone,

If no one is working on this, I would love to take this up. I do have a few questions regarding this; coco.py seems to be working with boxes, while the element-level and outline/detection metrics are more fit to be worked with masks. Is this assumption fine?

I believe the Dice per element metric was recently added here #8774 I would be happy to make a PR for the Centroid distance and then eventually work through the other 6 until all are implemented.

Thaks,
Aniketh

[csudre]

Dear Aniketh

I think most of the requirements for allocation and localisation using centroid distance or box overlap... is available on the MetricsReloaded project linked to MONAI - the best way would probably be to make a wrapper around what is available there - in the association_localization file

Could that be an option?

Best wishes

Carole