Metrics

Hyperparameter optimization and model evaluation

Which metric should be maximized?

By default, the confidence score threshold is set to maximize the F1 score, which is a weighted average of the precision and recall. Depending on your use case, you may want to optimize for either recall or precision:

  • Maximizing the recall will train the model to predict marginally likely objects, which is useful in cases where your classes of interest are under-represented in your training data.
  • Maximizing the precision will predict only the most likely objects, which is useful in cases where your classes of interest are well-represented in your training data.
Intersection Over Union (IOU)

Intersection Over Union (IOU) is a measure of the overlap between the predicted bounding box and the ground truth label with the same/correct class.

By comparing the IOU to the threshold*, we can tell if a detection is correct (True Positive) or wrong (False Positive).

IOU = 98% IOU = 40% IOU = 0%
≥ threshold* < threshold*
True Positive False Positive False Negative
*IOU threshold is usually set to 50%, 75% or 95%, and can be set when selecting the metric.

Average Precision is a useful metric for model accuracy. It represents the area under the precision-recall curve and measures how well your model performs across all score thresholds. The closer to 1.0 this score is, the better your model is performing on the test set; a model guessing at random for each label would get an average precision around 0.5.