Engineering for Annotation in the ML Pipeline

Introduction

Early collaboration between R&D and annotation saves time, reduces rework, and improves data quality.

Data annotation is the bedrock of machine learning, and in medical AI it needs to meet both scientific and regulatory expectations. Since annotations are made by people, they will always include some uncertainty. If we do not measure that uncertainty, we can overestimate model performance and miss inconsistencies between annotators.

In this part, we show how protocol-driven estimates of human inaccuracy in regression-style labels (like box coordinates) can be turned into simple, powerful statistical tests—tests that validate the dataset and reliably flag when the annotation process is drifting or inconsistent across annotators.

To make the discussion concrete, we will use a simple (and very common) annotation task: axis-aligned bounding boxes. For each image (or frame), an annotator draws a rectangle around a target object (e.g., a lesion, device, anatomical structure), with box edges parallel to the image axes. The label is typically stored as four numbers, for example

\[ (x_{\min}, y_{\min}, x_{\max}, y_{\max}), \]

with a guideline like: “include the full object, avoid background when possible, and be consistent about border cases.”

In this post, we will cover why engineers should work closely with the annotation team early on, and how engineering can help make the annotation process more reliable.

What Does the R&D Team Expect?

The best kind of errors are the ones that look like random noise.

Perfect agreement between annotations and “ground truth” is rarely realistic. The next best thing is that the remaining differences behave like random noise. In statistics, the ideal case is independent and identically distributed (IID) errors.

IID errors do not show patterns or correlations. If we see structure in the mistakes, it is a sign of something real: annotator bias, unclear guidelines, mismatched workflows, or even important signal the model is not capturing yet.

In bounding boxes, “structure” often looks like consistent geometry: always drawing boxes too loose (systematic padding), always shifting boxes in one direction (systematic offset), or consistently snapping to convenient landmarks. These are usually protocol problems, not model problems, and they are worth detecting early.

That is why these expectations should be set early, so we can design annotation protocols that are robust, measurable, and easy to validate.

How Can the R&D Team Help?

Simple checks that catch annotation artifacts early.

A common source of misalignment is that one annotator does not follow the labeling guidelines (or follows a different interpretation). To catch this early, we can ask a simple question: “Does annotator $A$ behave differently than the consensus?”

For bounding boxes, a practical way to define a “consensus” is to aggregate annotators per image (e.g., coordinate-wise median of $(x_{\min},y_{\min},x_{\max},y_{\max})$). Then we can look at how each annotator deviates from that consensus across many images.

If the protocol is well designed, we can often treat annotation errors as independent and identically distributed (IID) and approximately Normal. That assumption lets us use basic hypothesis testing to turn the question above into something we can measure and monitor.

In practice, we rely on the assumptions implied by the annotation protocol. For annotator $A$, we examine the residual between their annotation $(X_A, Y_A)$ and the team consensus $(X_C, Y_C)$:

\[ \varepsilon_A = (X_A, Y_A) – (X_C, Y_C). \]

(For bounding boxes, $(X,Y)$ can be the four box coordinates, or any equivalent parameterization such as center/size.)

Under a well-aligned protocol, we expect these residuals to behave like zero-mean Gaussian noise,

\[ \varepsilon_A \sim \mathcal{N}(0, \sigma I). \]

We can then test the null hypothesis, denoted by $H_0$, that annotator $A$ does not systematically diverge from the consensus. Using a test statistic,

\[ T_A = \frac{\|\varepsilon_A\|^2}{\sigma^2}, \]

we compute a $p$-value

\[ p = \Pr\!\big(T \ge T_A \,\big|\, H_0\big), \]

where $T$ is the corresponding random variable under $H_0$. A small $p$-value means that $A$’s residual is unlikely to have occurred by chance under the assumed noise model, so we reject $H_0$ and conclude that annotator $A$ is likely misaligned with the rest of the team.

Minimal working example (synthetic data + p-value, SciPy)

import numpy as np
from scipy.stats import chi2

rng = np.random.default_rng(0)
N, sigma = 200, 2.0

consensus = rng.normal([50, 50, 150, 150], [3, 3, 3, 3], size=(N, 4))
bias = np.array([-5., -5., +5., +5.])  # systematic padding (too loose)
ann_A = consensus + bias + rng.normal(0, sigma, size=(N, 4))

res = ann_A - consensus
T = np.sum((res / sigma) ** 2)
df = 4 * N
p = chi2.sf(T, df)  # survival function = P(Chi2 >= T)

print(f"T={T:.1f}, df={df}, p={p:.3g}")