How would you tell whether a new chunker is better than...

A weak approach relies on end-to-end evaluation, which fails to isolate the chunker’s impact and is often obscured by noise in small query sets. A strong approach moves measurement to token offsets. You should annotate gold excerpts independently of the chunking process and report token recall, precision, and IoU. It is critical to note that recall alone is insufficient because it can be artificially inflated by increasing chunk size. When using IoU, remember that it is related to F1 via the formula IoU = F1/(2 − F1), providing a harsher, more accurate scale for evaluation.