← All questions
EvaluationHard· Metrics· Copyright· Audit

Our audit reports mean token overlap with copyrighted w...

The mean is an inappropriate metric because, under lock-on, the distribution is bimodal; the mean is largely determined by generations that never locked on and is independent of the tail that exhibits memorization. You should instead request the maximum run at production k over prefixes from named works. If the maximum is too noisy, use a high-percentile variant as the tracked metric, while keeping the maximum as the gate.

Code playground (Python)