How do you choose a chunk size for a RAG system?

Choosing a chunk size based on ‘what everyone else uses’ (e.g., 512 tokens) is unfalsifiable and risky. Instead, identify the atomic unit of your corpus that must remain intact for the downstream task. Use the severance arithmetic (s - 1)/L and ensure the overlap ‘o’ is at least as large as the chunk size ‘s’ to maintain context, while considering the cost of the resulting index expansion.