← All questions
Data, Privacy & LegalHard· Data erasure· GDPR· System architecture

Legal now requires erasure within 24 hours, and your re...

The proposed solution of a nightly rebuild is fundamentally broken due to the scaling of your corpus. If the rebuild time scales linearly with the corpus size, a 50-fold increase in data will push the rebuild time to 50 hours, which exceeds the 24-hour legal requirement and the platform’s desired cadence.

Because a whole-index rebuild is no longer feasible under these constraints, you must shift your architecture. Instead of rebuilding the entire index to remove specific records, you should implement a mechanism that handles erasure as a per-record obligation, allowing for targeted deletions or updates without requiring a full re-indexing of the entire 500-million-record corpus.

Code playground (Python)