Index legacy archive adoption candidates
Replaced quadratic legacy-message matching with per-pass indexes keyed by source row and raw event identity while preserving archive order and claim semantics.
openclaw/wacrawl · #115
Migration performance fix
First-time WhatsApp archive adoption now indexes legacy candidates once instead of repeatedly scanning and re-encoding every archived message for each unmatched source message.
Problem
Validation and merge could each rescan and re-encode an entire legacy archive for every unmatched WhatsApp Desktop message. A reported 384,000-message upgrade ran for more than three hours, consumed substantial CPU and memory, and left the migrated archive unbound when interrupted.
Approach
Builds bounded candidate indexes once per pass, preserves matching precedence and archive order, and excludes events already claimed by the current store. Account checks, event identities, transaction boundaries, and repeat-import behavior remain unchanged.
Impact and scope
- Removes the quadratic matching path that made large first-time archive adoption appear hung.
- Preserves duplicate selection, already-claimed events, unmatched history, account isolation, and idempotent repeat imports.
- Improves the upgrade path without changing schema migration or transaction boundaries, and documents retry and rollback behavior.
- Turns a source-level scalability bottleneck into near-linear indexed matching while retaining the existing archive semantics.
Validation
- A built-CLI proof with 6,000 legacy and 6,000 unmatched Desktop messages exceeded 30 seconds on the baseline and completed in 1.502 seconds with the fix, retaining all 12,000 rows and passing SQLite integrity checks.
- Matching benchmarks for 500, 1,000, and 2,000 unmatched messages completed in 0.0064, 0.0132, and 0.0291 seconds; the reporter's private 384,000-message archive was not replayed.
- Full make check, race tests, the 85% coverage gate, analysis, dependency and vulnerability checks, six-platform packaging, release tests, secret scans, macOS tests, and CodeQL passed; the maintainer publicly credited the fix.