fix(retry): fence the queue lease, clean up after a kill, name dead-lettered posts

P2 (hardening) of the retry audit, closing the report's remaining findings.

- Lease fencing. `lease_next` now stamps a random `lease_token`, and every
  write-back a worker makes (the 30s heartbeat, `delete_row`, `reschedule`,
  `mark_done`) is guarded by it. A lease that expired while its holder was
  stalled and was then re-leased used to let *both* holders write the same row:
  one duplicated the send, the other silently discarded the new holder's retry
  (a 0-row update was not even logged). Now a worker that no longer holds the
  lease drops its attempt at the next heartbeat and writes nothing. Reaching
  existing databases needed a migration chain, which `db.rs` had been
  pre-committed to: `MIGRATIONS` + `migrate` track `PRAGMA user_version`, with
  `schema_init` as the version-0 baseline. Verified on a database created
  before this change: user_version 0 -> 1, column added, rows intact.
- Dead-letter notifications no longer mislabel an unparsable payload. A row
  whose payload no longer deserializes as a `Task` (an older version's shape,
  corruption) used to skip the cache invalidation *and* report "Forward failed
  permanently" for a send task, because both were derived from the parsed
  value. The identity now comes off the raw JSON, so the stale link-cache entry
  is dropped and the message names the post.
- Temp files are marked and swept. Every temp file/dir the project creates now
  carries `x_media::TEMP_FILE_PREFIX`, and startup removes entries with that
  prefix older than an hour — a killed process leaves its downloads (up to
  hundreds of MB) behind because no destructor runs, and the age gate keeps the
  sweep away from a second instance's in-flight files. Verified live: the log
  reports the sweep, an aged leftover goes, a fresh prefixed file and an
  unrelated file stay.
This commit is contained in:
2026-09-20 21:34:45 +08:00
parent 0a82ca5a42
commit 4cdf618c25
11 changed files with 363 additions and 58 deletions
+35 -5
View File
@@ -124,9 +124,39 @@ pub fn open_store(path: &str) -> rusqlite::Result<Arc<DbPool>> {
}
let conn = open_db(path)?;
schema_init(&conn)?;
migrate(&conn)?;
Ok(Arc::new(DbPool::new(path)))
}
/// Schema migrations, applied in order and tracked by `PRAGMA user_version`
/// (the index in this array + 1 is the version a statement brings the
/// database to). Append only — never edit or reorder an entry, or databases
/// already past it would skip or repeat work.
const MIGRATIONS: &[&str] = &[
// 1: lease fencing. A worker's write-backs (`delete`/`reschedule`/the
// lease heartbeat) are guarded by the token it was leased with, so a
// lease that expired and was re-leased by another worker can no longer be
// written by its former holder — which used to duplicate a send or drop
// the new holder's retry state, silently.
"ALTER TABLE tasks ADD COLUMN lease_token TEXT",
];
/// Brings an existing database up to [`MIGRATIONS`]. Idempotent: a database
/// already at the latest version does no work.
fn migrate(conn: &Connection) -> rusqlite::Result<()> {
let version: i64 = conn.query_row("PRAGMA user_version", [], |row| row.get(0))?;
for (index, statement) in MIGRATIONS.iter().enumerate() {
let target = index as i64 + 1;
if version >= target {
continue;
}
conn.execute_batch(statement)?;
// `PRAGMA` does not take bind parameters; the value is our own index.
conn.execute_batch(&format!("PRAGMA user_version = {target}"))?;
}
Ok(())
}
fn rusqlite_error(e: std::io::Error) -> rusqlite::Error {
rusqlite::Error::ToSqlConversionFailure(Box::new(e))
}
@@ -135,11 +165,11 @@ fn rusqlite_error(e: std::io::Error) -> rusqlite::Error {
/// The three stores used to own their own schema; keeping it in one place
/// means one initialization for the whole database file.
///
/// ⚠️ Schema-change reminder (deferred, see `docs/architecture-refactor.md`
/// §5): this is a plain `CREATE TABLE IF NOT EXISTS` with no versioning.
/// Before any column/table change that must migrate existing databases, land
/// the `PRAGMA user_version` migration chain first (`MIGRATIONS: &[&str]` +
/// `migrate(conn)`), then restructure this function.
/// This is the **baseline** schema (version 0): a fresh database is created
/// exactly like this, and anything that must *change* an existing one is
/// appended to [`MIGRATIONS`] instead of being edited in here — otherwise a
/// database created before the change would never gain the new column and a
/// freshly created one would try to apply the migration a second time.
pub fn schema_init(conn: &Connection) -> rusqlite::Result<()> {
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \