fix(retry): fence the queue lease, clean up after a kill, name dead-lettered posts

P2 (hardening) of the retry audit, closing the report's remaining findings.

- Lease fencing. `lease_next` now stamps a random `lease_token`, and every
  write-back a worker makes (the 30s heartbeat, `delete_row`, `reschedule`,
  `mark_done`) is guarded by it. A lease that expired while its holder was
  stalled and was then re-leased used to let *both* holders write the same row:
  one duplicated the send, the other silently discarded the new holder's retry
  (a 0-row update was not even logged). Now a worker that no longer holds the
  lease drops its attempt at the next heartbeat and writes nothing. Reaching
  existing databases needed a migration chain, which `db.rs` had been
  pre-committed to: `MIGRATIONS` + `migrate` track `PRAGMA user_version`, with
  `schema_init` as the version-0 baseline. Verified on a database created
  before this change: user_version 0 -> 1, column added, rows intact.
- Dead-letter notifications no longer mislabel an unparsable payload. A row
  whose payload no longer deserializes as a `Task` (an older version's shape,
  corruption) used to skip the cache invalidation *and* report "Forward failed
  permanently" for a send task, because both were derived from the parsed
  value. The identity now comes off the raw JSON, so the stale link-cache entry
  is dropped and the message names the post.
- Temp files are marked and swept. Every temp file/dir the project creates now
  carries `x_media::TEMP_FILE_PREFIX`, and startup removes entries with that
  prefix older than an hour — a killed process leaves its downloads (up to
  hundreds of MB) behind because no destructor runs, and the age gate keeps the
  sweep away from a second instance's in-flight files. Verified live: the log
  reports the sweep, an aged leftover goes, a fresh prefixed file and an
  unrelated file stay.
This commit is contained in:
2026-09-20 21:34:45 +08:00
parent 0a82ca5a42
commit 4cdf618c25
11 changed files with 363 additions and 58 deletions
+42 -6
View File
@@ -321,7 +321,7 @@ pub(crate) async fn post_send_actions(ctx: &AppContext<'_>, task: &Task, message
ctx.sender,
notify_chat_id,
notify_message_id,
&failure_text(Some(&task), "retry could not be queued"),
&failure_text(task.source_url(), "retry could not be queued"),
)
.await;
}
@@ -435,15 +435,34 @@ async fn send_media_or_animation(ctx: &AppContext<'_>, task: &Task) -> Result<Ve
/// User-facing text for a task that will never run again: which link died and
/// why. The raw error alone left the user guessing which post it was about.
pub(super) fn failure_text(task: Option<&Task>, message: &str) -> String {
match task.and_then(|task| task.source_url()).map(log_key) {
pub(super) fn failure_text(source_url: Option<&str>, message: &str) -> String {
match source_url.map(log_key) {
Some(key) => format!("Send failed permanently for {key}: {message}"),
// `ForwardMessages` carries no source URL: that failure is about the
// channel copy, not about a post.
// `ForwardMessages` carries no source URL (and neither does an
// unparsable payload): that failure is about the channel copy, not
// about a post.
None => format!("Forward failed permanently: {message}"),
}
}
/// The post a stored payload is about, without parsing it into a [`Task`]:
/// used when the payload no longer deserializes (written by an older version,
/// or corrupted) but its identity fields are still readable.
fn payload_source_url(payload: &serde_json::Value) -> Option<&str> {
payload.get("source_url").and_then(|v| v.as_str())
}
/// Whether a stored payload was a *cached* send (see `Task::is_cached_send`),
/// read straight off the JSON — the unparsable case still has to know whether
/// a link-cache entry may be holding the media that failed.
fn payload_is_cached_send(payload: &serde_json::Value) -> bool {
payload
.get("cache_data")
.and_then(|data| data.get("media"))
.and_then(|media| media.as_array())
.is_some_and(|media| !media.is_empty())
}
/// Dead-letter callback wired to the queue in main: settles the task and
/// notifies its chat.
pub(crate) async fn dead_letter_notify(
@@ -457,6 +476,18 @@ pub(crate) async fn dead_letter_notify(
let task = serde_json::from_value::<Task>(payload.clone()).ok();
if let Some(task) = &task {
settle_task(ctx, task, Settled::Failed).await;
} else {
// A payload that no longer parses (an older version's row shape, a
// corrupted one) still says which post it was about: drop the stale
// cache entry the same way, instead of leaving a bad file id to be
// re-sent forever — and name the post in the notification rather than
// reporting a *forward* failure for a send task.
if payload_is_cached_send(&payload)
&& let Some(key) = payload_source_url(&payload).and_then(x_media::site::cache_key)
{
log::debug!("removing stale link cache entry for [key={key}]");
ctx.link_cache.remove(&key).await;
}
}
let notify_chat_id = payload.get("notify_chat_id").and_then(|v| v.as_i64());
let notify_message_id = payload.get("notify_message_id").and_then(|v| v.as_i64());
@@ -464,7 +495,12 @@ pub(crate) async fn dead_letter_notify(
ctx.sender,
notify_chat_id,
notify_message_id,
&failure_text(task.as_ref(), &message),
&failure_text(
task.as_ref()
.and_then(|task| task.source_url())
.or_else(|| payload_source_url(&payload)),
&message,
),
)
.await;
}