mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-10-09 01:52:22 +00:00
fix(retry): fence the queue lease, clean up after a kill, name dead-lettered posts
P2 (hardening) of the retry audit, closing the report's remaining findings. - Lease fencing. `lease_next` now stamps a random `lease_token`, and every write-back a worker makes (the 30s heartbeat, `delete_row`, `reschedule`, `mark_done`) is guarded by it. A lease that expired while its holder was stalled and was then re-leased used to let *both* holders write the same row: one duplicated the send, the other silently discarded the new holder's retry (a 0-row update was not even logged). Now a worker that no longer holds the lease drops its attempt at the next heartbeat and writes nothing. Reaching existing databases needed a migration chain, which `db.rs` had been pre-committed to: `MIGRATIONS` + `migrate` track `PRAGMA user_version`, with `schema_init` as the version-0 baseline. Verified on a database created before this change: user_version 0 -> 1, column added, rows intact. - Dead-letter notifications no longer mislabel an unparsable payload. A row whose payload no longer deserializes as a `Task` (an older version's shape, corruption) used to skip the cache invalidation *and* report "Forward failed permanently" for a send task, because both were derived from the parsed value. The identity now comes off the raw JSON, so the stale link-cache entry is dropped and the message names the post. - Temp files are marked and swept. Every temp file/dir the project creates now carries `x_media::TEMP_FILE_PREFIX`, and startup removes entries with that prefix older than an hour — a killed process leaves its downloads (up to hundreds of MB) behind because no destructor runs, and the age gate keeps the sweep away from a second instance's in-flight files. Verified live: the log reports the sweep, an aged leftover goes, a fresh prefixed file and an unrelated file stay.
This commit is contained in:
@@ -321,7 +321,7 @@ pub(crate) async fn post_send_actions(ctx: &AppContext<'_>, task: &Task, message
|
||||
ctx.sender,
|
||||
notify_chat_id,
|
||||
notify_message_id,
|
||||
&failure_text(Some(&task), "retry could not be queued"),
|
||||
&failure_text(task.source_url(), "retry could not be queued"),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
@@ -435,15 +435,34 @@ async fn send_media_or_animation(ctx: &AppContext<'_>, task: &Task) -> Result<Ve
|
||||
|
||||
/// User-facing text for a task that will never run again: which link died and
|
||||
/// why. The raw error alone left the user guessing which post it was about.
|
||||
pub(super) fn failure_text(task: Option<&Task>, message: &str) -> String {
|
||||
match task.and_then(|task| task.source_url()).map(log_key) {
|
||||
pub(super) fn failure_text(source_url: Option<&str>, message: &str) -> String {
|
||||
match source_url.map(log_key) {
|
||||
Some(key) => format!("Send failed permanently for {key}: {message}"),
|
||||
// `ForwardMessages` carries no source URL: that failure is about the
|
||||
// channel copy, not about a post.
|
||||
// `ForwardMessages` carries no source URL (and neither does an
|
||||
// unparsable payload): that failure is about the channel copy, not
|
||||
// about a post.
|
||||
None => format!("Forward failed permanently: {message}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The post a stored payload is about, without parsing it into a [`Task`]:
|
||||
/// used when the payload no longer deserializes (written by an older version,
|
||||
/// or corrupted) but its identity fields are still readable.
|
||||
fn payload_source_url(payload: &serde_json::Value) -> Option<&str> {
|
||||
payload.get("source_url").and_then(|v| v.as_str())
|
||||
}
|
||||
|
||||
/// Whether a stored payload was a *cached* send (see `Task::is_cached_send`),
|
||||
/// read straight off the JSON — the unparsable case still has to know whether
|
||||
/// a link-cache entry may be holding the media that failed.
|
||||
fn payload_is_cached_send(payload: &serde_json::Value) -> bool {
|
||||
payload
|
||||
.get("cache_data")
|
||||
.and_then(|data| data.get("media"))
|
||||
.and_then(|media| media.as_array())
|
||||
.is_some_and(|media| !media.is_empty())
|
||||
}
|
||||
|
||||
/// Dead-letter callback wired to the queue in main: settles the task and
|
||||
/// notifies its chat.
|
||||
pub(crate) async fn dead_letter_notify(
|
||||
@@ -457,6 +476,18 @@ pub(crate) async fn dead_letter_notify(
|
||||
let task = serde_json::from_value::<Task>(payload.clone()).ok();
|
||||
if let Some(task) = &task {
|
||||
settle_task(ctx, task, Settled::Failed).await;
|
||||
} else {
|
||||
// A payload that no longer parses (an older version's row shape, a
|
||||
// corrupted one) still says which post it was about: drop the stale
|
||||
// cache entry the same way, instead of leaving a bad file id to be
|
||||
// re-sent forever — and name the post in the notification rather than
|
||||
// reporting a *forward* failure for a send task.
|
||||
if payload_is_cached_send(&payload)
|
||||
&& let Some(key) = payload_source_url(&payload).and_then(x_media::site::cache_key)
|
||||
{
|
||||
log::debug!("removing stale link cache entry for [key={key}]");
|
||||
ctx.link_cache.remove(&key).await;
|
||||
}
|
||||
}
|
||||
let notify_chat_id = payload.get("notify_chat_id").and_then(|v| v.as_i64());
|
||||
let notify_message_id = payload.get("notify_message_id").and_then(|v| v.as_i64());
|
||||
@@ -464,7 +495,12 @@ pub(crate) async fn dead_letter_notify(
|
||||
ctx.sender,
|
||||
notify_chat_id,
|
||||
notify_message_id,
|
||||
&failure_text(task.as_ref(), &message),
|
||||
&failure_text(
|
||||
task.as_ref()
|
||||
.and_then(|task| task.source_url())
|
||||
.or_else(|| payload_source_url(&payload)),
|
||||
&message,
|
||||
),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user