fix(retry): stop losing posts to transient failures and broken promises

P0 of the retry audit. The main finding: a Telegram 5xx was classified
Permanent, so one Telegram-side blip dead-lettered the post.

- `classify_request_error`: a server error is retryable again. teloxide sleeps
  10s on a 5xx and then parses the body, so the HTTP status is gone by the
  time the error arrives; it is recognised by shape instead — a JSON
  server-error description, or an `InvalidJson` whose raw body is not JSON
  (a proxy/error page). A JSON body of the wrong shape stays permanent, since
  retrying a type mismatch cannot help. Reproduced end to end: with the old
  classification a fake 502 (HTML body) logged "failed permanently" and
  dead-lettered; now it logs "queued for retry" and the retry delivers.
- The same class of mistake elsewhere: `is_media_fetch_failure` was missing
  `failed to get HTTP url content`, the description single-media URL sends
  answer with, so hotlink-rejected media failed permanently instead of going
  through the reupload fallback.
- `enqueue_retry` now reports whether the row was written, and the callers
  only promise a retry when it was — a failed enqueue (DB write) used to tell
  the user "retrying in Ns" and then deliver nothing, ever.
- A forward that fails retryably now settles the prompt instead of leaving it
  live: the queued row carries the message ids itself, and a live prompt let
  a second Confirm copy the same messages to the channel twice and let Skip
  answer "nothing was forwarded" while the row still delivered.
- A prompt that could not be sent no longer swallows the gated forward
  silently: the chat is told, since nothing would ever forward.
- `scaled_retry_delay` only scales up, so a server-asked `retry_after` above
  the 300s cap is honoured instead of retried early (which earned another 429
  and then dead-lettered the post).
- Download classification: a 4xx media download is permanent (the media is
  gone or refused) while transport errors and 429/5xx retry — previously every
  download error counted as retryable and burned the whole budget. A temp-file
  *write* failure retries too (resource exhaustion clears; a temp dir that
  cannot be created stays permanent).
- Site status mapping: 401/403 are `Blocked` (permanent) rather than
  `Transient`, so a refusal is reported at once instead of after three
  wasted attempts; and a twitter 200 that is not a tweet is no longer
  reported as withheld content (the empty `{}` withheld shape keeps
  `Sensitive`, which is what triggers the auth fallback).
This commit is contained in:
2026-09-20 20:46:18 +08:00
parent 36e5e8afe6
commit 4cf793cd7e
13 changed files with 312 additions and 60 deletions
+45 -8
View File
@@ -117,9 +117,24 @@ async fn handle_callback(
delay_seconds,
task,
}) => {
log::info!("forward queued for retry in {delay_seconds:.1}s");
send::enqueue_retry(ctx.task_queue, *task, delay_seconds).await;
("Forward queued for retry.".to_string(), false)
// The queued row owns the forward from here (it carries
// the message ids itself), so the prompt is settled
// either way: leaving it live let a second Confirm copy
// the same messages to the channel twice, and let Skip
// answer "nothing was forwarded" while the row still
// delivered it.
let queued =
send::enqueue_retry(ctx.task_queue, &task, delay_seconds).await;
if queued {
log::info!("forward queued for retry in {delay_seconds:.1}s");
("Forward queued for retry.".to_string(), true)
} else {
log::error!("forward retry could not be queued");
(
"Forward failed and the retry could not be queued.".to_string(),
true,
)
}
}
Err(send::SendError::Permanent { message, .. }) => {
log::error!("forward failed permanently: {message}");
@@ -314,7 +329,7 @@ mod tests {
}
#[tokio::test]
async fn retryable_forward_is_queued_and_keeps_the_prompt() {
async fn retryable_forward_is_queued_and_settles_the_prompt() {
use teloxide::types::Seconds;
let sender = MockSender::scripted(vec![Outcome::CopyErr], || {
RequestError::RetryAfter(Seconds::from_seconds(7))
@@ -325,23 +340,45 @@ mod tests {
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
// The queued row carries the message ids itself, so it owns the
// forward from here and the prompt is closed with it. Keeping it live
// (the old behaviour) let a second Confirm copy the same messages to
// the channel twice, and let Skip answer "nothing was forwarded" while
// the row still delivered it.
assert_eq!(
sender.calls(),
vec!["copy_messages", "answer_callback_query"]
vec!["copy_messages", "delete_message", "answer_callback_query"]
);
assert_eq!(
sender.answers(),
vec![Some("Forward queued for retry.".to_string())]
);
assert_eq!(stores.queued_tasks().await, 1);
// The prompt is not settled: the queued retry still needs the record.
assert!(
ctx.chat_store
!ctx.chat_store
.get(1)
.await
.edit_message
.contains_key(&PROMPT_ID)
.contains_key(&PROMPT_ID),
"the record must be dropped so the prompt cannot be used again"
);
// A second tap finds no record: it cannot enqueue a duplicate copy.
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(
sender.calls(),
vec![
"copy_messages",
"delete_message",
"answer_callback_query",
"answer_callback_query"
]
);
assert_eq!(
sender.answers().last().map(|a| a.as_deref()),
Some(Some("Expired"))
);
assert_eq!(stores.queued_tasks().await, 1, "no second forward row");
}
#[tokio::test]
+13 -9
View File
@@ -210,19 +210,23 @@ async fn dispatch_send(
"send for [key={}] chat={chat_id} failed after {ms}ms, queued for retry in {delay_seconds:.1}s",
log_key(url)
);
send::enqueue_retry(ctx.task_queue, *task, delay_seconds).await;
// Name the post and the wait: "queued for retry" alone left the
// user guessing which link it was and how long the wait is.
let _ = reply(
ctx.sender,
chat_id,
reply_to,
// user guessing which link it was and how long the wait is. The
// promise is made only when the retry was really persisted — an
// enqueue that failed (DB write) would leave the user waiting for
// a retry nothing can deliver.
let promised = if send::enqueue_retry(ctx.task_queue, &task, delay_seconds).await {
format!(
"Send failed for {} — retrying in {delay_seconds:.0}s.",
log_key(url)
),
)
.await;
)
} else {
format!(
"Send failed for {} and the retry could not be queued — please send the link again.",
log_key(url)
)
};
let _ = reply(ctx.sender, chat_id, reply_to, promised).await;
}
Err(send::SendError::Permanent {
message: err_message,