mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-10-01 00:32:05 +00:00
fix(retry): let slow downloads finish, and never re-run a finished task
P1 of the retry audit, from the report's "reliability and diagnosis" batch. - Media downloads no longer share the 30s *total* timeout of metadata fetches. The size caps allowed 10 MiB (reupload fallback) and 512 MiB (ugoira frame zip) while the clock allowed 30s, so a slow link made those posts impossible: `MEDIA_CLIENT` has no total timeout and instead bounds the response head and every chunk with a 30s *idle* window, which keeps the stalled-connection protection. Verified against a local probe: the old policy aborts a 40s download at 30.0s, the new one completes it (2 MiB, 40.1s), and a body that stops delivering still fails after exactly 30s. - A finished row's write-back is no longer best-effort. `delete_row` failing left the row `in_progress` with a live lease, so the next sweep flipped it back to `pending` and re-ran a completed task — a second album, a second prompt, a second channel copy. Both terminal writes are now retried, and a delete that still fails falls back to a `done` tombstone that neither the lease query nor the sweep looks at; reschedule (no safe tombstone: marking it done would drop the retry silently) logs what the sweep will do. - bsky and pixiv no longer present a *failed* video conversion as a post with no media: the remux/ugoira error propagates (pixiv keeps its retry class, bsky reports Transient), so the user sees the real cause and `fetch` gets its retries. bsky's "no ffmpeg" case stays a degradation — retrying a deployment gap cannot help. - pixiv's token exchange checks the HTTP status before parsing the body, so a 429/5xx from the OAuth endpoint stays retryable instead of becoming a permanent Api/Json error (via the shared `pixiv_error_is_retryable`), and startup validation only disables pixiv for a rejected credential — one 503 while the container came up used to turn every later pixiv link into "pixiv support is disabled".
This commit is contained in:
@@ -50,8 +50,14 @@ impl Site for PixivSite {
|
||||
match super::api::validate().await {
|
||||
Ok(()) => Ok(()),
|
||||
Err(e) => {
|
||||
// Keep the old behavior: a failed login disables pixiv
|
||||
// for the rest of this process.
|
||||
// A rejected credential disables pixiv for the rest of
|
||||
// this process (it will not fix itself). A bad *moment* —
|
||||
// a 5xx or a network error while the container comes up —
|
||||
// must not: disabling on any error turned every later
|
||||
// pixiv link into "pixiv support is disabled".
|
||||
if pixiv_error_is_retryable(&e) {
|
||||
return Err(format!("{e} (transient — pixiv stays enabled)"));
|
||||
}
|
||||
super::api::disable();
|
||||
Err(format!("{e}"))
|
||||
}
|
||||
@@ -84,18 +90,24 @@ pub fn cache_key(url: &str) -> Option<String> {
|
||||
pub fn is_retryable(err: &FetchError) -> bool {
|
||||
match err {
|
||||
FetchError::Http(_) | FetchError::Transient(_) => true,
|
||||
FetchError::Pixiv(e) => match e {
|
||||
PixivError::Http(_) => true,
|
||||
PixivError::Status(code) if *code == 429 || *code >= 500 => true,
|
||||
PixivError::Status(_)
|
||||
| PixivError::Api(_)
|
||||
| PixivError::Json(_)
|
||||
| PixivError::NoAuth => false,
|
||||
},
|
||||
FetchError::Pixiv(e) => pixiv_error_is_retryable(e),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// The pixiv-specific half of the retry policy, shared with startup
|
||||
/// validation: a bad moment (429/5xx, a network error) is retryable, a
|
||||
/// rejected credential is not.
|
||||
fn pixiv_error_is_retryable(err: &PixivError) -> bool {
|
||||
match err {
|
||||
PixivError::Http(_) => true,
|
||||
PixivError::Status(code) if *code == 429 || *code >= 500 => true,
|
||||
PixivError::Status(_) | PixivError::Api(_) | PixivError::Json(_) | PixivError::NoAuth => {
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// pximg.net is hotlink-protected: downloads must carry the pixiv Referer.
|
||||
/// The match is on the media host, not the site PATTERN — pixiv's PATTERN
|
||||
/// only matches `pixiv.net/artworks/...`, never `i.pximg.net`.
|
||||
@@ -410,6 +422,16 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn startup_validation_only_disables_on_a_definitive_failure() {
|
||||
// A bad moment: the site must stay enabled for later links.
|
||||
assert!(pixiv_error_is_retryable(&PixivError::Status(503)));
|
||||
assert!(pixiv_error_is_retryable(&PixivError::Status(429)));
|
||||
// A rejected credential is what `disable()` is for.
|
||||
assert!(!pixiv_error_is_retryable(&PixivError::Status(403)));
|
||||
assert!(!pixiv_error_is_retryable(&PixivError::NoAuth));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_retryable_classifies_transient_and_permanent() {
|
||||
// Transient: network errors, explicit transient, pixiv 429/5xx.
|
||||
|
||||
Reference in New Issue
Block a user