mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-10-08 01:42:05 +00:00
fix(fetch): jitter the retry backoff and honor a429's Retry-After
Two gaps in fetch_with_attempts: the sleep was the bare 1 << attempt, so every worker that failed together (a source coming back, a shared proxy blip) also recovered on the same tick and re-stamped the source; and a429 arrived as an anonymous Transient, its Retry-After header — the one place a source tells you exactly how long it wants silence — dropped on the floor.
retry_wait(attempt, roll, err) is the pure decision: doubling base plus a random slice of itself ([base, 2x base)), floored at a named Retry-After. status_error now takes the response, reads the seconds-form header and returns the new FetchError::RateLimited { site, retry_after_secs } (HTTP-date parses to None and stays transient); the pure table moved to classify_status so tests still build it from bare status codes. The delay is capped at MAX_RETRY_AFTER_SECS = 60 — the header is server-supplied and must not park one of the eight fetch slots.
Everywhere a Transient meant 'retryable' the new variant joins: the Site trait default, pixiv's override (plus its ugoira-zip mapping), bsky's HLS second attempt, the download classifier, and the user-facing message arm. Tests pin the429 rows (with and without the header, cap included) and retry_wait's math; AGENTS' retries bullet and variant list follow.
This commit is contained in:
@@ -91,7 +91,7 @@ pub fn cache_key(url: &str) -> Option<String> {
|
||||
/// API/auth errors, unparseable bodies and missing auth are not retried.
|
||||
pub fn is_retryable(err: &FetchError) -> bool {
|
||||
match err {
|
||||
FetchError::Http(_) | FetchError::Transient(_) => true,
|
||||
FetchError::Http(_) | FetchError::Transient(_) | FetchError::RateLimited { .. } => true,
|
||||
FetchError::Pixiv(e) => pixiv_error_is_retryable(e),
|
||||
_ => false,
|
||||
}
|
||||
@@ -464,6 +464,10 @@ mod tests {
|
||||
// Transient: network errors, explicit transient, pixiv 429/5xx, and a
|
||||
// failed media download (the frame zip's own bad moment).
|
||||
assert!(is_retryable(&FetchError::Transient("429".into())));
|
||||
assert!(is_retryable(&FetchError::RateLimited {
|
||||
site: "pixiv",
|
||||
retry_after_secs: 30
|
||||
}));
|
||||
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(429))));
|
||||
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(500))));
|
||||
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(503))));
|
||||
|
||||
Reference in New Issue
Block a user