YoursFunny/TelegramTwitterMediaBot
1
0
Fork 0
mirror of https://github.com/TheFunny/TelegramTwitterMediaBot.git synced 2026-10-05 01:12:06 +00:00
Code Releases Activity

fix(fetch): jitter the retry backoff and honor a429's Retry-After

Browse Source 
Two gaps in fetch_with_attempts: the sleep was the bare 1 << attempt, so every worker that failed together (a source coming back, a shared proxy blip) also recovered on the same tick and re-stamped the source; and a429 arrived as an anonymous Transient, its Retry-After header — the one place a source tells you exactly how long it wants silence — dropped on the floor.

retry_wait(attempt, roll, err) is the pure decision: doubling base plus a random slice of itself ([base, 2x base)), floored at a named Retry-After. status_error now takes the response, reads the seconds-form header and returns the new FetchError::RateLimited { site, retry_after_secs } (HTTP-date parses to None and stays transient); the pure table moved to classify_status so tests still build it from bare status codes. The delay is capped at MAX_RETRY_AFTER_SECS = 60 — the header is server-supplied and must not park one of the eight fetch slots.

Everywhere a Transient meant 'retryable' the new variant joins: the Site trait default, pixiv's override (plus its ugoira-zip mapping), bsky's HLS second attempt, the download classifier, and the user-facing message arm. Tests pin the429 rows (with and without the header, cap included) and retry_wait's math; AGENTS' retries bullet and variant list follow.
This commit is contained in:
YoursFunny
2026-09-24 15:35:55 +08:00
parent 053ae0ec25
commit d05450d88a
11 changed files with 132 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/x-media/src/site/bsky/interface.rs
+2 -2
View File
@@ -132,7 +132,7 @@ fn concat_list(files: &mut [(usize, std::path::PathBuf)]) -> String {
/// more, so the second attempt belongs on the request that actually failed.
async fn fetch_hls(url: &str, cap: u64) -> Result<bytes::Bytes, String> {
match crate::site::download_media_limited(url, cap, crate::site::DOWNLOAD_TOTAL_TIMEOUT).await {
Err(FetchError::Http(_) | FetchError::Transient(_)) => {
Err(FetchError::Http(_) | FetchError::Transient(_) | FetchError::RateLimited { .. }) => {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
crate::site::download_media_limited(url, cap, crate::site::DOWNLOAD_TOTAL_TIMEOUT)
.await
@@ -304,7 +304,7 @@ pub async fn fetch(handle: &str, rkey: &str) -> Result<Post, FetchError> {
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return Err(crate::site::status_error("bsky", status));
return Err(crate::site::status_error("bsky", &response));
}
let text = response.text().await?;
Post::from_json(&text, rkey.to_string())