Commit Graph
9 Commits
Author SHA1 Message Date
YoursFunny 4e0f1b834e fix: restrict media download hosts 2026-09-25 01:40:09 +08:00
YoursFunny 7b4e273533 fix: bound site JSON response bodies 2026-09-24 21:25:03 +08:00
YoursFunny d05450d88a fix(fetch): jitter the retry backoff and honor a429's Retry-After
Two gaps in fetch_with_attempts: the sleep was the bare 1 << attempt, so every worker that failed together (a source coming back, a shared proxy blip) also recovered on the same tick and re-stamped the source; and a429 arrived as an anonymous Transient, its Retry-After header — the one place a source tells you exactly how long it wants silence — dropped on the floor.

retry_wait(attempt, roll, err) is the pure decision: doubling base plus a random slice of itself ([base, 2x base)), floored at a named Retry-After. status_error now takes the response, reads the seconds-form header and returns the new FetchError::RateLimited { site, retry_after_secs } (HTTP-date parses to None and stays transient); the pure table moved to classify_status so tests still build it from bare status codes. The delay is capped at MAX_RETRY_AFTER_SECS = 60 — the header is server-supplied and must not park one of the eight fetch slots.

Everywhere a Transient meant 'retryable' the new variant joins: the Site trait default, pixiv's override (plus its ugoira-zip mapping), bsky's HLS second attempt, the download classifier, and the user-facing message arm. Tests pin the429 rows (with and without the header, cap included) and retry_wait's math; AGENTS' retries bullet and variant list follow.
2026-09-24 15:35:55 +08:00
YoursFunny 2b6cb506c7 refactor(download): fold apply_media_headers into media_request
apply_media_headers had exactly one caller — media_request, the documented 'one choke point' every download already goes through — so the site-header walk sits inside that choke point now instead of one indirection past it, and the two doc comments merge into the one that describes both rules (guard + headers).
2026-09-24 04:37:54 +08:00
YoursFunny a49bb1d1af test(x-media): gate the pixiv download test as live-network too
The pixiv download test was token-gated but not #[ignore]d, so a shell with PIXIV_REFRESH_TOKEN exported made every local cargo test --workspace hit i.pximg.net — where it demonstrably flaked (twice in one session: TLS EOF mid-body) — and CI's non-ignored step ran it as a side effect of having a secret. It carries the convention's #[ignore = "live network: …"] now and is renamed with the live_ prefix so the live job's --ignored live name filter actually picks it up (the old name did not contain 'live' and would have been filtered out). Its skip prints the SKIP prefix the next commit's live job greps, and AGENTS.md stops claiming a local workspace run is not fully offline.
2026-09-24 03:44:06 +08:00
YoursFunny 0e91753402 perf(x-media): give the slot-holding fallback download its own budget
download_media_limited had one hard-coded total (600s) for every caller, and its heaviest caller — the bot's upload fallback — holds a PREP slot (and its memory reservation) for the whole transfer: six slow-but-alive downloads (a byte every 29s satisfies the idle window) could stall the fallback chain for ten minutes, queue retries included. The budget is a parameter now: the fallback passes 300s of its own (50 MiB in 300s ≈ 1.4 Mbit/s; a slower link is better served by retrying toward the item's smaller URL than by pinning a slot), while bsky's in-fetch HLS segments keep the generous 600s DOWNLOAD_TOTAL_TIMEOUT, now pub(crate) and re-exported for them. download_too_slow reports whichever budget it got.
2026-09-24 03:37:38 +08:00
YoursFunny 8f254c3779 perf(x-media): stream the frame zip through a tokio file handle
download_media_to_file took &mut std::fs::File and wrote every network chunk with a sync write_all on the executor thread — for a ugoira frame zip (up to 512 MiB) that is the whole download stalling a runtime worker, while bsky's remux had already been moved to tokio::fs for exactly this reason (its comment: a multi-megabyte std::fs::write blocks the executor thread). The signature takes &mut tokio::fs::File now; the single caller (pixiv's ugoira path) clones the NamedTempFile's handle — the clone shares the file offset, so the ZipArchive extraction in spawn_blocking reads what the download wrote — and drops it after the download hands the bytes to the OS.
2026-09-24 03:33:23 +08:00
YoursFunny 3fb6b3b4da fix(fetch): one status table — a persistent 4xx is permanent everywhere
status_error's catch-all called every unlisted status Transient, so a 400/405/418/451 got three retries per link before the same answer (twitter syndication's broken-token 400 being the live example), and download_status_error plus misskey's and bilibili's local fallbacks each carried their own copy of the table — bilibili and misskey classifying a 404 as Transient while the center classified it NotFound. The center now makes any client error except 408/429 a refusal (permanent), the media path delegates to it as status_error("media", ...) and its duplicated fn is deleted, and misskey/bilibili fall through to the center after their own special statuses (misskey's 400 body, bilibili's 412). A table test pins every class.
2026-09-24 02:52:10 +08:00
YoursFunny 70d4f2ec88 refactor(x-media): move the download stack into site/download.rs
site/mod.rs held the registry, the Site trait, Fetched, the caption helpers,
FetchError and, at the end, the whole media-download stack: two HTTP clients
with different timeouts, the guard that refuses a URL inside the host's own
network (start URL and every redirect hop), and the two streaming entry
points. That stack is its own reason to change and moves whole, with its
tests; CLIENT and the download fns stay re-exported at site::… so no adapter
or bot call site moved. SITES becomes pub(crate) for the header rule.
2026-09-21 18:35:58 +08:00