mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-09-28 00:02:12 +00:00
perf: share one fetch between concurrent duplicates of a link
`link_cache` is only written *after* a send succeeds, so two chats posting the same link at the same moment each ran a full fetch: two sets of source requests, and for an ugoira or a bsky video two ffmpeg encodes of the same post — minutes of CPU for the second one. The same applies to a batch forward racing a queued retry. Within one message `dedupe_urls` already handled the duplicates; across calls nothing did. `fetch_shared` keys an in-flight fetch by the post's cache key: the first caller runs it, the rest subscribe and take its result. The entry is removed by a guard when the fetch settles (cancellation included), so this dedupes what is *concurrent* and never answers from an old result — a repeat later fetches again, and a failure is deliberately not cached: the user is told to try again, and a cached failure would answer that retry from a stale state. Two hazards that shape the code, each with a test: a broadcast channel lives while *any* sender does, so the waiter drops its own clone of the sender before waiting — otherwise a cancelled sharer would leave it waiting forever — and a waiter whose sharer vanished fetches for itself instead of failing a link that is perfectly fetchable. One fetched post can now serve several sends, which the temp files behind `Fetched::keep_alive` had to support: the field is `Arc<TempDir>` and the accessor hands out references (the bot's `KEEP_ALIVE` registry holds the same), so the ugoira/bsky MP4 stays on disk until the *last* task settles rather than the first. `take_keep_alive` is gone — a `take` could only ever serve one of the senders. Verified: 116 bot tests (three new sharing tests, including the cancelled sharer) and 91 x-media tests (a new one pinning the keep-alive refcount) pass, plus a live check that two concurrent `fetch_shared` calls for one tweet return the very same `Arc` after one fetch's worth of wall time. `cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings` and `cargo test --workspace --locked` clean.
This commit is contained in:
@@ -153,7 +153,7 @@ impl PixivAPI {
|
||||
url: mp4_path,
|
||||
thumbnail_url: model.image_urls.medium.clone(),
|
||||
});
|
||||
illustration._keep_alive = Some(_keep_alive);
|
||||
illustration._keep_alive = Some(std::sync::Arc::new(_keep_alive));
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(e) => {
|
||||
|
||||
Reference in New Issue
Block a user