mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-10-07 01:32:13 +00:00
perf: share one fetch between concurrent duplicates of a link
`link_cache` is only written *after* a send succeeds, so two chats posting the same link at the same moment each ran a full fetch: two sets of source requests, and for an ugoira or a bsky video two ffmpeg encodes of the same post — minutes of CPU for the second one. The same applies to a batch forward racing a queued retry. Within one message `dedupe_urls` already handled the duplicates; across calls nothing did. `fetch_shared` keys an in-flight fetch by the post's cache key: the first caller runs it, the rest subscribe and take its result. The entry is removed by a guard when the fetch settles (cancellation included), so this dedupes what is *concurrent* and never answers from an old result — a repeat later fetches again, and a failure is deliberately not cached: the user is told to try again, and a cached failure would answer that retry from a stale state. Two hazards that shape the code, each with a test: a broadcast channel lives while *any* sender does, so the waiter drops its own clone of the sender before waiting — otherwise a cancelled sharer would leave it waiting forever — and a waiter whose sharer vanished fetches for itself instead of failing a link that is perfectly fetchable. One fetched post can now serve several sends, which the temp files behind `Fetched::keep_alive` had to support: the field is `Arc<TempDir>` and the accessor hands out references (the bot's `KEEP_ALIVE` registry holds the same), so the ugoira/bsky MP4 stays on disk until the *last* task settles rather than the first. `take_keep_alive` is gone — a `take` could only ever serve one of the senders. Verified: 116 bot tests (three new sharing tests, including the cancelled sharer) and 91 x-media tests (a new one pinning the keep-alive refcount) pass, plus a live check that two concurrent `fetch_shared` calls for one tweet return the very same `Arc` after one fetch's worth of wall time. `cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings` and `cargo test --workspace --locked` clean.
This commit is contained in:
@@ -81,7 +81,7 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
url: mp4_path.to_string_lossy().into_owned(),
|
||||
thumbnail_url,
|
||||
});
|
||||
fetched._keep_alive = Some(keep_alive);
|
||||
fetched._keep_alive = Some(std::sync::Arc::new(keep_alive));
|
||||
}
|
||||
// No ffmpeg: a deployment gap, not a bad moment — retrying it
|
||||
// would only waste the fetch budget, so the post degrades (and an
|
||||
|
||||
Reference in New Issue
Block a user