mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-10-07 01:32:13 +00:00
perf: share one fetch between concurrent duplicates of a link
`link_cache` is only written *after* a send succeeds, so two chats posting the same link at the same moment each ran a full fetch: two sets of source requests, and for an ugoira or a bsky video two ffmpeg encodes of the same post — minutes of CPU for the second one. The same applies to a batch forward racing a queued retry. Within one message `dedupe_urls` already handled the duplicates; across calls nothing did. `fetch_shared` keys an in-flight fetch by the post's cache key: the first caller runs it, the rest subscribe and take its result. The entry is removed by a guard when the fetch settles (cancellation included), so this dedupes what is *concurrent* and never answers from an old result — a repeat later fetches again, and a failure is deliberately not cached: the user is told to try again, and a cached failure would answer that retry from a stale state. Two hazards that shape the code, each with a test: a broadcast channel lives while *any* sender does, so the waiter drops its own clone of the sender before waiting — otherwise a cancelled sharer would leave it waiting forever — and a waiter whose sharer vanished fetches for itself instead of failing a link that is perfectly fetchable. One fetched post can now serve several sends, which the temp files behind `Fetched::keep_alive` had to support: the field is `Arc<TempDir>` and the accessor hands out references (the bot's `KEEP_ALIVE` registry holds the same), so the ugoira/bsky MP4 stays on disk until the *last* task settles rather than the first. `take_keep_alive` is gone — a `take` could only ever serve one of the senders. Verified: 116 bot tests (three new sharing tests, including the cancelled sharer) and 91 x-media tests (a new one pinning the keep-alive refcount) pass, plus a live check that two concurrent `fetch_shared` calls for one tweet return the very same `Arc` after one fetch's worth of wall time. `cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings` and `cargo test --workspace --locked` clean.
This commit is contained in:
@@ -80,12 +80,14 @@ async fn invalidate_cache(cache: &LinkCache, task: &Task) {
|
||||
|
||||
/// Locally produced media files (ugoira MP4, bsky remux MP4) whose temp dirs
|
||||
/// must stay alive while their task may be retried by the queue. The fetch
|
||||
/// pipeline hands ownership here via
|
||||
/// [`x_media::site::Fetched::take_keep_alive`] before that
|
||||
/// [`x_media::site::Fetched`] is dropped; a queued retry runs after that drop,
|
||||
/// so without this the local file would be gone by the time the retry sends
|
||||
/// it. Entries are removed when the task settles (see [`release_keep_alive`]).
|
||||
pub(crate) static KEEP_ALIVE: LazyLock<parking_lot::Mutex<Vec<tempfile::TempDir>>> =
|
||||
/// pipeline hands a reference here via [`x_media::site::Fetched::keep_alive`]
|
||||
/// before that [`x_media::site::Fetched`] is dropped; a queued retry runs after
|
||||
/// that drop, so without this the local file would be gone by the time the
|
||||
/// retry sends it. `Arc` because one fetch can serve several tasks (a
|
||||
/// concurrent duplicate of the same link shares it): each holder keeps the
|
||||
/// directory alive until its own task settles. Entries are removed when the
|
||||
/// task settles (see [`release_keep_alive`]).
|
||||
pub(crate) static KEEP_ALIVE: LazyLock<parking_lot::Mutex<Vec<std::sync::Arc<tempfile::TempDir>>>> =
|
||||
LazyLock::new(|| parking_lot::Mutex::new(Vec::new()));
|
||||
|
||||
/// Drops the keep-alive temp dirs holding media referenced by `task` (matched
|
||||
|
||||
Reference in New Issue
Block a user