perf: share one fetch between concurrent duplicates of a link

`link_cache` is only written *after* a send succeeds, so two chats posting
the same link at the same moment each ran a full fetch: two sets of source
requests, and for an ugoira or a bsky video two ffmpeg encodes of the same
post — minutes of CPU for the second one. The same applies to a batch
forward racing a queued retry. Within one message `dedupe_urls` already
handled the duplicates; across calls nothing did.

`fetch_shared` keys an in-flight fetch by the post's cache key: the first
caller runs it, the rest subscribe and take its result. The entry is removed
by a guard when the fetch settles (cancellation included), so this dedupes
what is *concurrent* and never answers from an old result — a repeat later
fetches again, and a failure is deliberately not cached: the user is told to
try again, and a cached failure would answer that retry from a stale state.

Two hazards that shape the code, each with a test: a broadcast channel lives
while *any* sender does, so the waiter drops its own clone of the sender
before waiting — otherwise a cancelled sharer would leave it waiting forever
— and a waiter whose sharer vanished fetches for itself instead of failing a
link that is perfectly fetchable.

One fetched post can now serve several sends, which the temp files behind
`Fetched::keep_alive` had to support: the field is `Arc<TempDir>` and the
accessor hands out references (the bot's `KEEP_ALIVE` registry holds the
same), so the ugoira/bsky MP4 stays on disk until the *last* task settles
rather than the first. `take_keep_alive` is gone — a `take` could only ever
serve one of the senders.

Verified: 116 bot tests (three new sharing tests, including the cancelled
sharer) and 91 x-media tests (a new one pinning the keep-alive refcount)
pass, plus a live check that two concurrent `fetch_shared` calls for one
tweet return the very same `Arc` after one fetch's worth of wall time.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
This commit is contained in:
2026-09-21 13:42:52 +08:00
parent 994734b001
commit 0edaef56bd
7 changed files with 247 additions and 24 deletions
+43 -6
View File
@@ -52,8 +52,11 @@ pub struct Fetched {
/// Raw values (pre-escaped) for user-customizable caption formats.
pub(crate) render_data: Option<RenderData>,
/// Keeps temp files (e.g. an encoded ugoira MP4) alive until the caller
/// finishes uploading; not part of the public contract.
pub(crate) _keep_alive: Option<tempfile::TempDir>,
/// finishes uploading; not part of the public contract. Shared rather than
/// owned because one fetched post can serve several sends — the bot shares
/// one in-flight fetch between concurrent duplicates of the same link — and
/// the files have to outlive every one of them.
pub(crate) _keep_alive: Option<std::sync::Arc<tempfile::TempDir>>,
}
/// Values for the `{url} {author} {author_url} {title} {content} {tags}`
@@ -133,12 +136,14 @@ impl Fetched {
})
}
/// Hands over the temp dir keeping locally produced media (ugoira MP4,
/// A reference to the temp dir keeping locally produced media (ugoira MP4,
/// bsky remux MP4) alive. The bot keeps it while its task may still be
/// retried by the queue, which runs after this [`Fetched`] is dropped and
/// its temp files would otherwise be gone. `None` when no such dir exists.
pub fn take_keep_alive(&mut self) -> Option<tempfile::TempDir> {
self._keep_alive.take()
/// its temp files would otherwise be gone. `None` when no such dir exists;
/// each clone keeps the directory alive for as long as it lives, so two
/// sends of one post can each hold the same files.
pub fn keep_alive(&self) -> Option<std::sync::Arc<tempfile::TempDir>> {
self._keep_alive.clone()
}
}
@@ -742,6 +747,38 @@ pub async fn download_media_to_file(
mod tests {
use super::*;
/// Locally produced media (ugoira MP4, bsky remux MP4) lives in a temp dir
/// whose lifetime is refcounted: one fetch result can serve several sends
/// (the bot shares one in-flight fetch between concurrent duplicates), and
/// the files must outlive all of them — but no longer than the last one.
#[test]
fn a_keep_alive_clone_outlives_the_fetched() {
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("media.mp4");
std::fs::write(&file, b"mp4").unwrap();
let fetched = Fetched {
source_url: "https://x.com/u/status/1".into(),
caption: String::new(),
title: String::new(),
content: String::new(),
media: Vec::new(),
sensitive: false,
site_id: "twitter",
render_data: None,
_keep_alive: Some(std::sync::Arc::new(dir)),
};
let shared = fetched.keep_alive().expect("a temp dir to share");
drop(fetched);
assert!(file.exists(), "the file must survive the fetched post");
let second = shared.clone();
drop(shared);
assert!(file.exists(), "another holder keeps it alive");
drop(second);
assert!(!file.exists(), "the last holder releases the directory");
}
#[test]
fn cache_key_normalizes_domain_variants() {
assert_eq!(