mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-09-23 23:32:05 +00:00
perf: share one fetch between concurrent duplicates of a link
`link_cache` is only written *after* a send succeeds, so two chats posting the same link at the same moment each ran a full fetch: two sets of source requests, and for an ugoira or a bsky video two ffmpeg encodes of the same post — minutes of CPU for the second one. The same applies to a batch forward racing a queued retry. Within one message `dedupe_urls` already handled the duplicates; across calls nothing did. `fetch_shared` keys an in-flight fetch by the post's cache key: the first caller runs it, the rest subscribe and take its result. The entry is removed by a guard when the fetch settles (cancellation included), so this dedupes what is *concurrent* and never answers from an old result — a repeat later fetches again, and a failure is deliberately not cached: the user is told to try again, and a cached failure would answer that retry from a stale state. Two hazards that shape the code, each with a test: a broadcast channel lives while *any* sender does, so the waiter drops its own clone of the sender before waiting — otherwise a cancelled sharer would leave it waiting forever — and a waiter whose sharer vanished fetches for itself instead of failing a link that is perfectly fetchable. One fetched post can now serve several sends, which the temp files behind `Fetched::keep_alive` had to support: the field is `Arc<TempDir>` and the accessor hands out references (the bot's `KEEP_ALIVE` registry holds the same), so the ugoira/bsky MP4 stays on disk until the *last* task settles rather than the first. `take_keep_alive` is gone — a `take` could only ever serve one of the senders. Verified: 116 bot tests (three new sharing tests, including the cancelled sharer) and 91 x-media tests (a new one pinning the keep-alive refcount) pass, plus a live check that two concurrent `fetch_shared` calls for one tweet return the very same `Arc` after one fetch's worth of wall time. `cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings` and `cargo test --workspace --locked` clean.
This commit is contained in:
@@ -81,7 +81,7 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
url: mp4_path.to_string_lossy().into_owned(),
|
||||
thumbnail_url,
|
||||
});
|
||||
fetched._keep_alive = Some(keep_alive);
|
||||
fetched._keep_alive = Some(std::sync::Arc::new(keep_alive));
|
||||
}
|
||||
// No ffmpeg: a deployment gap, not a bad moment — retrying it
|
||||
// would only waste the fetch budget, so the post degrades (and an
|
||||
|
||||
@@ -52,8 +52,11 @@ pub struct Fetched {
|
||||
/// Raw values (pre-escaped) for user-customizable caption formats.
|
||||
pub(crate) render_data: Option<RenderData>,
|
||||
/// Keeps temp files (e.g. an encoded ugoira MP4) alive until the caller
|
||||
/// finishes uploading; not part of the public contract.
|
||||
pub(crate) _keep_alive: Option<tempfile::TempDir>,
|
||||
/// finishes uploading; not part of the public contract. Shared rather than
|
||||
/// owned because one fetched post can serve several sends — the bot shares
|
||||
/// one in-flight fetch between concurrent duplicates of the same link — and
|
||||
/// the files have to outlive every one of them.
|
||||
pub(crate) _keep_alive: Option<std::sync::Arc<tempfile::TempDir>>,
|
||||
}
|
||||
|
||||
/// Values for the `{url} {author} {author_url} {title} {content} {tags}`
|
||||
@@ -133,12 +136,14 @@ impl Fetched {
|
||||
})
|
||||
}
|
||||
|
||||
/// Hands over the temp dir keeping locally produced media (ugoira MP4,
|
||||
/// A reference to the temp dir keeping locally produced media (ugoira MP4,
|
||||
/// bsky remux MP4) alive. The bot keeps it while its task may still be
|
||||
/// retried by the queue, which runs after this [`Fetched`] is dropped and
|
||||
/// its temp files would otherwise be gone. `None` when no such dir exists.
|
||||
pub fn take_keep_alive(&mut self) -> Option<tempfile::TempDir> {
|
||||
self._keep_alive.take()
|
||||
/// its temp files would otherwise be gone. `None` when no such dir exists;
|
||||
/// each clone keeps the directory alive for as long as it lives, so two
|
||||
/// sends of one post can each hold the same files.
|
||||
pub fn keep_alive(&self) -> Option<std::sync::Arc<tempfile::TempDir>> {
|
||||
self._keep_alive.clone()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -742,6 +747,38 @@ pub async fn download_media_to_file(
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Locally produced media (ugoira MP4, bsky remux MP4) lives in a temp dir
|
||||
/// whose lifetime is refcounted: one fetch result can serve several sends
|
||||
/// (the bot shares one in-flight fetch between concurrent duplicates), and
|
||||
/// the files must outlive all of them — but no longer than the last one.
|
||||
#[test]
|
||||
fn a_keep_alive_clone_outlives_the_fetched() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let file = dir.path().join("media.mp4");
|
||||
std::fs::write(&file, b"mp4").unwrap();
|
||||
let fetched = Fetched {
|
||||
source_url: "https://x.com/u/status/1".into(),
|
||||
caption: String::new(),
|
||||
title: String::new(),
|
||||
content: String::new(),
|
||||
media: Vec::new(),
|
||||
sensitive: false,
|
||||
site_id: "twitter",
|
||||
render_data: None,
|
||||
_keep_alive: Some(std::sync::Arc::new(dir)),
|
||||
};
|
||||
|
||||
let shared = fetched.keep_alive().expect("a temp dir to share");
|
||||
drop(fetched);
|
||||
assert!(file.exists(), "the file must survive the fetched post");
|
||||
|
||||
let second = shared.clone();
|
||||
drop(shared);
|
||||
assert!(file.exists(), "another holder keeps it alive");
|
||||
drop(second);
|
||||
assert!(!file.exists(), "the last holder releases the directory");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cache_key_normalizes_domain_variants() {
|
||||
assert_eq!(
|
||||
|
||||
@@ -153,7 +153,7 @@ impl PixivAPI {
|
||||
url: mp4_path,
|
||||
thumbnail_url: model.image_urls.medium.clone(),
|
||||
});
|
||||
illustration._keep_alive = Some(_keep_alive);
|
||||
illustration._keep_alive = Some(std::sync::Arc::new(_keep_alive));
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(e) => {
|
||||
|
||||
@@ -181,7 +181,7 @@ pub struct Illustration {
|
||||
pub(crate) media: Vec<Media>,
|
||||
nsfw: bool,
|
||||
/// Keeps a temp dir (ugoira MP4) alive until the send completes.
|
||||
pub(crate) _keep_alive: Option<tempfile::TempDir>,
|
||||
pub(crate) _keep_alive: Option<std::sync::Arc<tempfile::TempDir>>,
|
||||
}
|
||||
|
||||
impl Illustration {
|
||||
|
||||
Reference in New Issue
Block a user