mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-09-23 23:32:05 +00:00
perf: bound upload-fallback preparation process-wide
Each batch's items were prepared under their own `Semaphore::new(3)`, which is not a memory bound: 8 URL workers and 4 queue workers can each be inside a batch, so a burst could have two dozen downloads in flight at once, each buffering a whole photo before it is processed. Nothing else on the media path bounds them — the send itself is paced by the rate limiter, but the download and the decode happen before it is charged. One process-wide `PREP_SLOTS` (6) replaces the per-batch semaphore, and the photo download gets its own cap: `MAX_PHOTO_DOWNLOAD_BYTES` (32 MiB) for the transfer, with `MAX_DECODE_BYTES` (512 MiB) left as the pre-allocation guard on a single decoded buffer. A photo over the download cap degrades to its smaller URL exactly as one over the decode budget does (`FallbackError::MediaTooLarge` → `fallback_url`) — never an error. Verified with the same throwaway proxy harness: 4 concurrent 10-item batches against a server that holds every response 150 ms peak at exactly 6 concurrent downloads (the per-batch three allowed 12) with all 40 items prepared. `cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D warnings` and `cargo test --workspace --locked` clean.
This commit is contained in:
@@ -26,9 +26,16 @@ pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900;
|
||||
/// to a smaller media URL instead.
|
||||
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024;
|
||||
/// Decode budget (bytes): a larger intermediate buffer is not worth the peak
|
||||
/// memory; the photo degrades to the smaller URL instead. Also the cap for
|
||||
/// downloading photos in the send fallback (they must be downloaded whole).
|
||||
/// memory; the photo degrades to the smaller URL instead.
|
||||
pub(crate) const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
|
||||
/// Cap for *downloading* a photo in the send fallback, kept separate from the
|
||||
/// decode budget above: the whole body is buffered before it is processed, once
|
||||
/// per download slot in flight, while the decode budget is about a single
|
||||
/// buffer. Telegram's upload cap is 10 MiB, so a photo this large can only be
|
||||
/// sent after a downscale that its reduced variant serves just as well — over
|
||||
/// the cap the item degrades to the smaller URL
|
||||
/// (`FallbackError::MediaTooLarge`), it is never an error.
|
||||
pub(crate) const MAX_PHOTO_DOWNLOAD_BYTES: u64 = 32 * 1024 * 1024;
|
||||
/// JPEG output quality (1-100).
|
||||
const JPEG_QUALITY: u8 = 90;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user