From c549a6d35e16db214ddcb9aa3824708ecb42fe80 Mon Sep 17 00:00:00 2001 From: YoursFunny Date: Mon, 21 Sep 2026 20:21:01 +0800 Subject: [PATCH] feat(upload): give videos and animations Telegram's real 50 MB cap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One MAX_UPLOAD_BYTES (10 MiB) bounded every upload, but that is the *photo* limit: Telegram's own docs say sendVideo/sendAnimation/sendDocument take up to 50 MB, and RequestEntityTooLarge is "larger than 50 MB". So a 10-50 MB video that Telegram refused to fetch by URL was refused a download too, and a video has no smaller variant — the post was lost. The non-photo cap is now MAX_MEDIA_UPLOAD_BYTES, and such a body charges the process-wide budget for the length of the preparation (one 64 MiB unit covers the cap), since PREP_SLOTS alone no longer bounds their added RAM. A const test pins both caps against Telegram's numbers. --- AGENTS.md | 2 +- crates/xmedia-bot/src/photo.rs | 12 ++++++---- crates/xmedia-bot/src/send/mod.rs | 11 +++++++++ crates/xmedia-bot/src/send/upload.rs | 34 ++++++++++++++++++++++------ 4 files changed, 46 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index bf724b1..076ba4b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -82,7 +82,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi |---|---| | `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) | | `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `mod.rs` also holds `apply_caption_edit`, the one place a caption edit is applied and its failure classified: a short retryable delay is retried once, anything else is reported to the user instead of being swallowed (`callback.rs`'s template button answers its toast with the failure and leaves the record alone); `commands.rs` = command dispatch (incl. `/test ` send-only, `/debug ` parse-only, the read-only `/settings` every chat member can read — unlike the admin-only `/bot_dict` raw dump — and template removal; `/start`/`/help` carry the guidance teloxide's `descriptions()` cannot render, and `/set_format` rejects unknown `{…}` placeholders, resetting with `-`); `urls.rs` = URL extraction + the per-URL pipeline (`url_media` takes a `PostSend` mode: chat settings vs `/test`'s suppressed actions); `inline.rs` = debounced inline queries (hotlink-protected and local media skipped); `callback.rs` = edit-before-forward buttons (dptree entry + testable `handle_callback` core, incl. `skip`) | -| `crates/xmedia-bot/src/send/` | `mod.rs`: constants `MAX_MEDIA_GROUP = 10` and the senders; `error.rs`: `classify_request_error` (5xx/non-JSON bodies retry, see the Retries bullet) and the media-fetch markers that route a URL send into the reupload fallback — including `failed to get HTTP url content`, the description single-media URL sends answer with. `upload.rs`: download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`), with a download's class from `classify_download_error` (transport/429/5xx retry; 4xx is permanent — the media itself is gone or refused — and a temp-file *write* failure retries, being resource exhaustion far more often than a broken temp dir). Item preparation is bounded **process-wide** (`PREP_SLOTS` in `upload.rs`: URL workers and queue workers can each be inside a batch, so a per-batch bound is not a memory bound), and the check that routes an oversized item to `fallback_url` is the download's own declared-Content-Length abort (`FetchError::TooLarge` → `MediaTooLarge`) — there is no separate size probe, which used to cost a second request per item. `post_send.rs`: settlement (`settle_task`), cache write, post-send actions (dead-letter text via `failure_text`: post key + cause, since the raw error alone does not say which link died), queue handlers. `input_media.rs`: payload → `InputMedia` | +| `crates/xmedia-bot/src/send/` | `mod.rs`: constants `MAX_MEDIA_GROUP = 10` and the senders; `error.rs`: `classify_request_error` (5xx/non-JSON bodies retry, see the Retries bullet) and the media-fetch markers that route a URL send into the reupload fallback — including `failed to get HTTP url content`, the description single-media URL sends answer with. `upload.rs`: download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`), the per-kind upload cap (`MAX_MEDIA_UPLOAD_BYTES` = 50 MB for video/animation/other, Telegram's multipart limit; `photo::MAX_UPLOAD_BYTES` stays the 10 MiB photo one) with a download's class from `classify_download_error` (transport/429/5xx retry; 4xx is permanent — the media itself is gone or refused — and a temp-file *write* failure retries, being resource exhaustion far more often than a broken temp dir). Item preparation is bounded **process-wide** (`PREP_SLOTS` in `upload.rs`: URL workers and queue workers can each be inside a batch, so a per-batch bound is not a memory bound), and the check that routes an oversized item to `fallback_url` is the download's own declared-Content-Length abort (`FetchError::TooLarge` → `MediaTooLarge`) — there is no separate size probe, which used to cost a second request per item. `post_send.rs`: settlement (`settle_task`), cache write, post-send actions (dead-letter text via `failure_text`: post key + cause, since the raw error alone does not say which link died), queue handlers. `input_media.rs`: payload → `InputMedia` | | `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL. Two budgets, not one: `MAX_PHOTO_DOWNLOAD_BYTES` (32 MiB) caps the *download* in the send fallback — the whole body is buffered, once per prep slot — while `MAX_DECODE_BYTES` (512 MiB) stays the pre-allocation guard that decides whether a decoded photo can be processed at all; over either one the item degrades to its smaller URL | | `crates/x-media/src/site/{mod,download}.rs` | `mod.rs`: dispatcher, `Fetched`/`FetchError`, `needs_media_headers` (the per-site rule, asked by the inline path to skip what Telegram cannot fetch). `download.rs`: the media-download stack — the metadata vs. media HTTP clients, the host-network guard (applied to the start URL and every redirect hop) and `download_media_limited`/`download_media_to_file` (which add the site's headers, e.g. `Referer: https://www.pixiv.net/` for `pximg.net`) | | `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` | diff --git a/crates/xmedia-bot/src/photo.rs b/crates/xmedia-bot/src/photo.rs index 42c7c00..3a693ca 100644 --- a/crates/xmedia-bot/src/photo.rs +++ b/crates/xmedia-bot/src/photo.rs @@ -23,8 +23,10 @@ use tempfile::NamedTempFile; pub const PHOTO_MAX_DIMENSION_SUM: u32 = 10000; /// Resize target with a safety margin so rounding cannot cross the cap. pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900; -/// Upload cap (bytes): files above this are not uploaded; the bot falls back -/// to a smaller media URL instead. +/// Photo upload cap (bytes): Telegram rejects a larger `sendPhoto`, so the bot +/// falls back to a smaller media URL instead. Videos and animations have their +/// own, larger cap — `send::upload::MAX_MEDIA_UPLOAD_BYTES` — and never become +/// photos. pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024; /// Decode budget (bytes): a larger intermediate buffer is not worth the peak /// memory; the photo degrades to the smaller URL instead. @@ -32,9 +34,9 @@ pub(crate) const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024; /// Cap for *downloading* a photo in the send fallback, kept separate from the /// decode budget above: the whole body is buffered before it is processed, once /// per download slot in flight, while the decode budget is about a single -/// buffer. Telegram's upload cap is 10 MiB, so a photo this large can only be -/// sent after a downscale that its reduced variant serves just as well — over -/// the cap the item degrades to the smaller URL +/// buffer. Telegram's *photo* upload cap is 10 MiB, so a photo this large can +/// only be sent after a downscale that its reduced variant serves just as +/// well — over the cap the item degrades to the smaller URL /// (`FallbackError::MediaTooLarge`), it is never an error. pub(crate) const MAX_PHOTO_DOWNLOAD_BYTES: u64 = 32 * 1024 * 1024; diff --git a/crates/xmedia-bot/src/send/mod.rs b/crates/xmedia-bot/src/send/mod.rs index d11bbd5..46e80f1 100644 --- a/crates/xmedia-bot/src/send/mod.rs +++ b/crates/xmedia-bot/src/send/mod.rs @@ -709,6 +709,17 @@ mod tests { use std::time::Duration; use teloxide::ApiError; + /// The two multipart upload caps, pinned where the bot draws them: photos + /// are the 10 MiB case, everything else the 50 MB one. A single cap for + /// both refused to download a 10–50 MB video that Telegram would have + /// accepted (and a video has no smaller variant to fall back to). + #[test] + fn upload_caps_match_telegrams_limits() { + const { assert!(crate::photo::MAX_UPLOAD_BYTES == 10 * 1024 * 1024) }; + const { assert!(super::upload::MAX_MEDIA_UPLOAD_BYTES == 50 * 1024 * 1024) }; + const { assert!(crate::photo::MAX_PHOTO_DOWNLOAD_BYTES <= crate::photo::MAX_DECODE_BYTES) }; + } + #[test] fn oversized_photo_boundary() { // The empirical Telegram limit: sum 10000 passes, 10001 fails. Pinned diff --git a/crates/xmedia-bot/src/send/upload.rs b/crates/xmedia-bot/src/send/upload.rs index b32f9d8..6dcd96a 100644 --- a/crates/xmedia-bot/src/send/upload.rs +++ b/crates/xmedia-bot/src/send/upload.rs @@ -7,7 +7,7 @@ use super::{ MediaItemPayload, MediaRef, SendError, Task, classify_to_send_error, retry_delay_seconds, }; use crate::media_sender::MediaSender; -use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep}; +use crate::photo::{self, PhotoPrep}; use std::sync::LazyLock; use teloxide::prelude::*; use teloxide::types::{ChatId, InputFile, InputMedia, MessageId}; @@ -25,6 +25,15 @@ const PREP_CONCURRENCY: usize = 6; static PREP_SLOTS: LazyLock = LazyLock::new(|| tokio::sync::Semaphore::new(PREP_CONCURRENCY)); +/// Telegram's multipart upload limit for everything that is not a photo: +/// its own docs on `sendVideo`/`sendAnimation`/`sendDocument` say 50 MB +/// (`RequestEntityTooLarge` is "larger than 50 MB"), while photos are the +/// 10 MiB [`photo::MAX_UPLOAD_BYTES`] case. Using the photo cap here refused +/// to even download a 10–50 MB video that Telegram itself would have +/// accepted, and a video has no smaller variant to fall back to — so the +/// post was lost. +pub(super) const MAX_MEDIA_UPLOAD_BYTES: u64 = 50 * 1024 * 1024; + /// Infers a file extension from magic bytes so Telegram detects the mime type /// on multipart uploads. pub(super) fn sniff_ext(bytes: &[u8]) -> &'static str { @@ -82,14 +91,25 @@ async fn download_to_temp( }; // Photos are downloaded even over the upload cap so `prepare_photo` can // downscale / transcode them, up to their own download cap; videos and - // animations are refused as soon as the declared size crosses the upload - // cap. The limit is that cap, not `cap + 1`: a file of exactly the cap is - // admitted (`len > max_bytes` is false), and one byte over is not — the - // same boundary the size probe this replaced drew. - let limit = if matches!(item, MediaItemPayload::Photo { .. }) { + // animations are refused as soon as the declared size crosses their own + // (larger) upload cap. The limit is that cap, not `cap + 1`: a file of + // exactly the cap is admitted (`len > max_bytes` is false), and one byte + // over is not — the same boundary the size probe this replaced drew. + let is_photo = matches!(item, MediaItemPayload::Photo { .. }); + let limit = if is_photo { photo::MAX_PHOTO_DOWNLOAD_BYTES } else { - MAX_UPLOAD_BYTES + MAX_MEDIA_UPLOAD_BYTES + }; + // A non-photo body is buffered whole and can now be 50 MB, so it charges + // the process-wide budget for as long as this function holds it (one + // 64 MiB unit covers the cap): `PREP_SLOTS` bounds how many are in flight, + // this bounds what they add up to. Photos charge their real buffer after + // the download, once their header predicts it. + let _budget = if is_photo { + None + } else { + Some(photo::reserve_memory(MAX_MEDIA_UPLOAD_BYTES).await) }; let bytes = match x_media::site::download_media_limited(media_url, limit).await { Ok(bytes) => bytes,