diff --git a/AGENTS.md b/AGENTS.md index 5486f75..05f60d4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,7 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr | Path | Purpose | |---|---| -| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample | +| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media*` (the streaming `download_media_to_file` and the capped `download_media_limited`, which is where a download's size is checked); `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample | | `crates/x-media/src/site//` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `Site` implementing `site::Site`, `From for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`; without the token a withheld tweet stays `FetchError::Sensitive` and the bot reports it as age-restricted instead of "no media"). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Bilibili fetches dynamics (images/animated images only — an attached video degrades to its cover, and its title stands in for the post text, which AV dynamics do not have) from `/x/polymer/web-dynamic/v1/detail` sent with `features=itemOpusStyle` (without that flag the legacy serialization drops an image/text post's body and headline entirely — `desc` comes back `null`; the adapter still parses the legacy `major.draw`/`desc`/`archive` shapes as a fallback). No WBI signature is involved; device cookies `buvid3`/`buvid4` are fetched automatically from `/x/frontend/finger/spi` because bilibili's `-352` risk control starts rejecting plain requests, `BILIBILI_COOKIE` is the escalation when an IP stays blocked; `b23.tv` short links are deliberately unmatched. Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`>` `<` `&` `'`) — so the stored text is raw and the caption escap… | `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands` — `setMyCommands` plus the profile description texts), shared `send::BOT` force-init, startup sweep of this project's leftover temp files (`x_media::TEMP_FILE_PREFIX` + an age gate, since a killed process runs no destructors), startup repair of queued retries whose local media did not survive a restart (`handlers::repair_lost_local_media`, before any worker can lease: those rows are re-fetched from their `source_url`), queue worker start, site login validation (`site::validate_all`), `periodic_sweep` (`SWEEP_INTERVAL` 300 s): expired prompts are rewritten in place to `EDIT_PROMPT_EXPIRED_TEXT` with an empty keyboard — an edit, never a new message, so a background timer cannot wake a chat — plus the link-cache prune, the idle rate-limit buckets and the idle inline-query entries, and the queue backlog line (only when non-empty). Takes its collaborators rather than the statics so its loop is testable with a paused clock, dptree handler tree, webhook vs polling dispatch | | `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` | @@ -82,7 +82,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi |---|---| | `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) | | `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `mod.rs` also holds `apply_caption_edit`, the one place a caption edit is applied and its failure classified: a short retryable delay is retried once, anything else is reported to the user instead of being swallowed (`callback.rs`'s template button answers its toast with the failure and leaves the record alone); `commands.rs` = command dispatch (incl. `/test ` send-only, `/debug ` parse-only, the read-only `/settings` every chat member can read — unlike the admin-only `/bot_dict` raw dump — and template removal; `/start`/`/help` carry the guidance teloxide's `descriptions()` cannot render, and `/set_format` rejects unknown `{…}` placeholders, resetting with `-`); `urls.rs` = URL extraction + the per-URL pipeline (`url_media` takes a `PostSend` mode: chat settings vs `/test`'s suppressed actions); `inline.rs` = debounced inline queries (hotlink-protected and local media skipped); `callback.rs` = edit-before-forward buttons (dptree entry + testable `handle_callback` core, incl. `skip`) | -| `crates/xmedia-bot/src/send/` | `mod.rs`: constants `MAX_MEDIA_GROUP = 10`; `classify_request_error` (5xx/non-JSON bodies retry, see the Retries bullet) and the media-fetch markers that route a URL send into the reupload fallback — including `failed to get HTTP url content`, the description single-media URL sends answer with; the senders. `upload.rs`: download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`), with a download's class from `classify_download_error` (transport/429/5xx retry; 4xx is permanent — the media itself is gone or refused — and a temp-file *write* failure retries, being resource exhaustion far more often than a broken temp dir). `post_send.rs`: settlement (`settle_task`), cache write, post-send actions (dead-letter text via `failure_text`: post key + cause, since the raw error alone does not say which link died), queue handlers. `input_media.rs`: payload → `InputMedia` | +| `crates/xmedia-bot/src/send/` | `mod.rs`: constants `MAX_MEDIA_GROUP = 10`; `classify_request_error` (5xx/non-JSON bodies retry, see the Retries bullet) and the media-fetch markers that route a URL send into the reupload fallback — including `failed to get HTTP url content`, the description single-media URL sends answer with; the senders. `upload.rs`: download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`), with a download's class from `classify_download_error` (transport/429/5xx retry; 4xx is permanent — the media itself is gone or refused — and a temp-file *write* failure retries, being resource exhaustion far more often than a broken temp dir). The check that routes an oversized item to `fallback_url` is the download's own declared-Content-Length abort (`FetchError::TooLarge` → `MediaTooLarge`) — there is no separate size probe, which used to cost a second request per item. `post_send.rs`: settlement (`settle_task`), cache write, post-send actions (dead-letter text via `failure_text`: post key + cause, since the raw error alone does not say which link died), queue handlers. `input_media.rs`: payload → `InputMedia` | | `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL | | `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection), `needs_media_headers` (the same per-site rule, asked by the inline path to skip what Telegram cannot fetch) | | `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` | diff --git a/crates/x-media/src/site/mod.rs b/crates/x-media/src/site/mod.rs index c4ee8f4..39f688e 100644 --- a/crates/x-media/src/site/mod.rs +++ b/crates/x-media/src/site/mod.rs @@ -656,21 +656,6 @@ fn apply_media_headers(mut request: reqwest::RequestBuilder, url: &str) -> reqwe request } -/// Downloads media bytes for the bot's upload fallback: when Telegram's own -/// fetch of a media URL is blocked (hotlink protection), the bot downloads -/// the file itself and uploads it via multipart. Site-appropriate headers -/// come from each site's `media_headers` (pixiv image hosts need `Referer`). -/// Returns the Content-Length of a media URL, or `None` when the server does -/// not report one. Used to check whether a file fits Telegram's size limits -/// before downloading/uploading it. -pub async fn media_size(url: &str) -> Result, FetchError> { - let response = apply_media_headers(CLIENT.get(url), url) - .send() - .await? - .error_for_status()?; - Ok(response.content_length()) -} - /// Maps a media download's HTTP status onto the same classes the site /// adapters use, so callers can tell "try again" from "this URL is dead": /// 4xx is a property of the media (gone, refused by the host), while 429/5xx @@ -687,7 +672,13 @@ fn download_status_error(status: reqwest::StatusCode) -> FetchError { /// Downloads a media file with a hard size cap: the body is streamed and the /// download aborts with [`FetchError::TooLarge`] the moment the cap is /// crossed (or when a declared Content-Length already exceeds it). Keeps the -/// bot from buffering arbitrarily large bodies into memory. +/// bot from buffering arbitrarily large bodies into memory — the size check +/// the bot's upload fallback needs is the one here, not a probe of its own. +/// +/// This is the bot's download path for the upload fallback: when Telegram +/// cannot fetch a media URL itself (hotlink protection), the bot downloads +/// the file and uploads it via multipart. Site-appropriate headers come from +/// each site's `media_headers` (pixiv image hosts need `Referer`). pub async fn download_media_limited(url: &str, max_bytes: u64) -> Result { let response = send_download(media_request(url)?).await?; if let Some(len) = response.content_length() diff --git a/crates/xmedia-bot/src/send/upload.rs b/crates/xmedia-bot/src/send/upload.rs index 39c865f..f091ba5 100644 --- a/crates/xmedia-bot/src/send/upload.rs +++ b/crates/xmedia-bot/src/send/upload.rs @@ -62,12 +62,14 @@ async fn download_to_temp( | MediaItemPayload::Animation { media, .. } => media, }; // Photos are downloaded even over the upload cap so `prepare_photo` can - // downscale / transcode them (cap = decode budget); videos/animations - // abort as soon as the upload cap is crossed mid-stream. + // downscale / transcode them (cap = decode budget); videos and animations + // are refused as soon as the declared size crosses the upload cap — the + // boundary the size probe this replaced drew: a file of exactly the cap is + // admitted (`len > max_bytes` is false), one byte over is not. let limit = if matches!(item, MediaItemPayload::Photo { .. }) { photo::MAX_DECODE_BYTES } else { - MAX_UPLOAD_BYTES + 1 + MAX_UPLOAD_BYTES }; let bytes = match x_media::site::download_media_limited(media_url, limit).await { Ok(bytes) => bytes, @@ -197,28 +199,13 @@ pub(super) async fn prepare_upload_item( keep_alive: None, }); } - // Size check before downloading/uploading: over the cap, use the - // smaller URL instead of the file. Photos are exempt — they are - // downloaded and processed (downscale / PNG→JPEG) before uploading. - let too_large = match x_media::site::media_size(media_url).await { - Ok(Some(size)) => size > MAX_UPLOAD_BYTES, - _ => false, - }; - let too_large = too_large && !matches!(item, MediaItemPayload::Photo { .. }); - if too_large { - let url = item - .fallback_url() - .ok_or_else(|| FallbackError::Permanent { - message: "media too large".into(), - })?; - let media = media_from_url(&item, url, caption, item.thumbnail_url()) - .map_err(|message| FallbackError::Permanent { message })?; - return Ok(PreparedItem { - index, - media, - keep_alive: None, - }); - } + // Whether a file is over the cap is settled by the download itself: + // `download_media_limited` reads the declared Content-Length before any + // body byte and aborts with `FetchError::TooLarge`, which arrives here as + // `FallbackError::MediaTooLarge` — turned into the item's smaller URL by + // the match below. A separate size probe used to issue a second GET of the + // same URL for an answer this path already has (and issued it for photos, + // whose answer was discarded one line later). match download_to_temp(&item).await { Ok((file, bytes)) => { if matches!(item, MediaItemPayload::Photo { .. }) {