diff --git a/AGENTS.md b/AGENTS.md index 8bc32ad..40cd3cd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,6 +20,8 @@ Telegram update → Dispatcher (polling or axum webhook) → dptree branches Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)` → `Fetched` → builds a `Task` → `send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel. +Debug command: `/test ` runs the same `x_media::site::fetch` and replies with `test_parse_report` (`handlers/commands.rs`) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars (Telegram's 4096 plain-text limit). It uses a custom `parse_test_arg` parser (whole remainder, trimmed) because teloxide's built-in `split` parser takes exactly one space-separated token. + The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → pixiv) and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec, sensitive, site_id, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`. ## Key Directories @@ -30,7 +32,7 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr | `crates/x-media/src/site//` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `Site` implementing `site::Site`, `From for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`) | | `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch | | `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` | -| `crates/xmedia-bot/src/handlers.rs` | `Command` enum (teloxide `BotCommands`), message/inline/callback handlers, URL extraction, global statics; per-URL work flows through a bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns (teloxide's per-chat workers are sequential — batch-forwards need concurrency) | +| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. the `/test ` parse-only debug command), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons), `statics.rs` (global statics) | | `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex` cache + SQLite write-through (`chat_state` table) | | `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure | | `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections | @@ -67,7 +69,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi | File | Why it matters | |---|---| | `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) | -| `crates/xmedia-bot/src/handlers.rs` | `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `data/task_queue.db` **relative to CWD**); command dispatch; URL extraction; retry enqueue | +| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `data/task_queue.db` **relative to CWD**); `commands.rs` = command dispatch (incl. the `/test ` parse-only debug command); `urls.rs` = URL extraction + retry enqueue; `inline.rs`/`callback.rs` = inline queries / edit-before-forward buttons | | `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`; fallback chain; `classify_request_error`; download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`) | | `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL | | `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) | diff --git a/README.en.md b/README.en.md index aa16469..2725b7b 100644 --- a/README.en.md +++ b/README.en.md @@ -113,6 +113,7 @@ Telegram only accepts ports 443/80/88/8443. | `/set_format ` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` | | `/clear_cache [link]` | Clear the link cache (admin only); with a link only that entry, otherwise everything | | `/bot_dict` | Show the current chat state (debugging) | +| `/test ` | Debug: parse a link and report the parse result only (site, title, author, tags, media list) — no media is sent | Link processing works only in private chats; commands work in any chat. diff --git a/README.md b/README.md index 58f30b8..b227200 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,7 @@ Telegram 只接受 443/80/88/8443 端口。 | `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点：`twitter` / `bsky` / `pixiv`。占位符：`{url}` `{author}` `{author_url}` `{title}` `{tags}` | | `/clear_cache [链接]` | 清空链接缓存（仅管理员）；带链接只清该条，否则清空全部 | | `/bot_dict` | 查看当前聊天状态（调试用） | +| `/test <链接>` | 调试：只解析链接并返回解析结果（站点、标题、作者、标签、媒体列表），不发送任何媒体 | 链接处理仅限私聊；命令在任意聊天可用。 diff --git a/crates/xmedia-bot/src/handlers/commands.rs b/crates/xmedia-bot/src/handlers/commands.rs index 377d14c..1d18ecd 100644 --- a/crates/xmedia-bot/src/handlers/commands.rs +++ b/crates/xmedia-bot/src/handlers/commands.rs @@ -1,11 +1,11 @@ //! Bot command parsing, the `/`-command executor and `setMyCommands` //! registration. URL/inline/callback flows live in their own modules. -use super::{CHAT_STORE, CONFIG, LINK_CACHE, reply}; +use super::{CHAT_STORE, CONFIG, LINK_CACHE, log_key, reply}; use teloxide::RequestError; use teloxide::prelude::*; use teloxide::types::{ChatId, Message, Recipient}; -use teloxide::utils::command::BotCommands; +use teloxide::utils::command::{BotCommands, ParseError}; #[derive(BotCommands, Clone)] #[command( @@ -40,6 +40,19 @@ pub(crate) enum Command { parse_with = "split" )] ClearCache(String), + #[command( + description = "Test link parsing (debug; no media sent)", + parse_with = parse_test_arg + )] + Test(String), +} + +/// `/test` argument parser: the whole remainder after the command name, +/// trimmed. The built-in `split` parser takes exactly one space-separated +/// token and rejects the rest, so a URL followed by a trailing space (or +/// pasted text) would silently fall through to the URL flow instead. +fn parse_test_arg(s: String) -> Result<(String,), ParseError> { + Ok((s.trim().to_string(),)) } enum SetForwardChannelError { @@ -302,6 +315,56 @@ pub(crate) async fn execute_command( .await?; } } + Command::Test(arg) => { + let url = arg.trim(); + if url.is_empty() { + reply( + bot, + message.chat.id.0, + message.id, + "Usage: /test ", + ) + .await?; + return Ok(()); + } + // Debug tool: report the parse result only — nothing is sent, + // cached or forwarded. Info level echoes the normalized key + // (never the raw URL) per the logging convention. + log::info!("test: parsing [key={}]", log_key(url)); + match x_media::site::fetch(url).await { + Ok(None) => { + reply( + bot, + message.chat.id.0, + message.id, + "No enabled site matches this link (twitter/x, pixiv or bsky).", + ) + .await?; + } + Err(e) => { + reply( + bot, + message.chat.id.0, + message.id, + format!("Fetch failed: {e}"), + ) + .await?; + } + Ok(Some(fetched)) => { + let report = test_parse_report( + url, + fetched.site_name(), + &fetched.source_url, + &fetched.title, + fetched.render_fields(), + fetched.sensitive, + &fetched.caption, + &fetched.media, + ); + reply(bot, message.chat.id.0, message.id, report).await?; + } + } + } } Ok(()) } @@ -319,3 +382,137 @@ pub async fn register_commands(bot: &Bot) -> Result<(), RequestError> { log::info!("registered {} commands", commands.len()); Ok(()) } + +/// Telegram's plain-text message limit is 4096 chars; the report stays under +/// it even for very large threads (many media lines + a long caption). +const MAX_TEST_REPORT_CHARS: usize = 4000; + +/// Builds the plain-text report for the `/test` command: what the parser +/// produced for a link (site, canonical URL, title/author/tags, caption and +/// the media list) — no media is sent and nothing is cached or forwarded. +/// Fields are passed individually so the formatter stays a pure function +/// testable without constructing a `Fetched` (its render fields are +/// `pub(crate)` to the x-media crate). +#[allow(clippy::too_many_arguments)] +fn test_parse_report( + url: &str, + site_id: &str, + source_url: &str, + title: &str, + render: Option<(&str, &str, &str, &str)>, + sensitive: bool, + caption: &str, + media: &[x_media::media::Media], +) -> String { + let mut lines = vec![ + format!("Parse result for {url}"), + format!("site: {site_id}"), + format!( + "key: {}", + x_media::site::cache_key(url).unwrap_or_else(|| "".to_string()) + ), + ]; + lines.push(format!("source_url: {source_url}")); + lines.push(format!("title: {title}")); + if let Some((author, author_url, _title, tags)) = render { + lines.push(format!("author: {author}")); + lines.push(format!("author_url: {author_url}")); + lines.push(format!("tags: {tags}")); + } + lines.push(format!("sensitive: {sensitive}")); + lines.push(format!( + "caption: {}", + x_media::site::truncate_caption(caption) + )); + lines.push(format!("media ({}):", media.len())); + for (i, item) in media.iter().enumerate() { + let kind = match item { + x_media::media::Media::Illustration { .. } => "image", + x_media::media::Media::Video { .. } => "video", + x_media::media::Media::Animated { .. } => "gif", + }; + lines.push(format!(" {}. {kind}: {}", i + 1, item.url())); + } + let mut out = lines.join( + " +", + ); + if out.chars().count() > MAX_TEST_REPORT_CHARS { + let end = out.floor_char_boundary(MAX_TEST_REPORT_CHARS - 1); + out = format!("{}…", &out[..end]); + } + out +} + +#[cfg(test)] +mod tests { + use super::{MAX_TEST_REPORT_CHARS, test_parse_report}; + use x_media::media::Media; + + #[test] + fn test_parse_report_renders_fields_and_media() { + let media = vec![ + Media::Illustration { + title: None, + url: "https://cdn.example/1.jpg".into(), + thumbnail_url: None, + fallback_url: None, + }, + Media::Video { + title: None, + url: "https://cdn.example/2.mp4".into(), + thumbnail_url: "https://cdn.example/2.jpg".into(), + }, + ]; + let report = test_parse_report( + "https://x.com/u/status/1", + "twitter", + "https://x.com/u/status/1", + "My title", + Some(("Author", "https://x.com/u", "My title", "tag1 tag2")), + false, + "Author · My title", + &media, + ); + assert!(report.contains("site: twitter"), "{report}"); + assert!(report.contains("key: twitter:1"), "{report}"); + assert!(report.contains("title: My title"), "{report}"); + assert!(report.contains("author: Author"), "{report}"); + assert!(report.contains("author_url: https://x.com/u"), "{report}"); + assert!(report.contains("tags: tag1 tag2"), "{report}"); + assert!(report.contains("sensitive: false"), "{report}"); + assert!(report.contains("media (2):"), "{report}"); + assert!( + report.contains("1. image: https://cdn.example/1.jpg"), + "{report}" + ); + assert!( + report.contains("2. video: https://cdn.example/2.mp4"), + "{report}" + ); + } + + #[test] + fn test_parse_report_without_render_data_and_no_media() { + let report = test_parse_report("u", "pixiv", "s", "t", None, true, "c", &[]); + assert!(!report.contains("author:"), "{report}"); + assert!(report.contains("sensitive: true"), "{report}"); + assert!(report.contains("media (0):"), "{report}"); + } + + #[test] + fn test_parse_report_is_capped() { + // 200 media lines ≈ 8 KB, comfortably over the cap. + let media: Vec = (0..200) + .map(|i| Media::Illustration { + title: None, + url: format!("https://cdn.example/{i}.jpg"), + thumbnail_url: None, + fallback_url: None, + }) + .collect(); + let report = test_parse_report("u", "twitter", "s", "t", None, false, "c", &media); + assert!(report.chars().count() <= MAX_TEST_REPORT_CHARS, "{report}"); + assert!(report.ends_with('…'), "{report}"); + } +}