From abdc27ed5e4e150db654d068a174159e93136c95 Mon Sep 17 00:00:00 2001 From: YoursFunny Date: Wed, 16 Sep 2026 23:28:46 +0800 Subject: [PATCH] docs: resync AGENTS.md and the doc comments with the code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md: - db.rs row claimed a per-store connection pool; there is one shared pool for all three tables (statics.rs builds it once). - retry enqueue moved to send.rs, noted in both handlers rows. - queue row now names both notifies (workers' + the sweep's). - Retries bullet documents fetch vs fetch_once. - test count ~125 -> ~135, the untested-files list no longer claims state.rs and handlers.rs are untested, and the live-test inventory mentions the token-gated, not-#[ignore]d pixiv download test that makes a local `cargo test --workspace` hit the network. - /bot_dict is admin-only now. Code docs: - site/mod.rs: the module doc pointed new sites at `fetch_once` (a name that did not exist then and now means a single-attempt fetch) -> `SITES`; the cache_key/SITES/Site docs still said "twitter -> bsky -> pixiv" (misskey is registered third); RenderData now documents which fields are escaped and why url/author_url are not. - state.rs, callback.rs: drop the pre-misskey site list and the `` that rustdoc read as an HTML tag. - Fixed the remaining rustdoc links/warnings: `cargo doc --workspace --no-deps` is now warning-free (was 8). - docs/site-registry-refactor.md: §1 describes the pre-refactor state; said so. No behavior change. fmt/clippy clean, 55 + 68 tests pass (the live pixiv download test flaked on a CDN body timeout, as before). --- AGENTS.md | 16 +++++----- crates/x-media/src/site/mod.rs | 36 +++++++++++++--------- crates/xmedia-bot/src/handlers/callback.rs | 4 +-- crates/xmedia-bot/src/link_cache.rs | 2 +- crates/xmedia-bot/src/send.rs | 9 +++--- crates/xmedia-bot/src/state.rs | 4 +-- docs/site-registry-refactor.md | 4 +++ 7 files changed, 44 insertions(+), 31 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6bbbe98..b63c098 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,11 +32,11 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr | `crates/x-media/src/site//` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `Site` implementing `site::Site`, `From for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`>` `<` `&` `'`) — so the stored text is raw and the caption escapes exactly once | | `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch | | `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` | -| `crates/xmedia-bot/src/db.rs` | `DbPool`: per-store SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) over `$DATA_DIR/task_queue.db` (default `data/`); `open_store` creates file + schema, `with_conn` runs all rusqlite I/O in `spawn_blocking` | -| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. the `/test ` parse-only debug command), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons), `statics.rs` (global statics) | +| `crates/xmedia-bot/src/db.rs` | `DbPool`: one shared SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) for all three tables over `$DATA_DIR/task_queue.db` (default `data/`) — the three stores share it; `open_store` creates file + schema, `with_conn` runs all rusqlite I/O in `spawn_blocking` | +| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. the `/test ` parse-only debug command and the admin-only `/bot_dict` state dump), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons), `statics.rs` (global statics) | | `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex` cache + SQLite write-through (`chat_state` table) | | `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure | -| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections | +| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `notify_one` worker wakeup plus a separate `Notify` for the 30 s lease-expiry sweep (a shared one let the sweep steal the workers' wakeup permit), `busy_timeout` on all connections | | `crates/xmedia-bot/src/send.rs` | Media senders, upload fallback, error classification, queue task handlers | | `crates/xmedia-bot/src/media_sender.rs` | `MediaSender` trait: the send surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`send_chat_action`) implemented by teloxide `Bot` (per-chat rate-limited) and by a scripted `MockSender` in tests | | `crates/xmedia-bot/src/rate_limit.rs` | Per-chat token bucket (`CAPACITY = 20`, ~20 msg/min refill) paced before sends reach the API so batch forwards don't trip flood control | @@ -64,7 +64,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi - **Site adapter convention**: each site module exports `PATTERN: LazyLock`, `enabled() -> bool`, `fetch_from_url(url) -> Result`, plus `cache_key`/`is_retryable`/`media_headers`, and a unit struct `Site` implementing `site::Site`; the central dispatcher (`site/mod.rs`) only iterates the `SITES` registry. Adding a site = new `site//{mod.rs,interface.rs,model.rs}` + one `Box::new(...)` entry in `SITES` — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return `SiteFuture` (a boxed `Pin>`) because `async fn` in traits is not dyn-compatible. - **Serde**: per-site `model.rs` are pure `Deserialize` DTOs mirroring API JSON; site structs in `interface.rs` have private fields, a `caption()` builder, and `impl From for Fetched`. Persisted payloads use internally-tagged enums (`#[serde(tag = "kind")]` / `type`). - **Naming**: module-per-concern, snake_case files, `CamelCase` types, `snake_case` fns. `//!` module docs and `///` docs on non-obvious logic (syndication token, ugoira encoding, `display_text_range`). -- **Retries**: only `x-media::site::fetch` retries (3 attempts, `1 << attempt` backoff, HTTP errors only). Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`). +- **Retries**: only `x-media::site::fetch` retries (3 attempts, `1 << attempt` backoff, HTTP errors only); `site::fetch_once` is the same code path with a single attempt, used by inline queries whose answer window is shorter than the backoff. Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`). - Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`). Level convention: `info` = lifecycle + per-post business results (`sent`/`forwarded`/`copied`), admin/operator actions and anomalies (fallback, retry enqueue, dead-letter is `error`); `debug` = per-request detail (message/command/URL extraction, `fetching`/`fetched`, batch sends, queue processing, photo processing, inline queries). Full user-submitted URLs and message text only appear at `debug`; at `info` and above links are printed via the normalized cache key (`handlers::log_key`, e.g. `[key=twitter:123...]`) so logs stay short and do not echo user data. ## Important Files @@ -72,7 +72,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi | File | Why it matters | |---|---| | `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) | -| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `commands.rs` = command dispatch (incl. the `/test ` parse-only debug command); `urls.rs` = URL extraction + retry enqueue; `inline.rs`/`callback.rs` = inline queries / edit-before-forward buttons | +| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `commands.rs` = command dispatch (incl. the `/test ` parse-only debug command and the admin-only `/bot_dict` state dump); `urls.rs` = URL extraction + the per-URL pipeline (`enqueue_retry` lives in `send.rs`); `inline.rs`/`callback.rs` = inline queries / edit-before-forward buttons | | `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`; fallback chain; `classify_request_error`; download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`) | | `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL | | `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) | @@ -96,10 +96,10 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi ## Testing & QA -- **~125 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv). +- **~135 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv). - No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches. -- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`. +- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/mod.rs` also has a **token-gated but not `#[ignore]`d** pixiv download test (`download_media_pixiv_original_with_referer`): it hits `i.pximg.net` whenever `PIXIV_REFRESH_TOKEN` is set, so a local `cargo test --workspace` is not fully offline and can flake on a pixiv CDN body timeout. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`. - Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`. - **CI** — `.github/workflows/ci.yml` runs `cargo fmt --check` + `cargo clippy --workspace --all-targets -- -D warnings` + `cargo test --workspace` + a `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN` from secrets, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds/pushes the image only. -- Untested and hard to test without a mock seam: `handlers.rs` (depends directly on teloxide `Bot`); `main.rs`, `config.rs`, `state.rs`; `media.rs`, `lib.rs`, all `model.rs`. +- Untested and hard to test without a mock seam: `main.rs`, `config.rs`, `db.rs`, `media_sender.rs` (holds `MockSender` itself); `handlers/mod.rs`, `handlers/callback.rs`, `handlers/statics.rs` (need a real teloxide `Bot`); `media.rs`, `lib.rs`, all `model.rs`. `handlers/urls.rs`/`handlers/commands.rs`/`send.rs`/`state.rs`/`queue.rs`/`link_cache.rs`/`rate_limit.rs` are covered through their injected stores and the scripted `MockSender`. - No coverage tracking. diff --git a/crates/x-media/src/site/mod.rs b/crates/x-media/src/site/mod.rs index a54c766..018045e 100644 --- a/crates/x-media/src/site/mod.rs +++ b/crates/x-media/src/site/mod.rs @@ -2,7 +2,7 @@ //! //! Dispatch order: twitter → bsky → misskey → pixiv. Each site module //! exports a `PATTERN`, `enabled()` and `fetch_from_url()`; a future site -//! plugs in by adding one guarded entry in [`fetch_once`]. +//! plugs in by adding one guarded entry in `SITES`. use std::future::Future; use std::pin::Pin; @@ -24,9 +24,9 @@ pub use pixiv::PixivError; /// media list and spoiler flag. Produced by [`fetch`]. #[derive(Debug)] pub struct Fetched { - /// Canonical URL: x.com/{author}/status/{id} | - /// https://www.pixiv.net/artworks/{id} | - /// https://bsky.app/profile/{handle}/post/{rkey} + /// Canonical URL: `x.com/{author}/status/{id}` | + /// `https://www.pixiv.net/artworks/{id}` | + /// `https://bsky.app/profile/{handle}/post/{rkey}` pub source_url: String, /// The exact HTML produced by the site's caption(). pub caption: String, @@ -46,8 +46,15 @@ pub struct Fetched { pub(crate) _keep_alive: Option, } -/// Pre-escaped values for `{url} {author} {author_url} {title} {tags}` -/// placeholders in user-supplied caption formats. +/// Values for the `{url} {author} {author_url} {title} {tags}` placeholders in +/// user-supplied caption formats, substituted by [`caption_from_fields`] as +/// HTML text (never as an attribute value). +/// +/// `author`, `title` and `tags` come from the site API (post text, display +/// names) and are HTML-escaped at construction. `url` and `author_url` stay +/// raw: they are canonical URLs the adapter builds from numeric ids and +/// API-constrained handles/DIDs, so they carry no escapable character — the +/// bot's `/test` report relies on that when it embeds them. #[derive(Debug)] pub(crate) struct RenderData { pub url: String, @@ -166,7 +173,7 @@ pub fn caption_from_fields( /// Stable per-post cache key derived from any supported URL, so variant /// domains (x.com / twitter.com / fxtwitter.com, mobile, `/photo/N` /// suffixes) map to the same post. Delegates to each registered site's -/// `cache_key` (dispatch order twitter → bsky → pixiv). +/// `cache_key` (in registry order). pub fn cache_key(url: &str) -> Option { SITES.iter().find_map(|site| site.cache_key(url)) } @@ -275,20 +282,21 @@ pub(crate) fn log_once_ffmpeg_missing() { } } -/// Site adapter: one impl per supported site (twitter / bsky / pixiv), -/// registered in [`SITES`]. All site-specific knowledge — URL pattern, +/// Site adapter: one impl per supported site (twitter / bsky / misskey / +/// pixiv), registered in `SITES`. All site-specific knowledge — URL pattern, /// cache-key format, fetch, retry policy, media-host headers, startup /// validation — lives in the site module; the central dispatcher only /// iterates the registry. /// -/// Async methods return a boxed future (see [`SiteFuture`]): `async fn` / +/// Async methods return a boxed future (see `SiteFuture`): `async fn` / /// RPITIT in traits are not dyn-compatible (verified on rustc 1.95), and /// `+ Send` is required since URL/queue workers spawn these futures. The /// site structs are stateless unit structs, so the boxed futures never /// borrow from `self` beyond the call's scope. pub trait Site: Send + Sync { - /// Stable site id (`"twitter"` / `"bsky"` / `"pixiv"`): caption-format - /// lookup, cache-key prefixes and the SetFormat whitelist derive from it. + /// Stable site id (`"twitter"` / `"bsky"` / `"misskey"` / `"pixiv"`): + /// caption-format lookup, cache-key prefixes and the SetFormat whitelist + /// derive from it. fn id(&self) -> &'static str; /// URL pattern; the dispatcher's first match wins (dispatch order). fn pattern(&self) -> &'static Regex; @@ -324,8 +332,8 @@ pub trait Site: Send + Sync { type SiteFuture<'a, T, E = FetchError> = Pin> + Send + 'a>>; /// The one registry of supported sites, in dispatch order (twitter → bsky → -/// pixiv). Adding a site = new module + one `Box::new(...)` entry here; the -/// bot crate never lists sites itself. +/// misskey → pixiv). Adding a site = new module + one `Box::new(...)` entry +/// here; the bot crate never lists sites itself. static SITES: LazyLock>> = LazyLock::new(|| { vec![ Box::new(twitter::TwitterSite), diff --git a/crates/xmedia-bot/src/handlers/callback.rs b/crates/xmedia-bot/src/handlers/callback.rs index a4244f7..b7f90b7 100644 --- a/crates/xmedia-bot/src/handlers/callback.rs +++ b/crates/xmedia-bot/src/handlers/callback.rs @@ -1,5 +1,5 @@ -//! Callback query handling: the edit-before-forward prompt's "forward" and -//! "template|" buttons. +//! Callback query handling: the edit-before-forward prompt's `"forward"` and +//! `"template|"` buttons. use super::{CHAT_STORE, CONFIG, TASK_QUEUE}; use crate::db::unix_now; diff --git a/crates/xmedia-bot/src/link_cache.rs b/crates/xmedia-bot/src/link_cache.rs index fc86510..c502466 100644 --- a/crates/xmedia-bot/src/link_cache.rs +++ b/crates/xmedia-bot/src/link_cache.rs @@ -5,7 +5,7 @@ //! key]. A repeated link is then answered entirely from local state — no //! re-fetch of the source site, no re-upload — and no media file is stored //! on disk (the file ids point at Telegram's servers). Entries expire after -//! [`Config::link_cache_ttl`]; a stale entry is dropped lazily on read and +//! `Config::link_cache_ttl`; a stale entry is dropped lazily on read and //! by the periodic prune in `main`. use crate::db::now_f64; diff --git a/crates/xmedia-bot/src/send.rs b/crates/xmedia-bot/src/send.rs index d166c85..d798bb0 100644 --- a/crates/xmedia-bot/src/send.rs +++ b/crates/xmedia-bot/src/send.rs @@ -271,10 +271,11 @@ pub async fn invalidate_cache_with(cache: &LinkCache, task: &Task) { /// Locally produced media files (ugoira MP4, bsky remux MP4) whose temp dirs /// must stay alive while their task may be retried by the queue. The fetch -/// pipeline hands ownership here via [`x_media::site::Fetched::take_keep_alive`] -/// before the [`Fetched`] is dropped; a queued retry runs after that drop, so -/// without this the local file would be gone by the time the retry sends it. -/// Entries are removed when the task settles (see [`release_keep_alive`]). +/// pipeline hands ownership here via +/// [`x_media::site::Fetched::take_keep_alive`] before that +/// [`x_media::site::Fetched`] is dropped; a queued retry runs after that drop, +/// so without this the local file would be gone by the time the retry sends +/// it. Entries are removed when the task settles (see [`release_keep_alive`]). pub static KEEP_ALIVE: LazyLock>> = LazyLock::new(|| parking_lot::Mutex::new(Vec::new())); diff --git a/crates/xmedia-bot/src/state.rs b/crates/xmedia-bot/src/state.rs index ab77b48..499ddb9 100644 --- a/crates/xmedia-bot/src/state.rs +++ b/crates/xmedia-bot/src/state.rs @@ -17,8 +17,8 @@ pub struct ChatData { pub edit_message: HashMap, /// name -> HTML template containing "[]" pub template: HashMap, - /// site name (twitter/bsky/pixiv) -> user-supplied caption format with - /// {url} {author} {author_url} {title} {tags} placeholders. + /// site name (twitter/bsky/misskey/pixiv) -> user-supplied caption format + /// with {url} {author} {author_url} {title} {tags} placeholders. pub message_format: HashMap, } diff --git a/docs/site-registry-refactor.md b/docs/site-registry-refactor.md index 448b9c7..6d61ff7 100644 --- a/docs/site-registry-refactor.md +++ b/docs/site-registry-refactor.md @@ -10,6 +10,10 @@ ## 1. 现状摩擦清单 +> ⚠️ 本节记录的是**重构前**的现状：其中的行号、以及 `site/mod.rs` 里的 +> `fetch_once` 分派函数（当时的实现）都已不存在，仅作历史记录。当前形态见 +> `site/mod.rs` 的 `SITES` 注册表——新增站点 = 新模块 + 注册一行。 + 以现有三站（twitter / bsky / pixiv）为基线，新增第 4 个站点（代号 `example`） 今天需要触碰的位置：