diff --git a/AGENTS.md b/AGENTS.md index 86a5226..25bdfca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr | Path | Purpose | |---|---| | `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample | -| `crates/x-media/src/site//` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `Site` implementing `site::Site`, `From for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Bilibili fetches dynamics (images/animated images only — an attached video degrades to its cover) from the anonymous `/x/polymer/web-dynamic/v1/detail` (no WBI signature; device cookies `buvid3`/`buvid4` are fetched automatically from `/x/frontend/finger/spi` because bilibili's `-352` risk control starts rejecting plain requests, `BILIBILI_COOKIE` is the escalation when an IP stays blocked; `b23.tv` short links are deliberately unmatched). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`>` `<` `&` `'`) — so the stored text is raw and the caption escap… +| `crates/x-media/src/site//` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `Site` implementing `site::Site`, `From for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Bilibili fetches dynamics (images/animated images only — an attached video degrades to its cover, and its title stands in for the post text, which AV dynamics do not have) from the anonymous `/x/polymer/web-dynamic/v1/detail` (no WBI signature; device cookies `buvid3`/`buvid4` are fetched automatically from `/x/frontend/finger/spi` because bilibili's `-352` risk control starts rejecting plain requests, `BILIBILI_COOKIE` is the escalation when an IP stays blocked; `b23.tv` short links are deliberately unmatched). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`>` `<` `&` `'`) — so the stored text is raw and the caption escap… | `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch | | `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` | | `crates/xmedia-bot/src/db.rs` | `DbPool`: one shared SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) for all three tables over `$DATA_DIR/task_queue.db` (default `data/`) — the three stores share it; `open_store` creates file + schema, `with_conn` runs all rusqlite I/O in `spawn_blocking` | @@ -101,7 +101,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi - **~150 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv). - No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches. -- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/bilibili/interface.rs` (3), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/mod.rs` also has a **token-gated but not `#[ignore]`d** pixiv download test (`download_media_pixiv_original_with_referer`): it hits `i.pximg.net` whenever `PIXIV_REFRESH_TOKEN` is set, so a local `cargo test --workspace` is not fully offline and can flake on a pixiv CDN body timeout. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail), and the bilibili live tests early-return when the API answers risk control (`-352`, which bilibili applies per IP by request volume). Run the full offline suite with `cargo test --workspace`. +- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/bilibili/interface.rs` (4), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/mod.rs` also has a **token-gated but not `#[ignore]`d** pixiv download test (`download_media_pixiv_original_with_referer`): it hits `i.pximg.net` whenever `PIXIV_REFRESH_TOKEN` is set, so a local `cargo test --workspace` is not fully offline and can flake on a pixiv CDN body timeout. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail), and the bilibili live tests early-return when the API answers risk control (`-352`, which bilibili applies per IP by request volume). Run the full offline suite with `cargo test --workspace`. - Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`. - **CI** — `.github/workflows/ci.yml` (actions pinned to commit SHAs, `--locked` on every cargo invocation, `concurrency` cancels superseded runs, `RUST_BACKTRACE=1`) runs `cargo fmt --check` + `cargo clippy --workspace --all-targets --locked -- -D warnings` + `cargo test --workspace --locked` + a release-profile `cargo build --release --locked` + an `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `-p x-media` since every network/secret-gated test lives there, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds and pushes the image on master/tag and runs a **build-only check on pull requests touching the build inputs** (`Dockerfile`, entrypoint, manifests, `.dockerignore`); a release tag must match both crate versions or the build stops, and `FFMPEG_URL`/`FFMPEG_SHA256` are taken from repository variables when set (a release can pin an exact ffmpeg build). `.github/dependabot.yml` keeps crates, the pinned actions and the Docker base images current. - Untested and hard to test without a mock seam: `main.rs`, `config.rs`, `db.rs`, `handlers/statics.rs`, `media_sender.rs` (holds the `MockSender` itself); in `x-media`: `media.rs`, `lib.rs`, all `model.rs`. The `commands.rs` *executor* needs a real `Bot` (only its pure report builder is tested). Everything else — `handlers/{mod,callback,inline,urls}.rs`, `send/*`, `ctx.rs`, `state.rs`, `queue.rs`, `link_cache.rs`, `rate_limit.rs` — is driven through `TestStores`/`ctx::test_support` and the scripted `MockSender`. diff --git a/BILIBILI_PLAN.md b/BILIBILI_PLAN.md index 1a65071..09dad91 100644 --- a/BILIBILI_PLAN.md +++ b/BILIBILI_PLAN.md @@ -68,6 +68,9 @@ | `playurl`（仅调研用，未采用） | `fnval=1` 匿名给 durl：720P=9.18 MiB / 360P=2.97 MiB；`fnval=4048` 匿名 DASH 上限仅 480P | | `dyn_archive` 字段 | 有 `aid/bvid/cover/title/duration_text`，**没有 `cid`**（所以发流要再来一次 `view` 请求） | | **风控阶梯（同一 IP 连续请求后实测）** | ① 无 cookie → `-352`；② 仅 `buvid3` → 仍 `-352`；③ `buvid3`+`buvid4`（取自匿名 `/x/frontend/finger/spi`）→ **`code:0` 恢复**；④ 继续高频请求后 → 连同 buvid 一起 `-352`（此时只有登录 cookie 或换 IP） | +| **正文位置（24 条真实动态逐条审计）** | 有正文的动态都在 `module_dynamic.desc.text`（图文/转发/纯文字，含 34–193 字样本）；**AV（视频投稿）动态 `desc` 恒为 `null`**，内容在 `major.archive.title` / `.desc` 卡片里 → 已做 title 回退 | +| feed 与 detail 的差异 | `feed/space` 的 item 会把 `desc.text` 挖空，**只有 detail 有正文** → 排查时不要用 feed 数据判断正文缺失 | +| 不存在的 19 位 id | `4101105 请求数据发生错误`（提示可重试，但只出现在不可能存在的 id 上）→ 仍归入永久错误，见 `code_error` 注释 | 测试样本（live 测试用）： @@ -117,6 +120,9 @@ crates/x-media/src/site/bilibili/model.rs # 纯 Deserialize DTO（全 Optio - `major.archive.cover` → 1 张 `Illustration`（视频不发流）。 - 转发且自身无媒体 → 递归取 `orig` 的媒体；正文拼 `//@{原作者}:\n{原文}`。 - 其他 major（PGC/ARTICLE/MUSIC/LIVE/COMMON）不建模 → 无媒体，走既有 "No media found"。 +- **正文 / title**：`module_dynamic.desc.text`；为空时回退到 **`major.archive.title`**。 + 实测（24 条真实动态审计 + 9 条 AV 动态）AV 动态（视频投稿）的 `desc` 恒为 `null`——它的"内容"就是卡片， + 不回退则所有视频动态的 `title`/`{title}` 都是空的。有正文的动态（图文/转发/纯文字）`desc.text` 实测正常。 - **caption**（与 misskey 同形）：`{opus 链接}\n{name}: {正文}`； `RenderData` 的 `{tags}` 来自话题名；正文由既有 `truncate_caption` 截断。 - **注册表**：`SITES` 末尾追加 → `/set_format` 白名单、链接缓存、启动校验、日志前缀全部自动生效。 diff --git a/crates/x-media/src/site/bilibili/interface.rs b/crates/x-media/src/site/bilibili/interface.rs index f6f05e2..ab88991 100644 --- a/crates/x-media/src/site/bilibili/interface.rs +++ b/crates/x-media/src/site/bilibili/interface.rs @@ -9,6 +9,9 @@ //! `x/player/playurl` and its size/quality chasing — the cover plus the post //! link is what the operator asked for. //! +//! Text: a post's own body (`module_dynamic.desc.text`); for a 视频投稿动态 +//! the body is empty by construction, so the archive card's title stands in. +//! //! `b23.tv` short links are not matched: most of them point at videos, which //! this adapter does not handle, and matching them would turn a silently //! ignored link into the bot's "Failed to fetch media" reply. @@ -221,7 +224,10 @@ pub async fn fetch(dynamic_id: &str) -> Result { /// error on purpose: the queue retries them with backoff instead of dropping /// the post. A removed or nonexistent dynamic answers `500` (verified /// 2026-09-17; the older `4101147` is kept because nazurin still documents -/// it) — permanent, so a dead link is not retried. +/// it) — permanent, so a dead link is not retried. Codes that were never +/// observed on a live post (e.g. `4101105 请求数据发生错误`, which only ever +/// came back for ids that cannot exist) stay on the permanent arm too: its +/// message hints at a retry, but the user-visible outcome is the same. fn code_error(code: i64, message: &str) -> Option { match code { 0 => None, @@ -301,6 +307,34 @@ fn desc_text(item: &model::Item) -> &str { .unwrap_or_default() } +fn archive_title(item: &model::Item) -> Option<&str> { + item.modules + .as_ref() + .and_then(|modules| modules.module_dynamic.as_ref()) + .and_then(|dynamic| dynamic.major.as_ref()) + .and_then(|major| major.archive.as_ref()) + .and_then(|archive| archive.title.as_deref()) + .filter(|title| !title.trim().is_empty()) +} + +/// The dynamic's own words: its body, or the attached video's title when the +/// body is empty. +/// +/// A 视频投稿动态 (`MAJOR_TYPE_ARCHIVE`) carries **no body at all** — `desc` +/// comes back `null`, the content being the archive card (verified on 9 live +/// AV dynamics 2026-09-17). Falling back to the card title is what keeps +/// `title` (and `{title}` in caption formats) populated for the most common +/// dynamic type, mirroring pixiv, whose `title` is the artwork title rather +/// than post text. +fn own_text(item: &model::Item) -> &str { + let body = desc_text(item); + if body.trim().is_empty() { + archive_title(item).unwrap_or_default() + } else { + body + } +} + fn topic_name(item: &model::Item) -> &str { item.modules .as_ref() @@ -313,11 +347,11 @@ fn topic_name(item: &model::Item) -> &str { /// The post's text: its own plus the quoted original's when it is a forward, /// marked the way bilibili's web UI does (`//@author:`). fn text_of(item: &model::Item) -> String { - let own = desc_text(item); + let own = own_text(item); let Some(orig) = item.orig.as_deref() else { return own.to_string(); }; - let orig_text = desc_text(orig); + let orig_text = own_text(orig); if orig_text.is_empty() { return own.to_string(); } @@ -616,6 +650,57 @@ mod tests { assert!(matches!(fetched.media[0], Media::Illustration { .. })); } + /// A 视频投稿动态 (`MAJOR_TYPE_ARCHIVE`) has **no body** — the API answers + /// `desc: null` — so the archive card's title is the post's content and + /// must fill `title` / `{title}` (regression: it used to stay empty). + #[test] + fn from_item_video_dynamic_uses_archive_title() { + let major = serde_json::json!({ + "type": "MAJOR_TYPE_ARCHIVE", + "archive": { + "bvid": "BV1JTtt6JEZu", + "cover": "http://i0.hdslb.com/bfs/archive/c.jpg", + "title": "GTX760游戏性能测试，二手显卡尚能战否？", + "desc": "入手一张2GB显存的七彩虹GTX760", + }, + }); + let item = { + let mut json = item_json(major, ""); + json["modules"]["module_dynamic"]["desc"] = serde_json::Value::Null; + json + }; + let fetched = parse(item.clone()); + + assert_eq!(fetched.title, "GTX760游戏性能测试，二手显卡尚能战否？"); + assert!( + fetched + .caption + .ends_with(": GTX760游戏性能测试，二手显卡尚能战否？"), + "{}", + fetched.caption + ); + assert_eq!( + fetched.render_fields().unwrap().2, + "GTX760游戏性能测试，二手显卡尚能战否？" + ); + assert_eq!(fetched.media.len(), 1); + + // Forwarding a video dynamic: the quoted card title lands after the + // `//@` marker, and the quoted cover becomes the media. + let mut forward = item_json(serde_json::Value::Null, ""); + forward["modules"]["module_dynamic"]["desc"] = serde_json::Value::Null; + forward["orig"] = item; + let fetched = parse(forward); + assert_eq!( + fetched.title, + "//@索尼音乐中国:\nGTX760游戏性能测试，二手显卡尚能战否？" + ); + assert_eq!( + fetched.media[0].url(), + "https://i0.hdslb.com/bfs/archive/c.jpg" + ); + } + /// A forward shell carries the quote's text and, when it has no media of /// its own, the quote's images. #[test] @@ -770,6 +855,24 @@ mod tests { assert!(!fetched.title.trim().is_empty()); } + #[tokio::test] + #[ignore = "live network: requires outbound HTTPS to api.bilibili.com"] + async fn live_fetch_video_dynamic_uses_archive_title() { + // 索尼音乐中国's AV dynamic: no body, so the video title is the text. + let Some(fetched) = live_fetch("https://t.bilibili.com/1248717597691609105").await else { + return; + }; + assert!(!fetched.title.trim().is_empty(), "{fetched:?}"); + assert!( + fetched.caption.contains(&fetched.title), + "{}", + fetched.caption + ); + let urls: Vec<&str> = fetched.media.iter().map(|m| m.url()).collect(); + assert_eq!(urls.len(), 1, "{urls:?}"); + assert!(urls[0].contains("/bfs/archive/"), "{urls:?}"); + } + /// Fetches a live dynamic, skipping the assertion when bilibili /// risk-controls this IP (the site blocks datacenter/over-used addresses /// with `-352` regardless of cookies — a real failure would surface as a diff --git a/crates/x-media/src/site/bilibili/model.rs b/crates/x-media/src/site/bilibili/model.rs index fe6333e..9a30118 100644 --- a/crates/x-media/src/site/bilibili/model.rs +++ b/crates/x-media/src/site/bilibili/model.rs @@ -97,6 +97,10 @@ pub(crate) struct Archive { /// video itself is deliberately not resolved, see the module docs). #[serde(default)] pub(crate) cover: Option, + /// The video's title. An AV dynamic has no body of its own (`desc` comes + /// back `null`), so this card title is the post's content. + #[serde(default)] + pub(crate) title: Option, } #[derive(Deserialize, Debug)]