fix(sites): use the archive title when a bilibili dynamic has no body

A 视频投稿动态 (`DYNAMIC_TYPE_AV`) carries no body at all: the API
answers `desc: null` and the content is the archive card, so `title`
(and `{title}` in caption formats) stayed empty for the most common
dynamic type. Audited 24 live dynamics: every dynamic that *has* text
(a 图文 post, a forward, a text post) keeps it in
`module_dynamic.desc.text` — only the AV card has none, so the video
title now stands in, mirroring pixiv whose `title` is the artwork title
rather than post text.

Also records two API observations in comments/docs: an id that cannot
exist answers `4101105 请求数据发生错误` (kept on the permanent arm), and
the feed endpoints strip `desc.text` so only the detail endpoint shows
whether a post has text.
This commit is contained in:
2026-09-17 22:12:53 +08:00
parent c1f5d3ca54
commit 5c51de217a
4 changed files with 118 additions and 5 deletions
+2 -2
View File
@@ -31,7 +31,7 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr
| Path | Purpose |
|---|---|
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky\|misskey\|bilibili>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Bilibili fetches dynamics (images/animated images only — an attached video degrades to its cover) from the anonymous `/x/polymer/web-dynamic/v1/detail` (no WBI signature; device cookies `buvid3`/`buvid4` are fetched automatically from `/x/frontend/finger/spi` because bilibili's `-352` risk control starts rejecting plain requests, `BILIBILI_COOKIE` is the escalation when an IP stays blocked; `b23.tv` short links are deliberately unmatched). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`&gt;` `&lt;` `&amp;` `&#39;`) — so the stored text is raw and the caption escap…
| `crates/x-media/src/site/<twitter\|pixiv\|bsky\|misskey\|bilibili>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Bilibili fetches dynamics (images/animated images only — an attached video degrades to its cover, and its title stands in for the post text, which AV dynamics do not have) from the anonymous `/x/polymer/web-dynamic/v1/detail` (no WBI signature; device cookies `buvid3`/`buvid4` are fetched automatically from `/x/frontend/finger/spi` because bilibili's `-352` risk control starts rejecting plain requests, `BILIBILI_COOKIE` is the escalation when an IP stays blocked; `b23.tv` short links are deliberately unmatched). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`&gt;` `&lt;` `&amp;` `&#39;`) — so the stored text is raw and the caption escap…
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
| `crates/xmedia-bot/src/db.rs` | `DbPool`: one shared SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) for all three tables over `$DATA_DIR/task_queue.db` (default `data/`) — the three stores share it; `open_store` creates file + schema, `with_conn` runs all rusqlite I/O in `spawn_blocking` |
@@ -101,7 +101,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
- **~150 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches.
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/bilibili/interface.rs` (3), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/mod.rs` also has a **token-gated but not `#[ignore]`d** pixiv download test (`download_media_pixiv_original_with_referer`): it hits `i.pximg.net` whenever `PIXIV_REFRESH_TOKEN` is set, so a local `cargo test --workspace` is not fully offline and can flake on a pixiv CDN body timeout. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""``is_err()` alone would run them tokenless and fail), and the bilibili live tests early-return when the API answers risk control (`-352`, which bilibili applies per IP by request volume). Run the full offline suite with `cargo test --workspace`.
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/bilibili/interface.rs` (4), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/mod.rs` also has a **token-gated but not `#[ignore]`d** pixiv download test (`download_media_pixiv_original_with_referer`): it hits `i.pximg.net` whenever `PIXIV_REFRESH_TOKEN` is set, so a local `cargo test --workspace` is not fully offline and can flake on a pixiv CDN body timeout. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""``is_err()` alone would run them tokenless and fail), and the bilibili live tests early-return when the API answers risk control (`-352`, which bilibili applies per IP by request volume). Run the full offline suite with `cargo test --workspace`.
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
- **CI** — `.github/workflows/ci.yml` (actions pinned to commit SHAs, `--locked` on every cargo invocation, `concurrency` cancels superseded runs, `RUST_BACKTRACE=1`) runs `cargo fmt --check` + `cargo clippy --workspace --all-targets --locked -- -D warnings` + `cargo test --workspace --locked` + a release-profile `cargo build --release --locked` + an `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `-p x-media` since every network/secret-gated test lives there, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds and pushes the image on master/tag and runs a **build-only check on pull requests touching the build inputs** (`Dockerfile`, entrypoint, manifests, `.dockerignore`); a release tag must match both crate versions or the build stops, and `FFMPEG_URL`/`FFMPEG_SHA256` are taken from repository variables when set (a release can pin an exact ffmpeg build). `.github/dependabot.yml` keeps crates, the pinned actions and the Docker base images current.
- Untested and hard to test without a mock seam: `main.rs`, `config.rs`, `db.rs`, `handlers/statics.rs`, `media_sender.rs` (holds the `MockSender` itself); in `x-media`: `media.rs`, `lib.rs`, all `model.rs`. The `commands.rs` *executor* needs a real `Bot` (only its pure report builder is tested). Everything else — `handlers/{mod,callback,inline,urls}.rs`, `send/*`, `ctx.rs`, `state.rs`, `queue.rs`, `link_cache.rs`, `rate_limit.rs` — is driven through `TestStores`/`ctx::test_support` and the scripted `MockSender`.
+6
View File
@@ -68,6 +68,9 @@
| `playurl`(仅调研用,未采用) | `fnval=1` 匿名给 durl720P=9.18 MiB / 360P=2.97 MiB`fnval=4048` 匿名 DASH 上限仅 480P |
| `dyn_archive` 字段 | 有 `aid/bvid/cover/title/duration_text`**没有 `cid`**(所以发流要再来一次 `view` 请求) |
| **风控阶梯(同一 IP 连续请求后实测)** | ① 无 cookie → `-352`;② 仅 `buvid3` → 仍 `-352`;③ `buvid3`+`buvid4`(取自匿名 `/x/frontend/finger/spi`)→ **`code:0` 恢复**;④ 继续高频请求后 → 连同 buvid 一起 `-352`(此时只有登录 cookie 或换 IP) |
| **正文位置(24 条真实动态逐条审计)** | 有正文的动态都在 `module_dynamic.desc.text`(图文/转发/纯文字,含 34–193 字样本);**AV(视频投稿)动态 `desc` 恒为 `null`**,内容在 `major.archive.title` / `.desc` 卡片里 → 已做 title 回退 |
| feed 与 detail 的差异 | `feed/space` 的 item 会把 `desc.text` 挖空,**只有 detail 有正文** → 排查时不要用 feed 数据判断正文缺失 |
| 不存在的 19 位 id | `4101105 请求数据发生错误`(提示可重试,但只出现在不可能存在的 id 上)→ 仍归入永久错误,见 `code_error` 注释 |
测试样本(live 测试用):
@@ -117,6 +120,9 @@ crates/x-media/src/site/bilibili/model.rs # 纯 Deserialize DTO(全 Optio
- `major.archive.cover` → 1 张 `Illustration`(视频不发流)。
- 转发且自身无媒体 → 递归取 `orig` 的媒体;正文拼 `//@{原作者}:\n{原文}`
- 其他 majorPGC/ARTICLE/MUSIC/LIVE/COMMON)不建模 → 无媒体,走既有 "No media found"。
- **正文 / title**`module_dynamic.desc.text`;为空时回退到 **`major.archive.title`**。
实测(24 条真实动态审计 + 9 条 AV 动态)AV 动态(视频投稿)的 `desc` 恒为 `null`——它的"内容"就是卡片,
不回退则所有视频动态的 `title`/`{title}` 都是空的。有正文的动态(图文/转发/纯文字)`desc.text` 实测正常。
- **caption**(与 misskey 同形):`{opus 链接}\n<a href="space.bilibili.com/{mid}">{name}</a>: {正文}`
`RenderData``{tags}` 来自话题名;正文由既有 `truncate_caption` 截断。
- **注册表**`SITES` 末尾追加 → `/set_format` 白名单、链接缓存、启动校验、日志前缀全部自动生效。
+106 -3
View File
@@ -9,6 +9,9 @@
//! `x/player/playurl` and its size/quality chasing — the cover plus the post
//! link is what the operator asked for.
//!
//! Text: a post's own body (`module_dynamic.desc.text`); for a 视频投稿动态
//! the body is empty by construction, so the archive card's title stands in.
//!
//! `b23.tv` short links are not matched: most of them point at videos, which
//! this adapter does not handle, and matching them would turn a silently
//! ignored link into the bot's "Failed to fetch media" reply.
@@ -221,7 +224,10 @@ pub async fn fetch(dynamic_id: &str) -> Result<model::Item, FetchError> {
/// error on purpose: the queue retries them with backoff instead of dropping
/// the post. A removed or nonexistent dynamic answers `500` (verified
/// 2026-09-17; the older `4101147` is kept because nazurin still documents
/// it) — permanent, so a dead link is not retried.
/// it) — permanent, so a dead link is not retried. Codes that were never
/// observed on a live post (e.g. `4101105 请求数据发生错误`, which only ever
/// came back for ids that cannot exist) stay on the permanent arm too: its
/// message hints at a retry, but the user-visible outcome is the same.
fn code_error(code: i64, message: &str) -> Option<FetchError> {
match code {
0 => None,
@@ -301,6 +307,34 @@ fn desc_text(item: &model::Item) -> &str {
.unwrap_or_default()
}
fn archive_title(item: &model::Item) -> Option<&str> {
item.modules
.as_ref()
.and_then(|modules| modules.module_dynamic.as_ref())
.and_then(|dynamic| dynamic.major.as_ref())
.and_then(|major| major.archive.as_ref())
.and_then(|archive| archive.title.as_deref())
.filter(|title| !title.trim().is_empty())
}
/// The dynamic's own words: its body, or the attached video's title when the
/// body is empty.
///
/// A 视频投稿动态 (`MAJOR_TYPE_ARCHIVE`) carries **no body at all** — `desc`
/// comes back `null`, the content being the archive card (verified on 9 live
/// AV dynamics 2026-09-17). Falling back to the card title is what keeps
/// `title` (and `{title}` in caption formats) populated for the most common
/// dynamic type, mirroring pixiv, whose `title` is the artwork title rather
/// than post text.
fn own_text(item: &model::Item) -> &str {
let body = desc_text(item);
if body.trim().is_empty() {
archive_title(item).unwrap_or_default()
} else {
body
}
}
fn topic_name(item: &model::Item) -> &str {
item.modules
.as_ref()
@@ -313,11 +347,11 @@ fn topic_name(item: &model::Item) -> &str {
/// The post's text: its own plus the quoted original's when it is a forward,
/// marked the way bilibili's web UI does (`//@author:`).
fn text_of(item: &model::Item) -> String {
let own = desc_text(item);
let own = own_text(item);
let Some(orig) = item.orig.as_deref() else {
return own.to_string();
};
let orig_text = desc_text(orig);
let orig_text = own_text(orig);
if orig_text.is_empty() {
return own.to_string();
}
@@ -616,6 +650,57 @@ mod tests {
assert!(matches!(fetched.media[0], Media::Illustration { .. }));
}
/// A 视频投稿动态 (`MAJOR_TYPE_ARCHIVE`) has **no body** — the API answers
/// `desc: null` — so the archive card's title is the post's content and
/// must fill `title` / `{title}` (regression: it used to stay empty).
#[test]
fn from_item_video_dynamic_uses_archive_title() {
let major = serde_json::json!({
"type": "MAJOR_TYPE_ARCHIVE",
"archive": {
"bvid": "BV1JTtt6JEZu",
"cover": "http://i0.hdslb.com/bfs/archive/c.jpg",
"title": "GTX760游戏性能测试,二手显卡尚能战否?",
"desc": "入手一张2GB显存的七彩虹GTX760",
},
});
let item = {
let mut json = item_json(major, "");
json["modules"]["module_dynamic"]["desc"] = serde_json::Value::Null;
json
};
let fetched = parse(item.clone());
assert_eq!(fetched.title, "GTX760游戏性能测试,二手显卡尚能战否?");
assert!(
fetched
.caption
.ends_with(": GTX760游戏性能测试,二手显卡尚能战否?"),
"{}",
fetched.caption
);
assert_eq!(
fetched.render_fields().unwrap().2,
"GTX760游戏性能测试,二手显卡尚能战否?"
);
assert_eq!(fetched.media.len(), 1);
// Forwarding a video dynamic: the quoted card title lands after the
// `//@` marker, and the quoted cover becomes the media.
let mut forward = item_json(serde_json::Value::Null, "");
forward["modules"]["module_dynamic"]["desc"] = serde_json::Value::Null;
forward["orig"] = item;
let fetched = parse(forward);
assert_eq!(
fetched.title,
"//@索尼音乐中国:\nGTX760游戏性能测试,二手显卡尚能战否?"
);
assert_eq!(
fetched.media[0].url(),
"https://i0.hdslb.com/bfs/archive/c.jpg"
);
}
/// A forward shell carries the quote's text and, when it has no media of
/// its own, the quote's images.
#[test]
@@ -770,6 +855,24 @@ mod tests {
assert!(!fetched.title.trim().is_empty());
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to api.bilibili.com"]
async fn live_fetch_video_dynamic_uses_archive_title() {
// 索尼音乐中国's AV dynamic: no body, so the video title is the text.
let Some(fetched) = live_fetch("https://t.bilibili.com/1248717597691609105").await else {
return;
};
assert!(!fetched.title.trim().is_empty(), "{fetched:?}");
assert!(
fetched.caption.contains(&fetched.title),
"{}",
fetched.caption
);
let urls: Vec<&str> = fetched.media.iter().map(|m| m.url()).collect();
assert_eq!(urls.len(), 1, "{urls:?}");
assert!(urls[0].contains("/bfs/archive/"), "{urls:?}");
}
/// Fetches a live dynamic, skipping the assertion when bilibili
/// risk-controls this IP (the site blocks datacenter/over-used addresses
/// with `-352` regardless of cookies — a real failure would surface as a
@@ -97,6 +97,10 @@ pub(crate) struct Archive {
/// video itself is deliberately not resolved, see the module docs).
#[serde(default)]
pub(crate) cover: Option<String>,
/// The video's title. An AV dynamic has no body of its own (`desc` comes
/// back `null`), so this card title is the post's content.
#[serde(default)]
pub(crate) title: Option<String>,
}
#[derive(Deserialize, Debug)]