Compare commits

...
4 Commits
Author SHA1 Message Date
YoursFunny 32254fa807 chore: bump version to 1.5.0 2026-09-07 21:26:24 +08:00
YoursFunny 11c04b66dc feat: add Misskey (misskey.io) fetch support
Fourth site adapter: POST /api/notes/show, renote-aware caption and
media normalization, DriveFile type → Illustration/Animated/Video.
Empty thumbnailUrl strings filtered out in thumbnail_for.
2026-09-07 21:25:52 +08:00
YoursFunny 2f741e5f4b refactor(send): apply ponytail audit cuts 2, 4, 6
- updated_sequence_task: clone the Task and mutate the two fields
  instead of rebuilding all 12 by hand (-22 lines; new fields no
  longer need a sync here)
- unify unix_now with db::now_f64 (unix_now() = now_f64() as i64),
  moved to db.rs next to its clock source
- classify_to_send_error takes the MediaFetchFailure label, folding
  the duplicated inline match in send_batch_via_upload (-8 lines)
2026-09-07 19:19:56 +08:00
YoursFunny 89c4642e1c fix(lint): resolve clippy warnings from rust 1.98
- photo.rs: chunks_exact(4)/(2) -> as_chunks::<N>().0
  (chunks_exact_to_as_chunks, the new lint prefers the
  compile-time-checked slice split)
- send.rs: box the Task inside SendError so the error fits the
  result_large_err limit (Task is ~400 bytes; the error now moves
  through Result as a pointer); unbox with *task at the two
  enqueue_retry call sites (handlers/urls.rs, handlers/callback.rs)

cargo clippy --workspace --all-targets is now warning-free; the
remaining proc-macro-error2 future-incompat note is upstream
(teloxide -> aquamarine) and unfixable locally. Full test suite passes.
2026-09-07 17:00:38 +08:00
17 changed files with 519 additions and 92 deletions
+7 -7
View File
@@ -2,11 +2,11 @@
## Project Overview
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README is in Chinese; user-facing bot strings are in English. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, Bluesky, and Misskey (misskey.io) into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README is in Chinese; user-facing bot strings are in English. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
Two-crate Cargo workspace (both v1.4.0, edition 2024, resolver 3):
Two-crate Cargo workspace (both v1.5.0, edition 2024, resolver 3):
- **`crates/x-media`** — library that fetches and normalizes media from the three sites. Pure, no Telegram knowledge.
- **`crates/x-media`** — library that fetches and normalizes media from the four sites. Pure, no Telegram knowledge.
- **`crates/xmedia-bot`** — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.
## Architecture & Data Flow
@@ -22,14 +22,14 @@ Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities,
Debug command: `/test <url>` runs the same `x_media::site::fetch` and replies with `test_parse_report` (`handlers/commands.rs`) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars and sent with HTML parse mode: raw fields are escaped, and the caption is wrapped in a `<blockquote>` so it renders exactly like the sent media caption (escaped text and links included). It uses a custom `parse_test_arg` parser (whole remainder, trimmed) because teloxide's built-in `split` parser takes exactly one space-separated token.
The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → pixiv) and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, site_id, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → misskey → pixiv) and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, site_id, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
## Key Directories
| Path | Purpose |
|---|---|
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`&gt;` `&lt;` `&amp;` `&#39;`) — so the stored text is raw and the caption escapes exactly once |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky\|misskey>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`&gt;` `&lt;` `&amp;` `&#39;`) — so the stored text is raw and the caption escapes exactly once |
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
| `crates/xmedia-bot/src/db.rs` | `DbPool`: per-store SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) over `$DATA_DIR/task_queue.db` (default `data/`); `open_store` creates file + schema, `with_conn` runs all rusqlite I/O in `spawn_blocking` |
@@ -96,9 +96,9 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
## Testing & QA
- **~115 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- **~125 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches.
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`.
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`.
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
- **CI** — `.github/workflows/ci.yml` runs `cargo fmt --check` + `cargo clippy --workspace --all-targets -- -D warnings` + `cargo test --workspace` + a `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN` from secrets, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds/pushes the image only.
- Untested and hard to test without a mock seam: `handlers.rs` (depends directly on teloxide `Bot`); `main.rs`, `config.rs`, `state.rs`; `media.rs`, `lib.rs`, all `model.rs`.
Generated
+2 -2
View File
@@ -2925,7 +2925,7 @@ checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
[[package]]
name = "x-media"
version = "1.4.0"
version = "1.5.0"
dependencies = [
"bytes",
"dotenv",
@@ -2945,7 +2945,7 @@ dependencies = [
[[package]]
name = "xmedia-bot"
version = "1.4.0"
version = "1.5.0"
dependencies = [
"bytes",
"dotenv",
+2 -2
View File
@@ -1,6 +1,6 @@
# TelegramXMediaBot
A Telegram bot that turns post links from X / Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags.
A Telegram bot that turns post links from X / Twitter, Pixiv, Bluesky, and Misskey (misskey.io) into media messages (images, video, GIF) with the post's title, author, and tags.
## Features
@@ -111,7 +111,7 @@ Telegram only accepts ports 443/80/88/8443.
| `/remove_forward_channel` | Remove the forward channel |
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or taps a template button to apply one) |
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward") |
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv` / `misskey`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/clear_cache [link]` | Clear the link cache (admin only); with a link only that entry, otherwise everything |
| `/bot_dict` | Show the current chat state (debugging) |
| `/test <link>` | Debug: parse a link and report the parse result only (site, title, author, tags, media list) — no media is sent |
+2 -2
View File
@@ -1,6 +1,6 @@
# TelegramXMediaBot
Telegram 机器人,将 X / Twitter、Pixiv、Bluesky 的帖子链接转换为媒体消息发送,附带帖子标题、作者与标签。
Telegram 机器人,将 X / Twitter、Pixiv、Bluesky、Misskey (misskey.io) 的帖子链接转换为媒体消息发送,附带帖子标题、作者与标签。
## 功能
@@ -111,7 +111,7 @@ Telegram 只接受 443/80/88/8443 端口。
| `/remove_forward_channel` | 取消转发频道 |
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv` / `misskey`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/clear_cache [链接]` | 清空链接缓存(仅管理员);带链接只清该条,否则清空全部 |
| `/bot_dict` | 查看当前聊天状态(调试用) |
| `/test <链接>` | 调试:只解析链接并返回解析结果(站点、标题、作者、标签、媒体列表),不发送任何媒体 |
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "x-media"
version = "1.4.0"
version = "1.5.0"
edition = "2024"
[dependencies]
@@ -0,0 +1,384 @@
//! Site adapter for misskey.io notes: URL pattern, API fetch and
//! normalization into [`Fetched`] (see [`crate::site::Site`]).
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched, RenderData, Site, SiteFuture};
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
const API_URL: &str = "https://misskey.io/api/notes/show";
/// Registry entry for the misskey.io adapter (see [`crate::site::Site`]).
pub struct MisskeySite;
impl Site for MisskeySite {
fn id(&self) -> &'static str {
"misskey"
}
fn pattern(&self) -> &'static Regex {
&PATTERN
}
fn cache_key(&self, url: &str) -> Option<String> {
cache_key(url)
}
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched> {
Box::pin(async move { fetch_from_url(url).await })
}
}
pub static PATTERN: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^(?:https?://)?misskey\.io/notes/([\w.\-~]+)").unwrap());
pub fn enabled() -> bool {
true
}
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
let caps = PATTERN.captures(url).ok_or(FetchError::NotFound)?;
let note_id = caps.get(1).ok_or(FetchError::NotFound)?.as_str();
let note = fetch(note_id).await?;
Ok(note.into())
}
/// Cache key for a misskey URL: `"misskey:<note id>"`. The prefix is the
/// site id used for caption-format lookup and link-cache keys.
pub fn cache_key(url: &str) -> Option<String> {
PATTERN
.captures(url)
.map(|caps| format!("misskey:{}", &caps[1]))
}
/// Misskey's fetch-retry policy: transient classes only. Not-found, blocked
/// and parse failures are permanent.
pub fn is_retryable(err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
}
/// misskey.io media hosts need no extra headers (verified: direct GET works).
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// Fetches a note from misskey.io by id. The API answers client failures
/// with HTTP 400 + `{"error":{"code":...}}` (NO_SUCH_NOTE → NotFound);
/// everything else non-success is transient and retried by [`crate::site::fetch`].
pub async fn fetch(note_id: &str) -> Result<model::Note, FetchError> {
let response = crate::site::CLIENT
.post(API_URL)
.json(&serde_json::json!({ "noteId": note_id }))
.send()
.await?;
let status = response.status();
if !status.is_success() {
return Err(match status.as_u16() {
400 => not_found_or_invalid(response).await,
_ => FetchError::Transient(format!("misskey status {status}")),
});
}
response.json().await.map_err(|e| FetchError::Site {
site: "misskey",
error: Box::new(e),
})
}
/// Maps a 400 response: NO_SUCH_NOTE is permanent NotFound, any other 400 is
/// a site error (permanent — retrying a rejected request cannot succeed).
async fn not_found_or_invalid(response: reqwest::Response) -> FetchError {
match response.json::<serde_json::Value>().await {
Ok(v) if v["error"]["code"] == "NO_SUCH_NOTE" => FetchError::NotFound,
_ => FetchError::Site {
site: "misskey",
error: "note rejected (invalid param or private note)".into(),
},
}
}
/// The note whose content matters: a renote shell has no text/files of its
/// own — the embedded renote carries them.
fn effective(note: &model::Note) -> &model::Note {
match &note.renote {
Some(renote) if note.files.is_empty() => renote,
_ => note,
}
}
impl From<model::Note> for Fetched {
fn from(note: model::Note) -> Self {
let note = &note;
let content = effective(note);
let url = format!("https://misskey.io/notes/{}", note.id);
let author = content
.user
.name
.as_deref()
.filter(|n| !n.is_empty())
.unwrap_or(&content.user.username)
.to_string();
let author_url = format!("https://misskey.io/@{}", content.user.username);
let cw = content.cw.as_deref().unwrap_or_default();
// Notes carry hashtags inline in the text (no structured tags array);
// a CW note gets the marker prefixed so recipients see the spoiler.
let mut title = cw.to_string();
if !cw.is_empty() && !title.ends_with(' ') {
title.push(' ');
}
title.push_str(content.text.as_deref().unwrap_or_default().trim());
let title = title.trim().to_string();
let caption = caption(&url, &author_url, &author, &title);
let sensitive = content.cw.is_some() || content.files.iter().any(|f| f.is_sensitive);
let media: Vec<Media> = content.files.iter().filter_map(media_from_file).collect();
Fetched {
source_url: url.clone(),
caption,
title: title.clone(),
media,
sensitive,
site_id: "misskey",
render_data: Some(RenderData {
url,
author: encode_text(&author).into_owned(),
author_url: author_url.clone(),
title: encode_text(&title).into_owned(),
tags: String::new(),
}),
_keep_alive: None,
}
}
}
fn caption(url: &str, author_url: &str, author: &str, text: &str) -> String {
let url = encode_double_quoted_attribute(url);
let author_url = encode_double_quoted_attribute(author_url);
let author = encode_text(author);
if text.is_empty() {
return format!("{url}\n<a href=\"{author_url}\">{author}</a>");
}
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
text = encode_text(text),
)
}
/// Maps a Misskey DriveFile to a [`Media`] item; unknown/audio/other types
/// are skipped (twitter's `_ => {}` precedent). GIF must be matched before
/// the generic image arm.
fn media_from_file(file: &model::DriveFile) -> Option<Media> {
let title = file.name.clone();
match file.mime_type.as_str() {
"image/gif" => Some(Media::Animated {
title,
url: file.url.clone(),
thumbnail_url: file.thumbnail_url.clone().unwrap_or_default(),
}),
mime if mime.starts_with("image/") => Some(Media::Illustration {
title,
url: file.url.clone(),
thumbnail_url: file.thumbnail_url.clone(),
fallback_url: None,
}),
mime if mime.starts_with("video/") => Some(Media::Video {
title,
url: file.url.clone(),
thumbnail_url: file.thumbnail_url.clone().unwrap_or_default(),
}),
_ => None,
}
}
#[cfg(test)]
mod tests {
use super::*;
fn note_json(json: serde_json::Value) -> model::Note {
serde_json::from_value(json).unwrap()
}
fn base_note() -> serde_json::Value {
serde_json::json!({
"id": "aotihl10lqrs015s",
"text": "hello",
"user": { "name": "ミロン", "username": "donyan47897", "host": null },
"files": []
})
}
#[test]
fn pattern_matches_misskey_note_urls() {
for url in [
"https://misskey.io/notes/aotihl10lqrs015s",
"http://misskey.io/notes/aotihl10lqrs015s",
"misskey.io/notes/aotihl10lqrs015s",
] {
assert!(PATTERN.is_match(url), "{url}");
}
for url in [
"https://misskey.io/",
"https://misskey.io/@user",
"https://misskey.io/notes/",
"https://x.com/user/status/123",
] {
assert!(!PATTERN.is_match(url), "{url}");
}
}
#[test]
fn cache_key_normalizes_variants() {
assert_eq!(
cache_key("https://misskey.io/notes/aotihl10lqrs015s"),
Some("misskey:aotihl10lqrs015s".to_string())
);
assert_eq!(x_media_site_id("misskey:abc"), "misskey");
}
fn x_media_site_id(key: &str) -> &'static str {
crate::site::site_id_from_key(key)
}
#[test]
fn from_json_image_file() {
let mut note = base_note();
note["files"] = serde_json::json!([{
"type": "image/webp",
"url": "https://media.misskeyusercontent.jp/io/a.webp",
"thumbnailUrl": "https://media.misskeyusercontent.jp/io/t.webp",
"isSensitive": true,
"name": "pic.webp"
}]);
let fetched: Fetched = note_json(note).into();
assert_eq!(
fetched.source_url,
"https://misskey.io/notes/aotihl10lqrs015s"
);
assert_eq!(fetched.site_id, "misskey");
assert_eq!(fetched.title, "hello");
assert!(fetched.sensitive);
assert_eq!(fetched.media.len(), 1);
match &fetched.media[0] {
Media::Illustration {
title,
url,
thumbnail_url,
fallback_url,
} => {
assert_eq!(title.as_deref(), Some("pic.webp"));
assert_eq!(url, "https://media.misskeyusercontent.jp/io/a.webp");
assert_eq!(
thumbnail_url.as_deref(),
Some("https://media.misskeyusercontent.jp/io/t.webp")
);
assert!(fallback_url.is_none());
}
other => panic!("expected illustration, got {other:?}"),
}
}
#[test]
fn from_json_gif_video_and_skip_audio() {
let mut note = base_note();
note["files"] = serde_json::json!([
{ "type": "audio/mpeg", "url": "https://m/a.mp3", "isSensitive": false },
{ "type": "image/gif", "url": "https://m/a.gif", "isSensitive": false },
{ "type": "video/webm", "url": "https://m/a.webm", "isSensitive": false }
]);
let fetched: Fetched = note_json(note).into();
assert_eq!(fetched.media.len(), 2);
assert!(
matches!(&fetched.media[0], Media::Animated { url, .. } if url == "https://m/a.gif")
);
assert!(matches!(&fetched.media[1], Media::Video { url, .. } if url == "https://m/a.webm"));
// No thumbnailUrl → empty string, not a broken URL.
match &fetched.media[1] {
Media::Video { thumbnail_url, .. } => assert_eq!(thumbnail_url, ""),
other => panic!("expected video, got {other:?}"),
}
assert!(!fetched.sensitive);
}
#[test]
fn from_json_cw_marks_sensitive_and_prefixes_title() {
let mut note = base_note();
note["cw"] = serde_json::json!("spoiler");
note["text"] = serde_json::json!("body");
let fetched: Fetched = note_json(note).into();
assert!(fetched.sensitive);
assert_eq!(fetched.title, "spoiler body");
}
#[test]
fn from_json_author_falls_back_to_username() {
let mut note = base_note();
note["user"] = serde_json::json!({ "name": null, "username": "donyan47897", "host": null });
let fetched: Fetched = note_json(note).into();
assert!(
fetched.caption.contains("donyan47897"),
"{}",
fetched.caption
);
assert!(fetched.caption.contains("https://misskey.io/@donyan47897"));
}
#[test]
fn from_json_renote_uses_embedded_content() {
let note = serde_json::json!({
"id": "shell0000000000",
"text": null,
"user": { "name": "shell", "username": "shelluser", "host": null },
"files": [],
"renote": {
"id": "inner000000000",
"text": "inner text",
"user": { "name": "inner", "username": "inneruser", "host": null },
"files": [
{ "type": "image/png", "url": "https://m/i.png", "isSensitive": false }
]
}
});
let fetched: Fetched = note_json(note).into();
assert_eq!(fetched.title, "inner text");
assert_eq!(fetched.media.len(), 1);
// The source URL still points at the renote shell the user posted.
assert_eq!(
fetched.source_url,
"https://misskey.io/notes/shell0000000000"
);
}
#[test]
fn caption_layout_matches_bsky() {
let fetched: Fetched = note_json(base_note()).into();
assert_eq!(
fetched.caption,
"https://misskey.io/notes/aotihl10lqrs015s\n<a href=\"https://misskey.io/@donyan47897\">ミロン</a>: hello"
);
}
#[test]
fn caption_without_text_has_no_dangling_colon() {
let mut note = base_note();
note["text"] = serde_json::json!(null);
let fetched: Fetched = note_json(note).into();
assert_eq!(
fetched.caption,
"https://misskey.io/notes/aotihl10lqrs015s\n<a href=\"https://misskey.io/@donyan47897\">ミロン</a>"
);
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to misskey.io"]
async fn live_fetch_reference_note() {
let fetched = fetch_from_url("https://misskey.io/notes/aotihl10lqrs015s")
.await
.unwrap();
assert_eq!(fetched.site_id, "misskey");
assert_eq!(fetched.media.len(), 1);
assert!(fetched.sensitive);
assert!(!fetched.caption.is_empty());
}
}
+6
View File
@@ -0,0 +1,6 @@
mod interface;
mod model;
pub use interface::{
MisskeySite, PATTERN, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
};
+35
View File
@@ -0,0 +1,35 @@
use serde::Deserialize;
#[derive(Deserialize, Debug)]
pub(crate) struct Note {
pub(crate) id: String,
pub(crate) text: Option<String>,
#[serde(default)]
pub(crate) cw: Option<String>,
pub(crate) user: User,
#[serde(default)]
pub(crate) files: Vec<DriveFile>,
/// Embedded original note when this note is a renote; the shell's own
/// text/files are usually empty and the content lives here.
#[serde(default)]
pub(crate) renote: Option<Box<Note>>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct User {
pub(crate) name: Option<String>,
pub(crate) username: String,
}
#[derive(Deserialize, Debug)]
pub(crate) struct DriveFile {
#[serde(rename = "type")]
pub(crate) mime_type: String,
pub(crate) url: String,
#[serde(default, rename = "thumbnailUrl")]
pub(crate) thumbnail_url: Option<String>,
#[serde(default, rename = "isSensitive")]
pub(crate) is_sensitive: bool,
#[serde(default)]
pub(crate) name: Option<String>,
}
+7 -5
View File
@@ -1,8 +1,8 @@
//! Site fetching dispatcher and unified result types.
//!
//! Dispatch order: twitter → bsky → pixiv. Each site module exports a
//! `PATTERN`, `enabled()` and `fetch_from_url()`; a future site plugs in by
//! adding one guarded entry in [`fetch_once`].
//! Dispatch order: twitter → bsky → misskey → pixiv. Each site module
//! exports a `PATTERN`, `enabled()` and `fetch_from_url()`; a future site
//! plugs in by adding one guarded entry in [`fetch_once`].
use std::future::Future;
use std::pin::Pin;
@@ -14,6 +14,7 @@ use regex::Regex;
use thiserror::Error;
pub mod bsky;
pub mod misskey;
pub mod pixiv;
pub mod twitter;
@@ -329,6 +330,7 @@ static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| {
vec![
Box::new(twitter::TwitterSite),
Box::new(bsky::BskySite),
Box::new(misskey::MisskeySite),
Box::new(pixiv::PixivSite),
]
});
@@ -531,10 +533,10 @@ mod tests {
#[test]
fn registry_lists_all_sites_in_dispatch_order() {
assert_eq!(site_ids(), vec!["twitter", "bsky", "pixiv"]);
assert_eq!(site_ids(), vec!["twitter", "bsky", "misskey", "pixiv"]);
// Enabled sites dispatch; unsupported URLs never match.
assert!(find_site("https://x.com/u/status/1").is_some());
assert!(find_site("https://bsky.app/profile/u/post/3x").is_some());
assert!(find_site("https://misskey.io/notes/abc").is_some());
assert!(find_site("https://example.com/x").is_none());
// Cache keys are pattern-driven, independent of the enabled() gate
// (pixiv is disabled in tests without PIXIV_REFRESH_TOKEN).
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "xmedia-bot"
version = "1.4.0"
version = "1.5.0"
edition = "2024"
[dependencies]
+6
View File
@@ -160,3 +160,9 @@ pub fn now_f64() -> f64 {
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
}
/// Unix timestamp in whole seconds. Same clock as [`now_f64`], for fields
/// that store integer seconds (chat-state expiry, edit prompts).
pub fn unix_now() -> i64 {
now_f64() as i64
}
+2 -2
View File
@@ -3,8 +3,8 @@
use super::urls::enqueue_retry;
use super::{CHAT_STORE, CONFIG, TASK_QUEUE};
use crate::db::unix_now;
use crate::send::{self, Task};
use crate::state::unix_now;
use teloxide::RequestError;
use teloxide::prelude::*;
use teloxide::types::{CallbackQuery, ChatId, MessageId, ParseMode};
@@ -83,7 +83,7 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
task,
}) => {
log::info!("forward queued for retry in {delay_seconds:.1}s");
enqueue_retry(&TASK_QUEUE, task, delay_seconds).await;
enqueue_retry(&TASK_QUEUE, *task, delay_seconds).await;
bot.answer_callback_query(callback_query_id)
.text("Forward queued for retry.")
.await?;
+3 -3
View File
@@ -253,7 +253,7 @@ pub(crate) async fn execute_command(
bot,
message.chat.id.0,
message.id,
"Unknown site. Use twitter, bsky or pixiv.",
"Unknown site. Use twitter, bsky, pixiv or misskey.",
)
.await?;
return Ok(());
@@ -294,7 +294,7 @@ pub(crate) async fn execute_command(
bot,
message.chat.id.0,
message.id,
"Unrecognized link. Use a twitter/x, pixiv or bsky post URL.",
"Unrecognized link. Use a twitter/x, pixiv, bsky or misskey post URL.",
)
.await?;
return Ok(());
@@ -337,7 +337,7 @@ pub(crate) async fn execute_command(
bot,
message.chat.id.0,
message.id,
"No enabled site matches this link (twitter/x, pixiv or bsky).",
"No enabled site matches this link (twitter/x, pixiv, bsky or misskey).",
)
.await?;
}
+7 -2
View File
@@ -135,7 +135,12 @@ pub fn extract_urls(message: &Message) -> Vec<String> {
fn thumbnail_for(media: &Media) -> Option<String> {
let url = media.url();
if url.starts_with("http://") || url.starts_with("https://") {
media.thumbnail_url().map(str::to_string)
// An empty thumbnail string (misskey video/gif files without a
// thumbnailUrl) must not reach Telegram; let it generate its own.
media
.thumbnail_url()
.map(str::to_string)
.filter(|t| !t.is_empty())
} else {
None
}
@@ -211,7 +216,7 @@ async fn dispatch_send(
"send for [key={}] failed, queued for retry in {delay_seconds:.1}s",
log_key(url)
);
enqueue_retry(ctx.task_queue, task, delay_seconds).await;
enqueue_retry(ctx.task_queue, *task, delay_seconds).await;
let _ = reply(
ctx.sender,
chat_id,
+4 -2
View File
@@ -122,7 +122,7 @@ fn output_channels(color: png::ColorType) -> usize {
/// white; 16-bit per channel was already stripped to 8-bit at decode.
fn flatten_rgba_to_rgb(rgba: &[u8]) -> Vec<u8> {
let mut rgb = Vec::with_capacity(rgba.len() / 4 * 3);
for px in rgba.chunks_exact(4) {
for px in rgba.as_chunks::<4>().0 {
let a = px[3] as u32;
for v in &px[..3] {
// Over white: C = C*a/255 + 255*(1 - a/255).
@@ -178,7 +178,9 @@ fn encode_jpeg(pix: &PixBuf, w: u32, h: u32) -> Result<Vec<u8>, String> {
PixBuf::GrayAlpha(v) => {
// JPEG has no alpha: composite onto white, output as gray.
let gray: Vec<u8> = v
.chunks_exact(2)
.as_chunks::<2>()
.0
.iter()
.map(|px| {
let (g, a) = (px[0] as u32, px[1] as u32);
((g * a + 255 * (255 - a)) / 255).min(255) as u8
+48 -55
View File
@@ -3,12 +3,13 @@
//! URL is blocked by hotlink protection; the bot downloads the file itself
//! and uploads it via multipart).
use crate::db::unix_now;
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE, log_key};
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost, LinkCache};
use crate::media_sender::MediaSender;
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
use crate::queue::QueueError;
use crate::state::{EditMessage, unix_now};
use crate::state::EditMessage;
use rand::Rng;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
@@ -386,22 +387,25 @@ pub fn classify_request_error(e: &RequestError) -> Classification {
}
}
/// Task boxed to keep the error size within `result_large_err` limits.
#[derive(Debug)]
pub enum SendError {
Retryable { delay_seconds: f64, task: Task },
Permanent { message: String, task: Task },
Retryable { delay_seconds: f64, task: Box<Task> },
Permanent { message: String, task: Box<Task> },
}
fn classify_to_send_error(e: &RequestError, task: Task) -> SendError {
fn classify_to_send_error(e: &RequestError, task: Task, fetch_failure_label: &str) -> SendError {
match classify_request_error(e) {
Classification::Retryable { delay_seconds } => SendError::Retryable {
delay_seconds,
task,
task: Box::new(task),
},
Classification::Permanent { message } => SendError::Permanent {
message,
task: Box::new(task),
},
Classification::Permanent { message } => SendError::Permanent { message, task },
Classification::MediaFetchFailure => SendError::Permanent {
message: "media fetch failed".into(),
task,
message: fetch_failure_label.into(),
task: Box::new(task),
},
}
}
@@ -415,9 +419,12 @@ impl SendError {
match f {
FallbackError::Retryable { delay_seconds } => SendError::Retryable {
delay_seconds,
task,
task: Box::new(task),
},
FallbackError::Permanent { message } => SendError::Permanent {
message,
task: Box::new(task),
},
FallbackError::Permanent { message } => SendError::Permanent { message, task },
FallbackError::MediaTooLarge => unreachable!("handled inside the upload fallback"),
}
}
@@ -847,7 +854,7 @@ async fn send_batch_via_upload(
Err(e) => {
return Err(SendError::Permanent {
message: format!("upload worker panicked: {e}"),
task,
task: Box::new(task),
});
}
};
@@ -872,51 +879,24 @@ async fn send_batch_via_upload(
drop(keep_alive);
match result {
Ok(messages) => Ok(messages),
Err(e) => Err(match classify_request_error(&e) {
Classification::Retryable { delay_seconds } => SendError::Retryable {
delay_seconds,
task: task.clone(),
},
Classification::Permanent { message } => SendError::Permanent { message, task },
Classification::MediaFetchFailure => SendError::Permanent {
message: "upload failed".into(),
task,
},
}),
Err(e) => Err(classify_to_send_error(&e, task, "upload failed")),
}
}
fn updated_sequence_task(task: &Task, batch_index: usize, sent_message_ids: Vec<i64>) -> Task {
match task {
let mut updated = task.clone();
match &mut updated {
Task::SendMediaSequence {
chat_id,
reply_to_message_id,
caption,
media_batches,
batch_index: _,
sent_message_ids: _,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
cache_data,
} => Task::SendMediaSequence {
chat_id: *chat_id,
reply_to_message_id: *reply_to_message_id,
caption: caption.clone(),
media_batches: media_batches.clone(),
batch_index,
sent_message_ids,
source_url: source_url.clone(),
edit_before_forward: *edit_before_forward,
forward_channel_id: *forward_channel_id,
notify_chat_id: *notify_chat_id,
notify_message_id: *notify_message_id,
cache_data: cache_data.clone(),
},
batch_index: index,
sent_message_ids: ids,
..
} => {
*index = batch_index;
*ids = sent_message_ids;
}
_ => unreachable!("updated_sequence_task requires a SendMediaSequence task"),
}
updated
}
/// Sends the media batches starting at `task.batch_index`, extending
@@ -957,7 +937,7 @@ pub async fn send_media_sequence(
Err(message) => {
return Err(SendError::Permanent {
message,
task: updated_sequence_task(task, idx, sent),
task: Box::new(updated_sequence_task(task, idx, sent)),
});
}
};
@@ -1004,6 +984,7 @@ pub async fn send_media_sequence(
return Err(classify_to_send_error(
&e,
updated_sequence_task(task, idx, sent),
"media fetch failed",
));
}
}
@@ -1060,7 +1041,7 @@ pub async fn send_animation(sender: &dyn MediaSender, task: &Task) -> Result<Vec
Err(message) => {
return Err(SendError::Permanent {
message,
task: task.clone(),
task: Box::new(task.clone()),
});
}
};
@@ -1105,13 +1086,21 @@ pub async fn send_animation(sender: &dyn MediaSender, task: &Task) -> Result<Vec
cache_animation_send(task, &message).await;
Ok(vec![id])
}
Err(e) => Err(classify_to_send_error(&e, task.clone())),
Err(e) => Err(classify_to_send_error(
&e,
task.clone(),
"media fetch failed",
)),
}
}
Err(e) => Err(SendError::from_fallback(e, task.clone())),
}
}
Err(e) => Err(classify_to_send_error(&e, task.clone())),
Err(e) => Err(classify_to_send_error(
&e,
task.clone(),
"media fetch failed",
)),
}
}
@@ -1148,7 +1137,11 @@ pub async fn forward_messages(sender: &dyn MediaSender, task: &Task) -> Result<(
);
Ok(())
}
Err(e) => Err(classify_to_send_error(&e, task.clone())),
Err(e) => Err(classify_to_send_error(
&e,
task.clone(),
"media fetch failed",
)),
}
}
+2 -8
View File
@@ -1,12 +1,13 @@
//! Per-chat state with SQLite persistence (table `chat_state` in
//! `data/task_queue.db`, shared with the task queue).
use crate::db::unix_now;
use parking_lot::Mutex;
use rusqlite::params;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use std::time::Duration;
#[derive(Serialize, Deserialize, Default, Clone, Debug)]
pub struct ChatData {
@@ -40,13 +41,6 @@ pub struct ChatStore {
pool: Arc<crate::db::DbPool>,
}
pub fn unix_now() -> i64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
.unwrap_or(0)
}
impl ChatStore {
/// Wraps the shared DB pool (schema initialized once by
/// [`crate::db::open_store`]; the `chat_state` table lives in the merged