mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-10-02 00:42:10 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
32254fa807
|
||
|
|
11c04b66dc
|
||
|
|
2f741e5f4b
|
||
|
|
89c4642e1c
|
||
|
|
3f6a0f034a
|
||
|
|
90a011e978
|
||
|
|
12a065846c
|
||
|
|
dca1eff1c9
|
||
|
|
894a9ebf4a
|
||
|
|
c968891ff6
|
||
|
|
0087bd01ac
|
||
|
|
4cb40909c5
|
||
|
|
f260f41755
|
||
|
|
af901caddb
|
@@ -4,8 +4,9 @@ name: CI
|
|||||||
# job that exercises the real source sites and the token-gated pixiv tests.
|
# job that exercises the real source sites and the token-gated pixiv tests.
|
||||||
#
|
#
|
||||||
# Layering:
|
# Layering:
|
||||||
# test — fmt + clippy + the full offline unit suite. Runs on every push
|
# test — fmt + clippy + the full offline unit suite + cargo-audit
|
||||||
# and PR, including forks (it needs no secrets).
|
# dependency gate. Runs on every push and PR, including forks
|
||||||
|
# (it needs no secrets).
|
||||||
# live — the #[ignore]d live-network tests plus the pixiv tests that are
|
# live — the #[ignore]d live-network tests plus the pixiv tests that are
|
||||||
# gated on PIXIV_REFRESH_TOKEN. Runs on schedule / manual dispatch
|
# gated on PIXIV_REFRESH_TOKEN. Runs on schedule / manual dispatch
|
||||||
# / tag pushes only, because pull requests from forks cannot read
|
# / tag pushes only, because pull requests from forks cannot read
|
||||||
@@ -42,6 +43,12 @@ jobs:
|
|||||||
run: cargo clippy --workspace --all-targets -- -D warnings
|
run: cargo clippy --workspace --all-targets -- -D warnings
|
||||||
- name: Run offline tests
|
- name: Run offline tests
|
||||||
run: cargo test --workspace
|
run: cargo test --workspace
|
||||||
|
# Dependency vulnerability gate: fails the build when a crate in
|
||||||
|
# Cargo.lock has an unfixed security advisory. Unmaintained/unsound
|
||||||
|
# *warnings* (dotenv, proc-macro-error2, anyhow transitive) do not fail
|
||||||
|
# the build by default; the advisory DB is cached across runs.
|
||||||
|
- name: Audit dependencies
|
||||||
|
uses: actions-rust-lang/audit@v1
|
||||||
|
|
||||||
live:
|
live:
|
||||||
needs: test
|
needs: test
|
||||||
|
|||||||
@@ -2,11 +2,11 @@
|
|||||||
|
|
||||||
## Project Overview
|
## Project Overview
|
||||||
|
|
||||||
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README and user-facing strings are in Chinese. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
|
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, Bluesky, and Misskey (misskey.io) into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README is in Chinese; user-facing bot strings are in English. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
|
||||||
|
|
||||||
Two-crate Cargo workspace (both v1.3.0, edition 2024, resolver 3):
|
Two-crate Cargo workspace (both v1.5.0, edition 2024, resolver 3):
|
||||||
|
|
||||||
- **`crates/x-media`** — library that fetches and normalizes media from the three sites. Pure, no Telegram knowledge.
|
- **`crates/x-media`** — library that fetches and normalizes media from the four sites. Pure, no Telegram knowledge.
|
||||||
- **`crates/xmedia-bot`** — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.
|
- **`crates/xmedia-bot`** — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.
|
||||||
|
|
||||||
## Architecture & Data Flow
|
## Architecture & Data Flow
|
||||||
@@ -20,23 +20,26 @@ Telegram update → Dispatcher (polling or axum webhook) → dptree branches
|
|||||||
|
|
||||||
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)` → `Fetched` → builds a `Task` → `send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
|
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)` → `Fetched` → builds a `Task` → `send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
|
||||||
|
|
||||||
Debug command: `/test <url>` runs the same `x_media::site::fetch` and replies with `test_parse_report` (`handlers/commands.rs`) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars (Telegram's 4096 plain-text limit). It uses a custom `parse_test_arg` parser (whole remainder, trimmed) because teloxide's built-in `split` parser takes exactly one space-separated token.
|
Debug command: `/test <url>` runs the same `x_media::site::fetch` and replies with `test_parse_report` (`handlers/commands.rs`) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars and sent with HTML parse mode: raw fields are escaped, and the caption is wrapped in a `<blockquote>` so it renders exactly like the sent media caption (escaped text and links included). It uses a custom `parse_test_arg` parser (whole remainder, trimmed) because teloxide's built-in `split` parser takes exactly one space-separated token.
|
||||||
|
|
||||||
The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → pixiv) and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, site_id, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
|
The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → misskey → pixiv) and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, site_id, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
|
||||||
|
|
||||||
## Key Directories
|
## Key Directories
|
||||||
|
|
||||||
| Path | Purpose |
|
| Path | Purpose |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
|
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
|
||||||
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`) |
|
| `crates/x-media/src/site/<twitter\|pixiv\|bsky\|misskey>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`>` `<` `&` `'`) — so the stored text is raw and the caption escapes exactly once |
|
||||||
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
|
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
|
||||||
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
|
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
|
||||||
|
| `crates/xmedia-bot/src/db.rs` | `DbPool`: per-store SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) over `$DATA_DIR/task_queue.db` (default `data/`); `open_store` creates file + schema, `with_conn` runs all rusqlite I/O in `spawn_blocking` |
|
||||||
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. the `/test <url>` parse-only debug command), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons), `statics.rs` (global statics) |
|
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. the `/test <url>` parse-only debug command), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons), `statics.rs` (global statics) |
|
||||||
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table) |
|
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table) |
|
||||||
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure |
|
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure |
|
||||||
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections |
|
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections |
|
||||||
| `crates/xmedia-bot/src/send.rs` | Media senders, upload fallback, error classification, queue task handlers |
|
| `crates/xmedia-bot/src/send.rs` | Media senders, upload fallback, error classification, queue task handlers |
|
||||||
|
| `crates/xmedia-bot/src/media_sender.rs` | `MediaSender` trait: the send surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`send_chat_action`) implemented by teloxide `Bot` (per-chat rate-limited) and by a scripted `MockSender` in tests |
|
||||||
|
| `crates/xmedia-bot/src/rate_limit.rs` | Per-chat token bucket (`CAPACITY = 20`, ~20 msg/min refill) paced before sends reach the API so batch forwards don't trip flood control |
|
||||||
|
|
||||||
## Development Commands
|
## Development Commands
|
||||||
|
|
||||||
@@ -55,7 +58,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
|
|||||||
## Code Conventions & Common Patterns
|
## Code Conventions & Common Patterns
|
||||||
|
|
||||||
- **Errors via `thiserror` derive** (no anyhow): the public, stringified errors — `FetchError` (`Http`/`Json`/`Pixiv`/`Site`/`NotFound`/`Blocked`) and `PixivError` — derive `thiserror::Error` with `#[from]` conversions; `Display`/`source()` come from the derive. The internal control-flow enums — `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `Classification`, `FallbackError` — carry no `Display` and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive `thiserror`, internal flow enums stay plain.
|
- **Errors via `thiserror` derive** (no anyhow): the public, stringified errors — `FetchError` (`Http`/`Json`/`Pixiv`/`Site`/`NotFound`/`Blocked`) and `PixivError` — derive `thiserror::Error` with `#[from]` conversions; `Display`/`source()` come from the derive. The internal control-flow enums — `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `Classification`, `FallbackError` — carry no `Display` and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive `thiserror`, internal flow enums stay plain.
|
||||||
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers share the process-wide `send::BOT` (`LazyLock<Bot>`, force-initialized in `main` so a missing token fails at startup).
|
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers/statics.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers share the process-wide `send::BOT` (`LazyLock<Bot>`, force-initialized in `main` so a missing token fails at startup).
|
||||||
- **Async**: tokio multi-thread runtime (`#[tokio::main]` default). All rusqlite I/O inside `tokio::task::spawn_blocking`. Long loops use `tokio::select!` with `tokio::sync::{watch, Notify}` stop/wake channels. No streams.
|
- **Async**: tokio multi-thread runtime (`#[tokio::main]` default). All rusqlite I/O inside `tokio::task::spawn_blocking`. Long loops use `tokio::select!` with `tokio::sync::{watch, Notify}` stop/wake channels. No streams.
|
||||||
- **Blocking sync primitives**: `parking_lot::Mutex` for hot caches, `tokio::sync::Mutex` for async-shared state (pixiv token cache), `AtomicBool` for feature gates.
|
- **Blocking sync primitives**: `parking_lot::Mutex` for hot caches, `tokio::sync::Mutex` for async-shared state (pixiv token cache), `AtomicBool` for feature gates.
|
||||||
- **Site adapter convention**: each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`, plus `cache_key`/`is_retryable`/`media_headers`, and a unit struct `<Name>Site` implementing `site::Site`; the central dispatcher (`site/mod.rs`) only iterates the `SITES` registry. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one `Box::new(...)` entry in `SITES` — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return `SiteFuture` (a boxed `Pin<Box<dyn Future + Send>>`) because `async fn` in traits is not dyn-compatible.
|
- **Site adapter convention**: each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`, plus `cache_key`/`is_retryable`/`media_headers`, and a unit struct `<Name>Site` implementing `site::Site`; the central dispatcher (`site/mod.rs`) only iterates the `SITES` registry. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one `Box::new(...)` entry in `SITES` — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return `SiteFuture` (a boxed `Pin<Box<dyn Future + Send>>`) because `async fn` in traits is not dyn-compatible.
|
||||||
@@ -69,7 +72,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
|
|||||||
| File | Why it matters |
|
| File | Why it matters |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) |
|
| `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) |
|
||||||
| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `data/task_queue.db` **relative to CWD**); `commands.rs` = command dispatch (incl. the `/test <url>` parse-only debug command); `urls.rs` = URL extraction + retry enqueue; `inline.rs`/`callback.rs` = inline queries / edit-before-forward buttons |
|
| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `commands.rs` = command dispatch (incl. the `/test <url>` parse-only debug command); `urls.rs` = URL extraction + retry enqueue; `inline.rs`/`callback.rs` = inline queries / edit-before-forward buttons |
|
||||||
| `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`; fallback chain; `classify_request_error`; download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`) |
|
| `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`; fallback chain; `classify_request_error`; download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`) |
|
||||||
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL |
|
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL |
|
||||||
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) |
|
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) |
|
||||||
@@ -85,18 +88,18 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
|
|||||||
- **Rust, stable, edition 2024**, workspace resolver 3. No `rust-version`/MSRV pin, no `rust-toolchain.toml` — recent stable is assumed. No nightly features.
|
- **Rust, stable, edition 2024**, workspace resolver 3. No `rust-version`/MSRV pin, no `rust-toolchain.toml` — recent stable is assumed. No nightly features.
|
||||||
- Package manager: **Cargo** (workspace with path dep `x-media` ← `xmedia-bot`). No `[workspace.package]`/shared deps — each crate lists deps independently.
|
- Package manager: **Cargo** (workspace with path dep `x-media` ← `xmedia-bot`). No `[workspace.package]`/shared deps — each crate lists deps independently.
|
||||||
- **TLS is rustls end-to-end** (no native-tls/openssl in the tree, no libssl in the Docker runtime image): `teloxide` is declared `default-features = false` with `["webhooks-axum", "macros", "rustls", "ctrlc_handler"]` (the removed `default` also carried `native-tls` and `ctrlc_handler` — the latter must stay); x-media's reqwest is `default-features = false` with `["json", "rustls-tls"]` (webpki-roots baked in, so the image ships no CA bundle). One reqwest 0.12.28 in the lock.
|
- **TLS is rustls end-to-end** (no native-tls/openssl in the tree, no libssl in the Docker runtime image): `teloxide` is declared `default-features = false` with `["webhooks-axum", "macros", "rustls", "ctrlc_handler"]` (the removed `default` also carried `native-tls` and `ctrlc_handler` — the latter must stay); x-media's reqwest is `default-features = false` with `["json", "rustls-tls"]` (webpki-roots baked in, so the image ships no CA bundle). One reqwest 0.12.28 in the lock.
|
||||||
- **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag (the tag push triggers the Docker Hub build).
|
- **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md`, `README.en.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag (the tag push triggers the Docker Hub build).
|
||||||
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
|
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `DATA_DIR` (default `data`, CWD-relative; the SQLite dir, auto-created), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
|
||||||
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `data/task_queue.db` is CWD-relative — run from the workspace root, or `/app` in Docker. Mount `./data` and `./cert` volumes.
|
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `$DATA_DIR/task_queue.db` (default `data/task_queue.db`, CWD-relative — run from the workspace root, or `/app` in Docker; set `DATA_DIR` to pin state anywhere). Mount `./data` and `./cert` volumes.
|
||||||
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `docker-compose.yml`, `/target`, `.idea/`.
|
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `docker-compose.yml`, `/target`, `.idea/`.
|
||||||
- Docs are in Chinese; user-facing bot strings too. Keep that convention when editing captions/templates/docs.
|
- Docs are in Chinese (README, AGENTS.md); user-facing bot strings are in English. Keep that split when editing user-facing strings and docs.
|
||||||
|
|
||||||
## Testing & QA
|
## Testing & QA
|
||||||
|
|
||||||
- **~80 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
|
- **~125 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
|
||||||
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches.
|
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches.
|
||||||
- Live-network tests exist in `site/twitter/interface.rs` (3), `site/bsky/interface.rs` (2), `site/pixiv/interface.rs`/`api.rs`. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`.
|
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`.
|
||||||
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
|
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
|
||||||
- **CI** — `.github/workflows/ci.yml` runs `cargo fmt --check` + `cargo clippy --workspace --all-targets -- -D warnings` + `cargo test --workspace` (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN` from secrets, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds/pushes the image only.
|
- **CI** — `.github/workflows/ci.yml` runs `cargo fmt --check` + `cargo clippy --workspace --all-targets -- -D warnings` + `cargo test --workspace` + a `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN` from secrets, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds/pushes the image only.
|
||||||
- Untested and hard to test without a mock seam: `handlers.rs` (depends directly on teloxide `Bot`); `main.rs`, `config.rs`, `state.rs`; `media.rs`, `lib.rs`, all `model.rs`.
|
- Untested and hard to test without a mock seam: `handlers.rs` (depends directly on teloxide `Bot`); `main.rs`, `config.rs`, `state.rs`; `media.rs`, `lib.rs`, all `model.rs`.
|
||||||
- No coverage tracking.
|
- No coverage tracking.
|
||||||
|
|||||||
Generated
+2
-2
@@ -2925,7 +2925,7 @@ checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "x-media"
|
name = "x-media"
|
||||||
version = "1.3.0"
|
version = "1.5.0"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"bytes",
|
"bytes",
|
||||||
"dotenv",
|
"dotenv",
|
||||||
@@ -2945,7 +2945,7 @@ dependencies = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "xmedia-bot"
|
name = "xmedia-bot"
|
||||||
version = "1.3.0"
|
version = "1.5.0"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"bytes",
|
"bytes",
|
||||||
"dotenv",
|
"dotenv",
|
||||||
|
|||||||
+4
-3
@@ -1,6 +1,6 @@
|
|||||||
# TelegramXMediaBot
|
# TelegramXMediaBot
|
||||||
|
|
||||||
A Telegram bot that turns post links from X / Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags.
|
A Telegram bot that turns post links from X / Twitter, Pixiv, Bluesky, and Misskey (misskey.io) into media messages (images, video, GIF) with the post's title, author, and tags.
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
@@ -30,7 +30,7 @@ docker build -t tgxmb .
|
|||||||
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
|
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
|
||||||
```
|
```
|
||||||
|
|
||||||
Environment variables: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `BOT_ADMIN`, `EDIT_MESSAGE_TTL_SECONDS`, `LINK_CACHE_TTL_SECONDS`, `RUST_LOG`, `WEBHOOK*`, `TWITTER_AUTH_TOKEN` (optional).
|
Environment variables: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `BOT_ADMIN`, `EDIT_MESSAGE_TTL_SECONDS`, `LINK_CACHE_TTL_SECONDS`, `RUST_LOG`, `TELOXIDE_PROXY`, `WEBHOOK*`, `TWITTER_AUTH_TOKEN` (optional).
|
||||||
|
|
||||||
NSFW tweets: the public syndication endpoint does not return sensitive content. Setting `TWITTER_AUTH_TOKEN` (the `auth_token` cookie value of a logged-in x.com session) lets the bot fetch NSFW media in the logged-in state only when it hits a withheld tweet; without it, the bot reports no media.
|
NSFW tweets: the public syndication endpoint does not return sensitive content. Setting `TWITTER_AUTH_TOKEN` (the `auth_token` cookie value of a logged-in x.com session) lets the bot fetch NSFW media in the logged-in state only when it hits a withheld tweet; without it, the bot reports no media.
|
||||||
|
|
||||||
@@ -84,6 +84,7 @@ Telegram only accepts ports 443/80/88/8443.
|
|||||||
| `BOT_ADMIN` | Admin chat IDs, comma-separated; receives start/stop notifications |
|
| `BOT_ADMIN` | Admin chat IDs, comma-separated; receives start/stop notifications |
|
||||||
| `EDIT_MESSAGE_TTL_SECONDS` | Edit-before-forward record expiry in seconds, default 86400 |
|
| `EDIT_MESSAGE_TTL_SECONDS` | Edit-before-forward record expiry in seconds, default 86400 |
|
||||||
| `LINK_CACHE_TTL_SECONDS` | Link-result cache expiry in seconds, default 604800 (7 days) |
|
| `LINK_CACHE_TTL_SECONDS` | Link-result cache expiry in seconds, default 604800 (7 days) |
|
||||||
|
| `DATA_DIR` | Data directory (where the SQLite `task_queue.db` lives), default `data` (relative to the working directory, created automatically) |
|
||||||
| `RUST_LOG` | Log level |
|
| `RUST_LOG` | Log level |
|
||||||
| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW) |
|
| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW) |
|
||||||
| `LOCAL_USER_ID` | UID the container runs as, default 9001 |
|
| `LOCAL_USER_ID` | UID the container runs as, default 9001 |
|
||||||
@@ -110,7 +111,7 @@ Telegram only accepts ports 443/80/88/8443.
|
|||||||
| `/remove_forward_channel` | Remove the forward channel |
|
| `/remove_forward_channel` | Remove the forward channel |
|
||||||
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or taps a template button to apply one) |
|
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or taps a template button to apply one) |
|
||||||
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward") |
|
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward") |
|
||||||
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` |
|
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv` / `misskey`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` |
|
||||||
| `/clear_cache [link]` | Clear the link cache (admin only); with a link only that entry, otherwise everything |
|
| `/clear_cache [link]` | Clear the link cache (admin only); with a link only that entry, otherwise everything |
|
||||||
| `/bot_dict` | Show the current chat state (debugging) |
|
| `/bot_dict` | Show the current chat state (debugging) |
|
||||||
| `/test <link>` | Debug: parse a link and report the parse result only (site, title, author, tags, media list) — no media is sent |
|
| `/test <link>` | Debug: parse a link and report the parse result only (site, title, author, tags, media list) — no media is sent |
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# TelegramXMediaBot
|
# TelegramXMediaBot
|
||||||
|
|
||||||
Telegram 机器人,将 X / Twitter、Pixiv、Bluesky 的帖子链接转换为媒体消息发送,附带帖子标题、作者与标签。
|
Telegram 机器人,将 X / Twitter、Pixiv、Bluesky、Misskey (misskey.io) 的帖子链接转换为媒体消息发送,附带帖子标题、作者与标签。
|
||||||
|
|
||||||
## 功能
|
## 功能
|
||||||
|
|
||||||
@@ -30,7 +30,7 @@ docker build -t tgxmb .
|
|||||||
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
|
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
|
||||||
```
|
```
|
||||||
|
|
||||||
环境变量:`TELOXIDE_TOKEN`(必填)、`PIXIV_REFRESH_TOKEN`、`BOT_ADMIN`、`EDIT_MESSAGE_TTL_SECONDS`、`LINK_CACHE_TTL_SECONDS`、`RUST_LOG`、`WEBHOOK*`、`TWITTER_AUTH_TOKEN`(可选)。
|
环境变量:`TELOXIDE_TOKEN`(必填)、`PIXIV_REFRESH_TOKEN`、`BOT_ADMIN`、`EDIT_MESSAGE_TTL_SECONDS`、`LINK_CACHE_TTL_SECONDS`、`RUST_LOG`、`TELOXIDE_PROXY`、`WEBHOOK*`、`TWITTER_AUTH_TOKEN`(可选)。
|
||||||
|
|
||||||
NSFW 推文:公开的 syndication 接口不返回敏感内容。设置 `TWITTER_AUTH_TOKEN`(登录 x.com 后浏览器 Cookie 里的 `auth_token` 值)后,bot 会仅在遇到 NSFW 推文时以登录态获取媒体;未设置则提示无媒体。
|
NSFW 推文:公开的 syndication 接口不返回敏感内容。设置 `TWITTER_AUTH_TOKEN`(登录 x.com 后浏览器 Cookie 里的 `auth_token` 值)后,bot 会仅在遇到 NSFW 推文时以登录态获取媒体;未设置则提示无媒体。
|
||||||
|
|
||||||
@@ -84,6 +84,7 @@ Telegram 只接受 443/80/88/8443 端口。
|
|||||||
| `BOT_ADMIN` | 管理员聊天 ID,逗号分隔;接收启动/停止通知 |
|
| `BOT_ADMIN` | 管理员聊天 ID,逗号分隔;接收启动/停止通知 |
|
||||||
| `EDIT_MESSAGE_TTL_SECONDS` | 转发前编辑记录过期秒数,默认 86400 |
|
| `EDIT_MESSAGE_TTL_SECONDS` | 转发前编辑记录过期秒数,默认 86400 |
|
||||||
| `LINK_CACHE_TTL_SECONDS` | 链接结果缓存过期秒数,默认 604800(7 天) |
|
| `LINK_CACHE_TTL_SECONDS` | 链接结果缓存过期秒数,默认 604800(7 天) |
|
||||||
|
| `DATA_DIR` | 数据目录(SQLite 数据库 `task_queue.db` 所在目录),默认 `data`(相对工作目录,会自动创建) |
|
||||||
| `RUST_LOG` | 日志级别 |
|
| `RUST_LOG` | 日志级别 |
|
||||||
| `TELOXIDE_PROXY` | HTTP 代理(如 `http://127.0.0.1:10808`);同时作用于 Telegram Bot API 与站点抓取请求,网络受限环境(如 GFW)必需 |
|
| `TELOXIDE_PROXY` | HTTP 代理(如 `http://127.0.0.1:10808`);同时作用于 Telegram Bot API 与站点抓取请求,网络受限环境(如 GFW)必需 |
|
||||||
| `LOCAL_USER_ID` | 容器内运行用户 UID,默认 9001 |
|
| `LOCAL_USER_ID` | 容器内运行用户 UID,默认 9001 |
|
||||||
@@ -110,7 +111,7 @@ Telegram 只接受 443/80/88/8443 端口。
|
|||||||
| `/remove_forward_channel` | 取消转发频道 |
|
| `/remove_forward_channel` | 取消转发频道 |
|
||||||
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
|
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
|
||||||
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
|
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
|
||||||
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
|
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv` / `misskey`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
|
||||||
| `/clear_cache [链接]` | 清空链接缓存(仅管理员);带链接只清该条,否则清空全部 |
|
| `/clear_cache [链接]` | 清空链接缓存(仅管理员);带链接只清该条,否则清空全部 |
|
||||||
| `/bot_dict` | 查看当前聊天状态(调试用) |
|
| `/bot_dict` | 查看当前聊天状态(调试用) |
|
||||||
| `/test <链接>` | 调试:只解析链接并返回解析结果(站点、标题、作者、标签、媒体列表),不发送任何媒体 |
|
| `/test <链接>` | 调试:只解析链接并返回解析结果(站点、标题、作者、标签、媒体列表),不发送任何媒体 |
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "x-media"
|
name = "x-media"
|
||||||
version = "1.3.0"
|
version = "1.5.0"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
|
|||||||
@@ -0,0 +1,384 @@
|
|||||||
|
//! Site adapter for misskey.io notes: URL pattern, API fetch and
|
||||||
|
//! normalization into [`Fetched`] (see [`crate::site::Site`]).
|
||||||
|
|
||||||
|
use super::model;
|
||||||
|
use crate::media::Media;
|
||||||
|
use crate::site::{FetchError, Fetched, RenderData, Site, SiteFuture};
|
||||||
|
use html_escape::{encode_double_quoted_attribute, encode_text};
|
||||||
|
use regex::Regex;
|
||||||
|
use std::sync::LazyLock;
|
||||||
|
|
||||||
|
const API_URL: &str = "https://misskey.io/api/notes/show";
|
||||||
|
|
||||||
|
/// Registry entry for the misskey.io adapter (see [`crate::site::Site`]).
|
||||||
|
pub struct MisskeySite;
|
||||||
|
|
||||||
|
impl Site for MisskeySite {
|
||||||
|
fn id(&self) -> &'static str {
|
||||||
|
"misskey"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn pattern(&self) -> &'static Regex {
|
||||||
|
&PATTERN
|
||||||
|
}
|
||||||
|
|
||||||
|
fn cache_key(&self, url: &str) -> Option<String> {
|
||||||
|
cache_key(url)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched> {
|
||||||
|
Box::pin(async move { fetch_from_url(url).await })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub static PATTERN: LazyLock<Regex> =
|
||||||
|
LazyLock::new(|| Regex::new(r"^(?:https?://)?misskey\.io/notes/([\w.\-~]+)").unwrap());
|
||||||
|
|
||||||
|
pub fn enabled() -> bool {
|
||||||
|
true
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||||
|
let caps = PATTERN.captures(url).ok_or(FetchError::NotFound)?;
|
||||||
|
let note_id = caps.get(1).ok_or(FetchError::NotFound)?.as_str();
|
||||||
|
let note = fetch(note_id).await?;
|
||||||
|
Ok(note.into())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cache key for a misskey URL: `"misskey:<note id>"`. The prefix is the
|
||||||
|
/// site id used for caption-format lookup and link-cache keys.
|
||||||
|
pub fn cache_key(url: &str) -> Option<String> {
|
||||||
|
PATTERN
|
||||||
|
.captures(url)
|
||||||
|
.map(|caps| format!("misskey:{}", &caps[1]))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Misskey's fetch-retry policy: transient classes only. Not-found, blocked
|
||||||
|
/// and parse failures are permanent.
|
||||||
|
pub fn is_retryable(err: &FetchError) -> bool {
|
||||||
|
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// misskey.io media hosts need no extra headers (verified: direct GET works).
|
||||||
|
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fetches a note from misskey.io by id. The API answers client failures
|
||||||
|
/// with HTTP 400 + `{"error":{"code":...}}` (NO_SUCH_NOTE → NotFound);
|
||||||
|
/// everything else non-success is transient and retried by [`crate::site::fetch`].
|
||||||
|
pub async fn fetch(note_id: &str) -> Result<model::Note, FetchError> {
|
||||||
|
let response = crate::site::CLIENT
|
||||||
|
.post(API_URL)
|
||||||
|
.json(&serde_json::json!({ "noteId": note_id }))
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
let status = response.status();
|
||||||
|
if !status.is_success() {
|
||||||
|
return Err(match status.as_u16() {
|
||||||
|
400 => not_found_or_invalid(response).await,
|
||||||
|
_ => FetchError::Transient(format!("misskey status {status}")),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
response.json().await.map_err(|e| FetchError::Site {
|
||||||
|
site: "misskey",
|
||||||
|
error: Box::new(e),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Maps a 400 response: NO_SUCH_NOTE is permanent NotFound, any other 400 is
|
||||||
|
/// a site error (permanent — retrying a rejected request cannot succeed).
|
||||||
|
async fn not_found_or_invalid(response: reqwest::Response) -> FetchError {
|
||||||
|
match response.json::<serde_json::Value>().await {
|
||||||
|
Ok(v) if v["error"]["code"] == "NO_SUCH_NOTE" => FetchError::NotFound,
|
||||||
|
_ => FetchError::Site {
|
||||||
|
site: "misskey",
|
||||||
|
error: "note rejected (invalid param or private note)".into(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The note whose content matters: a renote shell has no text/files of its
|
||||||
|
/// own — the embedded renote carries them.
|
||||||
|
fn effective(note: &model::Note) -> &model::Note {
|
||||||
|
match ¬e.renote {
|
||||||
|
Some(renote) if note.files.is_empty() => renote,
|
||||||
|
_ => note,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl From<model::Note> for Fetched {
|
||||||
|
fn from(note: model::Note) -> Self {
|
||||||
|
let note = ¬e;
|
||||||
|
let content = effective(note);
|
||||||
|
let url = format!("https://misskey.io/notes/{}", note.id);
|
||||||
|
let author = content
|
||||||
|
.user
|
||||||
|
.name
|
||||||
|
.as_deref()
|
||||||
|
.filter(|n| !n.is_empty())
|
||||||
|
.unwrap_or(&content.user.username)
|
||||||
|
.to_string();
|
||||||
|
let author_url = format!("https://misskey.io/@{}", content.user.username);
|
||||||
|
let cw = content.cw.as_deref().unwrap_or_default();
|
||||||
|
// Notes carry hashtags inline in the text (no structured tags array);
|
||||||
|
// a CW note gets the marker prefixed so recipients see the spoiler.
|
||||||
|
let mut title = cw.to_string();
|
||||||
|
if !cw.is_empty() && !title.ends_with(' ') {
|
||||||
|
title.push(' ');
|
||||||
|
}
|
||||||
|
title.push_str(content.text.as_deref().unwrap_or_default().trim());
|
||||||
|
let title = title.trim().to_string();
|
||||||
|
|
||||||
|
let caption = caption(&url, &author_url, &author, &title);
|
||||||
|
let sensitive = content.cw.is_some() || content.files.iter().any(|f| f.is_sensitive);
|
||||||
|
let media: Vec<Media> = content.files.iter().filter_map(media_from_file).collect();
|
||||||
|
|
||||||
|
Fetched {
|
||||||
|
source_url: url.clone(),
|
||||||
|
caption,
|
||||||
|
title: title.clone(),
|
||||||
|
media,
|
||||||
|
sensitive,
|
||||||
|
site_id: "misskey",
|
||||||
|
render_data: Some(RenderData {
|
||||||
|
url,
|
||||||
|
author: encode_text(&author).into_owned(),
|
||||||
|
author_url: author_url.clone(),
|
||||||
|
title: encode_text(&title).into_owned(),
|
||||||
|
tags: String::new(),
|
||||||
|
}),
|
||||||
|
_keep_alive: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn caption(url: &str, author_url: &str, author: &str, text: &str) -> String {
|
||||||
|
let url = encode_double_quoted_attribute(url);
|
||||||
|
let author_url = encode_double_quoted_attribute(author_url);
|
||||||
|
let author = encode_text(author);
|
||||||
|
if text.is_empty() {
|
||||||
|
return format!("{url}\n<a href=\"{author_url}\">{author}</a>");
|
||||||
|
}
|
||||||
|
format!(
|
||||||
|
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
|
||||||
|
text = encode_text(text),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Maps a Misskey DriveFile to a [`Media`] item; unknown/audio/other types
|
||||||
|
/// are skipped (twitter's `_ => {}` precedent). GIF must be matched before
|
||||||
|
/// the generic image arm.
|
||||||
|
fn media_from_file(file: &model::DriveFile) -> Option<Media> {
|
||||||
|
let title = file.name.clone();
|
||||||
|
match file.mime_type.as_str() {
|
||||||
|
"image/gif" => Some(Media::Animated {
|
||||||
|
title,
|
||||||
|
url: file.url.clone(),
|
||||||
|
thumbnail_url: file.thumbnail_url.clone().unwrap_or_default(),
|
||||||
|
}),
|
||||||
|
mime if mime.starts_with("image/") => Some(Media::Illustration {
|
||||||
|
title,
|
||||||
|
url: file.url.clone(),
|
||||||
|
thumbnail_url: file.thumbnail_url.clone(),
|
||||||
|
fallback_url: None,
|
||||||
|
}),
|
||||||
|
mime if mime.starts_with("video/") => Some(Media::Video {
|
||||||
|
title,
|
||||||
|
url: file.url.clone(),
|
||||||
|
thumbnail_url: file.thumbnail_url.clone().unwrap_or_default(),
|
||||||
|
}),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn note_json(json: serde_json::Value) -> model::Note {
|
||||||
|
serde_json::from_value(json).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn base_note() -> serde_json::Value {
|
||||||
|
serde_json::json!({
|
||||||
|
"id": "aotihl10lqrs015s",
|
||||||
|
"text": "hello",
|
||||||
|
"user": { "name": "ミロン", "username": "donyan47897", "host": null },
|
||||||
|
"files": []
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pattern_matches_misskey_note_urls() {
|
||||||
|
for url in [
|
||||||
|
"https://misskey.io/notes/aotihl10lqrs015s",
|
||||||
|
"http://misskey.io/notes/aotihl10lqrs015s",
|
||||||
|
"misskey.io/notes/aotihl10lqrs015s",
|
||||||
|
] {
|
||||||
|
assert!(PATTERN.is_match(url), "{url}");
|
||||||
|
}
|
||||||
|
for url in [
|
||||||
|
"https://misskey.io/",
|
||||||
|
"https://misskey.io/@user",
|
||||||
|
"https://misskey.io/notes/",
|
||||||
|
"https://x.com/user/status/123",
|
||||||
|
] {
|
||||||
|
assert!(!PATTERN.is_match(url), "{url}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn cache_key_normalizes_variants() {
|
||||||
|
assert_eq!(
|
||||||
|
cache_key("https://misskey.io/notes/aotihl10lqrs015s"),
|
||||||
|
Some("misskey:aotihl10lqrs015s".to_string())
|
||||||
|
);
|
||||||
|
assert_eq!(x_media_site_id("misskey:abc"), "misskey");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn x_media_site_id(key: &str) -> &'static str {
|
||||||
|
crate::site::site_id_from_key(key)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn from_json_image_file() {
|
||||||
|
let mut note = base_note();
|
||||||
|
note["files"] = serde_json::json!([{
|
||||||
|
"type": "image/webp",
|
||||||
|
"url": "https://media.misskeyusercontent.jp/io/a.webp",
|
||||||
|
"thumbnailUrl": "https://media.misskeyusercontent.jp/io/t.webp",
|
||||||
|
"isSensitive": true,
|
||||||
|
"name": "pic.webp"
|
||||||
|
}]);
|
||||||
|
let fetched: Fetched = note_json(note).into();
|
||||||
|
assert_eq!(
|
||||||
|
fetched.source_url,
|
||||||
|
"https://misskey.io/notes/aotihl10lqrs015s"
|
||||||
|
);
|
||||||
|
assert_eq!(fetched.site_id, "misskey");
|
||||||
|
assert_eq!(fetched.title, "hello");
|
||||||
|
assert!(fetched.sensitive);
|
||||||
|
assert_eq!(fetched.media.len(), 1);
|
||||||
|
match &fetched.media[0] {
|
||||||
|
Media::Illustration {
|
||||||
|
title,
|
||||||
|
url,
|
||||||
|
thumbnail_url,
|
||||||
|
fallback_url,
|
||||||
|
} => {
|
||||||
|
assert_eq!(title.as_deref(), Some("pic.webp"));
|
||||||
|
assert_eq!(url, "https://media.misskeyusercontent.jp/io/a.webp");
|
||||||
|
assert_eq!(
|
||||||
|
thumbnail_url.as_deref(),
|
||||||
|
Some("https://media.misskeyusercontent.jp/io/t.webp")
|
||||||
|
);
|
||||||
|
assert!(fallback_url.is_none());
|
||||||
|
}
|
||||||
|
other => panic!("expected illustration, got {other:?}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn from_json_gif_video_and_skip_audio() {
|
||||||
|
let mut note = base_note();
|
||||||
|
note["files"] = serde_json::json!([
|
||||||
|
{ "type": "audio/mpeg", "url": "https://m/a.mp3", "isSensitive": false },
|
||||||
|
{ "type": "image/gif", "url": "https://m/a.gif", "isSensitive": false },
|
||||||
|
{ "type": "video/webm", "url": "https://m/a.webm", "isSensitive": false }
|
||||||
|
]);
|
||||||
|
let fetched: Fetched = note_json(note).into();
|
||||||
|
assert_eq!(fetched.media.len(), 2);
|
||||||
|
assert!(
|
||||||
|
matches!(&fetched.media[0], Media::Animated { url, .. } if url == "https://m/a.gif")
|
||||||
|
);
|
||||||
|
assert!(matches!(&fetched.media[1], Media::Video { url, .. } if url == "https://m/a.webm"));
|
||||||
|
// No thumbnailUrl → empty string, not a broken URL.
|
||||||
|
match &fetched.media[1] {
|
||||||
|
Media::Video { thumbnail_url, .. } => assert_eq!(thumbnail_url, ""),
|
||||||
|
other => panic!("expected video, got {other:?}"),
|
||||||
|
}
|
||||||
|
assert!(!fetched.sensitive);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn from_json_cw_marks_sensitive_and_prefixes_title() {
|
||||||
|
let mut note = base_note();
|
||||||
|
note["cw"] = serde_json::json!("spoiler");
|
||||||
|
note["text"] = serde_json::json!("body");
|
||||||
|
let fetched: Fetched = note_json(note).into();
|
||||||
|
assert!(fetched.sensitive);
|
||||||
|
assert_eq!(fetched.title, "spoiler body");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn from_json_author_falls_back_to_username() {
|
||||||
|
let mut note = base_note();
|
||||||
|
note["user"] = serde_json::json!({ "name": null, "username": "donyan47897", "host": null });
|
||||||
|
let fetched: Fetched = note_json(note).into();
|
||||||
|
assert!(
|
||||||
|
fetched.caption.contains("donyan47897"),
|
||||||
|
"{}",
|
||||||
|
fetched.caption
|
||||||
|
);
|
||||||
|
assert!(fetched.caption.contains("https://misskey.io/@donyan47897"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn from_json_renote_uses_embedded_content() {
|
||||||
|
let note = serde_json::json!({
|
||||||
|
"id": "shell0000000000",
|
||||||
|
"text": null,
|
||||||
|
"user": { "name": "shell", "username": "shelluser", "host": null },
|
||||||
|
"files": [],
|
||||||
|
"renote": {
|
||||||
|
"id": "inner000000000",
|
||||||
|
"text": "inner text",
|
||||||
|
"user": { "name": "inner", "username": "inneruser", "host": null },
|
||||||
|
"files": [
|
||||||
|
{ "type": "image/png", "url": "https://m/i.png", "isSensitive": false }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
});
|
||||||
|
let fetched: Fetched = note_json(note).into();
|
||||||
|
assert_eq!(fetched.title, "inner text");
|
||||||
|
assert_eq!(fetched.media.len(), 1);
|
||||||
|
// The source URL still points at the renote shell the user posted.
|
||||||
|
assert_eq!(
|
||||||
|
fetched.source_url,
|
||||||
|
"https://misskey.io/notes/shell0000000000"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn caption_layout_matches_bsky() {
|
||||||
|
let fetched: Fetched = note_json(base_note()).into();
|
||||||
|
assert_eq!(
|
||||||
|
fetched.caption,
|
||||||
|
"https://misskey.io/notes/aotihl10lqrs015s\n<a href=\"https://misskey.io/@donyan47897\">ミロン</a>: hello"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn caption_without_text_has_no_dangling_colon() {
|
||||||
|
let mut note = base_note();
|
||||||
|
note["text"] = serde_json::json!(null);
|
||||||
|
let fetched: Fetched = note_json(note).into();
|
||||||
|
assert_eq!(
|
||||||
|
fetched.caption,
|
||||||
|
"https://misskey.io/notes/aotihl10lqrs015s\n<a href=\"https://misskey.io/@donyan47897\">ミロン</a>"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
#[ignore = "live network: requires outbound HTTPS to misskey.io"]
|
||||||
|
async fn live_fetch_reference_note() {
|
||||||
|
let fetched = fetch_from_url("https://misskey.io/notes/aotihl10lqrs015s")
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(fetched.site_id, "misskey");
|
||||||
|
assert_eq!(fetched.media.len(), 1);
|
||||||
|
assert!(fetched.sensitive);
|
||||||
|
assert!(!fetched.caption.is_empty());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
mod interface;
|
||||||
|
mod model;
|
||||||
|
|
||||||
|
pub use interface::{
|
||||||
|
MisskeySite, PATTERN, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
|
||||||
|
};
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
use serde::Deserialize;
|
||||||
|
|
||||||
|
#[derive(Deserialize, Debug)]
|
||||||
|
pub(crate) struct Note {
|
||||||
|
pub(crate) id: String,
|
||||||
|
pub(crate) text: Option<String>,
|
||||||
|
#[serde(default)]
|
||||||
|
pub(crate) cw: Option<String>,
|
||||||
|
pub(crate) user: User,
|
||||||
|
#[serde(default)]
|
||||||
|
pub(crate) files: Vec<DriveFile>,
|
||||||
|
/// Embedded original note when this note is a renote; the shell's own
|
||||||
|
/// text/files are usually empty and the content lives here.
|
||||||
|
#[serde(default)]
|
||||||
|
pub(crate) renote: Option<Box<Note>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize, Debug)]
|
||||||
|
pub(crate) struct User {
|
||||||
|
pub(crate) name: Option<String>,
|
||||||
|
pub(crate) username: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize, Debug)]
|
||||||
|
pub(crate) struct DriveFile {
|
||||||
|
#[serde(rename = "type")]
|
||||||
|
pub(crate) mime_type: String,
|
||||||
|
pub(crate) url: String,
|
||||||
|
#[serde(default, rename = "thumbnailUrl")]
|
||||||
|
pub(crate) thumbnail_url: Option<String>,
|
||||||
|
#[serde(default, rename = "isSensitive")]
|
||||||
|
pub(crate) is_sensitive: bool,
|
||||||
|
#[serde(default)]
|
||||||
|
pub(crate) name: Option<String>,
|
||||||
|
}
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
//! Site fetching dispatcher and unified result types.
|
//! Site fetching dispatcher and unified result types.
|
||||||
//!
|
//!
|
||||||
//! Dispatch order: twitter → bsky → pixiv. Each site module exports a
|
//! Dispatch order: twitter → bsky → misskey → pixiv. Each site module
|
||||||
//! `PATTERN`, `enabled()` and `fetch_from_url()`; a future site plugs in by
|
//! exports a `PATTERN`, `enabled()` and `fetch_from_url()`; a future site
|
||||||
//! adding one guarded entry in [`fetch_once`].
|
//! plugs in by adding one guarded entry in [`fetch_once`].
|
||||||
|
|
||||||
use std::future::Future;
|
use std::future::Future;
|
||||||
use std::pin::Pin;
|
use std::pin::Pin;
|
||||||
@@ -14,6 +14,7 @@ use regex::Regex;
|
|||||||
use thiserror::Error;
|
use thiserror::Error;
|
||||||
|
|
||||||
pub mod bsky;
|
pub mod bsky;
|
||||||
|
pub mod misskey;
|
||||||
pub mod pixiv;
|
pub mod pixiv;
|
||||||
pub mod twitter;
|
pub mod twitter;
|
||||||
|
|
||||||
@@ -329,6 +330,7 @@ static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| {
|
|||||||
vec![
|
vec![
|
||||||
Box::new(twitter::TwitterSite),
|
Box::new(twitter::TwitterSite),
|
||||||
Box::new(bsky::BskySite),
|
Box::new(bsky::BskySite),
|
||||||
|
Box::new(misskey::MisskeySite),
|
||||||
Box::new(pixiv::PixivSite),
|
Box::new(pixiv::PixivSite),
|
||||||
]
|
]
|
||||||
});
|
});
|
||||||
@@ -531,10 +533,10 @@ mod tests {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn registry_lists_all_sites_in_dispatch_order() {
|
fn registry_lists_all_sites_in_dispatch_order() {
|
||||||
assert_eq!(site_ids(), vec!["twitter", "bsky", "pixiv"]);
|
assert_eq!(site_ids(), vec!["twitter", "bsky", "misskey", "pixiv"]);
|
||||||
// Enabled sites dispatch; unsupported URLs never match.
|
// Enabled sites dispatch; unsupported URLs never match.
|
||||||
assert!(find_site("https://x.com/u/status/1").is_some());
|
assert!(find_site("https://x.com/u/status/1").is_some());
|
||||||
assert!(find_site("https://bsky.app/profile/u/post/3x").is_some());
|
assert!(find_site("https://misskey.io/notes/abc").is_some());
|
||||||
assert!(find_site("https://example.com/x").is_none());
|
assert!(find_site("https://example.com/x").is_none());
|
||||||
// Cache keys are pattern-driven, independent of the enabled() gate
|
// Cache keys are pattern-driven, independent of the enabled() gate
|
||||||
// (pixiv is disabled in tests without PIXIV_REFRESH_TOKEN).
|
// (pixiv is disabled in tests without PIXIV_REFRESH_TOKEN).
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
use super::model;
|
use super::model;
|
||||||
use crate::media::Media;
|
use crate::media::Media;
|
||||||
use crate::site::{FetchError, Fetched, Site, SiteFuture};
|
use crate::site::{FetchError, Fetched, Site, SiteFuture};
|
||||||
use html_escape::{encode_double_quoted_attribute, encode_text};
|
use html_escape::{decode_html_entities, encode_double_quoted_attribute, encode_text};
|
||||||
use regex::Regex;
|
use regex::Regex;
|
||||||
use std::sync::LazyLock;
|
use std::sync::LazyLock;
|
||||||
|
|
||||||
@@ -239,9 +239,17 @@ impl Tweet {
|
|||||||
// strip the appended media short link, mirroring FxEmbed's linkFixer
|
// strip the appended media short link, mirroring FxEmbed's linkFixer
|
||||||
// (no display_text_range arithmetic — see expand_links).
|
// (no display_text_range arithmetic — see expand_links).
|
||||||
let text = expand_links(&json.text, &json.entities.urls);
|
let text = expand_links(&json.text, &json.entities.urls);
|
||||||
|
// Twitter APIs (syndication AND GraphQL full_text) return the text
|
||||||
|
// pre-escaped for HTML (`>` `<` `&` `'` …): decode it so
|
||||||
|
// the stored text is raw. The caption's own escaping then produces
|
||||||
|
// the rendered form exactly once — without this, `>^ω^<` would
|
||||||
|
// be double-escaped to `&gt;^ω^&lt;` and the sent message
|
||||||
|
// would show literal `>^ω^<`.
|
||||||
|
let text = decode_html_entities(&text).into_owned();
|
||||||
// `name` is the display name, `screen_name` the handle (Python's
|
// `name` is the display name, `screen_name` the handle (Python's
|
||||||
// vxtwitter mapping: author = display name, author_id = handle).
|
// vxtwitter mapping: author = display name, author_id = handle).
|
||||||
let author = json.user.name;
|
// Display names can carry the same pre-escaped entities.
|
||||||
|
let author = decode_html_entities(&json.user.name).into_owned();
|
||||||
let author_id = json.user.screen_name;
|
let author_id = json.user.screen_name;
|
||||||
let mut media = vec![];
|
let mut media = vec![];
|
||||||
for item in json.media_details {
|
for item in json.media_details {
|
||||||
@@ -409,6 +417,41 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn syndication_text_is_unescaped_before_storing() {
|
||||||
|
// Real API shape: the text arrives pre-escaped for HTML — e.g. the
|
||||||
|
// tweet `>^ω^<` comes back as `>^ω^<` (fxtwitter's raw_text for
|
||||||
|
// 2060196388252827954) and apostrophes as `'`. Storing it raw and
|
||||||
|
// escaping once at caption build avoids the double-escape that would
|
||||||
|
// show literal `>`/`<`/`&` in the sent message.
|
||||||
|
let raw = serde_json::json!({
|
||||||
|
"__typename": "Tweet",
|
||||||
|
"id_str": "1",
|
||||||
|
"text": ">^ω^< & more 'quoted' https://t.co/abc123",
|
||||||
|
"user": { "name": "O'Brien", "screen_name": "h" },
|
||||||
|
"entities": { "urls": [] },
|
||||||
|
"mediaDetails": []
|
||||||
|
});
|
||||||
|
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
|
||||||
|
// The appended media short link is stripped, then entities decoded.
|
||||||
|
assert_eq!(tweet.text, ">^ω^< & more 'quoted'");
|
||||||
|
assert_eq!(tweet.author, "O'Brien");
|
||||||
|
let fetched: Fetched = tweet.into();
|
||||||
|
assert_eq!(fetched.title, ">^ω^< & more 'quoted'");
|
||||||
|
// The caption escapes the raw text exactly once (encode_text covers
|
||||||
|
// & < >; apostrophes stay literal — they are harmless in text).
|
||||||
|
assert!(
|
||||||
|
fetched.caption.contains(">^ω^< & more 'quoted'"),
|
||||||
|
"caption: {}",
|
||||||
|
fetched.caption
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!fetched.caption.contains("&gt;"),
|
||||||
|
"double-escaped text: {}",
|
||||||
|
fetched.caption
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn cache_key_prefixes_tweet_id() {
|
fn cache_key_prefixes_tweet_id() {
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "xmedia-bot"
|
name = "xmedia-bot"
|
||||||
version = "1.3.0"
|
version = "1.5.0"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
|
|||||||
@@ -160,3 +160,9 @@ pub fn now_f64() -> f64 {
|
|||||||
.map(|d| d.as_secs_f64())
|
.map(|d| d.as_secs_f64())
|
||||||
.unwrap_or(0.0)
|
.unwrap_or(0.0)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Unix timestamp in whole seconds. Same clock as [`now_f64`], for fields
|
||||||
|
/// that store integer seconds (chat-state expiry, edit prompts).
|
||||||
|
pub fn unix_now() -> i64 {
|
||||||
|
now_f64() as i64
|
||||||
|
}
|
||||||
|
|||||||
@@ -3,8 +3,8 @@
|
|||||||
|
|
||||||
use super::urls::enqueue_retry;
|
use super::urls::enqueue_retry;
|
||||||
use super::{CHAT_STORE, CONFIG, TASK_QUEUE};
|
use super::{CHAT_STORE, CONFIG, TASK_QUEUE};
|
||||||
|
use crate::db::unix_now;
|
||||||
use crate::send::{self, Task};
|
use crate::send::{self, Task};
|
||||||
use crate::state::unix_now;
|
|
||||||
use teloxide::RequestError;
|
use teloxide::RequestError;
|
||||||
use teloxide::prelude::*;
|
use teloxide::prelude::*;
|
||||||
use teloxide::types::{CallbackQuery, ChatId, MessageId, ParseMode};
|
use teloxide::types::{CallbackQuery, ChatId, MessageId, ParseMode};
|
||||||
@@ -83,7 +83,7 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
|
|||||||
task,
|
task,
|
||||||
}) => {
|
}) => {
|
||||||
log::info!("forward queued for retry in {delay_seconds:.1}s");
|
log::info!("forward queued for retry in {delay_seconds:.1}s");
|
||||||
enqueue_retry(&TASK_QUEUE, task, delay_seconds).await;
|
enqueue_retry(&TASK_QUEUE, *task, delay_seconds).await;
|
||||||
bot.answer_callback_query(callback_query_id)
|
bot.answer_callback_query(callback_query_id)
|
||||||
.text("Forward queued for retry.")
|
.text("Forward queued for retry.")
|
||||||
.await?;
|
.await?;
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
//! Bot command parsing, the `/`-command executor and `setMyCommands`
|
//! Bot command parsing, the `/`-command executor and `setMyCommands`
|
||||||
//! registration. URL/inline/callback flows live in their own modules.
|
//! registration. URL/inline/callback flows live in their own modules.
|
||||||
|
|
||||||
use super::{CHAT_STORE, CONFIG, LINK_CACHE, log_key, reply};
|
use super::{CHAT_STORE, CONFIG, LINK_CACHE, log_key, reply, reply_html};
|
||||||
use teloxide::RequestError;
|
use teloxide::RequestError;
|
||||||
use teloxide::prelude::*;
|
use teloxide::prelude::*;
|
||||||
use teloxide::types::{ChatId, Message, Recipient};
|
use teloxide::types::{ChatId, Message, Recipient};
|
||||||
@@ -253,7 +253,7 @@ pub(crate) async fn execute_command(
|
|||||||
bot,
|
bot,
|
||||||
message.chat.id.0,
|
message.chat.id.0,
|
||||||
message.id,
|
message.id,
|
||||||
"Unknown site. Use twitter, bsky or pixiv.",
|
"Unknown site. Use twitter, bsky, pixiv or misskey.",
|
||||||
)
|
)
|
||||||
.await?;
|
.await?;
|
||||||
return Ok(());
|
return Ok(());
|
||||||
@@ -294,7 +294,7 @@ pub(crate) async fn execute_command(
|
|||||||
bot,
|
bot,
|
||||||
message.chat.id.0,
|
message.chat.id.0,
|
||||||
message.id,
|
message.id,
|
||||||
"Unrecognized link. Use a twitter/x, pixiv or bsky post URL.",
|
"Unrecognized link. Use a twitter/x, pixiv, bsky or misskey post URL.",
|
||||||
)
|
)
|
||||||
.await?;
|
.await?;
|
||||||
return Ok(());
|
return Ok(());
|
||||||
@@ -337,7 +337,7 @@ pub(crate) async fn execute_command(
|
|||||||
bot,
|
bot,
|
||||||
message.chat.id.0,
|
message.chat.id.0,
|
||||||
message.id,
|
message.id,
|
||||||
"No enabled site matches this link (twitter/x, pixiv or bsky).",
|
"No enabled site matches this link (twitter/x, pixiv, bsky or misskey).",
|
||||||
)
|
)
|
||||||
.await?;
|
.await?;
|
||||||
}
|
}
|
||||||
@@ -361,7 +361,9 @@ pub(crate) async fn execute_command(
|
|||||||
&fetched.caption,
|
&fetched.caption,
|
||||||
&fetched.media,
|
&fetched.media,
|
||||||
);
|
);
|
||||||
reply(bot, message.chat.id.0, message.id, report).await?;
|
// HTML report: the caption renders inside a <blockquote>
|
||||||
|
// exactly as it will appear in the sent media message.
|
||||||
|
reply_html(bot, message.chat.id.0, message.id, report).await?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -387,12 +389,15 @@ pub async fn register_commands(bot: &Bot) -> Result<(), RequestError> {
|
|||||||
/// it even for very large threads (many media lines + a long caption).
|
/// it even for very large threads (many media lines + a long caption).
|
||||||
const MAX_TEST_REPORT_CHARS: usize = 4000;
|
const MAX_TEST_REPORT_CHARS: usize = 4000;
|
||||||
|
|
||||||
/// Builds the plain-text report for the `/test` command: what the parser
|
/// Builds the HTML report for the `/test` command: what the parser produced
|
||||||
/// produced for a link (site, canonical URL, title/author/tags, caption and
|
/// for a link (site, canonical URL, title/author/tags, caption and the media
|
||||||
/// the media list) — no media is sent and nothing is cached or forwarded.
|
/// list) — no media is sent and nothing is cached or forwarded. Sent with
|
||||||
/// Fields are passed individually so the formatter stays a pure function
|
/// HTML parse mode: raw fields are escaped, the pre-escaped render fields are
|
||||||
/// testable without constructing a `Fetched` (its render fields are
|
/// embedded as-is, and the caption is wrapped in a `<blockquote>` so it shows
|
||||||
/// `pub(crate)` to the x-media crate).
|
/// exactly as it will render in the sent media message. Fields are passed
|
||||||
|
/// individually so the formatter stays a pure function testable without
|
||||||
|
/// constructing a `Fetched` (its render fields are `pub(crate)` to the
|
||||||
|
/// x-media crate).
|
||||||
#[allow(clippy::too_many_arguments)]
|
#[allow(clippy::too_many_arguments)]
|
||||||
fn test_parse_report(
|
fn test_parse_report(
|
||||||
url: &str,
|
url: &str,
|
||||||
@@ -405,23 +410,37 @@ fn test_parse_report(
|
|||||||
media: &[x_media::media::Media],
|
media: &[x_media::media::Media],
|
||||||
) -> String {
|
) -> String {
|
||||||
let mut lines = vec![
|
let mut lines = vec![
|
||||||
format!("Parse result for {url}"),
|
format!("Parse result for {}", html_escape::encode_text(url)),
|
||||||
format!("site: {site_id}"),
|
format!("site: {site_id}"),
|
||||||
format!(
|
format!(
|
||||||
"key: {}",
|
"key: {}",
|
||||||
x_media::site::cache_key(url).unwrap_or_else(|| "<unsupported>".to_string())
|
html_escape::encode_text(
|
||||||
|
&x_media::site::cache_key(url).unwrap_or_else(|| "<unsupported>".to_string())
|
||||||
|
)
|
||||||
),
|
),
|
||||||
];
|
];
|
||||||
lines.push(format!("source_url: {source_url}"));
|
lines.push(format!(
|
||||||
lines.push(format!("title: {title}"));
|
"source_url: {}",
|
||||||
|
html_escape::encode_text(source_url)
|
||||||
|
));
|
||||||
|
lines.push(format!("title: {}", html_escape::encode_text(title)));
|
||||||
if let Some((author, author_url, _title, tags)) = render {
|
if let Some((author, author_url, _title, tags)) = render {
|
||||||
|
// The render fields are already pre-escaped for HTML captions; embed
|
||||||
|
// them as-is so the report renders them exactly like the final
|
||||||
|
// caption. `author_url` is raw and gets escaped here.
|
||||||
lines.push(format!("author: {author}"));
|
lines.push(format!("author: {author}"));
|
||||||
lines.push(format!("author_url: {author_url}"));
|
lines.push(format!(
|
||||||
|
"author_url: {}",
|
||||||
|
html_escape::encode_text(author_url)
|
||||||
|
));
|
||||||
lines.push(format!("tags: {tags}"));
|
lines.push(format!("tags: {tags}"));
|
||||||
}
|
}
|
||||||
lines.push(format!("sensitive: {sensitive}"));
|
lines.push(format!("sensitive: {sensitive}"));
|
||||||
|
// The caption is wrapped in a <blockquote> so the report (an HTML
|
||||||
|
// message) shows it exactly as it will render in the sent media caption
|
||||||
|
// — escaped text and links included.
|
||||||
lines.push(format!(
|
lines.push(format!(
|
||||||
"caption: {}",
|
"caption: <blockquote>{}</blockquote>",
|
||||||
x_media::site::truncate_caption(caption)
|
x_media::site::truncate_caption(caption)
|
||||||
));
|
));
|
||||||
lines.push(format!("media ({}):", media.len()));
|
lines.push(format!("media ({}):", media.len()));
|
||||||
@@ -431,7 +450,11 @@ fn test_parse_report(
|
|||||||
x_media::media::Media::Video { .. } => "video",
|
x_media::media::Media::Video { .. } => "video",
|
||||||
x_media::media::Media::Animated { .. } => "gif",
|
x_media::media::Media::Animated { .. } => "gif",
|
||||||
};
|
};
|
||||||
lines.push(format!(" {}. {kind}: {}", i + 1, item.url()));
|
lines.push(format!(
|
||||||
|
" {}. {kind}: {}",
|
||||||
|
i + 1,
|
||||||
|
html_escape::encode_text(item.url())
|
||||||
|
));
|
||||||
}
|
}
|
||||||
let mut out = lines.join(
|
let mut out = lines.join(
|
||||||
"
|
"
|
||||||
@@ -500,6 +523,45 @@ mod tests {
|
|||||||
assert!(report.contains("media (0):"), "{report}");
|
assert!(report.contains("media (0):"), "{report}");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_parse_report_wraps_caption_in_blockquote() {
|
||||||
|
// The report is an HTML message: raw fields are escaped, pre-escaped
|
||||||
|
// render fields are embedded as-is, and the caption is wrapped in a
|
||||||
|
// <blockquote> so it shows exactly as it will render in the sent
|
||||||
|
// media caption (escaped text and links included).
|
||||||
|
let report = test_parse_report(
|
||||||
|
"https://x.com/u/status/1",
|
||||||
|
"twitter",
|
||||||
|
"https://x.com/u/status/1",
|
||||||
|
"A & B <C>",
|
||||||
|
Some((
|
||||||
|
"A & B",
|
||||||
|
"https://x.com/u",
|
||||||
|
"A & B <C>",
|
||||||
|
"#a & #b",
|
||||||
|
)),
|
||||||
|
false,
|
||||||
|
"<a href=\"https://x.com/u\">A & B</a>: C <D> & E",
|
||||||
|
&[],
|
||||||
|
);
|
||||||
|
// Raw fields escaped (they render back to the original text in HTML).
|
||||||
|
assert!(report.contains("title: A & B <C>"), "{report}");
|
||||||
|
assert!(
|
||||||
|
report.contains("source_url: https://x.com/u/status/1"),
|
||||||
|
"{report}"
|
||||||
|
);
|
||||||
|
// Pre-escaped render fields embedded as-is.
|
||||||
|
assert!(report.contains("author: A & B"), "{report}");
|
||||||
|
assert!(report.contains("tags: #a & #b"), "{report}");
|
||||||
|
// Caption wrapped in a blockquote with its HTML preserved.
|
||||||
|
assert!(
|
||||||
|
report.contains(
|
||||||
|
"caption: <blockquote><a href=\"https://x.com/u\">A & B</a>: C <D> & E</blockquote>"
|
||||||
|
),
|
||||||
|
"{report}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_parse_report_is_capped() {
|
fn test_parse_report_is_capped() {
|
||||||
// 200 media lines ≈ 8 KB, comfortably over the cap.
|
// 200 media lines ≈ 8 KB, comfortably over the cap.
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ use crate::media_sender::MediaSender;
|
|||||||
use commands::{Command, execute_command};
|
use commands::{Command, execute_command};
|
||||||
use teloxide::RequestError;
|
use teloxide::RequestError;
|
||||||
use teloxide::prelude::*;
|
use teloxide::prelude::*;
|
||||||
use teloxide::types::{ChatId, ChatKind, Message, MessageId, ParseMode};
|
use teloxide::types::{ChatId, ChatKind, Message, MessageId, ParseMode, ReplyParameters};
|
||||||
use teloxide::utils::command::BotCommands;
|
use teloxide::utils::command::BotCommands;
|
||||||
use urls::{URL_JOBS, extract_urls};
|
use urls::{URL_JOBS, extract_urls};
|
||||||
|
|
||||||
@@ -42,6 +42,23 @@ where
|
|||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Reply to a message by id with HTML parse mode (same reply decoration as
|
||||||
|
/// [`reply`]). Used by `/test`, whose report is an HTML message (the caption
|
||||||
|
/// is wrapped in a `<blockquote>` to show it exactly as it will render).
|
||||||
|
pub(crate) async fn reply_html(
|
||||||
|
bot: &Bot,
|
||||||
|
chat_id: i64,
|
||||||
|
reply_to: MessageId,
|
||||||
|
text: String,
|
||||||
|
) -> Result<Message, RequestError> {
|
||||||
|
// `<Bot as Requester>::` disambiguates from the MediaSender trait's
|
||||||
|
// same-named method (see media_sender.rs).
|
||||||
|
<Bot as Requester>::send_message(bot, ChatId(chat_id), text)
|
||||||
|
.parse_mode(ParseMode::Html)
|
||||||
|
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply())
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
|
||||||
/// Log prefix tying the whole lifecycle of one link (fetch → send → cache →
|
/// Log prefix tying the whole lifecycle of one link (fetch → send → cache →
|
||||||
/// forward) together: the normalized cache key (`twitter:123…`, `pixiv:123`,
|
/// forward) together: the normalized cache key (`twitter:123…`, `pixiv:123`,
|
||||||
/// `bsky:handle/rkey`) instead of the raw URL, so logs stay short and do not
|
/// `bsky:handle/rkey`) instead of the raw URL, so logs stay short and do not
|
||||||
|
|||||||
@@ -12,8 +12,24 @@ use std::sync::{Arc, LazyLock};
|
|||||||
/// cache): a single pool bounds concurrent DB work on `data/task_queue.db`
|
/// cache): a single pool bounds concurrent DB work on `data/task_queue.db`
|
||||||
/// instead of three independent pools competing for the same file. The schema
|
/// instead of three independent pools competing for the same file. The schema
|
||||||
/// for all three tables is initialized once, here.
|
/// for all three tables is initialized once, here.
|
||||||
static DB: LazyLock<Arc<db::DbPool>> =
|
static DB: LazyLock<Arc<db::DbPool>> = LazyLock::new(|| {
|
||||||
LazyLock::new(|| db::open_store("data/task_queue.db").expect("failed to open database"));
|
let path = db_path();
|
||||||
|
db::open_store(&path.to_string_lossy()).expect("failed to open database")
|
||||||
|
});
|
||||||
|
|
||||||
|
/// DB file location: `$DATA_DIR/task_queue.db` (default `data`, relative to
|
||||||
|
/// the working directory — keeps the docker-compose `./data` mount and local
|
||||||
|
/// runs unchanged). The directory is created if missing: SQLite does not
|
||||||
|
/// create parent dirs, so the old hardcoded `data/task_queue.db` failed with
|
||||||
|
/// a confusing error when started from a directory without `data/`, and a
|
||||||
|
/// CWD-relative path is a footgun for systemd / cron deployments — `DATA_DIR`
|
||||||
|
/// lets them pin the state anywhere.
|
||||||
|
fn db_path() -> std::path::PathBuf {
|
||||||
|
let dir = std::env::var("DATA_DIR").unwrap_or_else(|_| "data".to_string());
|
||||||
|
let dir_path = std::path::Path::new(&dir);
|
||||||
|
std::fs::create_dir_all(dir_path).expect("failed to create data directory");
|
||||||
|
dir_path.join("task_queue.db")
|
||||||
|
}
|
||||||
|
|
||||||
pub static CHAT_STORE: LazyLock<ChatStore> = LazyLock::new(|| ChatStore::new(Arc::clone(&DB)));
|
pub static CHAT_STORE: LazyLock<ChatStore> = LazyLock::new(|| ChatStore::new(Arc::clone(&DB)));
|
||||||
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
|
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
|
||||||
|
|||||||
@@ -135,7 +135,12 @@ pub fn extract_urls(message: &Message) -> Vec<String> {
|
|||||||
fn thumbnail_for(media: &Media) -> Option<String> {
|
fn thumbnail_for(media: &Media) -> Option<String> {
|
||||||
let url = media.url();
|
let url = media.url();
|
||||||
if url.starts_with("http://") || url.starts_with("https://") {
|
if url.starts_with("http://") || url.starts_with("https://") {
|
||||||
media.thumbnail_url().map(str::to_string)
|
// An empty thumbnail string (misskey video/gif files without a
|
||||||
|
// thumbnailUrl) must not reach Telegram; let it generate its own.
|
||||||
|
media
|
||||||
|
.thumbnail_url()
|
||||||
|
.map(str::to_string)
|
||||||
|
.filter(|t| !t.is_empty())
|
||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
}
|
}
|
||||||
@@ -211,7 +216,7 @@ async fn dispatch_send(
|
|||||||
"send for [key={}] failed, queued for retry in {delay_seconds:.1}s",
|
"send for [key={}] failed, queued for retry in {delay_seconds:.1}s",
|
||||||
log_key(url)
|
log_key(url)
|
||||||
);
|
);
|
||||||
enqueue_retry(ctx.task_queue, task, delay_seconds).await;
|
enqueue_retry(ctx.task_queue, *task, delay_seconds).await;
|
||||||
let _ = reply(
|
let _ = reply(
|
||||||
ctx.sender,
|
ctx.sender,
|
||||||
chat_id,
|
chat_id,
|
||||||
|
|||||||
@@ -122,7 +122,7 @@ fn output_channels(color: png::ColorType) -> usize {
|
|||||||
/// white; 16-bit per channel was already stripped to 8-bit at decode.
|
/// white; 16-bit per channel was already stripped to 8-bit at decode.
|
||||||
fn flatten_rgba_to_rgb(rgba: &[u8]) -> Vec<u8> {
|
fn flatten_rgba_to_rgb(rgba: &[u8]) -> Vec<u8> {
|
||||||
let mut rgb = Vec::with_capacity(rgba.len() / 4 * 3);
|
let mut rgb = Vec::with_capacity(rgba.len() / 4 * 3);
|
||||||
for px in rgba.chunks_exact(4) {
|
for px in rgba.as_chunks::<4>().0 {
|
||||||
let a = px[3] as u32;
|
let a = px[3] as u32;
|
||||||
for v in &px[..3] {
|
for v in &px[..3] {
|
||||||
// Over white: C = C*a/255 + 255*(1 - a/255).
|
// Over white: C = C*a/255 + 255*(1 - a/255).
|
||||||
@@ -178,7 +178,9 @@ fn encode_jpeg(pix: &PixBuf, w: u32, h: u32) -> Result<Vec<u8>, String> {
|
|||||||
PixBuf::GrayAlpha(v) => {
|
PixBuf::GrayAlpha(v) => {
|
||||||
// JPEG has no alpha: composite onto white, output as gray.
|
// JPEG has no alpha: composite onto white, output as gray.
|
||||||
let gray: Vec<u8> = v
|
let gray: Vec<u8> = v
|
||||||
.chunks_exact(2)
|
.as_chunks::<2>()
|
||||||
|
.0
|
||||||
|
.iter()
|
||||||
.map(|px| {
|
.map(|px| {
|
||||||
let (g, a) = (px[0] as u32, px[1] as u32);
|
let (g, a) = (px[0] as u32, px[1] as u32);
|
||||||
((g * a + 255 * (255 - a)) / 255).min(255) as u8
|
((g * a + 255 * (255 - a)) / 255).min(255) as u8
|
||||||
|
|||||||
@@ -307,6 +307,11 @@ impl QueueWorker {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Processes one leased row, keeping the lease alive while the handler
|
||||||
|
/// runs. Without the heartbeat a task longer than [`LOCK_TTL_SECONDS`]
|
||||||
|
/// (slow download, ugoira encode, rate-limited batch forward) would have
|
||||||
|
/// its lease expire mid-run; the expiry sweep would flip the row back to
|
||||||
|
/// `pending` and another worker would process it again — duplicate sends.
|
||||||
async fn process(&self, row: LeasedRow) {
|
async fn process(&self, row: LeasedRow) {
|
||||||
let payload: Value = match serde_json::from_str(&row.payload) {
|
let payload: Value = match serde_json::from_str(&row.payload) {
|
||||||
Ok(value) => value,
|
Ok(value) => value,
|
||||||
@@ -318,7 +323,8 @@ impl QueueWorker {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
log::debug!("processing {} (attempt {})", row.id, row.attempts + 1);
|
log::debug!("processing {} (attempt {})", row.id, row.attempts + 1);
|
||||||
match (self.handler)(payload).await {
|
let outcome = self.run_with_lease(&row.id, payload).await;
|
||||||
|
match outcome {
|
||||||
Ok(()) => {
|
Ok(()) => {
|
||||||
log::debug!("task {} completed", row.id);
|
log::debug!("task {} completed", row.id);
|
||||||
self.delete_row(&row.id).await;
|
self.delete_row(&row.id).await;
|
||||||
@@ -351,6 +357,42 @@ impl QueueWorker {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Drives the handler to completion, refreshing the row's `locked_until`
|
||||||
|
/// every 30 s so the expiry sweep never re-leases a still-running task.
|
||||||
|
/// The heartbeat is part of this future, not a separate spawned task: if
|
||||||
|
/// the worker task dies (panic) the heartbeat dies with it and the sweep
|
||||||
|
/// recovers the row exactly as before.
|
||||||
|
async fn run_with_lease(&self, id: &str, payload: Value) -> Result<(), QueueError> {
|
||||||
|
let fut = (self.handler)(payload);
|
||||||
|
tokio::pin!(fut);
|
||||||
|
let mut interval = tokio::time::interval(Duration::from_secs(30));
|
||||||
|
// The first interval tick fires immediately; skip it (the lease was
|
||||||
|
// just set by lease_next).
|
||||||
|
interval.tick().await;
|
||||||
|
let id_owned = id.to_string();
|
||||||
|
loop {
|
||||||
|
tokio::select! {
|
||||||
|
result = &mut fut => return result,
|
||||||
|
_ = interval.tick() => {
|
||||||
|
let now = now_f64();
|
||||||
|
let id = id_owned.clone();
|
||||||
|
let result = self
|
||||||
|
.pool
|
||||||
|
.with_conn(move |conn| {
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE tasks SET locked_until=?1 WHERE id=?2 AND status='in_progress'",
|
||||||
|
params![now + LOCK_TTL_SECONDS, id],
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
if let Err(e) = result {
|
||||||
|
log::error!("queue lease heartbeat failed: {e}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async fn delete_row(&self, id: &str) {
|
async fn delete_row(&self, id: &str) {
|
||||||
let id = id.to_string();
|
let id = id.to_string();
|
||||||
let result = self
|
let result = self
|
||||||
|
|||||||
@@ -3,12 +3,13 @@
|
|||||||
//! URL is blocked by hotlink protection; the bot downloads the file itself
|
//! URL is blocked by hotlink protection; the bot downloads the file itself
|
||||||
//! and uploads it via multipart).
|
//! and uploads it via multipart).
|
||||||
|
|
||||||
|
use crate::db::unix_now;
|
||||||
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE, log_key};
|
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE, log_key};
|
||||||
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost, LinkCache};
|
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost, LinkCache};
|
||||||
use crate::media_sender::MediaSender;
|
use crate::media_sender::MediaSender;
|
||||||
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
|
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
|
||||||
use crate::queue::QueueError;
|
use crate::queue::QueueError;
|
||||||
use crate::state::{EditMessage, unix_now};
|
use crate::state::EditMessage;
|
||||||
use rand::Rng;
|
use rand::Rng;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
@@ -386,22 +387,25 @@ pub fn classify_request_error(e: &RequestError) -> Classification {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Task boxed to keep the error size within `result_large_err` limits.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub enum SendError {
|
pub enum SendError {
|
||||||
Retryable { delay_seconds: f64, task: Task },
|
Retryable { delay_seconds: f64, task: Box<Task> },
|
||||||
Permanent { message: String, task: Task },
|
Permanent { message: String, task: Box<Task> },
|
||||||
}
|
}
|
||||||
|
fn classify_to_send_error(e: &RequestError, task: Task, fetch_failure_label: &str) -> SendError {
|
||||||
fn classify_to_send_error(e: &RequestError, task: Task) -> SendError {
|
|
||||||
match classify_request_error(e) {
|
match classify_request_error(e) {
|
||||||
Classification::Retryable { delay_seconds } => SendError::Retryable {
|
Classification::Retryable { delay_seconds } => SendError::Retryable {
|
||||||
delay_seconds,
|
delay_seconds,
|
||||||
task,
|
task: Box::new(task),
|
||||||
|
},
|
||||||
|
Classification::Permanent { message } => SendError::Permanent {
|
||||||
|
message,
|
||||||
|
task: Box::new(task),
|
||||||
},
|
},
|
||||||
Classification::Permanent { message } => SendError::Permanent { message, task },
|
|
||||||
Classification::MediaFetchFailure => SendError::Permanent {
|
Classification::MediaFetchFailure => SendError::Permanent {
|
||||||
message: "media fetch failed".into(),
|
message: fetch_failure_label.into(),
|
||||||
task,
|
task: Box::new(task),
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -415,9 +419,12 @@ impl SendError {
|
|||||||
match f {
|
match f {
|
||||||
FallbackError::Retryable { delay_seconds } => SendError::Retryable {
|
FallbackError::Retryable { delay_seconds } => SendError::Retryable {
|
||||||
delay_seconds,
|
delay_seconds,
|
||||||
task,
|
task: Box::new(task),
|
||||||
|
},
|
||||||
|
FallbackError::Permanent { message } => SendError::Permanent {
|
||||||
|
message,
|
||||||
|
task: Box::new(task),
|
||||||
},
|
},
|
||||||
FallbackError::Permanent { message } => SendError::Permanent { message, task },
|
|
||||||
FallbackError::MediaTooLarge => unreachable!("handled inside the upload fallback"),
|
FallbackError::MediaTooLarge => unreachable!("handled inside the upload fallback"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -847,7 +854,7 @@ async fn send_batch_via_upload(
|
|||||||
Err(e) => {
|
Err(e) => {
|
||||||
return Err(SendError::Permanent {
|
return Err(SendError::Permanent {
|
||||||
message: format!("upload worker panicked: {e}"),
|
message: format!("upload worker panicked: {e}"),
|
||||||
task,
|
task: Box::new(task),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
@@ -872,51 +879,24 @@ async fn send_batch_via_upload(
|
|||||||
drop(keep_alive);
|
drop(keep_alive);
|
||||||
match result {
|
match result {
|
||||||
Ok(messages) => Ok(messages),
|
Ok(messages) => Ok(messages),
|
||||||
Err(e) => Err(match classify_request_error(&e) {
|
Err(e) => Err(classify_to_send_error(&e, task, "upload failed")),
|
||||||
Classification::Retryable { delay_seconds } => SendError::Retryable {
|
|
||||||
delay_seconds,
|
|
||||||
task: task.clone(),
|
|
||||||
},
|
|
||||||
Classification::Permanent { message } => SendError::Permanent { message, task },
|
|
||||||
Classification::MediaFetchFailure => SendError::Permanent {
|
|
||||||
message: "upload failed".into(),
|
|
||||||
task,
|
|
||||||
},
|
|
||||||
}),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn updated_sequence_task(task: &Task, batch_index: usize, sent_message_ids: Vec<i64>) -> Task {
|
fn updated_sequence_task(task: &Task, batch_index: usize, sent_message_ids: Vec<i64>) -> Task {
|
||||||
match task {
|
let mut updated = task.clone();
|
||||||
|
match &mut updated {
|
||||||
Task::SendMediaSequence {
|
Task::SendMediaSequence {
|
||||||
chat_id,
|
batch_index: index,
|
||||||
reply_to_message_id,
|
sent_message_ids: ids,
|
||||||
caption,
|
..
|
||||||
media_batches,
|
} => {
|
||||||
batch_index: _,
|
*index = batch_index;
|
||||||
sent_message_ids: _,
|
*ids = sent_message_ids;
|
||||||
source_url,
|
}
|
||||||
edit_before_forward,
|
|
||||||
forward_channel_id,
|
|
||||||
notify_chat_id,
|
|
||||||
notify_message_id,
|
|
||||||
cache_data,
|
|
||||||
} => Task::SendMediaSequence {
|
|
||||||
chat_id: *chat_id,
|
|
||||||
reply_to_message_id: *reply_to_message_id,
|
|
||||||
caption: caption.clone(),
|
|
||||||
media_batches: media_batches.clone(),
|
|
||||||
batch_index,
|
|
||||||
sent_message_ids,
|
|
||||||
source_url: source_url.clone(),
|
|
||||||
edit_before_forward: *edit_before_forward,
|
|
||||||
forward_channel_id: *forward_channel_id,
|
|
||||||
notify_chat_id: *notify_chat_id,
|
|
||||||
notify_message_id: *notify_message_id,
|
|
||||||
cache_data: cache_data.clone(),
|
|
||||||
},
|
|
||||||
_ => unreachable!("updated_sequence_task requires a SendMediaSequence task"),
|
_ => unreachable!("updated_sequence_task requires a SendMediaSequence task"),
|
||||||
}
|
}
|
||||||
|
updated
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Sends the media batches starting at `task.batch_index`, extending
|
/// Sends the media batches starting at `task.batch_index`, extending
|
||||||
@@ -957,7 +937,7 @@ pub async fn send_media_sequence(
|
|||||||
Err(message) => {
|
Err(message) => {
|
||||||
return Err(SendError::Permanent {
|
return Err(SendError::Permanent {
|
||||||
message,
|
message,
|
||||||
task: updated_sequence_task(task, idx, sent),
|
task: Box::new(updated_sequence_task(task, idx, sent)),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
@@ -1004,6 +984,7 @@ pub async fn send_media_sequence(
|
|||||||
return Err(classify_to_send_error(
|
return Err(classify_to_send_error(
|
||||||
&e,
|
&e,
|
||||||
updated_sequence_task(task, idx, sent),
|
updated_sequence_task(task, idx, sent),
|
||||||
|
"media fetch failed",
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1060,7 +1041,7 @@ pub async fn send_animation(sender: &dyn MediaSender, task: &Task) -> Result<Vec
|
|||||||
Err(message) => {
|
Err(message) => {
|
||||||
return Err(SendError::Permanent {
|
return Err(SendError::Permanent {
|
||||||
message,
|
message,
|
||||||
task: task.clone(),
|
task: Box::new(task.clone()),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
@@ -1105,13 +1086,21 @@ pub async fn send_animation(sender: &dyn MediaSender, task: &Task) -> Result<Vec
|
|||||||
cache_animation_send(task, &message).await;
|
cache_animation_send(task, &message).await;
|
||||||
Ok(vec![id])
|
Ok(vec![id])
|
||||||
}
|
}
|
||||||
Err(e) => Err(classify_to_send_error(&e, task.clone())),
|
Err(e) => Err(classify_to_send_error(
|
||||||
|
&e,
|
||||||
|
task.clone(),
|
||||||
|
"media fetch failed",
|
||||||
|
)),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Err(e) => Err(SendError::from_fallback(e, task.clone())),
|
Err(e) => Err(SendError::from_fallback(e, task.clone())),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Err(e) => Err(classify_to_send_error(&e, task.clone())),
|
Err(e) => Err(classify_to_send_error(
|
||||||
|
&e,
|
||||||
|
task.clone(),
|
||||||
|
"media fetch failed",
|
||||||
|
)),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1148,7 +1137,11 @@ pub async fn forward_messages(sender: &dyn MediaSender, task: &Task) -> Result<(
|
|||||||
);
|
);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
Err(e) => Err(classify_to_send_error(&e, task.clone())),
|
Err(e) => Err(classify_to_send_error(
|
||||||
|
&e,
|
||||||
|
task.clone(),
|
||||||
|
"media fetch failed",
|
||||||
|
)),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1321,18 +1314,6 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
let bot = BOT.clone();
|
let bot = BOT.clone();
|
||||||
// A resumed multi-batch send already ran post_send_actions (edit prompt /
|
|
||||||
// forward) when it first started; running them again on the resume would
|
|
||||||
// open a duplicate edit prompt and double-forward. SendAnimation is
|
|
||||||
// atomic (always a fresh run), so only SendMediaSequence can resume.
|
|
||||||
let resumed = matches!(
|
|
||||||
&task,
|
|
||||||
Task::SendMediaSequence {
|
|
||||||
batch_index,
|
|
||||||
sent_message_ids,
|
|
||||||
..
|
|
||||||
} if *batch_index > 0 || !sent_message_ids.is_empty()
|
|
||||||
);
|
|
||||||
match task {
|
match task {
|
||||||
Task::SendMediaSequence { .. } | Task::SendAnimation { .. } => {
|
Task::SendMediaSequence { .. } | Task::SendAnimation { .. } => {
|
||||||
let message_ids = match send_media_or_animation(&bot, &task).await {
|
let message_ids = match send_media_or_animation(&bot, &task).await {
|
||||||
@@ -1356,9 +1337,14 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
if !resumed {
|
// A task only reaches the queue after a failed send, so this
|
||||||
post_send_actions(&bot, &task, message_ids).await;
|
// successful run is the first time post_send_actions can fire —
|
||||||
}
|
// the fresh attempt failed before it ever got here. Run it
|
||||||
|
// unconditionally: `post_send_actions` executes once, after the
|
||||||
|
// whole sequence (every batch) completed, so the channel forward
|
||||||
|
// and the edit-before-forward prompt must not be lost just
|
||||||
|
// because the send needed a retry.
|
||||||
|
post_send_actions(&bot, &task, message_ids).await;
|
||||||
release_keep_alive(&task);
|
release_keep_alive(&task);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,12 +1,13 @@
|
|||||||
//! Per-chat state with SQLite persistence (table `chat_state` in
|
//! Per-chat state with SQLite persistence (table `chat_state` in
|
||||||
//! `data/task_queue.db`, shared with the task queue).
|
//! `data/task_queue.db`, shared with the task queue).
|
||||||
|
|
||||||
|
use crate::db::unix_now;
|
||||||
use parking_lot::Mutex;
|
use parking_lot::Mutex;
|
||||||
use rusqlite::params;
|
use rusqlite::params;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
use std::time::Duration;
|
||||||
|
|
||||||
#[derive(Serialize, Deserialize, Default, Clone, Debug)]
|
#[derive(Serialize, Deserialize, Default, Clone, Debug)]
|
||||||
pub struct ChatData {
|
pub struct ChatData {
|
||||||
@@ -40,13 +41,6 @@ pub struct ChatStore {
|
|||||||
pool: Arc<crate::db::DbPool>,
|
pool: Arc<crate::db::DbPool>,
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn unix_now() -> i64 {
|
|
||||||
SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.map(|d| d.as_secs() as i64)
|
|
||||||
.unwrap_or(0)
|
|
||||||
}
|
|
||||||
|
|
||||||
impl ChatStore {
|
impl ChatStore {
|
||||||
/// Wraps the shared DB pool (schema initialized once by
|
/// Wraps the shared DB pool (schema initialized once by
|
||||||
/// [`crate::db::open_store`]; the `chat_state` table lives in the merged
|
/// [`crate::db::open_store`]; the `chat_state` table lives in the merged
|
||||||
|
|||||||
Reference in New Issue
Block a user