mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-09-23 23:32:05 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3f6a0f034a
|
||
|
|
90a011e978
|
||
|
|
12a065846c
|
||
|
|
dca1eff1c9
|
||
|
|
894a9ebf4a
|
||
|
|
c968891ff6
|
||
|
|
0087bd01ac
|
||
|
|
4cb40909c5
|
||
|
|
f260f41755
|
||
|
|
af901caddb
|
||
|
|
c0af42b1cc
|
||
|
|
d0810217b4
|
||
|
|
e6ba178983
|
||
|
|
ae69d72930
|
||
|
|
1e30815a10
|
||
|
|
50206a9056
|
||
|
|
c9e72fda70
|
||
|
|
f6845b1b5c
|
||
|
|
fae8dc6f2d
|
||
|
|
ac72e414c3
|
||
|
|
8b3b2a246b
|
||
|
|
6f6898c245
|
||
|
|
69698992d5
|
||
|
|
1e77bb0478
|
||
|
|
8f2b0a1dcb
|
||
|
|
b65fb967c4
|
||
|
|
a8fd685777
|
||
|
|
5679a8c172
|
||
|
|
bf4e6159b3
|
||
|
|
5e23916b40
|
||
|
|
7ca8fd1da2
|
||
|
|
96c11becb9
|
||
|
|
5830a3f013
|
||
|
|
183bb7e435
|
||
|
|
2a8433a8d2
|
||
|
|
6b3e61881d
|
||
|
|
47935dd7c6
|
||
|
|
6911e9146e
|
||
|
|
aa705aef90
|
||
|
|
39260a8817
|
||
|
|
6b9640aa48
|
||
|
|
ad59f518ff
|
||
|
|
e21643063e
|
||
|
|
246fc989f0
|
||
|
|
2e2d1b3506
|
||
|
|
1747d321d8
|
||
|
|
505990e49e
|
||
|
|
95b475ff08
|
||
|
|
2297fdc91c
|
||
|
|
4580b79d4f
|
||
|
|
edb32c23b4
|
||
|
|
4a467641aa
|
||
|
|
bd43a12dee
|
||
|
|
c496e41c55
|
||
|
|
68f026c990
|
||
|
|
62d80c8905
|
||
|
|
78c9c841c6
|
||
|
|
e49d500d23
|
||
|
|
6feabd723b
|
@@ -0,0 +1,71 @@
|
||||
name: CI
|
||||
|
||||
# Test/lint gate (offline, no secrets) on every push/PR, plus a live-network
|
||||
# job that exercises the real source sites and the token-gated pixiv tests.
|
||||
#
|
||||
# Layering:
|
||||
# test — fmt + clippy + the full offline unit suite + cargo-audit
|
||||
# dependency gate. Runs on every push and PR, including forks
|
||||
# (it needs no secrets).
|
||||
# live — the #[ignore]d live-network tests plus the pixiv tests that are
|
||||
# gated on PIXIV_REFRESH_TOKEN. Runs on schedule / manual dispatch
|
||||
# / tag pushes only, because pull requests from forks cannot read
|
||||
# repository secrets. continue-on-error keeps a flaky external site
|
||||
# from blocking, while the run still records the outcome.
|
||||
#
|
||||
# Test gating convention (keep in sync with AGENTS.md "Testing & QA"):
|
||||
# - pure unit tests: plain #[test] / #[tokio::test], always run.
|
||||
# - live-network tests: #[ignore = "live network: ..."], only run here.
|
||||
# - token-gated tests (pixiv): #[tokio::test] with an early return when
|
||||
# PIXIV_REFRESH_TOKEN is absent or empty (empty = unset CI secret).
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
pull_request:
|
||||
schedule:
|
||||
# Weekly probe of the live endpoints, so external API changes surface.
|
||||
- cron: '0 3 * * 1'
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
components: clippy, rustfmt
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
- name: Check formatting
|
||||
run: cargo fmt --check
|
||||
- name: Lint (deny warnings)
|
||||
run: cargo clippy --workspace --all-targets -- -D warnings
|
||||
- name: Run offline tests
|
||||
run: cargo test --workspace
|
||||
# Dependency vulnerability gate: fails the build when a crate in
|
||||
# Cargo.lock has an unfixed security advisory. Unmaintained/unsound
|
||||
# *warnings* (dotenv, proc-macro-error2, anyhow transitive) do not fail
|
||||
# the build by default; the advisory DB is cached across runs.
|
||||
- name: Audit dependencies
|
||||
uses: actions-rust-lang/audit@v1
|
||||
|
||||
live:
|
||||
needs: test
|
||||
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/v')
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
env:
|
||||
PIXIV_REFRESH_TOKEN: ${{ secrets.PIXIV_REFRESH_TOKEN }}
|
||||
TWITTER_AUTH_TOKEN: ${{ secrets.TWITTER_AUTH_TOKEN }}
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
# Full suite: with the secret present, the pixiv token-gated tests run;
|
||||
# without it they skip themselves. Live tests stay #[ignore]d here.
|
||||
- name: Run token-gated tests
|
||||
run: cargo test --workspace
|
||||
# The live-network tests, by the "live" name filter (all #[ignore]d).
|
||||
- name: Run live-network tests
|
||||
run: cargo test --workspace -- --ignored live
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Project Overview
|
||||
|
||||
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README and user-facing strings are in Chinese. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
|
||||
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README is in Chinese; user-facing bot strings are in English. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
|
||||
|
||||
Two-crate Cargo workspace (both v1.0.3, edition 2024, resolver 3):
|
||||
Two-crate Cargo workspace (both v1.4.0, edition 2024, resolver 3):
|
||||
|
||||
- **`crates/x-media`** — library that fetches and normalizes media from the three sites. Pure, no Telegram knowledge.
|
||||
- **`crates/xmedia-bot`** — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.
|
||||
@@ -18,23 +18,28 @@ Telegram update → Dispatcher (polling or axum webhook) → dptree branches
|
||||
└─ callback_query → "forward" (copy to channel) / "template|<name>" (apply caption template)
|
||||
```
|
||||
|
||||
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)` → `Fetched` → builds a `Task` → `send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → single worker leases (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
|
||||
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)` → `Fetched` → builds a `Task` → `send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
|
||||
|
||||
The `x-media` library: `site::fetch(url)` dispatches (in order) twitter → bsky → pixiv via per-site regex `PATTERN` and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
|
||||
Debug command: `/test <url>` runs the same `x_media::site::fetch` and replies with `test_parse_report` (`handlers/commands.rs`) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars and sent with HTML parse mode: raw fields are escaped, and the caption is wrapped in a `<blockquote>` so it renders exactly like the sent media caption (escaped text and links included). It uses a custom `parse_test_arg` parser (whole remainder, trimmed) because teloxide's built-in `split` parser takes exactly one space-separated token.
|
||||
|
||||
The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → pixiv) and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, site_id, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
|
||||
|
||||
## Key Directories
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
|
||||
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, site struct, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`) |
|
||||
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, queue worker start, pixiv validation, 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
|
||||
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`>` `<` `&` `'`) — so the stored text is raw and the caption escapes exactly once |
|
||||
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
|
||||
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
|
||||
| `crates/xmedia-bot/src/handlers.rs` | `Command` enum (teloxide `BotCommands`), message/inline/callback handlers, URL extraction, global statics; per-URL work spawned with a `Semaphore(8)` cap (teloxide's per-chat workers are sequential — batch-forwards need concurrency) |
|
||||
| `crates/xmedia-bot/src/db.rs` | `DbPool`: per-store SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) over `$DATA_DIR/task_queue.db` (default `data/`); `open_store` creates file + schema, `with_conn` runs all rusqlite I/O in `spawn_blocking` |
|
||||
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. the `/test <url>` parse-only debug command), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons), `statics.rs` (global statics) |
|
||||
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table) |
|
||||
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure |
|
||||
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections |
|
||||
| `crates/xmedia-bot/src/send.rs` | Media senders, upload fallback, error classification, queue task handlers |
|
||||
| `crates/xmedia-bot/src/media_sender.rs` | `MediaSender` trait: the send surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`send_chat_action`) implemented by teloxide `Bot` (per-chat rate-limited) and by a scripted `MockSender` in tests |
|
||||
| `crates/xmedia-bot/src/rate_limit.rs` | Per-chat token bucket (`CAPACITY = 20`, ~20 msg/min refill) paced before sends reach the API so batch forwards don't trip flood control |
|
||||
|
||||
## Development Commands
|
||||
|
||||
@@ -48,31 +53,31 @@ cargo clippy --workspace --all-targets # lint (Clippy is the configured IDE lint
|
||||
cargo fmt --check # formatting
|
||||
```
|
||||
|
||||
Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb`. Runtime requires **ffmpeg** (built into the image).
|
||||
Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb`. Runtime requires **ffmpeg** (built into the image). The builder fetches crates.io + ffmpeg; on restricted networks pass proxy build args, e.g. `--build-arg HTTP_PROXY=http://host.docker.internal:10808 --build-arg HTTPS_PROXY=…` (Docker Desktop builds can't reach the host loopback — use `host.docker.internal`).
|
||||
|
||||
## Code Conventions & Common Patterns
|
||||
|
||||
- **No anyhow/thiserror.** Errors are hand-rolled enums with manual `Display`/`source()`/`From` impls: `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `FetchError` (`Http`/`Json`/`Pixiv`/`NotFound`/`Blocked`), `PixivError`, `Classification`. New errors should follow this pattern.
|
||||
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers rebuild `Bot::from_env()`.
|
||||
- **Errors via `thiserror` derive** (no anyhow): the public, stringified errors — `FetchError` (`Http`/`Json`/`Pixiv`/`Site`/`NotFound`/`Blocked`) and `PixivError` — derive `thiserror::Error` with `#[from]` conversions; `Display`/`source()` come from the derive. The internal control-flow enums — `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `Classification`, `FallbackError` — carry no `Display` and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive `thiserror`, internal flow enums stay plain.
|
||||
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers/statics.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers share the process-wide `send::BOT` (`LazyLock<Bot>`, force-initialized in `main` so a missing token fails at startup).
|
||||
- **Async**: tokio multi-thread runtime (`#[tokio::main]` default). All rusqlite I/O inside `tokio::task::spawn_blocking`. Long loops use `tokio::select!` with `tokio::sync::{watch, Notify}` stop/wake channels. No streams.
|
||||
- **Blocking sync primitives**: `parking_lot::Mutex` for hot caches, `tokio::sync::Mutex` for async-shared state (pixiv token cache), `AtomicBool` for feature gates.
|
||||
- **Site adapter convention** (no trait, no enum dispatch — follow the existing convention): each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`; `site/mod.rs` re-exports the site struct and `fetch_once` adds one guarded if-branch. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one branch in `fetch_once`.
|
||||
- **Site adapter convention**: each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`, plus `cache_key`/`is_retryable`/`media_headers`, and a unit struct `<Name>Site` implementing `site::Site`; the central dispatcher (`site/mod.rs`) only iterates the `SITES` registry. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one `Box::new(...)` entry in `SITES` — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return `SiteFuture` (a boxed `Pin<Box<dyn Future + Send>>`) because `async fn` in traits is not dyn-compatible.
|
||||
- **Serde**: per-site `model.rs` are pure `Deserialize` DTOs mirroring API JSON; site structs in `interface.rs` have private fields, a `caption()` builder, and `impl From<SiteStruct> for Fetched`. Persisted payloads use internally-tagged enums (`#[serde(tag = "kind")]` / `type`).
|
||||
- **Naming**: module-per-concern, snake_case files, `CamelCase` types, `snake_case` fns. `//!` module docs and `///` docs on non-obvious logic (syndication token, ugoira encoding, `display_text_range`).
|
||||
- **Retries**: only `x-media::site::fetch` retries (3 attempts, `1 << attempt` backoff, HTTP errors only). Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`).
|
||||
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`).
|
||||
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`). Level convention: `info` = lifecycle + per-post business results (`sent`/`forwarded`/`copied`), admin/operator actions and anomalies (fallback, retry enqueue, dead-letter is `error`); `debug` = per-request detail (message/command/URL extraction, `fetching`/`fetched`, batch sends, queue processing, photo processing, inline queries). Full user-submitted URLs and message text only appear at `debug`; at `info` and above links are printed via the normalized cache key (`handlers::log_key`, e.g. `[key=twitter:123...]`) so logs stay short and do not echo user data.
|
||||
|
||||
## Important Files
|
||||
|
||||
| File | Why it matters |
|
||||
|---|---|
|
||||
| `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) |
|
||||
| `crates/xmedia-bot/src/handlers.rs` | `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `data/task_queue.db` **relative to CWD**); command dispatch; URL extraction; retry enqueue |
|
||||
| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `commands.rs` = command dispatch (incl. the `/test <url>` parse-only debug command); `urls.rs` = URL extraction + retry enqueue; `inline.rs`/`callback.rs` = inline queries / edit-before-forward buttons |
|
||||
| `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`; fallback chain; `classify_request_error`; download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`) |
|
||||
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL |
|
||||
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) |
|
||||
| `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` |
|
||||
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime hack, static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint |
|
||||
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime bump (cargo's freshness is mtime-based and `cargo clean -p` removes 0 files — the touch is what forces the real sources to rebuild while deps stay cached), static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, optional `FFMPEG_SHA256` checksum, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint. Runtime ships **no libssl/libcrypto/CA bundle** — rustls webpki-roots handles all TLS, and the static ffmpeg only processes local files (downloads go through reqwest) |
|
||||
| `docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim) |
|
||||
| `docker-compose.yml.example` | Deployment env reference (real `docker-compose.yml` is gitignored). Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) |
|
||||
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master; **no test step**; buildx gha cache (`cache-from`/`cache-to`, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs |
|
||||
@@ -82,18 +87,19 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
|
||||
|
||||
- **Rust, stable, edition 2024**, workspace resolver 3. No `rust-version`/MSRV pin, no `rust-toolchain.toml` — recent stable is assumed. No nightly features.
|
||||
- Package manager: **Cargo** (workspace with path dep `x-media` ← `xmedia-bot`). No `[workspace.package]`/shared deps — each crate lists deps independently.
|
||||
- **Two reqwest versions coexist in the lock** (0.12.28 via teloxide, 0.13.3 in x-media) — don't unify casually.
|
||||
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
|
||||
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `data/task_queue.db` is CWD-relative — run from the workspace root, or `/app` in Docker. Mount `./data` and `./cert` volumes.
|
||||
- **TLS is rustls end-to-end** (no native-tls/openssl in the tree, no libssl in the Docker runtime image): `teloxide` is declared `default-features = false` with `["webhooks-axum", "macros", "rustls", "ctrlc_handler"]` (the removed `default` also carried `native-tls` and `ctrlc_handler` — the latter must stay); x-media's reqwest is `default-features = false` with `["json", "rustls-tls"]` (webpki-roots baked in, so the image ships no CA bundle). One reqwest 0.12.28 in the lock.
|
||||
- **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md`, `README.en.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag (the tag push triggers the Docker Hub build).
|
||||
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `DATA_DIR` (default `data`, CWD-relative; the SQLite dir, auto-created), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
|
||||
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `$DATA_DIR/task_queue.db` (default `data/task_queue.db`, CWD-relative — run from the workspace root, or `/app` in Docker; set `DATA_DIR` to pin state anywhere). Mount `./data` and `./cert` volumes.
|
||||
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `docker-compose.yml`, `/target`, `.idea/`.
|
||||
- Docs are in Chinese; user-facing bot strings too. Keep that convention when editing captions/templates/docs.
|
||||
- Docs are in Chinese (README, AGENTS.md); user-facing bot strings are in English. Keep that split when editing user-facing strings and docs.
|
||||
|
||||
## Testing & QA
|
||||
|
||||
- **~51 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
|
||||
- **~115 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
|
||||
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches.
|
||||
- Live-network tests exist in `site/twitter/interface.rs` (3), `site/bsky/interface.rs` (2), `site/pixiv/interface.rs`/`api.rs` (env-gated on `PIXIV_REFRESH_TOKEN`/dotenv, skip by early return). Run the full suite with `cargo test --workspace`.
|
||||
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`.
|
||||
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
|
||||
- **CI runs no tests** — `.github/workflows/docker.yml` only builds/pushes the image; verification is a local responsibility.
|
||||
- **CI** — `.github/workflows/ci.yml` runs `cargo fmt --check` + `cargo clippy --workspace --all-targets -- -D warnings` + `cargo test --workspace` + a `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN` from secrets, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds/pushes the image only.
|
||||
- Untested and hard to test without a mock seam: `handlers.rs` (depends directly on teloxide `Bot`); `main.rs`, `config.rs`, `state.rs`; `media.rs`, `lib.rs`, all `model.rs`.
|
||||
- No coverage tracking, no lint gate in CI.
|
||||
- No coverage tracking.
|
||||
|
||||
Generated
+162
-237
@@ -16,7 +16,7 @@ checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cipher",
|
||||
"cpufeatures",
|
||||
"cpufeatures 0.2.17",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -242,6 +242,23 @@ version = "1.0.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
|
||||
|
||||
[[package]]
|
||||
name = "cfg_aliases"
|
||||
version = "0.2.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
|
||||
|
||||
[[package]]
|
||||
name = "chacha20"
|
||||
version = "0.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.3.0",
|
||||
"rand_core 0.10.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "chrono"
|
||||
version = "0.4.44"
|
||||
@@ -279,26 +296,6 @@ version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6"
|
||||
|
||||
[[package]]
|
||||
name = "core-foundation"
|
||||
version = "0.9.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f"
|
||||
dependencies = [
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "core-foundation"
|
||||
version = "0.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
|
||||
dependencies = [
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "core-foundation-sys"
|
||||
version = "0.8.7"
|
||||
@@ -314,6 +311,15 @@ dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.3.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crc"
|
||||
version = "3.4.0"
|
||||
@@ -495,15 +501,6 @@ version = "1.15.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
|
||||
|
||||
[[package]]
|
||||
name = "encoding_rs"
|
||||
version = "0.8.35"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "env_logger"
|
||||
version = "0.10.2"
|
||||
@@ -598,33 +595,12 @@ dependencies = [
|
||||
"miniz_oxide",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fnv"
|
||||
version = "1.0.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
|
||||
|
||||
[[package]]
|
||||
name = "foldhash"
|
||||
version = "0.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
|
||||
|
||||
[[package]]
|
||||
name = "foreign-types"
|
||||
version = "0.3.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1"
|
||||
dependencies = [
|
||||
"foreign-types-shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "foreign-types-shared"
|
||||
version = "0.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b"
|
||||
|
||||
[[package]]
|
||||
name = "form_urlencoded"
|
||||
version = "1.2.2"
|
||||
@@ -739,8 +715,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"js-sys",
|
||||
"libc",
|
||||
"wasi",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -764,29 +742,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"js-sys",
|
||||
"libc",
|
||||
"r-efi 6.0.0",
|
||||
"rand_core 0.10.1",
|
||||
"wasip2",
|
||||
"wasip3",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "h2"
|
||||
version = "0.4.14"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "171fefbc92fe4a4de27e0698d6a5b392d6a0e333506bc49133760b3bcf948733"
|
||||
dependencies = [
|
||||
"atomic-waker",
|
||||
"bytes",
|
||||
"fnv",
|
||||
"futures-core",
|
||||
"futures-sink",
|
||||
"http",
|
||||
"indexmap 2.14.0",
|
||||
"slab",
|
||||
"tokio",
|
||||
"tokio-util",
|
||||
"tracing",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -925,7 +887,6 @@ dependencies = [
|
||||
"bytes",
|
||||
"futures-channel",
|
||||
"futures-core",
|
||||
"h2",
|
||||
"http",
|
||||
"http-body",
|
||||
"httparse",
|
||||
@@ -950,22 +911,7 @@ dependencies = [
|
||||
"tokio",
|
||||
"tokio-rustls",
|
||||
"tower-service",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hyper-tls"
|
||||
version = "0.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "70206fc6890eaca9fde8a0bf71caa2ddfc9fe045ac9e5c70df101a7dbde866e0"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"http-body-util",
|
||||
"hyper",
|
||||
"hyper-util",
|
||||
"native-tls",
|
||||
"tokio",
|
||||
"tokio-native-tls",
|
||||
"tower-service",
|
||||
"webpki-roots",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -986,11 +932,9 @@ dependencies = [
|
||||
"percent-encoding",
|
||||
"pin-project-lite",
|
||||
"socket2",
|
||||
"system-configuration",
|
||||
"tokio",
|
||||
"tower-service",
|
||||
"tracing",
|
||||
"windows-registry",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1299,6 +1243,12 @@ version = "0.4.29"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897"
|
||||
|
||||
[[package]]
|
||||
name = "lru-slab"
|
||||
version = "0.1.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
|
||||
|
||||
[[package]]
|
||||
name = "lzma-rs"
|
||||
version = "0.3.0"
|
||||
@@ -1369,23 +1319,6 @@ dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "native-tls"
|
||||
version = "0.2.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "465500e14ea162429d264d44189adc38b199b62b1c21eea9f69e4b73cb03bbf2"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"log",
|
||||
"openssl",
|
||||
"openssl-probe",
|
||||
"openssl-sys",
|
||||
"schannel",
|
||||
"security-framework",
|
||||
"security-framework-sys",
|
||||
"tempfile",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-conv"
|
||||
version = "0.2.1"
|
||||
@@ -1416,49 +1349,6 @@ version = "1.21.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
|
||||
|
||||
[[package]]
|
||||
name = "openssl"
|
||||
version = "0.10.79"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "bf0b434746ee2832f4f0baf10137e1cabb18cbe6912c69e2e33263c45250f542"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"cfg-if",
|
||||
"foreign-types",
|
||||
"libc",
|
||||
"openssl-macros",
|
||||
"openssl-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "openssl-macros"
|
||||
version = "0.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "openssl-probe"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
|
||||
|
||||
[[package]]
|
||||
name = "openssl-sys"
|
||||
version = "0.9.115"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "158fe5b292746440aa6e7a7e690e55aeb72d41505e2804c23c6973ad0e9c9781"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"libc",
|
||||
"pkg-config",
|
||||
"vcpkg",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "parking_lot"
|
||||
version = "0.12.5"
|
||||
@@ -1627,6 +1517,62 @@ dependencies = [
|
||||
"cc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quinn"
|
||||
version = "0.11.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"cfg_aliases",
|
||||
"pin-project-lite",
|
||||
"quinn-proto",
|
||||
"quinn-udp",
|
||||
"rustc-hash",
|
||||
"rustls",
|
||||
"socket2",
|
||||
"thiserror",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"web-time",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quinn-proto"
|
||||
version = "0.11.16"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"getrandom 0.4.2",
|
||||
"lru-slab",
|
||||
"rand 0.10.2",
|
||||
"rand_pcg",
|
||||
"ring",
|
||||
"rustc-hash",
|
||||
"rustls",
|
||||
"rustls-pki-types",
|
||||
"slab",
|
||||
"thiserror",
|
||||
"tinyvec",
|
||||
"tracing",
|
||||
"web-time",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quinn-udp"
|
||||
version = "0.5.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
|
||||
dependencies = [
|
||||
"cfg_aliases",
|
||||
"libc",
|
||||
"once_cell",
|
||||
"socket2",
|
||||
"tracing",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.45"
|
||||
@@ -1656,7 +1602,18 @@ checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"rand_chacha",
|
||||
"rand_core",
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand"
|
||||
version = "0.10.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
|
||||
dependencies = [
|
||||
"chacha20",
|
||||
"getrandom 0.4.2",
|
||||
"rand_core 0.10.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1666,7 +1623,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
|
||||
dependencies = [
|
||||
"ppv-lite86",
|
||||
"rand_core",
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1678,6 +1635,21 @@ dependencies = [
|
||||
"getrandom 0.2.17",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_core"
|
||||
version = "0.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
|
||||
|
||||
[[package]]
|
||||
name = "rand_pcg"
|
||||
version = "0.10.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a"
|
||||
dependencies = [
|
||||
"rand_core 0.10.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rc-box"
|
||||
version = "1.3.0"
|
||||
@@ -1753,31 +1725,28 @@ checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
|
||||
dependencies = [
|
||||
"base64",
|
||||
"bytes",
|
||||
"encoding_rs",
|
||||
"futures-core",
|
||||
"futures-util",
|
||||
"h2",
|
||||
"http",
|
||||
"http-body",
|
||||
"http-body-util",
|
||||
"hyper",
|
||||
"hyper-rustls",
|
||||
"hyper-tls",
|
||||
"hyper-util",
|
||||
"js-sys",
|
||||
"log",
|
||||
"mime",
|
||||
"mime_guess",
|
||||
"native-tls",
|
||||
"percent-encoding",
|
||||
"pin-project-lite",
|
||||
"quinn",
|
||||
"rustls",
|
||||
"rustls-pki-types",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"serde_urlencoded",
|
||||
"sync_wrapper",
|
||||
"tokio",
|
||||
"tokio-native-tls",
|
||||
"tokio-rustls",
|
||||
"tokio-util",
|
||||
"tower",
|
||||
"tower-http",
|
||||
@@ -1787,6 +1756,7 @@ dependencies = [
|
||||
"wasm-bindgen-futures",
|
||||
"wasm-streams",
|
||||
"web-sys",
|
||||
"webpki-roots",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1826,6 +1796,12 @@ dependencies = [
|
||||
"smallvec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustc-hash"
|
||||
version = "2.1.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
|
||||
|
||||
[[package]]
|
||||
name = "rustix"
|
||||
version = "1.1.4"
|
||||
@@ -1846,6 +1822,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b"
|
||||
dependencies = [
|
||||
"once_cell",
|
||||
"ring",
|
||||
"rustls-pki-types",
|
||||
"rustls-webpki",
|
||||
"subtle",
|
||||
@@ -1858,6 +1835,7 @@ version = "1.14.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9"
|
||||
dependencies = [
|
||||
"web-time",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
@@ -1884,15 +1862,6 @@ version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
|
||||
|
||||
[[package]]
|
||||
name = "schannel"
|
||||
version = "0.1.29"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
|
||||
dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "schemars"
|
||||
version = "0.9.0"
|
||||
@@ -1923,29 +1892,6 @@ version = "1.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
|
||||
|
||||
[[package]]
|
||||
name = "security-framework"
|
||||
version = "3.7.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"core-foundation 0.10.1",
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
"security-framework-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "security-framework-sys"
|
||||
version = "2.17.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
|
||||
dependencies = [
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "semver"
|
||||
version = "1.0.28"
|
||||
@@ -2057,7 +2003,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures",
|
||||
"cpufeatures 0.2.17",
|
||||
"digest",
|
||||
]
|
||||
|
||||
@@ -2167,27 +2113,6 @@ dependencies = [
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "system-configuration"
|
||||
version = "0.7.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"core-foundation 0.9.4",
|
||||
"system-configuration-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "system-configuration-sys"
|
||||
version = "0.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4"
|
||||
dependencies = [
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "take_mut"
|
||||
version = "0.2.2"
|
||||
@@ -2216,7 +2141,7 @@ dependencies = [
|
||||
"log",
|
||||
"mime",
|
||||
"pin-project",
|
||||
"rand",
|
||||
"rand 0.8.6",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"teloxide-core",
|
||||
@@ -2399,16 +2324,6 @@ dependencies = [
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tokio-native-tls"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2"
|
||||
dependencies = [
|
||||
"native-tls",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tokio-rustls"
|
||||
version = "0.26.4"
|
||||
@@ -2739,6 +2654,25 @@ dependencies = [
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "web-time"
|
||||
version = "1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
|
||||
dependencies = [
|
||||
"js-sys",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "webpki-roots"
|
||||
version = "1.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed"
|
||||
dependencies = [
|
||||
"rustls-pki-types",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "winapi-util"
|
||||
version = "0.1.11"
|
||||
@@ -2789,17 +2723,6 @@ version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
||||
|
||||
[[package]]
|
||||
name = "windows-registry"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720"
|
||||
dependencies = [
|
||||
"windows-link",
|
||||
"windows-result",
|
||||
"windows-strings",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-result"
|
||||
version = "0.4.1"
|
||||
@@ -3002,18 +2925,19 @@ checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
|
||||
|
||||
[[package]]
|
||||
name = "x-media"
|
||||
version = "1.1.0"
|
||||
version = "1.4.0"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"dotenv",
|
||||
"html-escape",
|
||||
"log",
|
||||
"rand",
|
||||
"rand 0.8.6",
|
||||
"regex",
|
||||
"reqwest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tempfile",
|
||||
"thiserror",
|
||||
"tokio",
|
||||
"url",
|
||||
"zip",
|
||||
@@ -3021,8 +2945,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "xmedia-bot"
|
||||
version = "1.1.0"
|
||||
version = "1.4.0"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"dotenv",
|
||||
"fast_image_resize",
|
||||
"html-escape",
|
||||
@@ -3031,7 +2956,7 @@ dependencies = [
|
||||
"parking_lot",
|
||||
"png",
|
||||
"pretty_env_logger",
|
||||
"rand",
|
||||
"rand 0.8.6",
|
||||
"rusqlite",
|
||||
"serde",
|
||||
"serde_json",
|
||||
|
||||
+6
-1
@@ -2,6 +2,11 @@
|
||||
members = ["crates/x-media", "crates/xmedia-bot"]
|
||||
resolver = "3"
|
||||
|
||||
# Smaller production binary; debug symbols are not shipped anyway.
|
||||
# Smaller/faster production binary: strip debug symbols, link-time
|
||||
# optimization across crates, and one codegen unit per crate (bigger LTO
|
||||
# wins). panic=abort is intentionally NOT set: queue workers and db
|
||||
# closures rely on JoinHandle catching panics, which abort would defeat.
|
||||
[profile.release]
|
||||
strip = true
|
||||
lto = "thin"
|
||||
codegen-units = 1
|
||||
|
||||
+12
-9
@@ -42,13 +42,15 @@ RUN wget -q -O /tmp/ffmpeg.zip "$FFMPEG_URL" \
|
||||
&& rm /tmp/ffmpeg.zip \
|
||||
&& /usr/local/bin/ffmpeg -version >/dev/null
|
||||
|
||||
# 3. Real sources last: only our crates recompile on source changes.
|
||||
# `cargo clean -p` drops the two crates' artifacts while keeping the
|
||||
# compiled dependency layer, forcing a deterministic rebuild of the real
|
||||
# sources. (The previous `touch`-mtimes hack silently shipped the stub
|
||||
# binary when host files carried future timestamps.)
|
||||
# 3. Real sources last: only our crates recompile on source changes. Cargo's
|
||||
# freshness check is mtime-based; the COPY'd host files usually predate the
|
||||
# step-1 stub build, so cargo would consider the stub up to date and never
|
||||
# compile the real sources. `touch` makes every .rs newer than the stub
|
||||
# artifacts, forcing a rebuild of just the two crates while the compiled
|
||||
# dependency layer stays cached. (`cargo clean -p` does NOT work here — it
|
||||
# removes 0 files and the stub binary silently ships.)
|
||||
COPY crates/ ./crates/
|
||||
RUN cargo clean -p xmedia-bot -p x-media \
|
||||
RUN find crates -type f -name '*.rs' -exec touch {} + \
|
||||
&& cargo build --release -p xmedia-bot
|
||||
|
||||
# ---------- runtime stage ----------
|
||||
@@ -62,9 +64,10 @@ LABEL org.opencontainers.image.title="${APP_NAME}"
|
||||
|
||||
# Everything is copied in — no apt in the runtime stage. Privilege dropping is
|
||||
# done by docker-entrypoint.sh with setpriv (util-linux, already in
|
||||
# bookworm-slim), so no gosu needed. (libssl3/libcrypto are already in
|
||||
# bookworm-slim; only ca-certificates and ffmpeg need copying.)
|
||||
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
|
||||
# bookworm-slim), so no gosu needed. TLS is rustls (webpki-roots baked in,
|
||||
# see Cargo.toml feature `rustls`/`rustls-tls`), so no system CA bundle or
|
||||
# libssl are needed; the static ffmpeg only processes local files (all
|
||||
# downloads go through reqwest).
|
||||
COPY --from=builder /usr/local/bin/ffmpeg /usr/local/bin/ffmpeg
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
+125
@@ -0,0 +1,125 @@
|
||||
# TelegramXMediaBot
|
||||
|
||||
A Telegram bot that turns post links from X / Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags.
|
||||
|
||||
## Features
|
||||
|
||||
- Sending a link in a private chat fetches and sends the images, videos and GIFs automatically; oversized media is split into batches
|
||||
- Text-only posts report "no media"; unsupported links are silently ignored
|
||||
- Inline queries (`@bot <link>`)
|
||||
- Bind a forward channel for automatic forwarding; edit the caption before forwarding and apply custom templates
|
||||
- Failed sends are retried automatically with persistence; the user is notified after retries are exhausted
|
||||
- Pixiv ugoira animations are transcoded to MP4; Bluesky videos are remuxed (HLS stream → MP4)
|
||||
- Photos exceeding Telegram's size/dimension limits are compressed automatically (original format kept, JPEG fallback only when needed)
|
||||
- Link-result cache: after a successful send the Telegram file ids and caption fields are cached locally, so a repeated link is re-sent from local state — no source-site request, no media file stored (expiry controlled by `LINK_CACHE_TTL_SECONDS`, default 7 days)
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Required: BotFather token; optional: PIXIV_REFRESH_TOKEN (Pixiv is disabled without it)
|
||||
export TELOXIDE_TOKEN=<token>
|
||||
export PIXIV_REFRESH_TOKEN=<token>
|
||||
|
||||
cargo run -p xmedia-bot
|
||||
```
|
||||
|
||||
Docker deployment (see `docker-compose.yml.example`):
|
||||
|
||||
```bash
|
||||
docker build -t tgxmb .
|
||||
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
|
||||
```
|
||||
|
||||
Environment variables: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `BOT_ADMIN`, `EDIT_MESSAGE_TTL_SECONDS`, `LINK_CACHE_TTL_SECONDS`, `RUST_LOG`, `TELOXIDE_PROXY`, `WEBHOOK*`, `TWITTER_AUTH_TOKEN` (optional).
|
||||
|
||||
NSFW tweets: the public syndication endpoint does not return sensitive content. Setting `TWITTER_AUTH_TOKEN` (the `auth_token` cookie value of a logged-in x.com session) lets the bot fetch NSFW media in the logged-in state only when it hits a withheld tweet; without it, the bot reports no media.
|
||||
|
||||
### Webhook deployment (needs a reverse proxy)
|
||||
|
||||
`docker-compose.yml.example` ships an [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) reverse-proxy orchestration. Pick one deployment shape:
|
||||
|
||||
**With a domain**
|
||||
1. Point a DNS A record at the server
|
||||
2. In compose set `VIRTUAL_HOST` and `WEBHOOK_URL` to the domain, and uncomment `ACME_HOST` (set it to the domain)
|
||||
3. acme-companion issues and renews certificates automatically — nothing manual
|
||||
|
||||
**IP only**
|
||||
Let's Encrypt can issue certificates for public IPs (available since 2026, validity ~7 days, requires the `shortlived` profile). Use [acme.sh](https://github.com/acmesh-official/acme.sh) to issue and renew automatically, no manual certificates:
|
||||
|
||||
1. Add an acme-ip service to compose (issue + daily auto-renewal check):
|
||||
```yaml
|
||||
acme-ip:
|
||||
image: neilpang/acme.sh
|
||||
container_name: acme-ip
|
||||
command: daemon
|
||||
restart: always
|
||||
volumes:
|
||||
- certs:/acme.sh
|
||||
- html:/usr/share/nginx/html
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
networks: [proxy]
|
||||
```
|
||||
2. First issuance (replace `<SERVER_IP>` with the server's public IP; IPv6 works too, repeat `-d` for more):
|
||||
```bash
|
||||
docker compose exec acme-ip acme.sh --issue --server letsencrypt \
|
||||
-d <SERVER_IP> --cert-profile shortlived --days 3 \
|
||||
--webroot /usr/share/nginx/html \
|
||||
--install-cert --cert-file /acme.sh/<SERVER_IP>.crt \
|
||||
--key-file /acme.sh/<SERVER_IP>.key \
|
||||
--reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP"
|
||||
```
|
||||
3. In compose set `VIRTUAL_HOST: '<SERVER_IP>'` and `WEBHOOK_URL: 'https://<SERVER_IP>/'`; no `WEBHOOK_CERT` needed. Renewal is handled by the acme.sh daemon (`--days 3` = renew every 3 days, buffer against the 7-day validity), and a successful renewal HUP-notifies nginx-proxy to load the new certificate.
|
||||
|
||||
Limitations: certificate validity ~7 days; only http-01/tls-alpn-01 validation (port 80 must be publicly reachable); no DNS-01, private IPs or IP ranges; at most 5 certificates per 168 hours for the same IP set. It is recommended to trial-issue with `--server letsencrypt_test` first, then switch to the production server.
|
||||
|
||||
Telegram only accepts ports 443/80/88/8443.
|
||||
|
||||
<details>
|
||||
<summary>Environment variables</summary>
|
||||
|
||||
| Variable | Description |
|
||||
|---|---|
|
||||
| `TELOXIDE_TOKEN` | Bot token (required) |
|
||||
| `PIXIV_REFRESH_TOKEN` | Pixiv refresh token; Pixiv is disabled without it |
|
||||
| `BOT_ADMIN` | Admin chat IDs, comma-separated; receives start/stop notifications |
|
||||
| `EDIT_MESSAGE_TTL_SECONDS` | Edit-before-forward record expiry in seconds, default 86400 |
|
||||
| `LINK_CACHE_TTL_SECONDS` | Link-result cache expiry in seconds, default 604800 (7 days) |
|
||||
| `DATA_DIR` | Data directory (where the SQLite `task_queue.db` lives), default `data` (relative to the working directory, created automatically) |
|
||||
| `RUST_LOG` | Log level |
|
||||
| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW) |
|
||||
| `LOCAL_USER_ID` | UID the container runs as, default 9001 |
|
||||
| `VIRTUAL_HOST` | Public domain or IP; nginx-proxy routes by this |
|
||||
| `VIRTUAL_PORT` | Port the bot listens on inside the container; nginx-proxy's forwarding target |
|
||||
| `ACME_HOST` | Domain deployment: when set to the domain, acme-companion issues/renews certificates automatically |
|
||||
| `DEFAULT_HOST` | nginx-proxy routes requests with unknown Host headers to this vhost (needed for IP access) |
|
||||
| `DEFAULT_EMAIL` | acme-companion certificate notification email |
|
||||
| `WEBHOOK` | `true` enables webhook mode (polling by default) |
|
||||
| `WEBHOOK_LISTEN` / `WEBHOOK_PORT` | Listen address/port inside the bot container |
|
||||
| `WEBHOOK_URL` | Public HTTPS URL (`https://domain/` or `https://IP/`) |
|
||||
| `WEBHOOK_CERT` | Optional; self-signed certificate path, only used for Telegram-side validation (TLS is terminated by the reverse proxy) |
|
||||
| `WEBHOOK_SECRET_TOKEN` | Update validation token (`X-Telegram-Bot-Api-Secret-Token`) |
|
||||
|
||||
</details>
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---|---|
|
||||
| `/start` | Welcome message |
|
||||
| `/help` | List all commands and usage (this command table) |
|
||||
| `/set_forward_channel <channel>` | Set the forward channel: `@channel` or channel ID; media messages are forwarded to it automatically afterwards |
|
||||
| `/remove_forward_channel` | Remove the forward channel |
|
||||
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or taps a template button to apply one) |
|
||||
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward") |
|
||||
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` |
|
||||
| `/clear_cache [link]` | Clear the link cache (admin only); with a link only that entry, otherwise everything |
|
||||
| `/bot_dict` | Show the current chat state (debugging) |
|
||||
| `/test <link>` | Debug: parse a link and report the parse result only (site, title, author, tags, media list) — no media is sent |
|
||||
|
||||
Link processing works only in private chats; commands work in any chat.
|
||||
|
||||
## Notes
|
||||
|
||||
- State is persisted in `data/task_queue.db`; compose deployments use the bind mount `./data` (keep it a directory for easy backups)
|
||||
- The runtime needs ffmpeg (built into the Docker image)
|
||||
- Tests: `cargo test --workspace`
|
||||
@@ -9,7 +9,8 @@ Telegram 机器人,将 X / Twitter、Pixiv、Bluesky 的帖子链接转换为
|
||||
- 支持内联查询(`@机器人 <链接>`)
|
||||
- 可绑定转发频道自动转发;支持转发前编辑 caption 与自定义模板
|
||||
- 发送失败自动重试并持久化,重试耗尽后通知用户
|
||||
- Pixiv ugoira 动图自动转码为 MP4
|
||||
- Pixiv ugoira 动图自动转码为 MP4;Bluesky 视频自动转码(HLS 流 → MP4)
|
||||
- 超过 Telegram 尺寸/大小限制的图片自动压缩(保持原格式,必要时转 JPEG)
|
||||
- 链接结果本地缓存:成功发送后缓存 Telegram file id 与 caption 等,再次收到相同链接直接本地重发,不再请求源站、不保存媒体文件(`LINK_CACHE_TTL_SECONDS` 控制过期,默认 7 天)
|
||||
|
||||
## 快速开始
|
||||
@@ -29,7 +30,7 @@ docker build -t tgxmb .
|
||||
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
|
||||
```
|
||||
|
||||
环境变量:`TELOXIDE_TOKEN`(必填)、`PIXIV_REFRESH_TOKEN`、`BOT_ADMIN`、`EDIT_MESSAGE_TTL_SECONDS`、`LINK_CACHE_TTL_SECONDS`、`RUST_LOG`、`WEBHOOK*`、`TWITTER_AUTH_TOKEN`(可选)。
|
||||
环境变量:`TELOXIDE_TOKEN`(必填)、`PIXIV_REFRESH_TOKEN`、`BOT_ADMIN`、`EDIT_MESSAGE_TTL_SECONDS`、`LINK_CACHE_TTL_SECONDS`、`RUST_LOG`、`TELOXIDE_PROXY`、`WEBHOOK*`、`TWITTER_AUTH_TOKEN`(可选)。
|
||||
|
||||
NSFW 推文:公开的 syndication 接口不返回敏感内容。设置 `TWITTER_AUTH_TOKEN`(登录 x.com 后浏览器 Cookie 里的 `auth_token` 值)后,bot 会仅在遇到 NSFW 推文时以登录态获取媒体;未设置则提示无媒体。
|
||||
|
||||
@@ -83,7 +84,9 @@ Telegram 只接受 443/80/88/8443 端口。
|
||||
| `BOT_ADMIN` | 管理员聊天 ID,逗号分隔;接收启动/停止通知 |
|
||||
| `EDIT_MESSAGE_TTL_SECONDS` | 转发前编辑记录过期秒数,默认 86400 |
|
||||
| `LINK_CACHE_TTL_SECONDS` | 链接结果缓存过期秒数,默认 604800(7 天) |
|
||||
| `DATA_DIR` | 数据目录(SQLite 数据库 `task_queue.db` 所在目录),默认 `data`(相对工作目录,会自动创建) |
|
||||
| `RUST_LOG` | 日志级别 |
|
||||
| `TELOXIDE_PROXY` | HTTP 代理(如 `http://127.0.0.1:10808`);同时作用于 Telegram Bot API 与站点抓取请求,网络受限环境(如 GFW)必需 |
|
||||
| `LOCAL_USER_ID` | 容器内运行用户 UID,默认 9001 |
|
||||
| `VIRTUAL_HOST` | 对外域名或 IP,nginx-proxy 按此路由 |
|
||||
| `VIRTUAL_PORT` | bot 容器内监听端口,nginx-proxy 的转发目标 |
|
||||
@@ -93,6 +96,7 @@ Telegram 只接受 443/80/88/8443 端口。
|
||||
| `WEBHOOK` | `true` 启用 webhook 模式(默认轮询) |
|
||||
| `WEBHOOK_LISTEN` / `WEBHOOK_PORT` | bot 容器内监听地址/端口 |
|
||||
| `WEBHOOK_URL` | 对外公网 HTTPS 地址(`https://域名/` 或 `https://IP/`) |
|
||||
| `WEBHOOK_CERT` | 可选;自签名证书路径,仅用于 Telegram 侧验证(TLS 由反向代理终止) |
|
||||
| `WEBHOOK_SECRET_TOKEN` | 更新校验令牌(`X-Telegram-Bot-Api-Secret-Token`) |
|
||||
|
||||
</details>
|
||||
@@ -108,7 +112,9 @@ Telegram 只接受 443/80/88/8443 端口。
|
||||
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
|
||||
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
|
||||
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
|
||||
| `/clear_cache [链接]` | 清空链接缓存(仅管理员);带链接只清该条,否则清空全部 |
|
||||
| `/bot_dict` | 查看当前聊天状态(调试用) |
|
||||
| `/test <链接>` | 调试:只解析链接并返回解析结果(站点、标题、作者、标签、媒体列表),不发送任何媒体 |
|
||||
|
||||
链接处理仅限私聊;命令在任意聊天可用。
|
||||
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
[package]
|
||||
name = "x-media"
|
||||
version = "1.1.0"
|
||||
version = "1.4.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
reqwest = { version = "0.12", features = ["json"] }
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
regex = "1.12"
|
||||
@@ -13,6 +13,7 @@ url = "2.5.2"
|
||||
bytes = "1"
|
||||
zip = "2"
|
||||
tempfile = "3"
|
||||
thiserror = "2"
|
||||
rand = "0.8"
|
||||
log = "0.4"
|
||||
tokio = { version = "1.40", features = ["time"] }
|
||||
|
||||
@@ -1,10 +1,31 @@
|
||||
use super::model;
|
||||
use crate::media::Media;
|
||||
use crate::site::{FetchError, Fetched};
|
||||
use crate::site::{FetchError, Fetched, Site, SiteFuture};
|
||||
use html_escape::{encode_double_quoted_attribute, encode_text};
|
||||
use regex::Regex;
|
||||
use std::sync::LazyLock;
|
||||
|
||||
/// Registry entry for the bluesky adapter (see [`crate::site::Site`]).
|
||||
pub struct BskySite;
|
||||
|
||||
impl Site for BskySite {
|
||||
fn id(&self) -> &'static str {
|
||||
"bsky"
|
||||
}
|
||||
|
||||
fn pattern(&self) -> &'static Regex {
|
||||
&PATTERN
|
||||
}
|
||||
|
||||
fn cache_key(&self, url: &str) -> Option<String> {
|
||||
cache_key(url)
|
||||
}
|
||||
|
||||
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched> {
|
||||
Box::pin(async move { fetch_from_url(url).await })
|
||||
}
|
||||
}
|
||||
|
||||
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
|
||||
Regex::new(r"^(?:https?://)?bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap()
|
||||
});
|
||||
@@ -59,6 +80,25 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
Ok(fetched)
|
||||
}
|
||||
|
||||
/// Cache key for a bsky URL: `"bsky:<handle>/<rkey>"`. The prefix is the
|
||||
/// site id used for caption-format lookup and link-cache keys.
|
||||
pub fn cache_key(url: &str) -> Option<String> {
|
||||
PATTERN
|
||||
.captures(url)
|
||||
.map(|caps| format!("bsky:{}/{}", &caps[1], &caps[2]))
|
||||
}
|
||||
|
||||
/// Bluesky's fetch-retry policy: transient classes only. Not-found, blocked
|
||||
/// and parse failures are permanent.
|
||||
pub fn is_retryable(err: &FetchError) -> bool {
|
||||
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
|
||||
}
|
||||
|
||||
/// bsky media (cdn.bsky.app) needs no extra headers.
|
||||
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
|
||||
None
|
||||
}
|
||||
|
||||
/// Downloads an HLS playlist (master or media) and remuxes its segments to a
|
||||
/// single MP4 via ffmpeg. Returns the MP4 path plus the temp dir that must
|
||||
/// stay alive until the file is uploaded. `Ok(None)` when ffmpeg is missing.
|
||||
@@ -299,6 +339,7 @@ impl From<Post> for Fetched {
|
||||
title: post.text.clone(),
|
||||
media: post.media,
|
||||
sensitive: post.sensitive,
|
||||
site_id: "bsky",
|
||||
render_data,
|
||||
_keep_alive: None,
|
||||
}
|
||||
@@ -416,6 +457,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
|
||||
async fn live_fetch_with_photos() {
|
||||
let fetched =
|
||||
fetch_from_url("https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m")
|
||||
@@ -429,6 +471,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
|
||||
async fn live_fetch_smoke() {
|
||||
let fetched =
|
||||
fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
mod interface;
|
||||
mod model;
|
||||
|
||||
pub use interface::{PATTERN, Post, enabled, fetch_from_url};
|
||||
pub use interface::{
|
||||
BskySite, PATTERN, Post, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
|
||||
};
|
||||
|
||||
+347
-119
@@ -4,11 +4,15 @@
|
||||
//! `PATTERN`, `enabled()` and `fetch_from_url()`; a future site plugs in by
|
||||
//! adding one guarded entry in [`fetch_once`].
|
||||
|
||||
use std::fmt;
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use std::sync::LazyLock;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
use regex::Regex;
|
||||
use thiserror::Error;
|
||||
|
||||
pub mod bsky;
|
||||
pub mod pixiv;
|
||||
pub mod twitter;
|
||||
@@ -30,6 +34,10 @@ pub struct Fetched {
|
||||
pub media: Vec<crate::media::Media>,
|
||||
/// Spoiler flag for all media of this post.
|
||||
pub sensitive: bool,
|
||||
/// Site id (`"twitter"` / `"bsky"` / `"pixiv"`): the single source of
|
||||
/// truth for site identity — caption-format lookup, cache-key prefix and
|
||||
/// the SetFormat whitelist all derive from it. Set by the producing site.
|
||||
pub site_id: &'static str,
|
||||
/// Raw values (pre-escaped) for user-customizable caption formats.
|
||||
pub(crate) render_data: Option<RenderData>,
|
||||
/// Keeps temp files (e.g. an encoded ugoira MP4) alive until the caller
|
||||
@@ -50,23 +58,18 @@ pub(crate) struct RenderData {
|
||||
|
||||
impl Fetched {
|
||||
/// The site this post came from (used for per-site format overrides).
|
||||
/// A thin alias over [`Fetched::site_id`] kept for callers that read the
|
||||
/// site off a fetched post.
|
||||
pub fn site_name(&self) -> &'static str {
|
||||
if self.source_url.contains("x.com") || self.source_url.contains("twitter.com") {
|
||||
"twitter"
|
||||
} else if self.source_url.contains("bsky.app") {
|
||||
"bsky"
|
||||
} else if self.source_url.contains("pixiv.net") {
|
||||
"pixiv"
|
||||
} else {
|
||||
"unknown"
|
||||
}
|
||||
self.site_id
|
||||
}
|
||||
|
||||
/// Renders a user-supplied caption format. The format string is
|
||||
/// HTML-escaped in full, then the (already-escaped) placeholder values
|
||||
/// are substituted — users can structure text but never inject raw HTML
|
||||
/// or attributes. An empty/unknown format falls back to the built-in
|
||||
/// caption.
|
||||
/// caption. The result is truncated to [`MAX_CAPTION_CHARS`] (Telegram's
|
||||
/// caption limit for HTML parse mode).
|
||||
pub fn caption_with(&self, format: &str) -> String {
|
||||
match (&self.render_data, format.is_empty()) {
|
||||
(Some(data), false) => caption_from_fields(
|
||||
@@ -78,7 +81,7 @@ impl Fetched {
|
||||
&data.title,
|
||||
&data.tags,
|
||||
),
|
||||
_ => self.caption.clone(),
|
||||
_ => truncate_caption(&self.caption),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -95,11 +98,47 @@ impl Fetched {
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
/// Hands over the temp dir keeping locally produced media (ugoira MP4,
|
||||
/// bsky remux MP4) alive. The bot keeps it while its task may still be
|
||||
/// retried by the queue, which runs after this [`Fetched`] is dropped and
|
||||
/// its temp files would otherwise be gone. `None` when no such dir exists.
|
||||
pub fn take_keep_alive(&mut self) -> Option<tempfile::TempDir> {
|
||||
self._keep_alive.take()
|
||||
}
|
||||
}
|
||||
|
||||
/// Telegram's caption length limit (chars) for HTML parse mode; longer
|
||||
/// captions are rejected with a 400.
|
||||
pub const MAX_CAPTION_CHARS: usize = 1024;
|
||||
|
||||
/// Truncates a caption to at most [`MAX_CAPTION_CHARS`] chars, appending an
|
||||
/// ellipsis when cut. Backs off to before an unclosed HTML entity (`&`
|
||||
/// without its `;` would be malformed HTML and rejected by Telegram).
|
||||
pub fn truncate_caption(caption: &str) -> String {
|
||||
if caption.chars().count() <= MAX_CAPTION_CHARS {
|
||||
return caption.to_string();
|
||||
}
|
||||
// Leave one char for the ellipsis; floor_char_boundary lands on a char
|
||||
// edge (byte index ≤ MAX-1, so chars ≤ MAX-1).
|
||||
let mut end = caption.floor_char_boundary(MAX_CAPTION_CHARS - 1);
|
||||
// Don't split an entity: if the last '&' before `end` has no closing ';'
|
||||
// inside the kept part, cut before it.
|
||||
if let Some(amp) = caption[..end].rfind('&')
|
||||
&& !caption[amp..end].contains(';')
|
||||
{
|
||||
end = amp;
|
||||
}
|
||||
let mut s = caption[..end].to_string();
|
||||
s.push('…');
|
||||
s
|
||||
}
|
||||
|
||||
/// Renders a user-supplied caption format from raw (already-escaped) field
|
||||
/// values with the same escaping/substitution rules as
|
||||
/// [`Fetched::caption_with`]. An empty format returns `built_in` unchanged.
|
||||
/// The result is truncated to [`MAX_CAPTION_CHARS`] (Telegram's caption
|
||||
/// limit for HTML parse mode).
|
||||
pub fn caption_from_fields(
|
||||
format: &str,
|
||||
built_in: &str,
|
||||
@@ -110,94 +149,77 @@ pub fn caption_from_fields(
|
||||
tags: &str,
|
||||
) -> String {
|
||||
if format.is_empty() {
|
||||
return built_in.to_string();
|
||||
return truncate_caption(built_in);
|
||||
}
|
||||
let escaped = html_escape::encode_text(format).into_owned();
|
||||
escaped
|
||||
.replace("{url}", url)
|
||||
.replace("{author}", author)
|
||||
.replace("{author_url}", author_url)
|
||||
.replace("{title}", title)
|
||||
.replace("{tags}", tags)
|
||||
truncate_caption(
|
||||
&escaped
|
||||
.replace("{url}", url)
|
||||
.replace("{author}", author)
|
||||
.replace("{author_url}", author_url)
|
||||
.replace("{title}", title)
|
||||
.replace("{tags}", tags),
|
||||
)
|
||||
}
|
||||
|
||||
/// Stable per-post cache key derived from any supported URL, so variant
|
||||
/// domains (x.com / twitter.com / fxtwitter.com, mobile, `/photo/N`
|
||||
/// suffixes) map to the same post. Returns `"twitter:<id>"`,
|
||||
/// `"pixiv:<id>"` or `"bsky:<handle>/<rkey>"`.
|
||||
/// suffixes) map to the same post. Delegates to each registered site's
|
||||
/// `cache_key` (dispatch order twitter → bsky → pixiv).
|
||||
pub fn cache_key(url: &str) -> Option<String> {
|
||||
if let Some(caps) = twitter::PATTERN.captures(url) {
|
||||
return Some(format!("twitter:{}", &caps[1]));
|
||||
}
|
||||
if let Some(caps) = pixiv::PATTERN.captures(url) {
|
||||
return Some(format!("pixiv:{}", &caps[1]));
|
||||
}
|
||||
if let Some(caps) = bsky::PATTERN.captures(url) {
|
||||
return Some(format!("bsky:{}/{}", &caps[1], &caps[2]));
|
||||
}
|
||||
None
|
||||
SITES.iter().find_map(|site| site.cache_key(url))
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
/// The site id carried by a cache key (`"twitter:123"` → `"twitter"`).
|
||||
/// Unknown prefixes fall back to `"unknown"`. The bot uses this on the
|
||||
/// link-cache hit path, where no [`Fetched`] is available — the same value
|
||||
/// a fresh fetch would read from [`Fetched::site_id`].
|
||||
pub fn site_id_from_key(key: &str) -> &'static str {
|
||||
let prefix = key.split(':').next().unwrap_or("");
|
||||
SITES
|
||||
.iter()
|
||||
.map(|site| site.id())
|
||||
.find(|id| *id == prefix)
|
||||
.unwrap_or("unknown")
|
||||
}
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum FetchError {
|
||||
Http(reqwest::Error),
|
||||
Json(serde_json::Error),
|
||||
Pixiv(PixivError),
|
||||
#[error("http error: {0}")]
|
||||
Http(#[from] reqwest::Error),
|
||||
#[error("json error: {0}")]
|
||||
Json(#[from] serde_json::Error),
|
||||
#[error("pixiv error: {0}")]
|
||||
Pixiv(#[from] PixivError),
|
||||
/// A site-specific error from a site that keeps its own error type.
|
||||
/// Permanent by default (sites that need retryable site errors convert
|
||||
/// them to [`FetchError::Http`] / [`FetchError::Transient`] before
|
||||
/// returning). Pixiv predates this and keeps the dedicated
|
||||
/// [`FetchError::Pixiv`] variant.
|
||||
#[error("{site} error: {error}")]
|
||||
Site {
|
||||
site: &'static str,
|
||||
#[source]
|
||||
error: Box<dyn std::error::Error + Send + Sync>,
|
||||
},
|
||||
#[error("not found")]
|
||||
NotFound,
|
||||
#[error("blocked")]
|
||||
Blocked,
|
||||
/// The post exists but its content is withheld (twitter NSFW /
|
||||
/// age-restricted tweets come back as an empty `{}` from syndication).
|
||||
#[error("content withheld (sensitive)")]
|
||||
Sensitive,
|
||||
/// A download exceeded the caller's size cap (see [`download_media_limited`]).
|
||||
#[error("media too large")]
|
||||
TooLarge,
|
||||
/// A transient server-side failure (429 / 5xx); [`fetch`] retries these.
|
||||
#[error("transient: {0}")]
|
||||
Transient(String),
|
||||
}
|
||||
|
||||
impl fmt::Display for FetchError {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match self {
|
||||
FetchError::Http(e) => write!(f, "http error: {e}"),
|
||||
FetchError::Json(e) => write!(f, "json error: {e}"),
|
||||
FetchError::Pixiv(e) => write!(f, "pixiv error: {e}"),
|
||||
FetchError::NotFound => write!(f, "not found"),
|
||||
FetchError::Blocked => write!(f, "blocked"),
|
||||
FetchError::Sensitive => write!(f, "content withheld (sensitive)"),
|
||||
FetchError::TooLarge => write!(f, "media too large"),
|
||||
FetchError::Transient(message) => write!(f, "transient: {message}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for FetchError {
|
||||
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
|
||||
match self {
|
||||
FetchError::Http(e) => Some(e),
|
||||
FetchError::Json(e) => Some(e),
|
||||
FetchError::Pixiv(e) => Some(e),
|
||||
FetchError::NotFound | FetchError::Blocked | FetchError::Sensitive => None,
|
||||
FetchError::TooLarge => None,
|
||||
FetchError::Transient(_) => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<reqwest::Error> for FetchError {
|
||||
fn from(e: reqwest::Error) -> Self {
|
||||
FetchError::Http(e)
|
||||
}
|
||||
}
|
||||
|
||||
impl From<serde_json::Error> for FetchError {
|
||||
fn from(e: serde_json::Error) -> Self {
|
||||
FetchError::Json(e)
|
||||
}
|
||||
}
|
||||
|
||||
impl From<PixivError> for FetchError {
|
||||
fn from(e: PixivError) -> Self {
|
||||
FetchError::Pixiv(e)
|
||||
}
|
||||
/// A local I/O failure while streaming a download to disk
|
||||
/// (see [`download_media_to_file`]).
|
||||
#[error("io error: {0}")]
|
||||
Io(std::io::Error),
|
||||
}
|
||||
|
||||
/// Shared HTTP client (browser User-Agent) for twitter/bsky fetches and
|
||||
@@ -252,65 +274,158 @@ pub(crate) fn log_once_ffmpeg_missing() {
|
||||
}
|
||||
}
|
||||
|
||||
/// Site adapter: one impl per supported site (twitter / bsky / pixiv),
|
||||
/// registered in [`SITES`]. All site-specific knowledge — URL pattern,
|
||||
/// cache-key format, fetch, retry policy, media-host headers, startup
|
||||
/// validation — lives in the site module; the central dispatcher only
|
||||
/// iterates the registry.
|
||||
///
|
||||
/// Async methods return a boxed future (see [`SiteFuture`]): `async fn` /
|
||||
/// RPITIT in traits are not dyn-compatible (verified on rustc 1.95), and
|
||||
/// `+ Send` is required since URL/queue workers spawn these futures. The
|
||||
/// site structs are stateless unit structs, so the boxed futures never
|
||||
/// borrow from `self` beyond the call's scope.
|
||||
pub trait Site: Send + Sync {
|
||||
/// Stable site id (`"twitter"` / `"bsky"` / `"pixiv"`): caption-format
|
||||
/// lookup, cache-key prefixes and the SetFormat whitelist derive from it.
|
||||
fn id(&self) -> &'static str;
|
||||
/// URL pattern; the dispatcher's first match wins (dispatch order).
|
||||
fn pattern(&self) -> &'static Regex;
|
||||
/// Whether the site is usable (env token present, not disabled).
|
||||
fn enabled(&self) -> bool {
|
||||
true
|
||||
}
|
||||
/// Normalized cache key for a URL of this site (`None` when the URL does
|
||||
/// not match this site).
|
||||
fn cache_key(&self, url: &str) -> Option<String>;
|
||||
/// Fetches and normalizes a post.
|
||||
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched>;
|
||||
/// Retry policy for fetch errors: transient classes only.
|
||||
fn is_retryable(&self, err: &FetchError) -> bool {
|
||||
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
|
||||
}
|
||||
/// Extra headers for downloading this site's media (hotlink protection,
|
||||
/// e.g. pixiv's Referer for pximg.net). Matched on the media URL, not
|
||||
/// the site pattern.
|
||||
fn media_headers(&self, _url: &str) -> Option<Vec<(&'static str, String)>> {
|
||||
None
|
||||
}
|
||||
/// Startup validation (token check etc.); failures are surfaced by
|
||||
/// [`validate_all`]. The default is a no-op.
|
||||
fn validate(&self) -> SiteFuture<'static, (), String> {
|
||||
Box::pin(async { Ok(()) })
|
||||
}
|
||||
}
|
||||
|
||||
/// A boxed, `Send` future produced by a [`Site`] async method. Boxed so the
|
||||
/// trait stays dyn-compatible; `Send` because URL/queue workers `tokio::spawn`
|
||||
/// these futures.
|
||||
type SiteFuture<'a, T, E = FetchError> = Pin<Box<dyn Future<Output = Result<T, E>> + Send + 'a>>;
|
||||
|
||||
/// The one registry of supported sites, in dispatch order (twitter → bsky →
|
||||
/// pixiv). Adding a site = new module + one `Box::new(...)` entry here; the
|
||||
/// bot crate never lists sites itself.
|
||||
static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| {
|
||||
vec![
|
||||
Box::new(twitter::TwitterSite),
|
||||
Box::new(bsky::BskySite),
|
||||
Box::new(pixiv::PixivSite),
|
||||
]
|
||||
});
|
||||
|
||||
/// The first enabled site whose pattern matches `url`, in dispatch order.
|
||||
fn find_site(url: &str) -> Option<&'static dyn Site> {
|
||||
SITES
|
||||
.iter()
|
||||
.find(|site| site.enabled() && site.pattern().is_match(url))
|
||||
.map(|site| site.as_ref())
|
||||
}
|
||||
|
||||
/// Every supported site id, in dispatch order. The bot's SetFormat whitelist
|
||||
/// derives from this list.
|
||||
pub fn site_ids() -> Vec<&'static str> {
|
||||
SITES.iter().map(|site| site.id()).collect()
|
||||
}
|
||||
|
||||
/// Runs every enabled site's startup validation and returns the failures
|
||||
/// (site id + message). The caller logs / notifies; failing sites disable
|
||||
/// themselves (pixiv disables on a bad token).
|
||||
pub async fn validate_all() -> Vec<(&'static str, String)> {
|
||||
let mut failures = Vec::new();
|
||||
for site in SITES.iter() {
|
||||
if !site.enabled() {
|
||||
continue;
|
||||
}
|
||||
if let Err(e) = site.validate().await {
|
||||
failures.push((site.id(), e));
|
||||
}
|
||||
}
|
||||
failures
|
||||
}
|
||||
|
||||
/// Fetches a post from its URL. Returns `Ok(None)` when no site pattern
|
||||
/// matches (unsupported links are silently ignored by the bot).
|
||||
///
|
||||
/// Transient network failures are retried: 3 total attempts with 1s then 2s
|
||||
/// delays. Retried classes: bare HTTP errors, [`FetchError::Transient`]
|
||||
/// (429/5xx from any site), and pixiv errors (its network failures arrive
|
||||
/// wrapped as `PixivError`). Non-retried: Json/NotFound/Blocked/Sensitive.
|
||||
/// Transient failures are retried: 3 total attempts with 1s then 2s delays.
|
||||
/// What counts as transient is the matched site's own policy (`is_retryable`
|
||||
/// — e.g. pixiv retries only network errors and 429/5xx). Permanent classes
|
||||
/// (not-found, blocked, sensitive, parse failures, pixiv 4xx/auth errors)
|
||||
/// are returned immediately; retrying them only wastes attempts against the
|
||||
/// source site.
|
||||
pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
|
||||
let Some(site) = find_site(url) else {
|
||||
return Ok(None);
|
||||
};
|
||||
for attempt in 0..3u32 {
|
||||
match fetch_once(url).await {
|
||||
Ok(Some(fetched)) => {
|
||||
log::info!(
|
||||
"fetched {url}: site {} returned {} media",
|
||||
match site.fetch_from_url(url).await {
|
||||
Ok(fetched) => {
|
||||
// Per-request detail: debug only, keyed by the post id.
|
||||
log::debug!(
|
||||
"fetched [key={}]: site {} returned {} media",
|
||||
cache_key(url).unwrap_or_else(|| "?".into()),
|
||||
fetched.site_name(),
|
||||
fetched.media.len()
|
||||
);
|
||||
return Ok(Some(fetched));
|
||||
}
|
||||
Ok(None) => return Ok(None),
|
||||
Err(e @ (FetchError::Http(_) | FetchError::Transient(_) | FetchError::Pixiv(_))) => {
|
||||
if attempt < 2 {
|
||||
Err(err) => {
|
||||
if site.is_retryable(&err) && attempt < 2 {
|
||||
tokio::time::sleep(Duration::from_secs(1 << attempt)).await;
|
||||
} else {
|
||||
return Err(e);
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
Err(other) => return Err(other),
|
||||
}
|
||||
}
|
||||
unreachable!("retry loop always returns")
|
||||
}
|
||||
|
||||
async fn fetch_once(url: &str) -> Result<Option<Fetched>, FetchError> {
|
||||
if twitter::enabled() && twitter::PATTERN.is_match(url) {
|
||||
return Ok(Some(twitter::fetch_from_url(url).await?));
|
||||
/// Applies every site's media-header rule to a download request (pixiv's
|
||||
/// `Referer` for pximg.net hotlink protection). Sites contribute via their
|
||||
/// `media_headers(url)` — the central download code carries no per-site logic.
|
||||
fn apply_media_headers(mut request: reqwest::RequestBuilder, url: &str) -> reqwest::RequestBuilder {
|
||||
for site in SITES.iter() {
|
||||
if let Some(headers) = site.media_headers(url) {
|
||||
for (name, value) in headers {
|
||||
request = request.header(name, value);
|
||||
}
|
||||
}
|
||||
}
|
||||
if bsky::enabled() && bsky::PATTERN.is_match(url) {
|
||||
return Ok(Some(bsky::fetch_from_url(url).await?));
|
||||
}
|
||||
if pixiv::enabled() && pixiv::PATTERN.is_match(url) {
|
||||
return Ok(Some(pixiv::fetch_from_url(url).await?));
|
||||
}
|
||||
Ok(None)
|
||||
request
|
||||
}
|
||||
|
||||
/// Downloads media bytes for the bot's upload fallback: when Telegram's own
|
||||
/// fetch of a media URL is blocked (hotlink protection), the bot downloads
|
||||
/// the file itself and uploads it via multipart. Site-appropriate headers:
|
||||
/// pixiv image hosts need the `Referer` header.
|
||||
/// the file itself and uploads it via multipart. Site-appropriate headers
|
||||
/// come from each site's `media_headers` (pixiv image hosts need `Referer`).
|
||||
/// Returns the Content-Length of a media URL, or `None` when the server does
|
||||
/// not report one. Used to check whether a file fits Telegram's size limits
|
||||
/// before downloading/uploading it.
|
||||
pub async fn media_size(url: &str) -> Result<Option<u64>, FetchError> {
|
||||
let mut request = CLIENT.get(url);
|
||||
let lower = url.to_ascii_lowercase();
|
||||
if lower.contains("pximg.net") {
|
||||
request = request.header("Referer", "https://www.pixiv.net/");
|
||||
}
|
||||
let response = request.send().await?.error_for_status()?;
|
||||
let response = apply_media_headers(CLIENT.get(url), url)
|
||||
.send()
|
||||
.await?
|
||||
.error_for_status()?;
|
||||
Ok(response.content_length())
|
||||
}
|
||||
|
||||
@@ -319,12 +434,10 @@ pub async fn media_size(url: &str) -> Result<Option<u64>, FetchError> {
|
||||
/// crossed (or when a declared Content-Length already exceeds it). Keeps the
|
||||
/// bot from buffering arbitrarily large bodies into memory.
|
||||
pub async fn download_media_limited(url: &str, max_bytes: u64) -> Result<bytes::Bytes, FetchError> {
|
||||
let mut request = CLIENT.get(url);
|
||||
let lower = url.to_ascii_lowercase();
|
||||
if lower.contains("pximg.net") {
|
||||
request = request.header("Referer", "https://www.pixiv.net/");
|
||||
}
|
||||
let response = request.send().await?.error_for_status()?;
|
||||
let response = apply_media_headers(CLIENT.get(url), url)
|
||||
.send()
|
||||
.await?
|
||||
.error_for_status()?;
|
||||
if let Some(len) = response.content_length()
|
||||
&& len > max_bytes
|
||||
{
|
||||
@@ -345,6 +458,39 @@ pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
|
||||
download_media_limited(url, u64::MAX).await
|
||||
}
|
||||
|
||||
/// Streams a download to `out`, aborting with [`FetchError::TooLarge`] the
|
||||
/// moment the body crosses `max_bytes` (or when a declared Content-Length
|
||||
/// already exceeds it). Unlike [`download_media_limited`] the body is never
|
||||
/// buffered in memory — used for large files (e.g. the pixiv ugoira frame
|
||||
/// zip, which can be hundreds of MB) that would otherwise spike RAM.
|
||||
/// Returns the number of bytes written.
|
||||
pub async fn download_media_to_file(
|
||||
url: &str,
|
||||
max_bytes: u64,
|
||||
out: &mut std::fs::File,
|
||||
) -> Result<u64, FetchError> {
|
||||
use std::io::Write;
|
||||
let response = apply_media_headers(CLIENT.get(url), url)
|
||||
.send()
|
||||
.await?
|
||||
.error_for_status()?;
|
||||
if let Some(len) = response.content_length()
|
||||
&& len > max_bytes
|
||||
{
|
||||
return Err(FetchError::TooLarge);
|
||||
}
|
||||
let mut response = response;
|
||||
let mut total: u64 = 0;
|
||||
while let Some(chunk) = response.chunk().await? {
|
||||
total += chunk.len() as u64;
|
||||
if total > max_bytes {
|
||||
return Err(FetchError::TooLarge);
|
||||
}
|
||||
out.write_all(&chunk).map_err(FetchError::Io)?;
|
||||
}
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -374,6 +520,43 @@ mod tests {
|
||||
assert_eq!(cache_key("https://example.com/not-a-post"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn site_id_from_key_parses_prefix() {
|
||||
assert_eq!(site_id_from_key("twitter:123"), "twitter");
|
||||
assert_eq!(site_id_from_key("pixiv:123"), "pixiv");
|
||||
assert_eq!(site_id_from_key("bsky:handle.example/3lorem"), "bsky");
|
||||
assert_eq!(site_id_from_key("unknown:1"), "unknown");
|
||||
assert_eq!(site_id_from_key("no-colon"), "unknown");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn registry_lists_all_sites_in_dispatch_order() {
|
||||
assert_eq!(site_ids(), vec!["twitter", "bsky", "pixiv"]);
|
||||
// Enabled sites dispatch; unsupported URLs never match.
|
||||
assert!(find_site("https://x.com/u/status/1").is_some());
|
||||
assert!(find_site("https://bsky.app/profile/u/post/3x").is_some());
|
||||
assert!(find_site("https://example.com/x").is_none());
|
||||
// Cache keys are pattern-driven, independent of the enabled() gate
|
||||
// (pixiv is disabled in tests without PIXIV_REFRESH_TOKEN).
|
||||
assert_eq!(
|
||||
cache_key("https://www.pixiv.net/artworks/1"),
|
||||
Some("pixiv:1".into())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn site_error_variant_displays_and_sources() {
|
||||
use std::error::Error as _;
|
||||
let err = FetchError::Site {
|
||||
site: "example",
|
||||
error: Box::new(std::io::Error::other("boom")),
|
||||
};
|
||||
assert_eq!(err.to_string(), "example error: boom");
|
||||
assert!(err.source().is_some());
|
||||
// Permanent by default: no site's is_retryable matches it.
|
||||
assert!(!twitter::is_retryable(&err));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn caption_from_fields_substitutes_and_escapes() {
|
||||
// The format string is escaped, the field values are substituted
|
||||
@@ -398,6 +581,45 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_caption_keeps_short_text() {
|
||||
assert_eq!(truncate_caption("short"), "short");
|
||||
// Exactly at the limit: untouched.
|
||||
let exact = "x".repeat(MAX_CAPTION_CHARS);
|
||||
assert_eq!(truncate_caption(&exact), exact);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_caption_cuts_long_text_with_ellipsis() {
|
||||
let long = "x".repeat(MAX_CAPTION_CHARS + 100);
|
||||
let out = truncate_caption(&long);
|
||||
assert!(
|
||||
out.chars().count() <= MAX_CAPTION_CHARS,
|
||||
"len {}",
|
||||
out.chars().count()
|
||||
);
|
||||
assert!(out.ends_with('…'));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_caption_does_not_split_an_html_entity() {
|
||||
// An entity crossing the cut must not be left half-open (& without ;).
|
||||
let mut long = "a".repeat(MAX_CAPTION_CHARS - 4);
|
||||
long.push_str("&bbbb");
|
||||
let out = truncate_caption(&long);
|
||||
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
|
||||
assert!(!out.contains("&"), "half entity left: {out:?}");
|
||||
assert!(!out.ends_with('&'));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_caption_handles_multibyte_boundary() {
|
||||
// Multi-byte chars near the cut must not panic (char-boundary cut).
|
||||
let long = "界".repeat(MAX_CAPTION_CHARS + 10);
|
||||
let out = truncate_caption(&long);
|
||||
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unsupported_url_returns_none() {
|
||||
let result = fetch("https://example.com/some/article").await;
|
||||
@@ -414,7 +636,13 @@ mod tests {
|
||||
async fn download_media_pixiv_original_with_referer() {
|
||||
// Proves the Referer header is attached for i.pximg.net: a header-less
|
||||
// GET to a pixiv original URL is rejected with 403.
|
||||
if std::env::var("PIXIV_REFRESH_TOKEN").is_err() {
|
||||
// Empty-string check too: an unset CI secret arrives as "" (GitHub
|
||||
// Actions), which would otherwise run the test tokenless and fail.
|
||||
if std::env::var("PIXIV_REFRESH_TOKEN")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty())
|
||||
.is_none()
|
||||
{
|
||||
eprintln!("skipping: no PIXIV_REFRESH_TOKEN");
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -8,11 +8,11 @@ use super::model::{IllustrationModel, TypeModel, UgoiraMetadataModel};
|
||||
use crate::media::Media;
|
||||
use crate::site::FetchError;
|
||||
use std::env;
|
||||
use std::fmt;
|
||||
use std::io::{Cursor, Read};
|
||||
use std::io::Read;
|
||||
use std::sync::LazyLock;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, SystemTime};
|
||||
use thiserror::Error;
|
||||
|
||||
const AUTH_TOKEN_URL: &str = "https://oauth.secure.pixiv.net/auth/token";
|
||||
const APP_API_URL: &str = "https://app-api.pixiv.net";
|
||||
@@ -23,48 +23,24 @@ const APP_USER_AGENT: &str = "PixivIOSApp/7.13.3 (iOS 14.6; iPhone13,2)";
|
||||
/// Token refresh safe margin (seconds).
|
||||
const TOKEN_REFRESH_SAFE_MARGIN: u64 = 300;
|
||||
|
||||
#[derive(Debug)]
|
||||
#[derive(Debug, Error)]
|
||||
pub enum PixivError {
|
||||
/// No refresh token available (PIXIV_REFRESH_TOKEN unset).
|
||||
#[error("pixiv: no authentication")]
|
||||
NoAuth,
|
||||
Http(reqwest::Error),
|
||||
Json(serde_json::Error),
|
||||
#[error("pixiv http error: {0}")]
|
||||
Http(#[from] reqwest::Error),
|
||||
#[error("pixiv json error: {0}")]
|
||||
Json(#[from] serde_json::Error),
|
||||
/// Non-2xx HTTP status from the app API. The code lets [`crate::site::fetch`]
|
||||
/// retry only transient classes (429 / 5xx) instead of burning attempts on
|
||||
/// permanent 4xx (bad token, forbidden, not found).
|
||||
#[error("pixiv status {0}")]
|
||||
Status(u16),
|
||||
#[error("pixiv api error: {0}")]
|
||||
Api(String),
|
||||
}
|
||||
|
||||
impl fmt::Display for PixivError {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match self {
|
||||
PixivError::NoAuth => write!(f, "pixiv: no authentication"),
|
||||
PixivError::Http(e) => write!(f, "pixiv http error: {e}"),
|
||||
PixivError::Json(e) => write!(f, "pixiv json error: {e}"),
|
||||
PixivError::Api(message) => write!(f, "pixiv api error: {message}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for PixivError {
|
||||
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
|
||||
match self {
|
||||
PixivError::Http(e) => Some(e),
|
||||
PixivError::Json(e) => Some(e),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<reqwest::Error> for PixivError {
|
||||
fn from(e: reqwest::Error) -> Self {
|
||||
PixivError::Http(e)
|
||||
}
|
||||
}
|
||||
|
||||
impl From<serde_json::Error> for PixivError {
|
||||
fn from(e: serde_json::Error) -> Self {
|
||||
PixivError::Json(e)
|
||||
}
|
||||
}
|
||||
|
||||
/// Native pixiv app-API client.
|
||||
pub struct PixivAPI {
|
||||
refresh_token: String,
|
||||
@@ -137,7 +113,7 @@ impl PixivAPI {
|
||||
.send()
|
||||
.await?;
|
||||
if !response.status().is_success() {
|
||||
return Err(PixivError::Api(format!("status {}", response.status())));
|
||||
return Err(PixivError::Status(response.status().as_u16()));
|
||||
}
|
||||
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
|
||||
if json.get("error").is_some() {
|
||||
@@ -190,7 +166,7 @@ impl PixivAPI {
|
||||
.send()
|
||||
.await?;
|
||||
if !response.status().is_success() {
|
||||
return Err(PixivError::Api(format!("status {}", response.status())));
|
||||
return Err(PixivError::Status(response.status().as_u16()));
|
||||
}
|
||||
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
|
||||
if json.get("error").is_some() {
|
||||
@@ -228,7 +204,14 @@ impl PixivAPI {
|
||||
let Some(zip_url) = zip_url else {
|
||||
return Ok(None);
|
||||
};
|
||||
let zip_bytes = crate::site::download_media_limited(&zip_url, 512 * 1024 * 1024)
|
||||
// Stream the frame zip to a temp file instead of buffering it in
|
||||
// memory: ugoira zips can be hundreds of MB, and the old
|
||||
// download_media_limited path spiked RAM up to the size cap.
|
||||
let mut zip_file = tempfile::Builder::new()
|
||||
.suffix(".zip")
|
||||
.tempfile()
|
||||
.map_err(|e| PixivError::Api(format!("temp zip failed: {e}")))?;
|
||||
crate::site::download_media_to_file(&zip_url, 512 * 1024 * 1024, zip_file.as_file_mut())
|
||||
.await
|
||||
.map_err(|e| match e {
|
||||
FetchError::Http(e) => PixivError::Http(e),
|
||||
@@ -241,9 +224,12 @@ impl PixivAPI {
|
||||
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
|
||||
|
||||
// Extract frames to canonical zero-padded names; pixiv ugoira
|
||||
// frames are uniformly jpg or png per artwork.
|
||||
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
|
||||
.map_err(|e| format!("unzip: {e}"))?;
|
||||
// frames are uniformly jpg or png per artwork. The zip is read
|
||||
// from disk; `zip_file` stays alive for the whole extraction.
|
||||
let mut archive = zip::ZipArchive::new(
|
||||
std::fs::File::open(zip_file.path()).map_err(|e| e.to_string())?,
|
||||
)
|
||||
.map_err(|e| format!("unzip: {e}"))?;
|
||||
if archive.is_empty() {
|
||||
return Err("empty frame zip".to_string());
|
||||
}
|
||||
@@ -392,16 +378,31 @@ mod tests {
|
||||
use super::*;
|
||||
use dotenv::dotenv;
|
||||
|
||||
/// Skips when `PIXIV_REFRESH_TOKEN` is absent or empty (CI without the
|
||||
/// secret must stay green; GitHub Actions exposes an unset secret as an
|
||||
/// empty string, so `is_err()` alone is not enough).
|
||||
fn require_pixiv_token() -> bool {
|
||||
std::env::var("PIXIV_REFRESH_TOKEN")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty())
|
||||
.is_some()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_fetch() {
|
||||
dotenv().ok();
|
||||
if !require_pixiv_token() {
|
||||
eprintln!("skipping: no PIXIV_REFRESH_TOKEN");
|
||||
return;
|
||||
}
|
||||
let result = fetch(126839080).await;
|
||||
assert!(result.is_ok());
|
||||
println!("{:#?}", result);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn validate_with_bogus_token_fails() {
|
||||
#[ignore = "live network: requires outbound HTTPS to oauth.secure.pixiv.net"]
|
||||
async fn live_validate_with_bogus_token_fails() {
|
||||
dotenv().ok();
|
||||
// A bogus token must surface as Api error (invalid_grant), not panic.
|
||||
let client = PixivAPI::new("bogus_token_for_testing".to_string());
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use super::model::{IllustrationModel, TypeModel};
|
||||
use crate::media::Media;
|
||||
use crate::site::{FetchError, Fetched};
|
||||
use crate::site::{FetchError, Fetched, PixivError, Site, SiteFuture};
|
||||
use html_escape::{encode_double_quoted_attribute, encode_text};
|
||||
use regex::Regex;
|
||||
use std::sync::LazyLock;
|
||||
@@ -13,6 +13,53 @@ pub fn enabled() -> bool {
|
||||
super::api::enabled()
|
||||
}
|
||||
|
||||
/// Registry entry for the pixiv adapter (see [`crate::site::Site`]).
|
||||
pub struct PixivSite;
|
||||
|
||||
impl Site for PixivSite {
|
||||
fn id(&self) -> &'static str {
|
||||
"pixiv"
|
||||
}
|
||||
|
||||
fn pattern(&self) -> &'static Regex {
|
||||
&PATTERN
|
||||
}
|
||||
|
||||
fn enabled(&self) -> bool {
|
||||
enabled()
|
||||
}
|
||||
|
||||
fn cache_key(&self, url: &str) -> Option<String> {
|
||||
cache_key(url)
|
||||
}
|
||||
|
||||
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched> {
|
||||
Box::pin(async move { fetch_from_url(url).await })
|
||||
}
|
||||
|
||||
fn is_retryable(&self, err: &FetchError) -> bool {
|
||||
is_retryable(err)
|
||||
}
|
||||
|
||||
fn media_headers(&self, url: &str) -> Option<Vec<(&'static str, String)>> {
|
||||
media_headers(url)
|
||||
}
|
||||
|
||||
fn validate(&self) -> SiteFuture<'static, (), String> {
|
||||
Box::pin(async {
|
||||
match super::api::validate().await {
|
||||
Ok(()) => Ok(()),
|
||||
Err(e) => {
|
||||
// Keep the old behavior: a failed login disables pixiv
|
||||
// for the rest of this process.
|
||||
super::api::disable();
|
||||
Err(format!("{e}"))
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
let id = PATTERN
|
||||
.captures(url)
|
||||
@@ -23,6 +70,43 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
Ok(super::api::fetch(id).await?.into())
|
||||
}
|
||||
|
||||
/// Cache key for a pixiv URL: `"pixiv:<id>"`. The prefix is the site id used
|
||||
/// for caption-format lookup and link-cache keys.
|
||||
pub fn cache_key(url: &str) -> Option<String> {
|
||||
PATTERN
|
||||
.captures(url)
|
||||
.map(|caps| format!("pixiv:{}", &caps[1]))
|
||||
}
|
||||
|
||||
/// Pixiv's fetch-retry policy: transient classes only — network errors and
|
||||
/// HTTP 429/5xx. Permanent 4xx (bad/expired token, forbidden, not found),
|
||||
/// API/auth errors, unparseable bodies and missing auth are not retried.
|
||||
pub fn is_retryable(err: &FetchError) -> bool {
|
||||
match err {
|
||||
FetchError::Http(_) | FetchError::Transient(_) => true,
|
||||
FetchError::Pixiv(e) => match e {
|
||||
PixivError::Http(_) => true,
|
||||
PixivError::Status(code) if *code == 429 || *code >= 500 => true,
|
||||
PixivError::Status(_)
|
||||
| PixivError::Api(_)
|
||||
| PixivError::Json(_)
|
||||
| PixivError::NoAuth => false,
|
||||
},
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// pximg.net is hotlink-protected: downloads must carry the pixiv Referer.
|
||||
/// The match is on the media host, not the site PATTERN — pixiv's PATTERN
|
||||
/// only matches `pixiv.net/artworks/...`, never `i.pximg.net`.
|
||||
pub fn media_headers(url: &str) -> Option<Vec<(&'static str, String)>> {
|
||||
if url.to_ascii_lowercase().contains("pximg.net") {
|
||||
Some(vec![("Referer", "https://www.pixiv.net/".to_string())])
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct Illustration {
|
||||
id: String,
|
||||
@@ -142,6 +226,7 @@ impl From<Illustration> for Fetched {
|
||||
title: illustration.title.clone(),
|
||||
media: illustration.media,
|
||||
sensitive: illustration.nsfw,
|
||||
site_id: "pixiv",
|
||||
render_data,
|
||||
_keep_alive: illustration._keep_alive,
|
||||
}
|
||||
@@ -232,6 +317,43 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_retryable_classifies_transient_and_permanent() {
|
||||
// Transient: network errors, explicit transient, pixiv 429/5xx.
|
||||
assert!(is_retryable(&FetchError::Transient("429".into())));
|
||||
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(429))));
|
||||
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(500))));
|
||||
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(503))));
|
||||
// Permanent: pixiv 4xx (bad/expired token, forbidden, not found),
|
||||
// api/auth errors, unparseable bodies, not-found/blocked/sensitive.
|
||||
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(400))));
|
||||
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(401))));
|
||||
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(403))));
|
||||
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(404))));
|
||||
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Api(
|
||||
"invalid_grant".into()
|
||||
))));
|
||||
assert!(!is_retryable(&FetchError::Pixiv(PixivError::NoAuth)));
|
||||
let json_err = serde_json::from_str::<serde_json::Value>("x").unwrap_err();
|
||||
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Json(
|
||||
json_err
|
||||
))));
|
||||
assert!(!is_retryable(&FetchError::NotFound));
|
||||
assert!(!is_retryable(&FetchError::Blocked));
|
||||
assert!(!is_retryable(&FetchError::Sensitive));
|
||||
assert!(!is_retryable(&FetchError::TooLarge));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn media_headers_adds_referer_only_for_pximg() {
|
||||
assert_eq!(
|
||||
media_headers("https://i.pximg.net/img-original/img/1.png"),
|
||||
Some(vec![("Referer", "https://www.pixiv.net/".to_string())])
|
||||
);
|
||||
assert_eq!(media_headers("https://www.pixiv.net/artworks/1"), None);
|
||||
assert_eq!(media_headers("https://x.com/u/status/1"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ugoira_yields_empty_media() {
|
||||
let v = illust_json(
|
||||
|
||||
@@ -3,4 +3,7 @@ mod interface;
|
||||
mod model;
|
||||
|
||||
pub use api::{PixivAPI, PixivError, disable, fetch, validate};
|
||||
pub use interface::{Illustration, PATTERN, enabled, fetch_from_url};
|
||||
pub use interface::{
|
||||
Illustration, PATTERN, PixivSite, cache_key, enabled, fetch_from_url, is_retryable,
|
||||
media_headers,
|
||||
};
|
||||
|
||||
@@ -1,10 +1,31 @@
|
||||
use super::model;
|
||||
use crate::media::Media;
|
||||
use crate::site::{FetchError, Fetched};
|
||||
use html_escape::{encode_double_quoted_attribute, encode_text};
|
||||
use crate::site::{FetchError, Fetched, Site, SiteFuture};
|
||||
use html_escape::{decode_html_entities, encode_double_quoted_attribute, encode_text};
|
||||
use regex::Regex;
|
||||
use std::sync::LazyLock;
|
||||
|
||||
/// Registry entry for the twitter adapter (see [`crate::site::Site`]).
|
||||
pub struct TwitterSite;
|
||||
|
||||
impl Site for TwitterSite {
|
||||
fn id(&self) -> &'static str {
|
||||
"twitter"
|
||||
}
|
||||
|
||||
fn pattern(&self) -> &'static Regex {
|
||||
&PATTERN
|
||||
}
|
||||
|
||||
fn cache_key(&self, url: &str) -> Option<String> {
|
||||
cache_key(url)
|
||||
}
|
||||
|
||||
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched> {
|
||||
Box::pin(async move { fetch_from_url(url).await })
|
||||
}
|
||||
}
|
||||
|
||||
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
|
||||
Regex::new(r"^(?:https?://)?(?:www\.|mobile\.)?(?:x|twitter|fixvx|vxtwitter|fixupx|fxtwitter)\.com/[^.]+/status/(\d+)").unwrap()
|
||||
});
|
||||
@@ -29,13 +50,19 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
if super::auth::enabled() {
|
||||
match super::auth::fetch(id).await {
|
||||
Ok(tweet) => Ok(tweet.into()),
|
||||
// The tweet is genuinely gone (deleted / suspended /
|
||||
// tombstoned): report it instead of degrading to an
|
||||
// empty result ("No media found"). Only unexpected
|
||||
// fallback failures (network, parse) keep the NSFW
|
||||
// placeholder.
|
||||
Err(FetchError::NotFound) => Err(FetchError::NotFound),
|
||||
Err(e) => {
|
||||
log::warn!("twitter auth fallback failed for {id}: {e}");
|
||||
Ok(empty_fetched(url))
|
||||
}
|
||||
}
|
||||
} else {
|
||||
log::info!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
|
||||
log::debug!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
|
||||
Ok(empty_fetched(url))
|
||||
}
|
||||
}
|
||||
@@ -43,6 +70,26 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Cache key for a twitter URL: `"twitter:<id>"`. The prefix is the site id
|
||||
/// used for caption-format lookup and link-cache keys.
|
||||
pub fn cache_key(url: &str) -> Option<String> {
|
||||
PATTERN
|
||||
.captures(url)
|
||||
.map(|caps| format!("twitter:{}", &caps[1]))
|
||||
}
|
||||
|
||||
/// Twitter's fetch-retry policy: transient classes only. Not-found, blocked,
|
||||
/// sensitive (NSFW withholding) and parse failures are permanent — retrying
|
||||
/// them only wastes attempts against the syndication endpoint.
|
||||
pub fn is_retryable(err: &FetchError) -> bool {
|
||||
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
|
||||
}
|
||||
|
||||
/// twimg URLs need no extra headers (no hotlink protection).
|
||||
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
|
||||
None
|
||||
}
|
||||
|
||||
/// A Fetched with no media for withheld tweets: the bot replies
|
||||
/// "No media found" and moves on instead of erroring.
|
||||
fn empty_fetched(url: &str) -> Fetched {
|
||||
@@ -54,13 +101,15 @@ fn empty_fetched(url: &str) -> Fetched {
|
||||
title: String::new(),
|
||||
media: vec![],
|
||||
sensitive: true,
|
||||
site_id: "twitter",
|
||||
render_data: None,
|
||||
_keep_alive: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Fetches a tweet from the syndication endpoint. Deleted/blocked tweets
|
||||
/// surface as `FetchError::NotFound`.
|
||||
/// surface as `FetchError::NotFound`; withheld content (empty tombstone,
|
||||
/// age-restricted) as `FetchError::Sensitive`.
|
||||
pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
|
||||
let id_num = id.parse::<u64>().map_err(|_| FetchError::NotFound)?;
|
||||
let response = crate::site::CLIENT
|
||||
@@ -79,23 +128,46 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
|
||||
};
|
||||
}
|
||||
let text = response.text().await?;
|
||||
// Deleted tweets answer with {"errors": [...]} instead of a tweet.
|
||||
if serde_json::from_str::<serde_json::Value>(&text)
|
||||
.map(|v| v.get("errors").is_some())
|
||||
.unwrap_or(false)
|
||||
{
|
||||
// Classify before parsing the tweet (see [`parse_syndication_body`]).
|
||||
parse_syndication_body(&text)?;
|
||||
Tweet::from_syndication_json(&text).map_err(FetchError::Json)
|
||||
}
|
||||
|
||||
/// Parses and classifies a syndication response body. `Ok` means the body is
|
||||
/// a real tweet payload; `Err` carries the permanent error class:
|
||||
/// - `NotFound`: an `errors` array (deleted/blocked) or a `TweetTombstone`
|
||||
/// **with a reason** — "This Post was deleted by the Post author." /
|
||||
/// "This Post is from a suspended account." (the tweet is gone).
|
||||
/// - `Sensitive`: content withheld **without a deletion reason** — the empty
|
||||
/// `{}` shape or an *empty* `TweetTombstone` (`{"__typename":
|
||||
/// "TweetTombstone","tombstone":{}}`). Live tweets in restricted contexts
|
||||
/// surface this way; treating them as deleted is a regression (a normal
|
||||
/// tweet must not report "deleted"). Age-restricted tombstones route here
|
||||
/// too so the logged-in GraphQL fallback can fetch the real tweet.
|
||||
/// - `Json`: an unparseable body.
|
||||
fn parse_syndication_body(text: &str) -> Result<serde_json::Value, FetchError> {
|
||||
let body: serde_json::Value = serde_json::from_str(text)?;
|
||||
if body.get("errors").is_some() {
|
||||
return Err(FetchError::NotFound);
|
||||
}
|
||||
// NSFW / age-restricted tweets exist but are served as an empty `{}` —
|
||||
// they surface as FetchError::Sensitive so the caller can retry as a
|
||||
// logged-in user.
|
||||
if serde_json::from_str::<serde_json::Value>(&text)
|
||||
.map(|v| v.get("id_str").is_none())
|
||||
.unwrap_or(false)
|
||||
{
|
||||
if let Some(tombstone) = body.get("tombstone") {
|
||||
// Only a tombstone with an explicit reason means the tweet is gone;
|
||||
// a missing reason (empty `tombstone: {}`) or an age-restricted
|
||||
// reason means the tweet exists but is withheld.
|
||||
let reason = tombstone
|
||||
.get("text")
|
||||
.and_then(|t| t.get("text"))
|
||||
.and_then(|t| t.as_str())
|
||||
.unwrap_or("");
|
||||
if reason.is_empty() || reason.to_ascii_lowercase().contains("age-restricted") {
|
||||
return Err(FetchError::Sensitive);
|
||||
}
|
||||
return Err(FetchError::NotFound);
|
||||
}
|
||||
if body.get("id_str").is_none() {
|
||||
return Err(FetchError::Sensitive);
|
||||
}
|
||||
Tweet::from_syndication_json(&text).map_err(FetchError::Json)
|
||||
Ok(body)
|
||||
}
|
||||
|
||||
/// The syndication token: JS `((id / 1e15) * PI).toString(36)` (the
|
||||
@@ -167,9 +239,17 @@ impl Tweet {
|
||||
// strip the appended media short link, mirroring FxEmbed's linkFixer
|
||||
// (no display_text_range arithmetic — see expand_links).
|
||||
let text = expand_links(&json.text, &json.entities.urls);
|
||||
// Twitter APIs (syndication AND GraphQL full_text) return the text
|
||||
// pre-escaped for HTML (`>` `<` `&` `'` …): decode it so
|
||||
// the stored text is raw. The caption's own escaping then produces
|
||||
// the rendered form exactly once — without this, `>^ω^<` would
|
||||
// be double-escaped to `&gt;^ω^&lt;` and the sent message
|
||||
// would show literal `>^ω^<`.
|
||||
let text = decode_html_entities(&text).into_owned();
|
||||
// `name` is the display name, `screen_name` the handle (Python's
|
||||
// vxtwitter mapping: author = display name, author_id = handle).
|
||||
let author = json.user.name;
|
||||
// Display names can carry the same pre-escaped entities.
|
||||
let author = decode_html_entities(&json.user.name).into_owned();
|
||||
let author_id = json.user.screen_name;
|
||||
let mut media = vec![];
|
||||
for item in json.media_details {
|
||||
@@ -285,6 +365,7 @@ impl From<Tweet> for Fetched {
|
||||
title: tweet.text.clone(),
|
||||
media: tweet.media,
|
||||
sensitive: tweet.sensitive,
|
||||
site_id: "twitter",
|
||||
render_data,
|
||||
_keep_alive: None,
|
||||
}
|
||||
@@ -336,6 +417,64 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_text_is_unescaped_before_storing() {
|
||||
// Real API shape: the text arrives pre-escaped for HTML — e.g. the
|
||||
// tweet `>^ω^<` comes back as `>^ω^<` (fxtwitter's raw_text for
|
||||
// 2060196388252827954) and apostrophes as `'`. Storing it raw and
|
||||
// escaping once at caption build avoids the double-escape that would
|
||||
// show literal `>`/`<`/`&` in the sent message.
|
||||
let raw = serde_json::json!({
|
||||
"__typename": "Tweet",
|
||||
"id_str": "1",
|
||||
"text": ">^ω^< & more 'quoted' https://t.co/abc123",
|
||||
"user": { "name": "O'Brien", "screen_name": "h" },
|
||||
"entities": { "urls": [] },
|
||||
"mediaDetails": []
|
||||
});
|
||||
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
|
||||
// The appended media short link is stripped, then entities decoded.
|
||||
assert_eq!(tweet.text, ">^ω^< & more 'quoted'");
|
||||
assert_eq!(tweet.author, "O'Brien");
|
||||
let fetched: Fetched = tweet.into();
|
||||
assert_eq!(fetched.title, ">^ω^< & more 'quoted'");
|
||||
// The caption escapes the raw text exactly once (encode_text covers
|
||||
// & < >; apostrophes stay literal — they are harmless in text).
|
||||
assert!(
|
||||
fetched.caption.contains(">^ω^< & more 'quoted'"),
|
||||
"caption: {}",
|
||||
fetched.caption
|
||||
);
|
||||
assert!(
|
||||
!fetched.caption.contains("&gt;"),
|
||||
"double-escaped text: {}",
|
||||
fetched.caption
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cache_key_prefixes_tweet_id() {
|
||||
assert_eq!(
|
||||
cache_key("https://x.com/user/status/1234567890"),
|
||||
Some("twitter:1234567890".into())
|
||||
);
|
||||
assert_eq!(cache_key("https://example.com/1"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_retryable_classifies_transient_and_permanent() {
|
||||
// Transient: network errors and explicit transient statuses (the
|
||||
// `Http` arm shares this match arm with `Transient`).
|
||||
assert!(is_retryable(&FetchError::Transient("429".into())));
|
||||
// Permanent: gone, blocked, withheld, oversized, unparseable.
|
||||
assert!(!is_retryable(&FetchError::NotFound));
|
||||
assert!(!is_retryable(&FetchError::Blocked));
|
||||
assert!(!is_retryable(&FetchError::Sensitive));
|
||||
assert!(!is_retryable(&FetchError::TooLarge));
|
||||
let json_err = serde_json::from_str::<serde_json::Value>("x").unwrap_err();
|
||||
assert!(!is_retryable(&FetchError::Json(json_err)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_json_converts_to_fetched() {
|
||||
let raw = fixture(serde_json::json!([
|
||||
@@ -572,19 +711,93 @@ mod tests {
|
||||
assert!(token.starts_with("236.v"), "got {token}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_tombstone_maps_to_not_found() {
|
||||
// Deleted tweets answer HTTP 200 with a TweetTombstone carrying a
|
||||
// reason (no `errors`, no `id_str`); they must not fall through to
|
||||
// Sensitive, which would make the bot reply "No media found" for a
|
||||
// deleted tweet.
|
||||
let raw = serde_json::json!({
|
||||
"__typename": "TweetTombstone",
|
||||
"tombstone": {
|
||||
"text": { "rtl": false, "text": "This Post was deleted by the Post author. Learn more" }
|
||||
}
|
||||
});
|
||||
assert!(matches!(
|
||||
parse_syndication_body(&raw.to_string()),
|
||||
Err(FetchError::NotFound)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_empty_tombstone_maps_to_sensitive() {
|
||||
// Regression: live tweets in restricted contexts answer with an
|
||||
// EMPTY tombstone (`{"__typename":"TweetTombstone","tombstone":{}}`)
|
||||
// — no deletion reason. They must not be reported as deleted.
|
||||
let raw = serde_json::json!({ "__typename": "TweetTombstone", "tombstone": {} });
|
||||
assert!(matches!(
|
||||
parse_syndication_body(&raw.to_string()),
|
||||
Err(FetchError::Sensitive)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_age_restricted_tombstone_maps_to_sensitive() {
|
||||
// An age-restricted tombstone withholds a live tweet; route it to
|
||||
// the logged-in fallback instead of reporting it as gone.
|
||||
let raw = serde_json::json!({
|
||||
"__typename": "TweetTombstone",
|
||||
"tombstone": {
|
||||
"text": { "rtl": false, "text": "Age-restricted adult content" }
|
||||
}
|
||||
});
|
||||
assert!(matches!(
|
||||
parse_syndication_body(&raw.to_string()),
|
||||
Err(FetchError::Sensitive)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_errors_maps_to_not_found() {
|
||||
// The classic gone shape: {"errors": [...]}.
|
||||
let raw = serde_json::json!({ "errors": [{ "message": "Couldn't find Tweet" }] });
|
||||
assert!(matches!(
|
||||
parse_syndication_body(&raw.to_string()),
|
||||
Err(FetchError::NotFound)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_empty_object_maps_to_sensitive() {
|
||||
// NSFW / age-restricted withholding: an empty `{}`.
|
||||
assert!(matches!(
|
||||
parse_syndication_body("{}"),
|
||||
Err(FetchError::Sensitive)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_tweet_body_passes() {
|
||||
let raw = fixture(serde_json::json!([]));
|
||||
assert!(parse_syndication_body(&raw.to_string()).is_ok());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
|
||||
async fn live_fetch_with_photos() {
|
||||
let fetched = fetch("861627479294746624").await.unwrap();
|
||||
assert_eq!(fetched.media.len(), 4);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
|
||||
async fn live_fetch_text_only() {
|
||||
let fetched = fetch("1992471125734142256").await.unwrap();
|
||||
assert!(fetched.media.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
|
||||
async fn live_fetch_deleted_tweet_is_not_found() {
|
||||
// Deleted tweet: the syndication endpoint answers with errors.
|
||||
let result = fetch("0").await;
|
||||
@@ -593,4 +806,30 @@ mod tests {
|
||||
"got {result:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
|
||||
async fn live_fetch_tombstone_deleted_tweet_is_not_found() {
|
||||
// Regression: a real deleted tweet answering with a TweetTombstone
|
||||
// (HTTP 200, no errors/id_str) used to surface as Sensitive and
|
||||
// degrade to an empty result ("No media found").
|
||||
let result = fetch("2085948045967986859").await;
|
||||
assert!(
|
||||
matches!(result, Err(FetchError::NotFound)),
|
||||
"got {result:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
|
||||
async fn live_fetch_empty_tombstone_is_sensitive() {
|
||||
// Regression: a LIVE tweet (verified via a third-party API) answers
|
||||
// syndication with an empty TweetTombstone; it must surface as
|
||||
// Sensitive (withheld), never as NotFound (deleted).
|
||||
let result = fetch("2087851366253555752").await;
|
||||
assert!(
|
||||
matches!(result, Err(FetchError::Sensitive)),
|
||||
"got {result:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,4 +2,6 @@ mod auth;
|
||||
mod interface;
|
||||
mod model;
|
||||
|
||||
pub use interface::{PATTERN, Tweet, enabled, fetch_from_url};
|
||||
pub use interface::{
|
||||
PATTERN, Tweet, TwitterSite, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
|
||||
};
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
[package]
|
||||
name = "xmedia-bot"
|
||||
version = "1.1.0"
|
||||
version = "1.4.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
teloxide = { version = "0.17", features = ["webhooks-axum", "macros"] }
|
||||
tokio = { version = "1.40", features = ["rt-multi-thread", "macros"] }
|
||||
teloxide = { version = "0.17", default-features = false, features = ["webhooks-axum", "macros", "rustls", "ctrlc_handler"] }
|
||||
tokio = { version = "1.40", features = ["rt-multi-thread", "macros", "time"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
log = "0.4"
|
||||
@@ -17,8 +17,12 @@ rusqlite = { version = "0.32", features = ["bundled"] }
|
||||
rand = "0.8"
|
||||
tempfile = "3"
|
||||
parking_lot = "0.12"
|
||||
bytes = "1"
|
||||
png = "0.18"
|
||||
zune-jpeg = "0.5"
|
||||
fast_image_resize = "6"
|
||||
jpeg-encoder = "0.7"
|
||||
x-media = { path = "../x-media" }
|
||||
|
||||
[dev-dependencies]
|
||||
tokio = { version = "1.40", features = ["test-util"] }
|
||||
|
||||
@@ -23,32 +23,64 @@ pub struct Config {
|
||||
|
||||
impl Config {
|
||||
pub fn load() -> Config {
|
||||
let admin_ids = env::var("BOT_ADMIN")
|
||||
.ok()
|
||||
.map(|s| {
|
||||
s.split(',')
|
||||
.filter_map(|part| part.trim().parse::<i64>().ok())
|
||||
// Fail-fast helpers: a misspelled value must not silently fall back
|
||||
// to a default and run with different behavior than the operator
|
||||
// intended — log a loud warning naming the variable instead.
|
||||
fn parse_u64(name: &str, default: u64) -> u64 {
|
||||
match env::var(name) {
|
||||
Ok(v) => v.parse::<u64>().unwrap_or_else(|_| {
|
||||
log::warn!("invalid {name}={v:?}; using default {default}");
|
||||
default
|
||||
}),
|
||||
Err(_) => default,
|
||||
}
|
||||
}
|
||||
|
||||
let admin_ids = match env::var("BOT_ADMIN") {
|
||||
Ok(s) => {
|
||||
let (ids, bad): (Vec<_>, Vec<_>) = s
|
||||
.split(',')
|
||||
.map(str::trim)
|
||||
.filter(|part| !part.is_empty())
|
||||
.partition(|part| part.parse::<i64>().is_ok());
|
||||
if !bad.is_empty() {
|
||||
log::warn!("BOT_ADMIN: ignoring non-numeric ids: {bad:?}");
|
||||
}
|
||||
ids.into_iter()
|
||||
.filter_map(|p| p.parse::<i64>().ok())
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
}
|
||||
Err(_) => Vec::new(),
|
||||
};
|
||||
|
||||
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.map(Duration::from_secs)
|
||||
.unwrap_or(Duration::from_secs(86400));
|
||||
|
||||
let link_cache_ttl = env::var("LINK_CACHE_TTL_SECONDS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.map(Duration::from_secs)
|
||||
.unwrap_or(Duration::from_secs(7 * 24 * 3600));
|
||||
let edit_message_ttl =
|
||||
Duration::from_secs(parse_u64("EDIT_MESSAGE_TTL_SECONDS", 24 * 3600));
|
||||
let link_cache_ttl =
|
||||
Duration::from_secs(parse_u64("LINK_CACHE_TTL_SECONDS", 7 * 24 * 3600));
|
||||
|
||||
let webhook_enabled = env::var("WEBHOOK")
|
||||
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
|
||||
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
|
||||
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
|
||||
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
|
||||
// The webhook settings are consumed by `.expect()` in main when
|
||||
// WEBHOOK=true, so an unparseable value fails fast at startup with a
|
||||
// clear message; still log here for the WEBHOOK=false case.
|
||||
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| {
|
||||
s.parse::<url::Url>().ok().or_else(|| {
|
||||
log::warn!("invalid WEBHOOK_URL={s:?}");
|
||||
None
|
||||
})
|
||||
});
|
||||
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| {
|
||||
s.parse::<IpAddr>().ok().or_else(|| {
|
||||
log::warn!("invalid WEBHOOK_LISTEN={s:?}");
|
||||
None
|
||||
})
|
||||
});
|
||||
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| {
|
||||
s.parse::<u16>().ok().or_else(|| {
|
||||
log::warn!("invalid WEBHOOK_PORT={s:?}");
|
||||
None
|
||||
})
|
||||
});
|
||||
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
|
||||
// value that would otherwise come from `.env`).
|
||||
let webhook_cert = env::var("WEBHOOK_CERT").ok().filter(|s| !s.is_empty());
|
||||
|
||||
+135
-23
@@ -2,16 +2,106 @@
|
||||
//! (`tasks` in queue.rs, `chat_state` in state.rs, `link_cache` in
|
||||
//! link_cache.rs).
|
||||
//!
|
||||
//! Every operation opens its own short-lived connection with a busy timeout:
|
||||
//! handler tasks enqueue while workers lease/update rows concurrently, and
|
||||
//! without the timeout a concurrent write fails immediately with SQLITE_BUSY
|
||||
//! and the operation is lost. All I/O runs inside `spawn_blocking` via
|
||||
//! [`with_conn`] — rusqlite connections are not Send-friendly to hold across
|
||||
//! an await point, and blocking the async executor stalls every handler.
|
||||
//! All I/O runs inside `spawn_blocking` via [`DbPool::with_conn`] — rusqlite
|
||||
//! connections are not Send-friendly to hold across an await point, and
|
||||
//! blocking the async executor stalls every handler. Connections are reused
|
||||
//! through a small per-store pool instead of opening a fresh connection per
|
||||
//! operation: WAL lets readers run alongside writer leases, and the pool's
|
||||
//! semaphore bounds how many DB operations run concurrently, giving natural
|
||||
//! backpressure on hot paths (every message / URL / callback touches
|
||||
//! chat_state or the link cache).
|
||||
|
||||
use parking_lot::Mutex;
|
||||
use rusqlite::Connection;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
/// Upper bound on pooled (reused) connections and on concurrent DB
|
||||
/// operations per store. Small on purpose: the queue's `BEGIN IMMEDIATE`
|
||||
/// leases serialize writes anyway, and WAL readers rarely need more.
|
||||
const POOL_SIZE: usize = 4;
|
||||
|
||||
/// A tiny connection pool for one SQLite file. Connections are checked out
|
||||
/// on a blocking thread and returned afterwards; `acquire` opens a new
|
||||
/// connection only when the idle list is empty, so the steady-state cost of
|
||||
/// an operation is a list pop instead of a fresh open (+ busy timeout + WAL
|
||||
/// pragma). The semaphore caps the number of concurrent operations, so a
|
||||
/// burst of handlers queues up instead of opening unbounded connections.
|
||||
pub struct DbPool {
|
||||
// Arc so [`DbPool::with_conn`] can hand an owned handle to
|
||||
// `spawn_blocking` without borrowing across the await point.
|
||||
inner: Arc<PoolInner>,
|
||||
}
|
||||
|
||||
struct PoolInner {
|
||||
path: String,
|
||||
permits: tokio::sync::Semaphore,
|
||||
idle: Mutex<Vec<Connection>>,
|
||||
}
|
||||
|
||||
impl DbPool {
|
||||
pub fn new(path: &str) -> Self {
|
||||
DbPool {
|
||||
inner: Arc::new(PoolInner {
|
||||
path: path.to_string(),
|
||||
permits: tokio::sync::Semaphore::new(POOL_SIZE),
|
||||
idle: Mutex::new(Vec::new()),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs `f` against a pooled connection on a blocking thread, returning
|
||||
/// the closure's result. Owns the semaphore + `spawn_blocking` +
|
||||
/// `expect` ceremony shared by every table access; the caller maps
|
||||
/// errors to its own log line.
|
||||
pub async fn with_conn<T, F>(&self, f: F) -> rusqlite::Result<T>
|
||||
where
|
||||
T: Send + 'static,
|
||||
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
|
||||
{
|
||||
let _permit = self
|
||||
.inner
|
||||
.permits
|
||||
.acquire()
|
||||
.await
|
||||
.expect("db pool semaphore closed");
|
||||
let inner = Arc::clone(&self.inner);
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let mut conn = inner.acquire()?;
|
||||
let result = f(&mut conn);
|
||||
inner.release(conn);
|
||||
result
|
||||
})
|
||||
.await
|
||||
.expect("db worker panicked")
|
||||
}
|
||||
|
||||
/// The database file this pool serves (used by tests that need a raw
|
||||
/// connection, e.g. to seed rows directly).
|
||||
#[cfg(test)]
|
||||
pub fn path(&self) -> &str {
|
||||
&self.inner.path
|
||||
}
|
||||
}
|
||||
|
||||
impl PoolInner {
|
||||
/// Reuses an idle connection or opens a fresh one.
|
||||
fn acquire(&self) -> rusqlite::Result<Connection> {
|
||||
if let Some(conn) = self.idle.lock().pop() {
|
||||
return Ok(conn);
|
||||
}
|
||||
open_db(&self.path)
|
||||
}
|
||||
|
||||
/// Returns a connection to the pool (dropped when the pool is full).
|
||||
fn release(&self, conn: Connection) {
|
||||
let mut idle = self.idle.lock();
|
||||
if idle.len() < POOL_SIZE {
|
||||
idle.push(conn);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Opens the shared DB with a busy timeout.
|
||||
pub fn open_db(path: &str) -> rusqlite::Result<Connection> {
|
||||
let conn = Connection::open(path)?;
|
||||
@@ -23,6 +113,45 @@ pub fn open_db(path: &str) -> rusqlite::Result<Connection> {
|
||||
Ok(conn)
|
||||
}
|
||||
|
||||
/// Opens the shared DB file, runs the merged schema for all three tables and
|
||||
/// returns a pool for it. One call per process in production (the stores
|
||||
/// share the returned pool); tests call it per tempdir.
|
||||
pub fn open_store(path: &str) -> rusqlite::Result<Arc<DbPool>> {
|
||||
if let Some(parent) = std::path::Path::new(path).parent()
|
||||
&& !parent.as_os_str().is_empty()
|
||||
{
|
||||
std::fs::create_dir_all(parent).map_err(rusqlite_error)?;
|
||||
}
|
||||
let conn = open_db(path)?;
|
||||
schema_init(&conn)?;
|
||||
Ok(Arc::new(DbPool::new(path)))
|
||||
}
|
||||
|
||||
fn rusqlite_error(e: std::io::Error) -> rusqlite::Error {
|
||||
rusqlite::Error::ToSqlConversionFailure(Box::new(e))
|
||||
}
|
||||
|
||||
/// Creates the `tasks`, `chat_state` and `link_cache` tables (idempotent).
|
||||
/// The three stores used to own their own schema; keeping it in one place
|
||||
/// means one initialization for the whole database file.
|
||||
///
|
||||
/// ⚠️ Schema-change reminder (deferred, see `docs/architecture-refactor.md`
|
||||
/// §5): this is a plain `CREATE TABLE IF NOT EXISTS` with no versioning.
|
||||
/// Before any column/table change that must migrate existing databases, land
|
||||
/// the `PRAGMA user_version` migration chain first (`MIGRATIONS: &[&str]` +
|
||||
/// `migrate(conn)`), then restructure this function.
|
||||
pub fn schema_init(conn: &Connection) -> rusqlite::Result<()> {
|
||||
conn.execute_batch(
|
||||
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
|
||||
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
|
||||
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
|
||||
CREATE INDEX IF NOT EXISTS idx_tasks_pending ON tasks(status, run_after); \
|
||||
CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL); \
|
||||
CREATE TABLE IF NOT EXISTS link_cache (url TEXT PRIMARY KEY, payload TEXT NOT NULL, \
|
||||
created_at REAL NOT NULL);",
|
||||
)
|
||||
}
|
||||
|
||||
/// Unix timestamp in fractional seconds. Shared by the queue, chat store and
|
||||
/// link cache (previously four private copies).
|
||||
pub fn now_f64() -> f64 {
|
||||
@@ -31,20 +160,3 @@ pub fn now_f64() -> f64 {
|
||||
.map(|d| d.as_secs_f64())
|
||||
.unwrap_or(0.0)
|
||||
}
|
||||
|
||||
/// Runs `f` against a fresh connection on a blocking thread, returning the
|
||||
/// closure's result. Owns the `spawn_blocking` + `expect` ceremony shared by
|
||||
/// every table access; the caller maps errors to its own log line.
|
||||
pub async fn with_conn<T, F>(path: &str, f: F) -> rusqlite::Result<T>
|
||||
where
|
||||
T: Send + 'static,
|
||||
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
|
||||
{
|
||||
let path = path.to_string();
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let mut conn = open_db(&path)?;
|
||||
f(&mut conn)
|
||||
})
|
||||
.await
|
||||
.expect("db worker panicked")
|
||||
}
|
||||
|
||||
@@ -1,952 +0,0 @@
|
||||
use crate::config::Config;
|
||||
use crate::db::now_f64;
|
||||
use crate::link_cache::{CachedMediaKind, CachedPost, LinkCache};
|
||||
use crate::queue::PersistentTaskQueue;
|
||||
use crate::send::{self, MediaItemPayload, Task};
|
||||
use crate::state::{ChatData, ChatStore, unix_now};
|
||||
use std::collections::HashSet;
|
||||
use std::sync::LazyLock;
|
||||
use teloxide::RequestError;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::types::{
|
||||
CallbackQuery, ChatAction, ChatId, ChatKind, InlineQuery, InlineQueryResult,
|
||||
InlineQueryResultMpeg4Gif, InlineQueryResultPhoto, InlineQueryResultVideo, Message,
|
||||
MessageEntityKind, MessageId, ParseMode, Recipient, ReplyParameters,
|
||||
};
|
||||
use teloxide::utils::command::BotCommands;
|
||||
use x_media::media::Media;
|
||||
|
||||
/// One URL job: bot handle + the message + the extracted URL.
|
||||
type UrlJob = (Bot, Message, String);
|
||||
/// Bounded channel of URL jobs drained by [`start_url_workers`]. The bound
|
||||
/// caps both queued memory and shutdown backlog; a full channel applies
|
||||
/// backpressure to the per-chat handler instead of spawning unbounded tasks.
|
||||
static URL_JOBS: LazyLock<parking_lot::Mutex<Option<tokio::sync::mpsc::Sender<UrlJob>>>> =
|
||||
LazyLock::new(|| parking_lot::Mutex::new(None));
|
||||
/// Set by main's shutdown sequence; workers stop pulling new jobs.
|
||||
static URL_STOP: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
|
||||
|
||||
/// Worker count draining URL jobs; keeps the old 8-permit concurrency cap
|
||||
/// while bounding how many jobs can be queued at all.
|
||||
const URL_WORKERS: usize = 8;
|
||||
|
||||
/// Starts the URL job workers (called once from main after the queue starts).
|
||||
/// teloxide dispatches updates to a per-chat worker that handles them
|
||||
/// sequentially, so a batch-forward of many messages would otherwise be
|
||||
/// processed one at a time (fetch + send each, roughly a second per
|
||||
/// message); the workers add throughput, and FIFO order preserves per-message
|
||||
/// URL order.
|
||||
pub async fn start_url_workers() {
|
||||
let (tx, rx) = tokio::sync::mpsc::channel::<UrlJob>(256);
|
||||
*URL_JOBS.lock() = Some(tx);
|
||||
let rx = std::sync::Arc::new(tokio::sync::Mutex::new(rx));
|
||||
for _ in 0..URL_WORKERS {
|
||||
let rx = std::sync::Arc::clone(&rx);
|
||||
tokio::spawn(async move {
|
||||
while !URL_STOP.load(std::sync::atomic::Ordering::Relaxed) {
|
||||
let job = rx.lock().await.recv().await;
|
||||
match job {
|
||||
Some((bot, message, url)) => url_media(bot, &message, &url).await,
|
||||
None => break,
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// Stops URL workers (drains up to the 256 queued jobs, then exits).
|
||||
pub fn stop_url_workers() {
|
||||
URL_STOP.store(true, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub static CHAT_STORE: LazyLock<ChatStore> =
|
||||
LazyLock::new(|| ChatStore::open("data/task_queue.db").expect("failed to open chat store"));
|
||||
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
|
||||
LazyLock::new(|| PersistentTaskQueue::new("data/task_queue.db"));
|
||||
pub static LINK_CACHE: LazyLock<LinkCache> =
|
||||
LazyLock::new(|| LinkCache::open("data/task_queue.db"));
|
||||
pub static CONFIG: LazyLock<Config> = LazyLock::new(Config::load);
|
||||
|
||||
#[derive(BotCommands, Clone)]
|
||||
#[command(
|
||||
rename_rule = "snake_case",
|
||||
description = "Turn X/Pixiv/Bluesky links into media messages"
|
||||
)]
|
||||
enum Command {
|
||||
#[command(description = "Get started")]
|
||||
Start,
|
||||
#[command(description = "Show command help")]
|
||||
Help,
|
||||
#[command(
|
||||
description = "Set forward channel (@channel or ID)",
|
||||
parse_with = "split"
|
||||
)]
|
||||
SetForwardChannel(String),
|
||||
#[command(description = "Remove forward channel")]
|
||||
RemoveForwardChannel,
|
||||
#[command(description = "Toggle edit-before-forward")]
|
||||
EditBeforeForward,
|
||||
#[command(
|
||||
description = "Reply with [] to save as template",
|
||||
parse_with = "split"
|
||||
)]
|
||||
SetTemplate(String),
|
||||
#[command(description = "Show chat state (debug)")]
|
||||
BotDict,
|
||||
#[command(description = "Set site caption format", parse_with = "split")]
|
||||
SetFormat(String),
|
||||
#[command(
|
||||
description = "Clear link cache (admin; optional URL, else all)",
|
||||
parse_with = "split"
|
||||
)]
|
||||
ClearCache(String),
|
||||
}
|
||||
|
||||
async fn reply<T>(bot: Bot, message: Message, text: T) -> Result<Message, RequestError>
|
||||
where
|
||||
T: Into<String>,
|
||||
{
|
||||
bot.send_message(message.chat.id, text)
|
||||
.reply_parameters(ReplyParameters::new(message.id).allow_sending_without_reply())
|
||||
.await
|
||||
}
|
||||
|
||||
/// Extracts URL and text-link entities (text + caption), deduped in order.
|
||||
pub fn extract_urls(message: &Message) -> Vec<String> {
|
||||
let mut urls = Vec::new();
|
||||
for entity in message.parse_entities().into_iter().flatten() {
|
||||
match entity.kind() {
|
||||
MessageEntityKind::Url => urls.push(entity.text().to_string()),
|
||||
MessageEntityKind::TextLink { url } => urls.push(url.to_string()),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
for entity in message.parse_caption_entities().into_iter().flatten() {
|
||||
match entity.kind() {
|
||||
MessageEntityKind::Url => urls.push(entity.text().to_string()),
|
||||
MessageEntityKind::TextLink { url } => urls.push(url.to_string()),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
let mut seen = HashSet::new();
|
||||
// Dedup by the normalized post id so variant URLs of the same post
|
||||
// (/status/1 vs /status/1/photo/1) are sent once; unsupported URLs fall
|
||||
// back to exact-string dedup.
|
||||
urls.retain(|url| seen.insert(x_media::site::cache_key(url).unwrap_or_else(|| url.clone())));
|
||||
urls
|
||||
}
|
||||
|
||||
/// Edit-before-forward: a reply to the prompt swaps the caption of the first
|
||||
/// forwarded message. Returns true when the message was consumed as an edit.
|
||||
async fn edit_message_handler(bot: &Bot, message: &Message) -> bool {
|
||||
let Some(reply) = message.reply_to_message() else {
|
||||
return false;
|
||||
};
|
||||
let chat_id = message.chat.id.0;
|
||||
let Some(text) = message.text() else {
|
||||
return false;
|
||||
};
|
||||
let chat_data = CHAT_STORE.get(chat_id).await;
|
||||
let Some(edit) = chat_data.edit_message.get(&(reply.id.0 as i64)) else {
|
||||
return false;
|
||||
};
|
||||
let Some(first_forward_id) = edit.forward_message_ids.first() else {
|
||||
return false;
|
||||
};
|
||||
let link = format!(
|
||||
"<a href=\"{0}\">{1}</a>",
|
||||
html_escape::encode_double_quoted_attribute(&edit.url),
|
||||
html_escape::encode_text(text)
|
||||
);
|
||||
let new_text = if edit.template.is_empty() {
|
||||
link
|
||||
} else {
|
||||
chat_data
|
||||
.template
|
||||
.get(&edit.template)
|
||||
.map(|template| template.replace("[]", &link))
|
||||
.unwrap_or(link)
|
||||
};
|
||||
let result = bot
|
||||
.edit_message_caption(ChatId(chat_id), MessageId(*first_forward_id as i32))
|
||||
.caption(new_text)
|
||||
.parse_mode(ParseMode::Html)
|
||||
.await;
|
||||
match result {
|
||||
Ok(_) => log::info!(
|
||||
"edit-before-forward: caption swapped on message {first_forward_id} for prompt {}",
|
||||
reply.id.0
|
||||
),
|
||||
Err(e) => log::error!("edit_message_caption failed: {e}"),
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
enum SetForwardChannelError {
|
||||
EmptyParameter,
|
||||
NotChannel,
|
||||
NotAdmin,
|
||||
NotBotAdmin(RequestError),
|
||||
NotBotCanPost,
|
||||
}
|
||||
|
||||
async fn set_forward_channel_handler(
|
||||
bot: &Bot,
|
||||
message: &Message,
|
||||
channel: String,
|
||||
) -> Result<i64, SetForwardChannelError> {
|
||||
if channel.is_empty() {
|
||||
return Err(SetForwardChannelError::EmptyParameter);
|
||||
}
|
||||
let channel = match channel.parse::<i64>() {
|
||||
Ok(id) => Recipient::Id(ChatId(id)),
|
||||
Err(_) => Recipient::ChannelUsername(channel),
|
||||
};
|
||||
if let Some(from) = &message.from {
|
||||
log::info!(
|
||||
"Set forward channel for {} ({}) to {}",
|
||||
from.full_name(),
|
||||
message.chat.id,
|
||||
channel
|
||||
);
|
||||
}
|
||||
let chat = match bot.get_chat(channel.clone()).await {
|
||||
Err(e) => {
|
||||
log::error!("Failed to get channel {}: {}", channel, e);
|
||||
return Err(SetForwardChannelError::NotBotAdmin(e));
|
||||
}
|
||||
Ok(chat) => chat,
|
||||
};
|
||||
if !chat.is_channel() {
|
||||
return Err(SetForwardChannelError::NotChannel);
|
||||
}
|
||||
let channel_id = chat.id.0;
|
||||
// The sender must be a channel administrator. Compare against the
|
||||
// sender's user id, NOT the chat id (they only coincide in private
|
||||
// chats, so the old check broke group usage).
|
||||
let Some(sender) = message.from.as_ref() else {
|
||||
return Err(SetForwardChannelError::NotAdmin);
|
||||
};
|
||||
match bot.get_chat_administrators(channel.clone()).await {
|
||||
Err(e) => {
|
||||
log::error!("Failed to get channel administrators {}: {}", channel, e);
|
||||
return Err(SetForwardChannelError::NotBotAdmin(e));
|
||||
}
|
||||
Ok(admins) => {
|
||||
if !admins.iter().any(|admin| admin.user.id == sender.id) {
|
||||
return Err(SetForwardChannelError::NotAdmin);
|
||||
}
|
||||
// The bot itself must be an admin that can post; a missing
|
||||
// bot entry must not pass silently (copy would fail later).
|
||||
let bot_id = match bot.get_me().await {
|
||||
Ok(me) => me.user.id,
|
||||
Err(e) => return Err(SetForwardChannelError::NotBotAdmin(e)),
|
||||
};
|
||||
let bot_ok = admins
|
||||
.iter()
|
||||
.any(|admin| admin.user.id == bot_id && admin.can_post_messages());
|
||||
if !bot_ok {
|
||||
return Err(SetForwardChannelError::NotBotCanPost);
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(channel_id)
|
||||
}
|
||||
|
||||
async fn execute_command(
|
||||
bot: &Bot,
|
||||
message: &Message,
|
||||
command: Command,
|
||||
) -> Result<(), RequestError> {
|
||||
match command {
|
||||
Command::Start => {
|
||||
bot.send_message(message.chat.id, "Hello!").await?;
|
||||
}
|
||||
Command::Help => {
|
||||
bot.send_message(message.chat.id, Command::descriptions().to_string())
|
||||
.await?;
|
||||
}
|
||||
Command::SetForwardChannel(channel) => {
|
||||
let result = match set_forward_channel_handler(bot, message, channel).await {
|
||||
Ok(channel_id) => {
|
||||
CHAT_STORE
|
||||
.update(message.chat.id.0, |data| {
|
||||
data.forward_channel_id = Some(channel_id);
|
||||
})
|
||||
.await;
|
||||
"Add successfully.".to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::EmptyParameter) => {
|
||||
"Receive empty parameter.\nYou should enter a channel id or username"
|
||||
.to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::NotChannel) => {
|
||||
"Given id / username is not a channel".to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::NotAdmin) => {
|
||||
"You are not an administrator of the channel".to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::NotBotAdmin(e)) => {
|
||||
e.to_string() + "\nPlease add the bot as an admin to the channel"
|
||||
}
|
||||
Err(SetForwardChannelError::NotBotCanPost) => {
|
||||
"Bot can't post messages to the channel".to_string()
|
||||
}
|
||||
};
|
||||
reply(bot.clone(), message.clone(), result).await?;
|
||||
}
|
||||
Command::RemoveForwardChannel => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let text = CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
if data.forward_channel_id.is_some() {
|
||||
data.forward_channel_id = None;
|
||||
"Remove successfully.".to_string()
|
||||
} else {
|
||||
"No channel to remove.".to_string()
|
||||
}
|
||||
})
|
||||
.await;
|
||||
reply(bot.clone(), message.clone(), text).await?;
|
||||
}
|
||||
Command::EditBeforeForward => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let text = CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
if data.forward_channel_id.is_none() {
|
||||
"Please enable forward channel first.".to_string()
|
||||
} else if data.edit_before_forward {
|
||||
data.edit_before_forward = false;
|
||||
data.edit_message.clear();
|
||||
"Disable edit before forward.".to_string()
|
||||
} else {
|
||||
data.edit_before_forward = true;
|
||||
"Enable edit before forward.".to_string()
|
||||
}
|
||||
})
|
||||
.await;
|
||||
reply(bot.clone(), message.clone(), text).await?;
|
||||
}
|
||||
Command::SetTemplate(name) => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let text = match message.reply_to_message() {
|
||||
None => "Please reply to a message to set as template.".to_string(),
|
||||
Some(reply) => {
|
||||
let reply_text = reply.text().unwrap_or_default();
|
||||
if !reply_text.contains("[]") {
|
||||
"Please reply to a message with [] to set as template.".to_string()
|
||||
} else if name.is_empty() {
|
||||
"Please provide a name for the template.".to_string()
|
||||
} else {
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
data.template.insert(
|
||||
name,
|
||||
html_escape::encode_text(reply_text).into_owned(),
|
||||
);
|
||||
})
|
||||
.await;
|
||||
"Template set.".to_string()
|
||||
}
|
||||
}
|
||||
};
|
||||
reply(bot.clone(), message.clone(), text).await?;
|
||||
}
|
||||
Command::BotDict => {
|
||||
let chat_data = CHAT_STORE.get(message.chat.id.0).await;
|
||||
let debug = format!("{chat_data:?}");
|
||||
let text = html_escape::encode_text(&debug).into_owned();
|
||||
reply(bot.clone(), message.clone(), text).await?;
|
||||
}
|
||||
Command::SetFormat(arg) => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let (site, format) = match arg.split_once(char::is_whitespace) {
|
||||
Some((site, format)) if !format.trim().is_empty() => {
|
||||
(site.trim(), format.trim().to_string())
|
||||
}
|
||||
_ => {
|
||||
reply(
|
||||
bot.clone(),
|
||||
message.clone(),
|
||||
"Usage: /set_format <site> <format>",
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
if !["twitter", "bsky", "pixiv"].contains(&site) {
|
||||
reply(
|
||||
bot.clone(),
|
||||
message.clone(),
|
||||
"Unknown site. Use twitter, bsky or pixiv.",
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
data.message_format.insert(site.to_string(), format);
|
||||
})
|
||||
.await;
|
||||
reply(bot.clone(), message.clone(), "Format set.").await?;
|
||||
}
|
||||
Command::ClearCache(arg) => {
|
||||
let sender_id = message
|
||||
.from
|
||||
.as_ref()
|
||||
.map(|user| user.id.0 as i64)
|
||||
.unwrap_or(-1);
|
||||
if !CONFIG.admin_ids.contains(&sender_id) {
|
||||
reply(bot.clone(), message.clone(), "Admin only.").await?;
|
||||
return Ok(());
|
||||
}
|
||||
let arg = arg.trim();
|
||||
if arg.is_empty() {
|
||||
let removed = LINK_CACHE.clear(None).await;
|
||||
log::info!("cache cleared by {sender_id}: {removed} entries");
|
||||
reply(
|
||||
bot.clone(),
|
||||
message.clone(),
|
||||
format!("Cleared {removed} cached entr{}.", plural(removed)),
|
||||
)
|
||||
.await?;
|
||||
} else {
|
||||
let key = match x_media::site::cache_key(arg) {
|
||||
Some(key) => key,
|
||||
None => {
|
||||
reply(
|
||||
bot.clone(),
|
||||
message.clone(),
|
||||
"Unrecognized link. Use a twitter/x, pixiv or bsky post URL.",
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
let removed = LINK_CACHE.clear(Some(&key)).await;
|
||||
log::info!("cache entry cleared by {sender_id}: {key} ({removed} rows)");
|
||||
reply(
|
||||
bot.clone(),
|
||||
message.clone(),
|
||||
format!(
|
||||
"Cleared cache for {arg} ({} entr{}).",
|
||||
removed,
|
||||
plural(removed)
|
||||
),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `"y"` for one, `"ies"` for anything else — "1 entry" / "2 entries".
|
||||
fn plural(n: usize) -> &'static str {
|
||||
if n == 1 { "y" } else { "ies" }
|
||||
}
|
||||
|
||||
/// Registers the bot's command list with Telegram so clients show it in the
|
||||
/// `/` menu (Bot API `setMyCommands`).
|
||||
pub async fn register_commands(bot: &Bot) -> Result<(), RequestError> {
|
||||
let commands = Command::bot_commands();
|
||||
bot.set_my_commands(commands.clone()).await?;
|
||||
log::info!("registered {} commands", commands.len());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// For locally produced media (encoded ugoira MP4) the thumbnail URL is a
|
||||
/// hotlink-protected remote URL Telegram may not fetch; let Telegram generate
|
||||
/// its own thumbnail instead.
|
||||
fn thumbnail_for(media: &Media) -> Option<String> {
|
||||
let url = media.url();
|
||||
if url.starts_with("http://") || url.starts_with("https://") {
|
||||
media.thumbnail_url().map(str::to_string)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn media_to_payload(media: &Media, sensitive: bool) -> MediaItemPayload {
|
||||
let fallback_url = media.smaller_url().map(str::to_string);
|
||||
match media {
|
||||
// A gif inside a group becomes a video item; a lone gif takes the
|
||||
// animation path (see url_media).
|
||||
Media::Illustration { .. } => MediaItemPayload::Photo {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
Media::Video { .. } => MediaItemPayload::Video {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
thumbnail: thumbnail_for(media),
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
Media::Animated { .. } => MediaItemPayload::Video {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
thumbnail: thumbnail_for(media),
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
async fn enqueue_retry(task: Task, delay_seconds: f64) {
|
||||
let payload = serde_json::to_value(task).expect("task serializes");
|
||||
let run_after = now_f64() + delay_seconds;
|
||||
if let Err(e) = TASK_QUEUE.enqueue(payload, run_after).await {
|
||||
log::error!("failed to enqueue retry: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Sends a task and handles the outcome: post-send actions on success, retry
|
||||
/// enqueue on retryable failure, reply + link-cache invalidation on
|
||||
/// permanent failure (a stale cached file id must not repeat forever).
|
||||
async fn dispatch_send(bot: Bot, message: &Message, task: &Task, url: &str) {
|
||||
let result = match task {
|
||||
Task::SendAnimation { .. } => send::send_animation(&bot, task).await,
|
||||
Task::SendMediaSequence { .. } => send::send_media_sequence(&bot, task).await,
|
||||
Task::ForwardMessages { .. } => unreachable!(),
|
||||
};
|
||||
match result {
|
||||
Ok(message_ids) => {
|
||||
log::info!("sent {} message(s) for {url}", message_ids.len());
|
||||
send::post_send_actions(&bot, task, message_ids).await;
|
||||
}
|
||||
Err(send::SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => {
|
||||
log::info!("send for {url} failed, queued for retry in {delay_seconds:.1}s");
|
||||
enqueue_retry(task, delay_seconds).await;
|
||||
let _ = reply(bot, message.clone(), "Send failed. Task queued for retry.").await;
|
||||
}
|
||||
Err(send::SendError::Permanent {
|
||||
message: err_message,
|
||||
task,
|
||||
}) => {
|
||||
send::invalidate_cache(&task).await;
|
||||
log::error!("send for {url} failed permanently: {err_message}");
|
||||
let _ = reply(bot, message.clone(), format!("Send failed: {err_message}")).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Builds the send task from ready-made items, sharing the payload shape
|
||||
/// between the fresh-fetch and link-cache paths.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn build_send_task(
|
||||
chat_data: &ChatData,
|
||||
message: &Message,
|
||||
source_url: String,
|
||||
caption: String,
|
||||
items: Vec<MediaItemPayload>,
|
||||
cache_data: Option<CachedPost>,
|
||||
) -> Task {
|
||||
let chat_id = message.chat.id.0;
|
||||
if items.len() == 1 && matches!(items[0], MediaItemPayload::Animation { .. }) {
|
||||
Task::SendAnimation {
|
||||
chat_id,
|
||||
reply_to_message_id: message.id.0 as i64,
|
||||
caption,
|
||||
animation: items.into_iter().next().unwrap(),
|
||||
source_url,
|
||||
edit_before_forward: chat_data.edit_before_forward,
|
||||
forward_channel_id: chat_data.forward_channel_id,
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(message.id.0 as i64),
|
||||
cache_data,
|
||||
}
|
||||
} else {
|
||||
Task::SendMediaSequence {
|
||||
chat_id,
|
||||
reply_to_message_id: message.id.0 as i64,
|
||||
caption,
|
||||
media_batches: send::chunk_media_items(items),
|
||||
batch_index: 0,
|
||||
sent_message_ids: vec![],
|
||||
source_url,
|
||||
edit_before_forward: chat_data.edit_before_forward,
|
||||
forward_channel_id: chat_data.forward_channel_id,
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(message.id.0 as i64),
|
||||
cache_data,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn url_media(bot: Bot, message: &Message, url: &str) {
|
||||
let chat_id = message.chat.id.0;
|
||||
if let Err(e) = bot
|
||||
.send_chat_action(ChatId(chat_id), ChatAction::Typing)
|
||||
.await
|
||||
{
|
||||
log::error!("send_chat_action failed: {e}");
|
||||
}
|
||||
|
||||
// Link cache: a post sent before is re-sent from Telegram file ids —
|
||||
// no source-site request, no download, no upload. Keyed by the
|
||||
// normalized post id so x.com / fxtwitter / /photo/N variants collide.
|
||||
if let Some(key) = x_media::site::cache_key(url)
|
||||
&& let Some(cached) = LINK_CACHE.get(&key, CONFIG.link_cache_ttl).await
|
||||
{
|
||||
log::info!("link cache hit for {url}");
|
||||
let chat_data = CHAT_STORE.get(chat_id).await;
|
||||
let site = key.split(':').next().unwrap_or("unknown");
|
||||
let format = chat_data
|
||||
.message_format
|
||||
.get(site)
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
let caption = if format.is_empty() {
|
||||
cached.caption.clone()
|
||||
} else {
|
||||
x_media::site::caption_from_fields(
|
||||
&format,
|
||||
"",
|
||||
&cached.url,
|
||||
&cached.author,
|
||||
&cached.author_url,
|
||||
&cached.title,
|
||||
&cached.tags,
|
||||
)
|
||||
};
|
||||
let items: Vec<MediaItemPayload> = cached
|
||||
.media
|
||||
.iter()
|
||||
.map(|m| match m.kind {
|
||||
CachedMediaKind::Photo => MediaItemPayload::Photo {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
fallback_url: None,
|
||||
file_id: true,
|
||||
},
|
||||
CachedMediaKind::Video => MediaItemPayload::Video {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
thumbnail: None,
|
||||
fallback_url: None,
|
||||
file_id: true,
|
||||
},
|
||||
CachedMediaKind::Animation => MediaItemPayload::Animation {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
file_id: true,
|
||||
},
|
||||
})
|
||||
.collect();
|
||||
let task = build_send_task(
|
||||
&chat_data,
|
||||
message,
|
||||
cached.url.clone(),
|
||||
caption,
|
||||
items,
|
||||
Some(cached),
|
||||
);
|
||||
dispatch_send(bot, message, &task, url).await;
|
||||
return;
|
||||
}
|
||||
|
||||
log::info!("fetching {url}");
|
||||
match x_media::site::fetch(url).await {
|
||||
// Unsupported links are ignored silently (Python parity).
|
||||
Ok(None) => {
|
||||
log::info!("no site pattern matches {url}; ignoring");
|
||||
}
|
||||
// Retries exhausted: notify the user (Rust-only requirement 3).
|
||||
Err(e) => {
|
||||
log::error!("fetch {url}: {e}");
|
||||
let _ = reply(
|
||||
bot,
|
||||
message.clone(),
|
||||
"Failed to fetch media from this link.",
|
||||
)
|
||||
.await;
|
||||
}
|
||||
Ok(Some(fetched)) => {
|
||||
if fetched.media.is_empty() {
|
||||
let _ = reply(
|
||||
bot,
|
||||
message.clone(),
|
||||
"No media found or media type is not supported.",
|
||||
)
|
||||
.await;
|
||||
return;
|
||||
}
|
||||
let chat_data = CHAT_STORE.get(chat_id).await;
|
||||
// Per-site caption format override (empty -> built-in caption).
|
||||
let format = chat_data
|
||||
.message_format
|
||||
.get(fetched.site_name())
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
let caption = fetched.caption_with(&format);
|
||||
// Raw render data for the link cache; the send fills in the
|
||||
// Telegram file ids and persists the entry.
|
||||
let cache_data = fetched
|
||||
.render_fields()
|
||||
.map(|(author, author_url, title, tags)| CachedPost {
|
||||
url: fetched.source_url.clone(),
|
||||
caption: fetched.caption.clone(),
|
||||
title: title.to_string(),
|
||||
author: author.to_string(),
|
||||
author_url: author_url.to_string(),
|
||||
tags: tags.to_string(),
|
||||
sensitive: fetched.sensitive,
|
||||
media: vec![],
|
||||
});
|
||||
let items: Vec<MediaItemPayload> = fetched
|
||||
.media
|
||||
.iter()
|
||||
.map(|media| media_to_payload(media, fetched.sensitive))
|
||||
.collect();
|
||||
let task = build_send_task(
|
||||
&chat_data,
|
||||
message,
|
||||
fetched.source_url.clone(),
|
||||
caption,
|
||||
items,
|
||||
cache_data,
|
||||
);
|
||||
dispatch_send(bot, message, &task, url).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestError> {
|
||||
let is_private = matches!(message.chat.kind, ChatKind::Private(_));
|
||||
let sender = message
|
||||
.from
|
||||
.as_ref()
|
||||
.map(|from| from.full_name())
|
||||
.unwrap_or_else(|| "unknown".to_string());
|
||||
let text_preview = message
|
||||
.text()
|
||||
.map(|t| {
|
||||
let end = t.floor_char_boundary(120.min(t.len()));
|
||||
&t[..end]
|
||||
})
|
||||
.unwrap_or("<no text>");
|
||||
log::info!(
|
||||
"message from {sender} in {} (private={is_private}): {text_preview}",
|
||||
message.chat.id
|
||||
);
|
||||
// URL/edit flows only run in private chats; commands run in any chat.
|
||||
if is_private && edit_message_handler(&bot, &message).await {
|
||||
return respond(());
|
||||
}
|
||||
if let Some(text) = message.text()
|
||||
&& let Ok(command) = Command::parse(text, "")
|
||||
{
|
||||
log::info!("command from {}: {text_preview}", message.chat.id);
|
||||
execute_command(&bot, &message, command).await?;
|
||||
return respond(());
|
||||
}
|
||||
if is_private {
|
||||
let urls = extract_urls(&message);
|
||||
if !urls.is_empty() {
|
||||
log::info!("extracted {} URL(s): {urls:?}", urls.len());
|
||||
}
|
||||
for url in urls {
|
||||
// Clone out of the lock: the parking_lot guard is !Send and must
|
||||
// not be held across the await below.
|
||||
let Some(tx) = URL_JOBS.lock().clone() else {
|
||||
log::warn!("url workers not started; dropping link");
|
||||
break;
|
||||
};
|
||||
let _ = tx.send((bot.clone(), message.clone(), url)).await;
|
||||
}
|
||||
}
|
||||
respond(())
|
||||
}
|
||||
|
||||
pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), RequestError> {
|
||||
if query.query.is_empty() {
|
||||
return respond(());
|
||||
}
|
||||
// Telegram fires an inline query on every keystroke; only run a fetch
|
||||
// (3 attempts!) for something that is actually a supported post URL, so
|
||||
// typing does not hammer the source sites.
|
||||
if x_media::site::cache_key(&query.query).is_none() {
|
||||
return respond(());
|
||||
}
|
||||
log::info!("inline query: {}", query.query);
|
||||
match x_media::site::fetch(&query.query).await {
|
||||
Ok(Some(fetched)) => {
|
||||
let mut results: Vec<InlineQueryResult> = Vec::new();
|
||||
for (i, media) in fetched.media.iter().enumerate() {
|
||||
let id = format!("{i}");
|
||||
let Some(url) = url::Url::parse(media.url()).ok() else {
|
||||
continue;
|
||||
};
|
||||
let thumbnail = media
|
||||
.thumbnail_url()
|
||||
.and_then(|t| url::Url::parse(t).ok())
|
||||
.unwrap_or_else(|| url.clone());
|
||||
let caption = fetched.caption.clone();
|
||||
let result = match media {
|
||||
Media::Illustration { .. } => {
|
||||
// Inline photo results have their own (smaller) size
|
||||
// cap; use the reduced variant when one exists.
|
||||
let photo_url = media
|
||||
.smaller_url()
|
||||
.and_then(|u| url::Url::parse(u).ok())
|
||||
.unwrap_or_else(|| url.clone());
|
||||
InlineQueryResult::Photo(
|
||||
InlineQueryResultPhoto::new(id, photo_url, thumbnail)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html),
|
||||
)
|
||||
}
|
||||
Media::Video { .. } => InlineQueryResult::Video(
|
||||
InlineQueryResultVideo::new(
|
||||
id,
|
||||
url,
|
||||
"video/mp4".parse().expect("valid mime"),
|
||||
thumbnail,
|
||||
fetched.title.clone(),
|
||||
)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html),
|
||||
),
|
||||
Media::Animated { .. } => InlineQueryResult::Mpeg4Gif(
|
||||
InlineQueryResultMpeg4Gif::new(id, url, thumbnail)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html),
|
||||
),
|
||||
};
|
||||
results.push(result);
|
||||
}
|
||||
if !results.is_empty() {
|
||||
bot.answer_inline_query(query.id, results).await?;
|
||||
}
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(e) => log::error!("inline fetch {}: {e}", query.query),
|
||||
}
|
||||
respond(())
|
||||
}
|
||||
|
||||
pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<(), RequestError> {
|
||||
let callback_query_id = query.id;
|
||||
let data = query.data.clone();
|
||||
let Some(message) = &query.message else {
|
||||
return respond(());
|
||||
};
|
||||
let chat_id = message.chat().id.0;
|
||||
let prompt_message_id = message.id().0 as i64;
|
||||
let ttl_secs = CONFIG.edit_message_ttl.as_secs() as i64;
|
||||
let chat_data = CHAT_STORE.get(chat_id).await;
|
||||
let edit = chat_data.edit_message.get(&prompt_message_id).cloned();
|
||||
let Some(edit) = edit else {
|
||||
log::info!(
|
||||
"callback from {}: no edit record for prompt {prompt_message_id}",
|
||||
chat_id
|
||||
);
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("Expired")
|
||||
.await?;
|
||||
return respond(());
|
||||
};
|
||||
// Lazy expiry: a stale record (past the TTL, not yet swept) is dropped.
|
||||
if edit.created_at + ttl_secs <= unix_now() {
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
data.edit_message.remove(&prompt_message_id);
|
||||
})
|
||||
.await;
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("Expired")
|
||||
.await?;
|
||||
return respond(());
|
||||
}
|
||||
|
||||
let Some(data) = data else {
|
||||
return respond(());
|
||||
};
|
||||
log::info!(
|
||||
"callback from {} on prompt {prompt_message_id}: {data}",
|
||||
chat_id
|
||||
);
|
||||
if data == "forward" {
|
||||
match chat_data.forward_channel_id {
|
||||
Some(channel_id) => {
|
||||
let forward_task = Task::ForwardMessages {
|
||||
from_chat_id: edit.chat_id,
|
||||
to_chat_id: channel_id,
|
||||
message_ids: edit.forward_message_ids.clone(),
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(prompt_message_id),
|
||||
};
|
||||
match send::forward_messages(&bot, &forward_task).await {
|
||||
Ok(()) => {
|
||||
log::info!(
|
||||
"forwarded {} message(s) to channel {channel_id}",
|
||||
edit.forward_message_ids.len()
|
||||
);
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("✅ Forwarded")
|
||||
.await?;
|
||||
let _ = bot
|
||||
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
|
||||
.await;
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
data.edit_message.remove(&prompt_message_id);
|
||||
})
|
||||
.await;
|
||||
}
|
||||
Err(send::SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => {
|
||||
log::info!("forward queued for retry in {delay_seconds:.1}s");
|
||||
enqueue_retry(task, delay_seconds).await;
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("Forward queued for retry.")
|
||||
.await?;
|
||||
}
|
||||
Err(send::SendError::Permanent { message, .. }) => {
|
||||
log::error!("forward failed permanently: {message}");
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text(format!("Forward failed: {message}"))
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
None => {
|
||||
log::info!("forward callback without a forward channel set");
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("No forward channel set.")
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
return respond(());
|
||||
}
|
||||
if let Some(name) = data.strip_prefix("template|") {
|
||||
if let Some(template_html) = chat_data.template.get(name).cloned()
|
||||
&& let Some(first_forward_id) = edit.forward_message_ids.first().copied()
|
||||
{
|
||||
// Raw template including the [] placeholder (Python parity).
|
||||
let _ = bot
|
||||
.edit_message_caption(ChatId(chat_id), MessageId(first_forward_id as i32))
|
||||
.caption(template_html)
|
||||
.parse_mode(ParseMode::Html)
|
||||
.await;
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
if let Some(entry) = data.edit_message.get_mut(&prompt_message_id) {
|
||||
entry.template = name.to_string();
|
||||
}
|
||||
})
|
||||
.await;
|
||||
log::info!("template '{name}' applied to prompt {prompt_message_id}");
|
||||
}
|
||||
bot.answer_callback_query(callback_query_id).await?;
|
||||
}
|
||||
respond(())
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
//! Callback query handling: the edit-before-forward prompt's "forward" and
|
||||
//! "template|<name>" buttons.
|
||||
|
||||
use super::urls::enqueue_retry;
|
||||
use super::{CHAT_STORE, CONFIG, TASK_QUEUE};
|
||||
use crate::send::{self, Task};
|
||||
use crate::state::unix_now;
|
||||
use teloxide::RequestError;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::types::{CallbackQuery, ChatId, MessageId, ParseMode};
|
||||
|
||||
pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<(), RequestError> {
|
||||
let callback_query_id = query.id;
|
||||
let data = query.data.clone();
|
||||
let Some(message) = &query.message else {
|
||||
return respond(());
|
||||
};
|
||||
let chat_id = message.chat().id.0;
|
||||
let prompt_message_id = message.id().0 as i64;
|
||||
let ttl_secs = CONFIG.edit_message_ttl.as_secs() as i64;
|
||||
let chat_data = CHAT_STORE.get(chat_id).await;
|
||||
let edit = chat_data.edit_message.get(&prompt_message_id).cloned();
|
||||
let Some(edit) = edit else {
|
||||
log::debug!(
|
||||
"callback from {}: no edit record for prompt {prompt_message_id}",
|
||||
chat_id
|
||||
);
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("Expired")
|
||||
.await?;
|
||||
return respond(());
|
||||
};
|
||||
// Lazy expiry: a stale record (past the TTL, not yet swept) is dropped.
|
||||
if edit.created_at + ttl_secs <= unix_now() {
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
data.edit_message.remove(&prompt_message_id);
|
||||
})
|
||||
.await;
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("Expired")
|
||||
.await?;
|
||||
return respond(());
|
||||
}
|
||||
|
||||
let Some(data) = data else {
|
||||
return respond(());
|
||||
};
|
||||
log::info!(
|
||||
"callback from {} on prompt {prompt_message_id}: {data}",
|
||||
chat_id
|
||||
);
|
||||
if data == "forward" {
|
||||
match chat_data.forward_channel_id {
|
||||
Some(channel_id) => {
|
||||
let forward_task = Task::ForwardMessages {
|
||||
from_chat_id: edit.chat_id,
|
||||
to_chat_id: channel_id,
|
||||
message_ids: edit.forward_message_ids.clone(),
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(prompt_message_id),
|
||||
};
|
||||
match send::forward_messages(&bot, &forward_task).await {
|
||||
Ok(()) => {
|
||||
log::info!(
|
||||
"forwarded {} message(s) to channel {channel_id}",
|
||||
edit.forward_message_ids.len()
|
||||
);
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("✅ Forwarded")
|
||||
.await?;
|
||||
let _ = bot
|
||||
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
|
||||
.await;
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
data.edit_message.remove(&prompt_message_id);
|
||||
})
|
||||
.await;
|
||||
}
|
||||
Err(send::SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => {
|
||||
log::info!("forward queued for retry in {delay_seconds:.1}s");
|
||||
enqueue_retry(&TASK_QUEUE, task, delay_seconds).await;
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("Forward queued for retry.")
|
||||
.await?;
|
||||
}
|
||||
Err(send::SendError::Permanent { message, .. }) => {
|
||||
log::error!("forward failed permanently: {message}");
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text(format!("Forward failed: {message}"))
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
None => {
|
||||
log::debug!("forward callback without a forward channel set");
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("No forward channel set.")
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
return respond(());
|
||||
}
|
||||
if let Some(name) = data.strip_prefix("template|") {
|
||||
if let Some(template_html) = chat_data.template.get(name).cloned()
|
||||
&& let Some(first_forward_id) = edit.forward_message_ids.first().copied()
|
||||
{
|
||||
// Raw template including the [] placeholder (Python parity).
|
||||
let _ = bot
|
||||
.edit_message_caption(ChatId(chat_id), MessageId(first_forward_id as i32))
|
||||
.caption(template_html)
|
||||
.parse_mode(ParseMode::Html)
|
||||
.await;
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
if let Some(entry) = data.edit_message.get_mut(&prompt_message_id) {
|
||||
entry.template = name.to_string();
|
||||
}
|
||||
})
|
||||
.await;
|
||||
log::info!("template '{name}' applied to prompt {prompt_message_id}");
|
||||
}
|
||||
bot.answer_callback_query(callback_query_id).await?;
|
||||
}
|
||||
respond(())
|
||||
}
|
||||
@@ -0,0 +1,580 @@
|
||||
//! Bot command parsing, the `/`-command executor and `setMyCommands`
|
||||
//! registration. URL/inline/callback flows live in their own modules.
|
||||
|
||||
use super::{CHAT_STORE, CONFIG, LINK_CACHE, log_key, reply, reply_html};
|
||||
use teloxide::RequestError;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::types::{ChatId, Message, Recipient};
|
||||
use teloxide::utils::command::{BotCommands, ParseError};
|
||||
|
||||
#[derive(BotCommands, Clone)]
|
||||
#[command(
|
||||
rename_rule = "snake_case",
|
||||
description = "Turn X/Pixiv/Bluesky links into media messages"
|
||||
)]
|
||||
pub(crate) enum Command {
|
||||
#[command(description = "Get started")]
|
||||
Start,
|
||||
#[command(description = "Show command help")]
|
||||
Help,
|
||||
#[command(
|
||||
description = "Set forward channel (@channel or ID)",
|
||||
parse_with = "split"
|
||||
)]
|
||||
SetForwardChannel(String),
|
||||
#[command(description = "Remove forward channel")]
|
||||
RemoveForwardChannel,
|
||||
#[command(description = "Toggle edit-before-forward")]
|
||||
EditBeforeForward,
|
||||
#[command(
|
||||
description = "Reply with [] to save as template",
|
||||
parse_with = "split"
|
||||
)]
|
||||
SetTemplate(String),
|
||||
#[command(description = "Show chat state (debug)")]
|
||||
BotDict,
|
||||
#[command(description = "Set site caption format", parse_with = "split")]
|
||||
SetFormat(String),
|
||||
#[command(
|
||||
description = "Clear link cache (admin; optional URL, else all)",
|
||||
parse_with = "split"
|
||||
)]
|
||||
ClearCache(String),
|
||||
#[command(
|
||||
description = "Test link parsing (debug; no media sent)",
|
||||
parse_with = parse_test_arg
|
||||
)]
|
||||
Test(String),
|
||||
}
|
||||
|
||||
/// `/test` argument parser: the whole remainder after the command name,
|
||||
/// trimmed. The built-in `split` parser takes exactly one space-separated
|
||||
/// token and rejects the rest, so a URL followed by a trailing space (or
|
||||
/// pasted text) would silently fall through to the URL flow instead.
|
||||
fn parse_test_arg(s: String) -> Result<(String,), ParseError> {
|
||||
Ok((s.trim().to_string(),))
|
||||
}
|
||||
|
||||
enum SetForwardChannelError {
|
||||
EmptyParameter,
|
||||
NotChannel,
|
||||
NotAdmin,
|
||||
NotBotAdmin(RequestError),
|
||||
NotBotCanPost,
|
||||
}
|
||||
|
||||
async fn set_forward_channel_handler(
|
||||
bot: &Bot,
|
||||
message: &Message,
|
||||
channel: String,
|
||||
) -> Result<i64, SetForwardChannelError> {
|
||||
if channel.is_empty() {
|
||||
return Err(SetForwardChannelError::EmptyParameter);
|
||||
}
|
||||
let channel = match channel.parse::<i64>() {
|
||||
Ok(id) => Recipient::Id(ChatId(id)),
|
||||
Err(_) => Recipient::ChannelUsername(channel),
|
||||
};
|
||||
if let Some(from) = &message.from {
|
||||
log::info!(
|
||||
"Set forward channel for {} ({}) to {}",
|
||||
from.full_name(),
|
||||
message.chat.id,
|
||||
channel
|
||||
);
|
||||
}
|
||||
let chat = match bot.get_chat(channel.clone()).await {
|
||||
Err(e) => {
|
||||
log::error!("Failed to get channel {}: {}", channel, e);
|
||||
return Err(SetForwardChannelError::NotBotAdmin(e));
|
||||
}
|
||||
Ok(chat) => chat,
|
||||
};
|
||||
if !chat.is_channel() {
|
||||
return Err(SetForwardChannelError::NotChannel);
|
||||
}
|
||||
let channel_id = chat.id.0;
|
||||
// The sender must be a channel administrator. Compare against the
|
||||
// sender's user id, NOT the chat id (they only coincide in private
|
||||
// chats, so the old check broke group usage).
|
||||
let Some(sender) = message.from.as_ref() else {
|
||||
return Err(SetForwardChannelError::NotAdmin);
|
||||
};
|
||||
match bot.get_chat_administrators(channel.clone()).await {
|
||||
Err(e) => {
|
||||
log::error!("Failed to get channel administrators {}: {}", channel, e);
|
||||
return Err(SetForwardChannelError::NotBotAdmin(e));
|
||||
}
|
||||
Ok(admins) => {
|
||||
if !admins.iter().any(|admin| admin.user.id == sender.id) {
|
||||
return Err(SetForwardChannelError::NotAdmin);
|
||||
}
|
||||
// The bot itself must be an admin that can post; a missing
|
||||
// bot entry must not pass silently (copy would fail later).
|
||||
let bot_id = match bot.get_me().await {
|
||||
Ok(me) => me.user.id,
|
||||
Err(e) => return Err(SetForwardChannelError::NotBotAdmin(e)),
|
||||
};
|
||||
let bot_ok = admins
|
||||
.iter()
|
||||
.any(|admin| admin.user.id == bot_id && admin.can_post_messages());
|
||||
if !bot_ok {
|
||||
return Err(SetForwardChannelError::NotBotCanPost);
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(channel_id)
|
||||
}
|
||||
|
||||
pub(crate) async fn execute_command(
|
||||
bot: &Bot,
|
||||
message: &Message,
|
||||
command: Command,
|
||||
) -> Result<(), RequestError> {
|
||||
match command {
|
||||
Command::Start => {
|
||||
bot.send_message(message.chat.id, "Hello!").await?;
|
||||
}
|
||||
Command::Help => {
|
||||
bot.send_message(message.chat.id, Command::descriptions().to_string())
|
||||
.await?;
|
||||
}
|
||||
Command::SetForwardChannel(channel) => {
|
||||
let result = match set_forward_channel_handler(bot, message, channel).await {
|
||||
Ok(channel_id) => {
|
||||
CHAT_STORE
|
||||
.update(message.chat.id.0, |data| {
|
||||
data.forward_channel_id = Some(channel_id);
|
||||
})
|
||||
.await;
|
||||
"Add successfully.".to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::EmptyParameter) => {
|
||||
"Receive empty parameter.\nYou should enter a channel id or username"
|
||||
.to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::NotChannel) => {
|
||||
"Given id / username is not a channel".to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::NotAdmin) => {
|
||||
"You are not an administrator of the channel".to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::NotBotAdmin(e)) => {
|
||||
e.to_string() + "\nPlease add the bot as an admin to the channel"
|
||||
}
|
||||
Err(SetForwardChannelError::NotBotCanPost) => {
|
||||
"Bot can't post messages to the channel".to_string()
|
||||
}
|
||||
};
|
||||
reply(bot, message.chat.id.0, message.id, result).await?;
|
||||
}
|
||||
Command::RemoveForwardChannel => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let text = CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
if data.forward_channel_id.is_some() {
|
||||
data.forward_channel_id = None;
|
||||
"Remove successfully.".to_string()
|
||||
} else {
|
||||
"No channel to remove.".to_string()
|
||||
}
|
||||
})
|
||||
.await;
|
||||
reply(bot, message.chat.id.0, message.id, text).await?;
|
||||
}
|
||||
Command::EditBeforeForward => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let text = CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
if data.forward_channel_id.is_none() {
|
||||
"Please enable forward channel first.".to_string()
|
||||
} else if data.edit_before_forward {
|
||||
data.edit_before_forward = false;
|
||||
data.edit_message.clear();
|
||||
"Disable edit before forward.".to_string()
|
||||
} else {
|
||||
data.edit_before_forward = true;
|
||||
"Enable edit before forward.".to_string()
|
||||
}
|
||||
})
|
||||
.await;
|
||||
reply(bot, message.chat.id.0, message.id, text).await?;
|
||||
}
|
||||
Command::SetTemplate(name) => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let text = match message.reply_to_message() {
|
||||
None => "Please reply to a message to set as template.".to_string(),
|
||||
Some(reply) => {
|
||||
let reply_text = reply.text().unwrap_or_default();
|
||||
if !reply_text.contains("[]") {
|
||||
"Please reply to a message with [] to set as template.".to_string()
|
||||
} else if name.is_empty() {
|
||||
"Please provide a name for the template.".to_string()
|
||||
} else {
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
data.template.insert(
|
||||
name,
|
||||
html_escape::encode_text(reply_text).into_owned(),
|
||||
);
|
||||
})
|
||||
.await;
|
||||
"Template set.".to_string()
|
||||
}
|
||||
}
|
||||
};
|
||||
reply(bot, message.chat.id.0, message.id, text).await?;
|
||||
}
|
||||
Command::BotDict => {
|
||||
let chat_data = CHAT_STORE.get(message.chat.id.0).await;
|
||||
let debug = format!("{chat_data:?}");
|
||||
let text = html_escape::encode_text(&debug).into_owned();
|
||||
reply(bot, message.chat.id.0, message.id, text).await?;
|
||||
}
|
||||
Command::SetFormat(arg) => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let (site, format) = match arg.split_once(char::is_whitespace) {
|
||||
Some((site, format)) if !format.trim().is_empty() => {
|
||||
(site.trim(), format.trim().to_string())
|
||||
}
|
||||
_ => {
|
||||
reply(
|
||||
bot,
|
||||
message.chat.id.0,
|
||||
message.id,
|
||||
"Usage: /set_format <site> <format>",
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
if !x_media::site::site_ids().contains(&site) {
|
||||
reply(
|
||||
bot,
|
||||
message.chat.id.0,
|
||||
message.id,
|
||||
"Unknown site. Use twitter, bsky or pixiv.",
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
CHAT_STORE
|
||||
.update(chat_id, |data| {
|
||||
data.message_format.insert(site.to_string(), format);
|
||||
})
|
||||
.await;
|
||||
reply(bot, message.chat.id.0, message.id, "Format set.").await?;
|
||||
}
|
||||
Command::ClearCache(arg) => {
|
||||
let sender_id = message
|
||||
.from
|
||||
.as_ref()
|
||||
.map(|user| user.id.0 as i64)
|
||||
.unwrap_or(-1);
|
||||
if !CONFIG.admin_ids.contains(&sender_id) {
|
||||
reply(bot, message.chat.id.0, message.id, "Admin only.").await?;
|
||||
return Ok(());
|
||||
}
|
||||
let arg = arg.trim();
|
||||
if arg.is_empty() {
|
||||
let removed = LINK_CACHE.clear(None).await;
|
||||
log::info!("cache cleared by {sender_id}: {removed} entries");
|
||||
reply(
|
||||
bot,
|
||||
message.chat.id.0,
|
||||
message.id,
|
||||
format!("Cleared {removed} cached entr{}.", plural(removed)),
|
||||
)
|
||||
.await?;
|
||||
} else {
|
||||
let key = match x_media::site::cache_key(arg) {
|
||||
Some(key) => key,
|
||||
None => {
|
||||
reply(
|
||||
bot,
|
||||
message.chat.id.0,
|
||||
message.id,
|
||||
"Unrecognized link. Use a twitter/x, pixiv or bsky post URL.",
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
let removed = LINK_CACHE.clear(Some(&key)).await;
|
||||
log::info!("cache entry cleared by {sender_id}: {key} ({removed} rows)");
|
||||
reply(
|
||||
bot,
|
||||
message.chat.id.0,
|
||||
message.id,
|
||||
format!(
|
||||
"Cleared cache for {arg} ({} entr{}).",
|
||||
removed,
|
||||
plural(removed)
|
||||
),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
Command::Test(arg) => {
|
||||
let url = arg.trim();
|
||||
if url.is_empty() {
|
||||
reply(
|
||||
bot,
|
||||
message.chat.id.0,
|
||||
message.id,
|
||||
"Usage: /test <post url>",
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
// Debug tool: report the parse result only — nothing is sent,
|
||||
// cached or forwarded. Info level echoes the normalized key
|
||||
// (never the raw URL) per the logging convention.
|
||||
log::info!("test: parsing [key={}]", log_key(url));
|
||||
match x_media::site::fetch(url).await {
|
||||
Ok(None) => {
|
||||
reply(
|
||||
bot,
|
||||
message.chat.id.0,
|
||||
message.id,
|
||||
"No enabled site matches this link (twitter/x, pixiv or bsky).",
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
Err(e) => {
|
||||
reply(
|
||||
bot,
|
||||
message.chat.id.0,
|
||||
message.id,
|
||||
format!("Fetch failed: {e}"),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
Ok(Some(fetched)) => {
|
||||
let report = test_parse_report(
|
||||
url,
|
||||
fetched.site_name(),
|
||||
&fetched.source_url,
|
||||
&fetched.title,
|
||||
fetched.render_fields(),
|
||||
fetched.sensitive,
|
||||
&fetched.caption,
|
||||
&fetched.media,
|
||||
);
|
||||
// HTML report: the caption renders inside a <blockquote>
|
||||
// exactly as it will appear in the sent media message.
|
||||
reply_html(bot, message.chat.id.0, message.id, report).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `"y"` for one, `"ies"` for anything else — "1 entry" / "2 entries".
|
||||
fn plural(n: usize) -> &'static str {
|
||||
if n == 1 { "y" } else { "ies" }
|
||||
}
|
||||
|
||||
/// Registers the bot's command list with Telegram so clients show it in the
|
||||
/// `/` menu (Bot API `setMyCommands`).
|
||||
pub async fn register_commands(bot: &Bot) -> Result<(), RequestError> {
|
||||
let commands = Command::bot_commands();
|
||||
bot.set_my_commands(commands.clone()).await?;
|
||||
log::info!("registered {} commands", commands.len());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Telegram's plain-text message limit is 4096 chars; the report stays under
|
||||
/// it even for very large threads (many media lines + a long caption).
|
||||
const MAX_TEST_REPORT_CHARS: usize = 4000;
|
||||
|
||||
/// Builds the HTML report for the `/test` command: what the parser produced
|
||||
/// for a link (site, canonical URL, title/author/tags, caption and the media
|
||||
/// list) — no media is sent and nothing is cached or forwarded. Sent with
|
||||
/// HTML parse mode: raw fields are escaped, the pre-escaped render fields are
|
||||
/// embedded as-is, and the caption is wrapped in a `<blockquote>` so it shows
|
||||
/// exactly as it will render in the sent media message. Fields are passed
|
||||
/// individually so the formatter stays a pure function testable without
|
||||
/// constructing a `Fetched` (its render fields are `pub(crate)` to the
|
||||
/// x-media crate).
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn test_parse_report(
|
||||
url: &str,
|
||||
site_id: &str,
|
||||
source_url: &str,
|
||||
title: &str,
|
||||
render: Option<(&str, &str, &str, &str)>,
|
||||
sensitive: bool,
|
||||
caption: &str,
|
||||
media: &[x_media::media::Media],
|
||||
) -> String {
|
||||
let mut lines = vec![
|
||||
format!("Parse result for {}", html_escape::encode_text(url)),
|
||||
format!("site: {site_id}"),
|
||||
format!(
|
||||
"key: {}",
|
||||
html_escape::encode_text(
|
||||
&x_media::site::cache_key(url).unwrap_or_else(|| "<unsupported>".to_string())
|
||||
)
|
||||
),
|
||||
];
|
||||
lines.push(format!(
|
||||
"source_url: {}",
|
||||
html_escape::encode_text(source_url)
|
||||
));
|
||||
lines.push(format!("title: {}", html_escape::encode_text(title)));
|
||||
if let Some((author, author_url, _title, tags)) = render {
|
||||
// The render fields are already pre-escaped for HTML captions; embed
|
||||
// them as-is so the report renders them exactly like the final
|
||||
// caption. `author_url` is raw and gets escaped here.
|
||||
lines.push(format!("author: {author}"));
|
||||
lines.push(format!(
|
||||
"author_url: {}",
|
||||
html_escape::encode_text(author_url)
|
||||
));
|
||||
lines.push(format!("tags: {tags}"));
|
||||
}
|
||||
lines.push(format!("sensitive: {sensitive}"));
|
||||
// The caption is wrapped in a <blockquote> so the report (an HTML
|
||||
// message) shows it exactly as it will render in the sent media caption
|
||||
// — escaped text and links included.
|
||||
lines.push(format!(
|
||||
"caption: <blockquote>{}</blockquote>",
|
||||
x_media::site::truncate_caption(caption)
|
||||
));
|
||||
lines.push(format!("media ({}):", media.len()));
|
||||
for (i, item) in media.iter().enumerate() {
|
||||
let kind = match item {
|
||||
x_media::media::Media::Illustration { .. } => "image",
|
||||
x_media::media::Media::Video { .. } => "video",
|
||||
x_media::media::Media::Animated { .. } => "gif",
|
||||
};
|
||||
lines.push(format!(
|
||||
" {}. {kind}: {}",
|
||||
i + 1,
|
||||
html_escape::encode_text(item.url())
|
||||
));
|
||||
}
|
||||
let mut out = lines.join(
|
||||
"
|
||||
",
|
||||
);
|
||||
if out.chars().count() > MAX_TEST_REPORT_CHARS {
|
||||
let end = out.floor_char_boundary(MAX_TEST_REPORT_CHARS - 1);
|
||||
out = format!("{}…", &out[..end]);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{MAX_TEST_REPORT_CHARS, test_parse_report};
|
||||
use x_media::media::Media;
|
||||
|
||||
#[test]
|
||||
fn test_parse_report_renders_fields_and_media() {
|
||||
let media = vec![
|
||||
Media::Illustration {
|
||||
title: None,
|
||||
url: "https://cdn.example/1.jpg".into(),
|
||||
thumbnail_url: None,
|
||||
fallback_url: None,
|
||||
},
|
||||
Media::Video {
|
||||
title: None,
|
||||
url: "https://cdn.example/2.mp4".into(),
|
||||
thumbnail_url: "https://cdn.example/2.jpg".into(),
|
||||
},
|
||||
];
|
||||
let report = test_parse_report(
|
||||
"https://x.com/u/status/1",
|
||||
"twitter",
|
||||
"https://x.com/u/status/1",
|
||||
"My title",
|
||||
Some(("Author", "https://x.com/u", "My title", "tag1 tag2")),
|
||||
false,
|
||||
"<a href=\"https://x.com/u\">Author</a> · My title",
|
||||
&media,
|
||||
);
|
||||
assert!(report.contains("site: twitter"), "{report}");
|
||||
assert!(report.contains("key: twitter:1"), "{report}");
|
||||
assert!(report.contains("title: My title"), "{report}");
|
||||
assert!(report.contains("author: Author"), "{report}");
|
||||
assert!(report.contains("author_url: https://x.com/u"), "{report}");
|
||||
assert!(report.contains("tags: tag1 tag2"), "{report}");
|
||||
assert!(report.contains("sensitive: false"), "{report}");
|
||||
assert!(report.contains("media (2):"), "{report}");
|
||||
assert!(
|
||||
report.contains("1. image: https://cdn.example/1.jpg"),
|
||||
"{report}"
|
||||
);
|
||||
assert!(
|
||||
report.contains("2. video: https://cdn.example/2.mp4"),
|
||||
"{report}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_report_without_render_data_and_no_media() {
|
||||
let report = test_parse_report("u", "pixiv", "s", "t", None, true, "c", &[]);
|
||||
assert!(!report.contains("author:"), "{report}");
|
||||
assert!(report.contains("sensitive: true"), "{report}");
|
||||
assert!(report.contains("media (0):"), "{report}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_report_wraps_caption_in_blockquote() {
|
||||
// The report is an HTML message: raw fields are escaped, pre-escaped
|
||||
// render fields are embedded as-is, and the caption is wrapped in a
|
||||
// <blockquote> so it shows exactly as it will render in the sent
|
||||
// media caption (escaped text and links included).
|
||||
let report = test_parse_report(
|
||||
"https://x.com/u/status/1",
|
||||
"twitter",
|
||||
"https://x.com/u/status/1",
|
||||
"A & B <C>",
|
||||
Some((
|
||||
"A & B",
|
||||
"https://x.com/u",
|
||||
"A & B <C>",
|
||||
"#a & #b",
|
||||
)),
|
||||
false,
|
||||
"<a href=\"https://x.com/u\">A & B</a>: C <D> & E",
|
||||
&[],
|
||||
);
|
||||
// Raw fields escaped (they render back to the original text in HTML).
|
||||
assert!(report.contains("title: A & B <C>"), "{report}");
|
||||
assert!(
|
||||
report.contains("source_url: https://x.com/u/status/1"),
|
||||
"{report}"
|
||||
);
|
||||
// Pre-escaped render fields embedded as-is.
|
||||
assert!(report.contains("author: A & B"), "{report}");
|
||||
assert!(report.contains("tags: #a & #b"), "{report}");
|
||||
// Caption wrapped in a blockquote with its HTML preserved.
|
||||
assert!(
|
||||
report.contains(
|
||||
"caption: <blockquote><a href=\"https://x.com/u\">A & B</a>: C <D> & E</blockquote>"
|
||||
),
|
||||
"{report}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_report_is_capped() {
|
||||
// 200 media lines ≈ 8 KB, comfortably over the cap.
|
||||
let media: Vec<Media> = (0..200)
|
||||
.map(|i| Media::Illustration {
|
||||
title: None,
|
||||
url: format!("https://cdn.example/{i}.jpg"),
|
||||
thumbnail_url: None,
|
||||
fallback_url: None,
|
||||
})
|
||||
.collect();
|
||||
let report = test_parse_report("u", "twitter", "s", "t", None, false, "c", &media);
|
||||
assert!(report.chars().count() <= MAX_TEST_REPORT_CHARS, "{report}");
|
||||
assert!(report.ends_with('…'), "{report}");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
//! Inline query handling with a keystroke debounce: only a query stable for
|
||||
//! [`INLINE_DEBOUNCE`] triggers a fetch, and repeats are served by Telegram's
|
||||
//! inline cache instead of re-fetching.
|
||||
|
||||
use super::log_key;
|
||||
use std::sync::LazyLock;
|
||||
use teloxide::RequestError;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::types::{
|
||||
InlineQuery, InlineQueryResult, InlineQueryResultMpeg4Gif, InlineQueryResultPhoto,
|
||||
InlineQueryResultVideo, ParseMode,
|
||||
};
|
||||
use x_media::media::Media;
|
||||
|
||||
/// Debounce window for inline queries: Telegram fires an inline query on
|
||||
/// every keystroke, and each prefix of a pasted URL (e.g. `.../status/12`,
|
||||
/// `.../status/123`, ...) already matches the site patterns. Without a
|
||||
/// debounce every keystroke triggers a fetch (3 attempts!) of a half-typed
|
||||
/// post id. Only answer once the query has been stable for this long.
|
||||
const INLINE_DEBOUNCE: std::time::Duration = std::time::Duration::from_millis(800);
|
||||
|
||||
/// Last seen inline query and whether it was already answered. Guards the
|
||||
/// debounce timer: a repeat of an answered query is served by Telegram's
|
||||
/// inline cache (see `cache_time`), not by another fetch.
|
||||
struct InlineDebounceState {
|
||||
query: String,
|
||||
answered: bool,
|
||||
}
|
||||
|
||||
static INLINE_DEBOUNCE_STATE: LazyLock<parking_lot::Mutex<Option<InlineDebounceState>>> =
|
||||
LazyLock::new(|| parking_lot::Mutex::new(None));
|
||||
|
||||
pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), RequestError> {
|
||||
if query.query.is_empty() {
|
||||
return respond(());
|
||||
}
|
||||
// Only run a fetch for something that is actually a supported post URL.
|
||||
if x_media::site::cache_key(&query.query).is_none() {
|
||||
return respond(());
|
||||
}
|
||||
// Debounce: record the query and answer only after it has been stable for
|
||||
// INLINE_DEBOUNCE (the timer below). An already-answered repeat of the
|
||||
// same query is left to Telegram's inline cache instead of re-fetching.
|
||||
{
|
||||
let mut state = INLINE_DEBOUNCE_STATE.lock();
|
||||
if let Some(prev) = state.as_ref()
|
||||
&& prev.query == query.query
|
||||
&& prev.answered
|
||||
{
|
||||
return respond(());
|
||||
}
|
||||
*state = Some(InlineDebounceState {
|
||||
query: query.query.clone(),
|
||||
answered: false,
|
||||
});
|
||||
}
|
||||
let query_text = query.query.clone();
|
||||
tokio::spawn(async move {
|
||||
tokio::time::sleep(INLINE_DEBOUNCE).await;
|
||||
// Only the last query of a typing burst survives: earlier timers see
|
||||
// the query changed and give up without answering.
|
||||
{
|
||||
let mut state = INLINE_DEBOUNCE_STATE.lock();
|
||||
let Some(state) = state.as_mut() else {
|
||||
return;
|
||||
};
|
||||
if state.query != query_text || state.answered {
|
||||
return;
|
||||
}
|
||||
// Claim the answer so a repeat of the same query cannot start a
|
||||
// second fetch; reset below when no answer was produced.
|
||||
state.answered = true;
|
||||
}
|
||||
match answer_inline_query(bot, query).await {
|
||||
Ok(true) => {}
|
||||
// No results produced (or nothing to answer): let a repeat of the
|
||||
// same query retry the fetch.
|
||||
Ok(false) | Err(_) => {
|
||||
let mut state = INLINE_DEBOUNCE_STATE.lock();
|
||||
if let Some(state) = state.as_mut()
|
||||
&& state.query == query_text
|
||||
{
|
||||
state.answered = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
respond(())
|
||||
}
|
||||
|
||||
/// Fetches the post behind an inline query and answers it. The caller has
|
||||
/// already applied the debounce. Returns `true` when an answer was sent.
|
||||
async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, RequestError> {
|
||||
log::debug!(
|
||||
"inline query: {} [key={}]",
|
||||
query.query,
|
||||
log_key(&query.query)
|
||||
);
|
||||
match x_media::site::fetch(&query.query).await {
|
||||
Ok(Some(fetched)) => {
|
||||
let mut results: Vec<InlineQueryResult> = Vec::new();
|
||||
// Inline results have the same 1024-char caption limit as regular
|
||||
// messages; truncate once here for all items.
|
||||
let caption = x_media::site::truncate_caption(&fetched.caption);
|
||||
for (i, media) in fetched.media.iter().enumerate() {
|
||||
let id = format!("{i}");
|
||||
let Some(url) = url::Url::parse(media.url()).ok() else {
|
||||
continue;
|
||||
};
|
||||
let thumbnail = media
|
||||
.thumbnail_url()
|
||||
.and_then(|t| url::Url::parse(t).ok())
|
||||
.unwrap_or_else(|| url.clone());
|
||||
let caption = caption.clone();
|
||||
let result = match media {
|
||||
Media::Illustration { .. } => {
|
||||
// Inline photo results have their own (smaller) size
|
||||
// cap; use the reduced variant when one exists.
|
||||
let photo_url = media
|
||||
.smaller_url()
|
||||
.and_then(|u| url::Url::parse(u).ok())
|
||||
.unwrap_or_else(|| url.clone());
|
||||
InlineQueryResult::Photo(
|
||||
InlineQueryResultPhoto::new(id, photo_url, thumbnail)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html),
|
||||
)
|
||||
}
|
||||
Media::Video { .. } => InlineQueryResult::Video(
|
||||
InlineQueryResultVideo::new(
|
||||
id,
|
||||
url,
|
||||
"video/mp4".parse().expect("valid mime"),
|
||||
thumbnail,
|
||||
fetched.title.clone(),
|
||||
)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html),
|
||||
),
|
||||
Media::Animated { .. } => InlineQueryResult::Mpeg4Gif(
|
||||
InlineQueryResultMpeg4Gif::new(id, url, thumbnail)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html),
|
||||
),
|
||||
};
|
||||
results.push(result);
|
||||
}
|
||||
if !results.is_empty() {
|
||||
// Explicit cache window: repeats of the same query within 5
|
||||
// minutes are served by Telegram without hitting the bot.
|
||||
bot.answer_inline_query(query.id, results)
|
||||
.cache_time(300)
|
||||
.await?;
|
||||
return Ok(true);
|
||||
}
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(e) => log::error!("inline fetch {}: {e}", query.query),
|
||||
}
|
||||
Ok(false)
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
//! Update handlers and the per-URL media pipeline.
|
||||
//!
|
||||
//! Split into per-concern modules: [`commands`] (the `/`-command executor),
|
||||
//! [`urls`] (URL extraction + the bounded worker pool + send dispatch),
|
||||
//! [`inline`] (debounced inline queries), [`callback`] (edit-before-forward
|
||||
//! buttons) and [`statics`] (the shared process-wide stores). This module
|
||||
//! holds the message entry point and the helpers the others share.
|
||||
|
||||
mod callback;
|
||||
mod commands;
|
||||
mod inline;
|
||||
mod statics;
|
||||
mod urls;
|
||||
|
||||
pub use callback::callback_query_handler;
|
||||
pub use commands::register_commands;
|
||||
pub use inline::inline_query_handler;
|
||||
pub use statics::{CHAT_STORE, CONFIG, LINK_CACHE, TASK_QUEUE};
|
||||
pub use urls::{start_url_workers, stop_url_workers};
|
||||
|
||||
use crate::media_sender::MediaSender;
|
||||
use commands::{Command, execute_command};
|
||||
use teloxide::RequestError;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::types::{ChatId, ChatKind, Message, MessageId, ParseMode, ReplyParameters};
|
||||
use teloxide::utils::command::BotCommands;
|
||||
use urls::{URL_JOBS, extract_urls};
|
||||
|
||||
/// Reply to a message by id, keeping the reply decoration even if the
|
||||
/// original was already deleted.
|
||||
pub(crate) async fn reply<T>(
|
||||
sender: &dyn MediaSender,
|
||||
chat_id: i64,
|
||||
reply_to: MessageId,
|
||||
text: T,
|
||||
) -> Result<Message, RequestError>
|
||||
where
|
||||
T: Into<String>,
|
||||
{
|
||||
sender
|
||||
.send_message(ChatId(chat_id), text.into(), Some(reply_to), None)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Reply to a message by id with HTML parse mode (same reply decoration as
|
||||
/// [`reply`]). Used by `/test`, whose report is an HTML message (the caption
|
||||
/// is wrapped in a `<blockquote>` to show it exactly as it will render).
|
||||
pub(crate) async fn reply_html(
|
||||
bot: &Bot,
|
||||
chat_id: i64,
|
||||
reply_to: MessageId,
|
||||
text: String,
|
||||
) -> Result<Message, RequestError> {
|
||||
// `<Bot as Requester>::` disambiguates from the MediaSender trait's
|
||||
// same-named method (see media_sender.rs).
|
||||
<Bot as Requester>::send_message(bot, ChatId(chat_id), text)
|
||||
.parse_mode(ParseMode::Html)
|
||||
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply())
|
||||
.await
|
||||
}
|
||||
|
||||
/// Log prefix tying the whole lifecycle of one link (fetch → send → cache →
|
||||
/// forward) together: the normalized cache key (`twitter:123…`, `pixiv:123`,
|
||||
/// `bsky:handle/rkey`) instead of the raw URL, so logs stay short and do not
|
||||
/// echo full user-submitted URLs at info level.
|
||||
pub fn log_key(url: &str) -> String {
|
||||
x_media::site::cache_key(url).unwrap_or_else(|| "<unsupported>".to_string())
|
||||
}
|
||||
|
||||
/// Edit-before-forward: a reply to the prompt swaps the caption of the first
|
||||
/// forwarded message. Returns true when the message was consumed as an edit.
|
||||
async fn edit_message_handler(bot: &Bot, message: &Message) -> bool {
|
||||
let Some(reply) = message.reply_to_message() else {
|
||||
return false;
|
||||
};
|
||||
let chat_id = message.chat.id.0;
|
||||
let Some(text) = message.text() else {
|
||||
return false;
|
||||
};
|
||||
let chat_data = CHAT_STORE.get(chat_id).await;
|
||||
let Some(edit) = chat_data.edit_message.get(&(reply.id.0 as i64)) else {
|
||||
return false;
|
||||
};
|
||||
let Some(first_forward_id) = edit.forward_message_ids.first() else {
|
||||
return false;
|
||||
};
|
||||
let link = format!(
|
||||
"<a href=\"{0}\">{1}</a>",
|
||||
html_escape::encode_double_quoted_attribute(&edit.url),
|
||||
html_escape::encode_text(text)
|
||||
);
|
||||
let new_text = if edit.template.is_empty() {
|
||||
link
|
||||
} else {
|
||||
chat_data
|
||||
.template
|
||||
.get(&edit.template)
|
||||
.map(|template| template.replace("[]", &link))
|
||||
.unwrap_or(link)
|
||||
};
|
||||
let result = bot
|
||||
.edit_message_caption(ChatId(chat_id), MessageId(*first_forward_id as i32))
|
||||
.caption(new_text)
|
||||
.parse_mode(ParseMode::Html)
|
||||
.await;
|
||||
match result {
|
||||
Ok(_) => log::info!(
|
||||
"edit-before-forward: caption swapped on message {first_forward_id} for prompt {}",
|
||||
reply.id.0
|
||||
),
|
||||
Err(e) => log::error!("edit_message_caption failed: {e}"),
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestError> {
|
||||
let is_private = matches!(message.chat.kind, ChatKind::Private(_));
|
||||
let sender = message
|
||||
.from
|
||||
.as_ref()
|
||||
.map(|from| from.full_name())
|
||||
.unwrap_or_else(|| "unknown".to_string());
|
||||
let text_preview = message
|
||||
.text()
|
||||
.map(|t| {
|
||||
let end = t.floor_char_boundary(120.min(t.len()));
|
||||
&t[..end]
|
||||
})
|
||||
.unwrap_or("<no text>");
|
||||
// Per-request detail: debug only (message text is user data).
|
||||
log::debug!(
|
||||
"message from {sender} in {} (private={is_private}): {text_preview}",
|
||||
message.chat.id
|
||||
);
|
||||
// URL/edit flows only run in private chats; commands run in any chat.
|
||||
if is_private && edit_message_handler(&bot, &message).await {
|
||||
return respond(());
|
||||
}
|
||||
if let Some(text) = message.text()
|
||||
&& let Ok(command) = Command::parse(text, "")
|
||||
{
|
||||
log::debug!("command from {}: {text_preview}", message.chat.id);
|
||||
execute_command(&bot, &message, command).await?;
|
||||
return respond(());
|
||||
}
|
||||
if is_private {
|
||||
let urls = extract_urls(&message);
|
||||
if !urls.is_empty() {
|
||||
// Debug only, and echo the normalized keys instead of the raw URLs.
|
||||
let keys: Vec<String> = urls.iter().map(|u| log_key(u)).collect();
|
||||
log::debug!("extracted {} URL(s): {keys:?}", urls.len());
|
||||
}
|
||||
for url in urls {
|
||||
// Clone out of the lock: the parking_lot guard is !Send and must
|
||||
// not be held across the await below.
|
||||
let Some(tx) = URL_JOBS.lock().clone() else {
|
||||
log::warn!("url workers not started; dropping link");
|
||||
break;
|
||||
};
|
||||
let _ = tx.send((message.clone(), url)).await;
|
||||
}
|
||||
}
|
||||
respond(())
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
//! Process-wide singletons shared by the handler modules: the one SQLite
|
||||
//! pool (and the three stores built on it) plus the configuration.
|
||||
|
||||
use crate::config::Config;
|
||||
use crate::db::{self};
|
||||
use crate::link_cache::LinkCache;
|
||||
use crate::queue::PersistentTaskQueue;
|
||||
use crate::state::ChatStore;
|
||||
use std::sync::{Arc, LazyLock};
|
||||
|
||||
/// One shared SQLite pool for the three stores (chat state, task queue, link
|
||||
/// cache): a single pool bounds concurrent DB work on `data/task_queue.db`
|
||||
/// instead of three independent pools competing for the same file. The schema
|
||||
/// for all three tables is initialized once, here.
|
||||
static DB: LazyLock<Arc<db::DbPool>> = LazyLock::new(|| {
|
||||
let path = db_path();
|
||||
db::open_store(&path.to_string_lossy()).expect("failed to open database")
|
||||
});
|
||||
|
||||
/// DB file location: `$DATA_DIR/task_queue.db` (default `data`, relative to
|
||||
/// the working directory — keeps the docker-compose `./data` mount and local
|
||||
/// runs unchanged). The directory is created if missing: SQLite does not
|
||||
/// create parent dirs, so the old hardcoded `data/task_queue.db` failed with
|
||||
/// a confusing error when started from a directory without `data/`, and a
|
||||
/// CWD-relative path is a footgun for systemd / cron deployments — `DATA_DIR`
|
||||
/// lets them pin the state anywhere.
|
||||
fn db_path() -> std::path::PathBuf {
|
||||
let dir = std::env::var("DATA_DIR").unwrap_or_else(|_| "data".to_string());
|
||||
let dir_path = std::path::Path::new(&dir);
|
||||
std::fs::create_dir_all(dir_path).expect("failed to create data directory");
|
||||
dir_path.join("task_queue.db")
|
||||
}
|
||||
|
||||
pub static CHAT_STORE: LazyLock<ChatStore> = LazyLock::new(|| ChatStore::new(Arc::clone(&DB)));
|
||||
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
|
||||
LazyLock::new(|| PersistentTaskQueue::new(Arc::clone(&DB)));
|
||||
pub static LINK_CACHE: LazyLock<LinkCache> = LazyLock::new(|| LinkCache::new(Arc::clone(&DB)));
|
||||
pub static CONFIG: LazyLock<Config> = LazyLock::new(Config::load);
|
||||
@@ -0,0 +1,560 @@
|
||||
//! URL extraction and the per-URL media pipeline: bounded job channel +
|
||||
//! worker pool, link-cache fast path, fetch, task build and send dispatch.
|
||||
|
||||
use super::{CHAT_STORE, CONFIG, LINK_CACHE, TASK_QUEUE, log_key, reply};
|
||||
use crate::config::Config;
|
||||
use crate::db::now_f64;
|
||||
use crate::link_cache::{CachedMediaKind, CachedPost, LinkCache};
|
||||
use crate::media_sender::MediaSender;
|
||||
use crate::queue::PersistentTaskQueue;
|
||||
use crate::send::{self, MediaItemPayload, Task};
|
||||
use crate::state::{ChatData, ChatStore};
|
||||
use std::collections::HashSet;
|
||||
use std::sync::LazyLock;
|
||||
use teloxide::types::{ChatAction, ChatId, Message, MessageEntityKind, MessageId};
|
||||
use x_media::media::Media;
|
||||
|
||||
/// One URL job: the message + the extracted URL (the sender and stores come
|
||||
/// from the shared [`AppContext`], assembled from statics inside the worker).
|
||||
type UrlJob = (Message, String);
|
||||
/// Bounded channel of URL jobs drained by [`start_url_workers`]. The bound
|
||||
/// caps both queued memory and shutdown backlog; a full channel applies
|
||||
/// backpressure to the per-chat handler instead of spawning unbounded tasks.
|
||||
pub(crate) static URL_JOBS: LazyLock<
|
||||
parking_lot::Mutex<Option<tokio::sync::mpsc::Sender<UrlJob>>>,
|
||||
> = LazyLock::new(|| parking_lot::Mutex::new(None));
|
||||
/// Set by main's shutdown sequence; workers stop pulling new jobs.
|
||||
pub(crate) static URL_STOP: std::sync::atomic::AtomicBool =
|
||||
std::sync::atomic::AtomicBool::new(false);
|
||||
|
||||
/// JoinHandles of the URL workers, awaited by [`stop_url_workers`].
|
||||
static URL_WORKER_HANDLES: LazyLock<parking_lot::Mutex<Option<Vec<tokio::task::JoinHandle<()>>>>> =
|
||||
LazyLock::new(|| parking_lot::Mutex::new(None));
|
||||
|
||||
/// Worker count draining URL jobs; keeps the old 8-permit concurrency cap
|
||||
/// while bounding how many jobs can be queued at all.
|
||||
const URL_WORKERS: usize = 8;
|
||||
|
||||
/// Dependencies of the per-URL pipeline, injected so tests can substitute a
|
||||
/// mock sender and tempdir-backed stores.
|
||||
pub(crate) struct AppContext<'a> {
|
||||
pub sender: &'a dyn MediaSender,
|
||||
pub chat_store: &'a ChatStore,
|
||||
pub task_queue: &'a PersistentTaskQueue,
|
||||
pub link_cache: &'a LinkCache,
|
||||
pub config: &'a Config,
|
||||
}
|
||||
|
||||
/// Assembles the production context from the process-wide statics.
|
||||
fn app_context() -> AppContext<'static> {
|
||||
AppContext {
|
||||
sender: &*crate::send::BOT,
|
||||
chat_store: &CHAT_STORE,
|
||||
task_queue: &TASK_QUEUE,
|
||||
link_cache: &LINK_CACHE,
|
||||
config: &CONFIG,
|
||||
}
|
||||
}
|
||||
|
||||
/// Starts the URL job workers (called once from main after the queue starts).
|
||||
/// teloxide dispatches updates to a per-chat worker that handles them
|
||||
/// sequentially, so a batch-forward of many messages would otherwise be
|
||||
/// processed one at a time (fetch + send each, roughly a second per
|
||||
/// message); the workers add throughput, and FIFO order preserves per-message
|
||||
/// URL order.
|
||||
pub async fn start_url_workers() {
|
||||
let (tx, rx) = tokio::sync::mpsc::channel::<UrlJob>(256);
|
||||
*URL_JOBS.lock() = Some(tx);
|
||||
let rx = std::sync::Arc::new(tokio::sync::Mutex::new(rx));
|
||||
let mut handles = Vec::with_capacity(URL_WORKERS);
|
||||
for _ in 0..URL_WORKERS {
|
||||
let rx = std::sync::Arc::clone(&rx);
|
||||
handles.push(tokio::spawn(async move {
|
||||
let ctx = app_context();
|
||||
while !URL_STOP.load(std::sync::atomic::Ordering::Relaxed) {
|
||||
let job = rx.lock().await.recv().await;
|
||||
match job {
|
||||
Some((message, url)) => {
|
||||
url_media(&ctx, message.chat.id.0, message.id.0 as i64, &url).await
|
||||
}
|
||||
None => break,
|
||||
}
|
||||
}
|
||||
}));
|
||||
}
|
||||
*URL_WORKER_HANDLES.lock() = Some(handles);
|
||||
}
|
||||
|
||||
/// Stops the URL workers: sets the stop flag, drops the job channel (so
|
||||
/// workers blocked in \`recv()\` wake with \`None\` and exit) and awaits the
|
||||
/// worker tasks. Each worker finishes its in-flight job first; jobs still
|
||||
/// queued in the channel are abandoned (the old implementation neither
|
||||
/// drained them nor woke blocked workers — it only set a flag checked
|
||||
/// between jobs).
|
||||
pub async fn stop_url_workers() {
|
||||
URL_STOP.store(true, std::sync::atomic::Ordering::Relaxed);
|
||||
// Dropping the sender makes every worker's recv() return None.
|
||||
*URL_JOBS.lock() = None;
|
||||
// Take the handles first so the lock guard drops before the awaits.
|
||||
let handles = URL_WORKER_HANDLES.lock().take();
|
||||
if let Some(handles) = handles {
|
||||
for handle in handles {
|
||||
let _ = handle.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Extracts URL and text-link entities (text + caption), deduped in order.
|
||||
pub fn extract_urls(message: &Message) -> Vec<String> {
|
||||
let mut urls = Vec::new();
|
||||
for entity in message.parse_entities().into_iter().flatten() {
|
||||
match entity.kind() {
|
||||
MessageEntityKind::Url => urls.push(entity.text().to_string()),
|
||||
MessageEntityKind::TextLink { url } => urls.push(url.to_string()),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
for entity in message.parse_caption_entities().into_iter().flatten() {
|
||||
match entity.kind() {
|
||||
MessageEntityKind::Url => urls.push(entity.text().to_string()),
|
||||
MessageEntityKind::TextLink { url } => urls.push(url.to_string()),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
let mut seen = HashSet::new();
|
||||
// Dedup by the normalized post id so variant URLs of the same post
|
||||
// (/status/1 vs /status/1/photo/1) are sent once; unsupported URLs fall
|
||||
// back to exact-string dedup.
|
||||
urls.retain(|url| seen.insert(x_media::site::cache_key(url).unwrap_or_else(|| url.clone())));
|
||||
urls
|
||||
}
|
||||
|
||||
/// For locally produced media (encoded ugoira MP4) the thumbnail URL is a
|
||||
/// hotlink-protected remote URL Telegram may not fetch; let Telegram generate
|
||||
/// its own thumbnail instead.
|
||||
fn thumbnail_for(media: &Media) -> Option<String> {
|
||||
let url = media.url();
|
||||
if url.starts_with("http://") || url.starts_with("https://") {
|
||||
media.thumbnail_url().map(str::to_string)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
fn media_to_payload(media: &Media, sensitive: bool) -> MediaItemPayload {
|
||||
let fallback_url = media.smaller_url().map(str::to_string);
|
||||
match media {
|
||||
// A gif inside a group becomes a video item; a lone gif takes the
|
||||
// animation path (see url_media).
|
||||
Media::Illustration { .. } => MediaItemPayload::Photo {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
Media::Video { .. } => MediaItemPayload::Video {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
thumbnail: thumbnail_for(media),
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
Media::Animated { .. } => MediaItemPayload::Video {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
thumbnail: thumbnail_for(media),
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_retry(queue: &PersistentTaskQueue, task: Task, delay_seconds: f64) {
|
||||
let payload = serde_json::to_value(task).expect("task serializes");
|
||||
let run_after = now_f64() + delay_seconds;
|
||||
if let Err(e) = queue.enqueue(payload, run_after).await {
|
||||
log::error!("failed to enqueue retry: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Sends a task and handles the outcome: post-send actions on success, retry
|
||||
/// enqueue on retryable failure, reply + link-cache invalidation on
|
||||
/// permanent failure (a stale cached file id must not repeat forever).
|
||||
async fn dispatch_send(
|
||||
ctx: &AppContext<'_>,
|
||||
chat_id: i64,
|
||||
reply_to: MessageId,
|
||||
task: &Task,
|
||||
url: &str,
|
||||
) {
|
||||
let result = match task {
|
||||
Task::SendAnimation { .. } => send::send_animation(ctx.sender, task).await,
|
||||
Task::SendMediaSequence { .. } => send::send_media_sequence(ctx.sender, task).await,
|
||||
Task::ForwardMessages { .. } => unreachable!(),
|
||||
};
|
||||
match result {
|
||||
Ok(message_ids) => {
|
||||
log::info!(
|
||||
"sent {} message(s) for [key={}]",
|
||||
message_ids.len(),
|
||||
log_key(url)
|
||||
);
|
||||
send::post_send_actions(ctx.sender, task, message_ids).await;
|
||||
// The task settled: drop any keep-alive temp media.
|
||||
send::release_keep_alive(task);
|
||||
}
|
||||
Err(send::SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => {
|
||||
log::info!(
|
||||
"send for [key={}] failed, queued for retry in {delay_seconds:.1}s",
|
||||
log_key(url)
|
||||
);
|
||||
enqueue_retry(ctx.task_queue, task, delay_seconds).await;
|
||||
let _ = reply(
|
||||
ctx.sender,
|
||||
chat_id,
|
||||
reply_to,
|
||||
"Send failed. Task queued for retry.",
|
||||
)
|
||||
.await;
|
||||
}
|
||||
Err(send::SendError::Permanent {
|
||||
message: err_message,
|
||||
task,
|
||||
}) => {
|
||||
send::invalidate_cache_with(ctx.link_cache, &task).await;
|
||||
send::release_keep_alive(&task);
|
||||
log::error!("send for {url} failed permanently: {err_message}");
|
||||
let _ = reply(
|
||||
ctx.sender,
|
||||
chat_id,
|
||||
reply_to,
|
||||
format!("Send failed: {err_message}"),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Builds the send task from ready-made items, sharing the payload shape
|
||||
/// between the fresh-fetch and link-cache paths.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn build_send_task(
|
||||
chat_data: &ChatData,
|
||||
chat_id: i64,
|
||||
reply_to_message_id: i64,
|
||||
source_url: String,
|
||||
caption: String,
|
||||
items: Vec<MediaItemPayload>,
|
||||
cache_data: Option<CachedPost>,
|
||||
) -> Task {
|
||||
if items.len() == 1 && matches!(items[0], MediaItemPayload::Animation { .. }) {
|
||||
Task::SendAnimation {
|
||||
chat_id,
|
||||
reply_to_message_id,
|
||||
caption,
|
||||
animation: items.into_iter().next().unwrap(),
|
||||
source_url,
|
||||
edit_before_forward: chat_data.edit_before_forward,
|
||||
forward_channel_id: chat_data.forward_channel_id,
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(reply_to_message_id),
|
||||
cache_data,
|
||||
}
|
||||
} else {
|
||||
Task::SendMediaSequence {
|
||||
chat_id,
|
||||
reply_to_message_id,
|
||||
caption,
|
||||
// Photos first so a mixed photo+video group starts with a photo
|
||||
// (Telegram's sendMediaGroup rule); order within each kind is kept.
|
||||
media_batches: send::chunk_media_items(send::photos_first(items)),
|
||||
batch_index: 0,
|
||||
sent_message_ids: vec![],
|
||||
source_url,
|
||||
edit_before_forward: chat_data.edit_before_forward,
|
||||
forward_channel_id: chat_data.forward_channel_id,
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(reply_to_message_id),
|
||||
cache_data,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn url_media(ctx: &AppContext<'_>, chat_id: i64, reply_to_message_id: i64, url: &str) {
|
||||
let reply_to = MessageId(reply_to_message_id as i32);
|
||||
if let Err(e) = ctx
|
||||
.sender
|
||||
.send_chat_action(ChatId(chat_id), ChatAction::Typing)
|
||||
.await
|
||||
{
|
||||
log::error!("send_chat_action failed: {e}");
|
||||
}
|
||||
|
||||
// Link cache: a post sent before is re-sent from Telegram file ids —
|
||||
// no source-site request, no download, no upload. Keyed by the
|
||||
// normalized post id so x.com / fxtwitter / /photo/N variants collide.
|
||||
if let Some(key) = x_media::site::cache_key(url)
|
||||
&& let Some(cached) = ctx.link_cache.get(&key, ctx.config.link_cache_ttl).await
|
||||
{
|
||||
log::debug!("link cache hit for {key}");
|
||||
let chat_data = ctx.chat_store.get(chat_id).await;
|
||||
// Cache keys are prefixed with the site id ("twitter:…"), matching
|
||||
// the value a fresh fetch would read from Fetched::site_id.
|
||||
let site = x_media::site::site_id_from_key(&key);
|
||||
let format = chat_data
|
||||
.message_format
|
||||
.get(site)
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
let caption = if format.is_empty() {
|
||||
x_media::site::truncate_caption(&cached.caption)
|
||||
} else {
|
||||
x_media::site::caption_from_fields(
|
||||
&format,
|
||||
"",
|
||||
&cached.url,
|
||||
&cached.author,
|
||||
&cached.author_url,
|
||||
&cached.title,
|
||||
&cached.tags,
|
||||
)
|
||||
};
|
||||
let items: Vec<MediaItemPayload> = cached
|
||||
.media
|
||||
.iter()
|
||||
.map(|m| match m.kind {
|
||||
CachedMediaKind::Photo => MediaItemPayload::Photo {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
fallback_url: None,
|
||||
file_id: true,
|
||||
},
|
||||
CachedMediaKind::Video => MediaItemPayload::Video {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
thumbnail: None,
|
||||
fallback_url: None,
|
||||
file_id: true,
|
||||
},
|
||||
CachedMediaKind::Animation => MediaItemPayload::Animation {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
file_id: true,
|
||||
},
|
||||
})
|
||||
.collect();
|
||||
let task = build_send_task(
|
||||
&chat_data,
|
||||
chat_id,
|
||||
reply_to_message_id,
|
||||
cached.url.clone(),
|
||||
caption,
|
||||
items,
|
||||
Some(cached),
|
||||
);
|
||||
dispatch_send(ctx, chat_id, reply_to, &task, url).await;
|
||||
return;
|
||||
}
|
||||
|
||||
log::debug!("fetching {url} [key={}]", log_key(url));
|
||||
match x_media::site::fetch(url).await {
|
||||
// Unsupported links are ignored silently (Python parity).
|
||||
Ok(None) => {
|
||||
log::debug!("no site pattern matches {url}; ignoring");
|
||||
}
|
||||
// Retries exhausted: notify the user (Rust-only requirement 3).
|
||||
Err(e) => {
|
||||
log::error!("fetch {url}: {e}");
|
||||
let _ = reply(
|
||||
ctx.sender,
|
||||
chat_id,
|
||||
reply_to,
|
||||
"Failed to fetch media from this link.",
|
||||
)
|
||||
.await;
|
||||
}
|
||||
Ok(Some(mut fetched)) => {
|
||||
if fetched.media.is_empty() {
|
||||
let _ = reply(
|
||||
ctx.sender,
|
||||
chat_id,
|
||||
reply_to,
|
||||
"No media found or media type is not supported.",
|
||||
)
|
||||
.await;
|
||||
return;
|
||||
}
|
||||
let chat_data = ctx.chat_store.get(chat_id).await;
|
||||
// Per-site caption format override (empty -> built-in caption).
|
||||
let format = chat_data
|
||||
.message_format
|
||||
.get(fetched.site_name())
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
let caption = fetched.caption_with(&format);
|
||||
// Raw render data for the link cache; the send fills in the
|
||||
// Telegram file ids and persists the entry.
|
||||
let cache_data = fetched
|
||||
.render_fields()
|
||||
.map(|(author, author_url, title, tags)| CachedPost {
|
||||
url: fetched.source_url.clone(),
|
||||
caption: fetched.caption.clone(),
|
||||
title: title.to_string(),
|
||||
author: author.to_string(),
|
||||
author_url: author_url.to_string(),
|
||||
tags: tags.to_string(),
|
||||
sensitive: fetched.sensitive,
|
||||
media: vec![],
|
||||
});
|
||||
let items: Vec<MediaItemPayload> = fetched
|
||||
.media
|
||||
.iter()
|
||||
.map(|media| media_to_payload(media, fetched.sensitive))
|
||||
.collect();
|
||||
let task = build_send_task(
|
||||
&chat_data,
|
||||
chat_id,
|
||||
reply_to_message_id,
|
||||
fetched.source_url.clone(),
|
||||
caption,
|
||||
items,
|
||||
cache_data,
|
||||
);
|
||||
// Hand the keep-alive temp dir (ugoira / bsky remux MP4) to the
|
||||
// retry registry: a queued retry runs after this function returns
|
||||
// and the fetch's own TempDir is dropped, so without this the
|
||||
// local file would be gone by the time the retry sends it.
|
||||
if let Some(dir) = fetched.take_keep_alive() {
|
||||
send::KEEP_ALIVE.lock().push(dir);
|
||||
}
|
||||
dispatch_send(ctx, chat_id, reply_to, &task, url).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::db;
|
||||
use crate::link_cache::CachedMedia;
|
||||
use crate::media_sender::test_support::{MockSender, Outcome};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use teloxide::{ApiError, RequestError};
|
||||
|
||||
fn permanent_error() -> RequestError {
|
||||
RequestError::Api(ApiError::Unknown(
|
||||
"Bad Request: message is not modified".into(),
|
||||
))
|
||||
}
|
||||
|
||||
fn cached_photo_entry() -> CachedPost {
|
||||
CachedPost {
|
||||
url: "https://x.com/u/status/1".into(),
|
||||
caption: "cap".into(),
|
||||
title: "t".into(),
|
||||
author: "a".into(),
|
||||
author_url: "au".into(),
|
||||
tags: "".into(),
|
||||
sensitive: false,
|
||||
media: vec![CachedMedia {
|
||||
kind: CachedMediaKind::Photo,
|
||||
file_id: "file-1".into(),
|
||||
}],
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cache_hit_sends_file_ids_and_invalidates_on_permanent_failure() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let pool = db::open_store(dir.path().join("t.db").to_str().unwrap()).unwrap();
|
||||
let chat_store = ChatStore::new(Arc::clone(&pool));
|
||||
let task_queue = PersistentTaskQueue::new(Arc::clone(&pool));
|
||||
let link_cache = LinkCache::new(Arc::clone(&pool));
|
||||
let config = Config::load();
|
||||
let sender = MockSender::scripted(
|
||||
vec![Outcome::GroupErr, Outcome::MessageErr],
|
||||
permanent_error,
|
||||
);
|
||||
let ctx = AppContext {
|
||||
sender: &sender,
|
||||
chat_store: &chat_store,
|
||||
task_queue: &task_queue,
|
||||
link_cache: &link_cache,
|
||||
config: &config,
|
||||
};
|
||||
link_cache.put("twitter:1", &cached_photo_entry()).await;
|
||||
|
||||
url_media(&ctx, 1, 2, "https://x.com/u/status/1").await;
|
||||
|
||||
// The cached file id went out as a group send; the permanent failure
|
||||
// then triggered the fire-and-forget reply (its mock error is fine).
|
||||
assert_eq!(
|
||||
sender.calls(),
|
||||
vec!["send_chat_action", "send_media_group", "send_message"]
|
||||
);
|
||||
// The stale cache entry was invalidated so the next request re-fetches.
|
||||
assert!(
|
||||
link_cache
|
||||
.get("twitter:1", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cache_hit_success_keeps_the_cache_entry() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let pool = db::open_store(dir.path().join("t.db").to_str().unwrap()).unwrap();
|
||||
let chat_store = ChatStore::new(Arc::clone(&pool));
|
||||
let task_queue = PersistentTaskQueue::new(Arc::clone(&pool));
|
||||
let link_cache = LinkCache::new(Arc::clone(&pool));
|
||||
let config = Config::load();
|
||||
let sender = MockSender::scripted(vec![Outcome::GroupOk], permanent_error);
|
||||
let ctx = AppContext {
|
||||
sender: &sender,
|
||||
chat_store: &chat_store,
|
||||
task_queue: &task_queue,
|
||||
link_cache: &link_cache,
|
||||
config: &config,
|
||||
};
|
||||
link_cache.put("twitter:1", &cached_photo_entry()).await;
|
||||
|
||||
url_media(&ctx, 1, 2, "https://x.com/u/status/1").await;
|
||||
|
||||
assert_eq!(sender.calls(), vec!["send_chat_action", "send_media_group"]);
|
||||
// Success must not evict the entry.
|
||||
assert!(
|
||||
link_cache
|
||||
.get("twitter:1", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_some()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unsupported_url_is_ignored_silently() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let pool = db::open_store(dir.path().join("t.db").to_str().unwrap()).unwrap();
|
||||
let chat_store = ChatStore::new(Arc::clone(&pool));
|
||||
let task_queue = PersistentTaskQueue::new(Arc::clone(&pool));
|
||||
let link_cache = LinkCache::new(Arc::clone(&pool));
|
||||
let config = Config::load();
|
||||
let sender = MockSender::scripted(vec![], permanent_error);
|
||||
let ctx = AppContext {
|
||||
sender: &sender,
|
||||
chat_store: &chat_store,
|
||||
task_queue: &task_queue,
|
||||
link_cache: &link_cache,
|
||||
config: &config,
|
||||
};
|
||||
|
||||
// No cache key → the fetch dispatcher returns Ok(None) without any
|
||||
// network; nothing is sent or replied.
|
||||
url_media(&ctx, 1, 2, "https://example.com/not-a-post").await;
|
||||
assert_eq!(sender.calls(), vec!["send_chat_action"]);
|
||||
}
|
||||
}
|
||||
@@ -9,8 +9,9 @@
|
||||
//! by the periodic prune in `main`.
|
||||
|
||||
use crate::db::now_f64;
|
||||
use rusqlite::{Connection, params};
|
||||
use rusqlite::params;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)]
|
||||
@@ -45,24 +46,16 @@ pub struct CachedPost {
|
||||
}
|
||||
|
||||
/// SQLite-backed cache sharing `data/task_queue.db` with the queue and chat
|
||||
/// state (same `open_db` pattern: busy timeout, `spawn_blocking` I/O).
|
||||
/// state (same shared pool, see [`crate::db::open_store`]).
|
||||
pub struct LinkCache {
|
||||
db_path: String,
|
||||
pool: Arc<crate::db::DbPool>,
|
||||
}
|
||||
|
||||
impl LinkCache {
|
||||
pub fn open(db_path: &str) -> Self {
|
||||
if let Ok(conn) = Connection::open(db_path)
|
||||
&& let Err(e) = conn.execute_batch(
|
||||
"CREATE TABLE IF NOT EXISTS link_cache (url TEXT PRIMARY KEY, \
|
||||
payload TEXT NOT NULL, created_at REAL NOT NULL);",
|
||||
)
|
||||
{
|
||||
log::error!("failed to initialize link cache schema: {e}");
|
||||
}
|
||||
Self {
|
||||
db_path: db_path.to_string(),
|
||||
}
|
||||
/// Wraps the shared DB pool (the `link_cache` table lives in the merged
|
||||
/// schema alongside `tasks` and `chat_state`).
|
||||
pub fn new(pool: Arc<crate::db::DbPool>) -> Self {
|
||||
LinkCache { pool }
|
||||
}
|
||||
|
||||
/// Returns the cached post if present and not expired; a stale entry is
|
||||
@@ -70,24 +63,26 @@ impl LinkCache {
|
||||
pub async fn get(&self, key: &str, ttl: Duration) -> Option<CachedPost> {
|
||||
let key = key.to_string();
|
||||
let ttl = ttl.as_secs_f64();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
|
||||
let mut rows = stmt.query(params![key])?;
|
||||
let Some(row) = rows.next()? else {
|
||||
return Ok(None);
|
||||
};
|
||||
let payload: String = row.get(0)?;
|
||||
let created_at: f64 = row.get(1)?;
|
||||
if now_f64() - created_at > ttl {
|
||||
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
|
||||
return Ok(None);
|
||||
}
|
||||
Ok(Some(serde_json::from_str::<CachedPost>(&payload).map_err(
|
||||
|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)),
|
||||
)?))
|
||||
})
|
||||
.await;
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(move |conn| {
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
|
||||
let mut rows = stmt.query(params![key])?;
|
||||
let Some(row) = rows.next()? else {
|
||||
return Ok(None);
|
||||
};
|
||||
let payload: String = row.get(0)?;
|
||||
let created_at: f64 = row.get(1)?;
|
||||
if now_f64() - created_at > ttl {
|
||||
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
|
||||
return Ok(None);
|
||||
}
|
||||
Ok(Some(serde_json::from_str::<CachedPost>(&payload).map_err(
|
||||
|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)),
|
||||
)?))
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(v) => v,
|
||||
Err(e) => {
|
||||
@@ -100,14 +95,16 @@ impl LinkCache {
|
||||
pub async fn put(&self, key: &str, post: &CachedPost) {
|
||||
let key = key.to_string();
|
||||
let payload = serde_json::to_string(post).expect("cached post serializes");
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(move |conn| {
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
|
||||
params![key, payload, now_f64()],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("link cache write failed: {e}");
|
||||
}
|
||||
@@ -116,11 +113,13 @@ impl LinkCache {
|
||||
/// Drops an entry (e.g. a cached file id that turned out invalid).
|
||||
pub async fn remove(&self, key: &str) {
|
||||
let key = key.to_string();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(move |conn| {
|
||||
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("link cache delete failed: {e}");
|
||||
}
|
||||
@@ -129,13 +128,15 @@ impl LinkCache {
|
||||
/// Removes expired entries; returns how many were deleted.
|
||||
pub async fn prune(&self, ttl: Duration) -> usize {
|
||||
let cutoff = now_f64() - ttl.as_secs_f64();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
"DELETE FROM link_cache WHERE created_at < ?1",
|
||||
params![cutoff],
|
||||
)
|
||||
})
|
||||
.await;
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(move |conn| {
|
||||
conn.execute(
|
||||
"DELETE FROM link_cache WHERE created_at < ?1",
|
||||
params![cutoff],
|
||||
)
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(n) => n,
|
||||
Err(e) => {
|
||||
@@ -149,11 +150,13 @@ impl LinkCache {
|
||||
/// `key` is `None`. Returns how many rows were removed.
|
||||
pub async fn clear(&self, key: Option<&str>) -> usize {
|
||||
let key = key.map(str::to_string);
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| match &key {
|
||||
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
|
||||
None => conn.execute("DELETE FROM link_cache", []),
|
||||
})
|
||||
.await;
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(move |conn| match &key {
|
||||
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
|
||||
None => conn.execute("DELETE FROM link_cache", []),
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(n) => n,
|
||||
Err(e) => {
|
||||
@@ -187,7 +190,9 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn put_get_roundtrip() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
|
||||
let cache = LinkCache::new(
|
||||
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
|
||||
);
|
||||
cache.put("twitter:1", &entry()).await;
|
||||
let got = cache.get("twitter:1", Duration::from_secs(3600)).await;
|
||||
assert!(got.is_some());
|
||||
@@ -199,11 +204,13 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn expired_entry_removed_on_read() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
|
||||
let cache = LinkCache::new(
|
||||
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
|
||||
);
|
||||
cache.put("twitter:1", &entry()).await;
|
||||
// Force the row into the past so a 1s TTL expires it.
|
||||
{
|
||||
let conn = Connection::open(dir.path().join("c.db")).unwrap();
|
||||
let conn = rusqlite::Connection::open(dir.path().join("c.db")).unwrap();
|
||||
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
|
||||
.unwrap();
|
||||
}
|
||||
@@ -224,7 +231,9 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn remove_and_prune() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
|
||||
let cache = LinkCache::new(
|
||||
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
|
||||
);
|
||||
cache.put("twitter:1", &entry()).await;
|
||||
cache.put("pixiv:2", &entry()).await;
|
||||
cache.remove("twitter:1").await;
|
||||
@@ -241,7 +250,7 @@ mod tests {
|
||||
.is_some()
|
||||
);
|
||||
{
|
||||
let conn = Connection::open(dir.path().join("c.db")).unwrap();
|
||||
let conn = rusqlite::Connection::open(dir.path().join("c.db")).unwrap();
|
||||
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
|
||||
.unwrap();
|
||||
}
|
||||
@@ -257,7 +266,9 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn clear_one_entry_or_all() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
|
||||
let cache = LinkCache::new(
|
||||
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
|
||||
);
|
||||
cache.put("twitter:1", &entry()).await;
|
||||
cache.put("pixiv:2", &entry()).await;
|
||||
// By key: only the matching row is removed.
|
||||
|
||||
@@ -11,8 +11,10 @@ mod config;
|
||||
mod db;
|
||||
mod handlers;
|
||||
mod link_cache;
|
||||
mod media_sender;
|
||||
mod photo;
|
||||
mod queue;
|
||||
mod rate_limit;
|
||||
mod send;
|
||||
mod state;
|
||||
|
||||
@@ -69,19 +71,18 @@ async fn main() {
|
||||
handlers::start_url_workers().await;
|
||||
log::info!("url workers started");
|
||||
|
||||
// Pixiv login validation (user request): a failed login notifies the
|
||||
// admin and disables pixiv for this process.
|
||||
if site::pixiv::enabled() {
|
||||
match site::pixiv::validate().await {
|
||||
Ok(()) => log::info!("pixiv login validated"),
|
||||
Err(e) => {
|
||||
log::error!("pixiv login failed: {e}");
|
||||
if let Some(admin) = CONFIG.admin_ids.first() {
|
||||
let _ = bot
|
||||
.send_message(ChatId(*admin), format!("Pixiv login failed: {e}"))
|
||||
.await;
|
||||
}
|
||||
site::pixiv::disable();
|
||||
// Site login validation (user request): a failed login notifies the
|
||||
// admin and the site disables itself for this process (pixiv).
|
||||
let failures = site::validate_all().await;
|
||||
if failures.is_empty() {
|
||||
log::info!("site logins validated");
|
||||
} else {
|
||||
for (site_id, message) in &failures {
|
||||
log::error!("{site_id} login failed: {message}");
|
||||
if let Some(admin) = CONFIG.admin_ids.first() {
|
||||
let _ = bot
|
||||
.send_message(ChatId(*admin), format!("{site_id} login failed: {message}"))
|
||||
.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -178,13 +179,25 @@ async fn main() {
|
||||
.await;
|
||||
}
|
||||
|
||||
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin, drain the queue.
|
||||
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin,
|
||||
// drain the queue. Bounded: a worker mid-download (30 s timeout) or a
|
||||
// long ugoira encode must not hold the shutdown hostage forever.
|
||||
log::info!("Stopping bot");
|
||||
let _ = stop_tx.send(true);
|
||||
handlers::stop_url_workers();
|
||||
if let Some(admin) = CONFIG.admin_ids.first() {
|
||||
let _ = bot.send_message(ChatId(*admin), "Shutting down...").await;
|
||||
const SHUTDOWN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||
let shutdown = async {
|
||||
let _ = stop_tx.send(true);
|
||||
handlers::stop_url_workers().await;
|
||||
if let Some(admin) = CONFIG.admin_ids.first() {
|
||||
let _ = bot.send_message(ChatId(*admin), "Shutting down...").await;
|
||||
}
|
||||
TASK_QUEUE.stop().await;
|
||||
};
|
||||
if tokio::time::timeout(SHUTDOWN_TIMEOUT, shutdown)
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
log::warn!("graceful shutdown timed out after {SHUTDOWN_TIMEOUT:?}; exiting");
|
||||
} else {
|
||||
log::info!("Bot stopped");
|
||||
}
|
||||
TASK_QUEUE.stop().await;
|
||||
log::info!("Bot stopped");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
//! Send abstraction: the message-sending surface [`send`](crate::send)
|
||||
//! needs, so the send pipeline can be tested with a scripted mock instead of
|
||||
//! a live teloxide `Bot`.
|
||||
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use teloxide::RequestError;
|
||||
use teloxide::prelude::Requester;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::types::{
|
||||
ChatAction, ChatId, InlineKeyboardMarkup, InputFile, InputMedia, Message, MessageId, ParseMode,
|
||||
ReplyParameters,
|
||||
};
|
||||
|
||||
/// Boxed, `Send` future returned by a [`MediaSender`] method (`async fn` in
|
||||
/// traits is not dyn-compatible).
|
||||
type BoxFuture<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
|
||||
|
||||
/// The message-sending surface the send pipeline uses. The production
|
||||
/// implementation is teloxide's [`Bot`]; tests inject a scripted mock to
|
||||
/// cover the fallback and classification logic without touching the
|
||||
/// Telegram API.
|
||||
pub trait MediaSender: Send + Sync {
|
||||
/// Sends a media group, replying to `reply_to`.
|
||||
fn send_media_group(
|
||||
&self,
|
||||
chat_id: ChatId,
|
||||
reply_to: MessageId,
|
||||
items: Vec<InputMedia>,
|
||||
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>>;
|
||||
|
||||
/// Sends a lone animation, replying to `reply_to`.
|
||||
fn send_animation<'a>(
|
||||
&'a self,
|
||||
chat_id: ChatId,
|
||||
reply_to: MessageId,
|
||||
caption: &'a str,
|
||||
spoiler: bool,
|
||||
file: InputFile,
|
||||
) -> BoxFuture<'a, Result<Message, RequestError>>;
|
||||
|
||||
/// Copies messages between chats (forward to channel).
|
||||
fn copy_messages(
|
||||
&self,
|
||||
to: ChatId,
|
||||
from: ChatId,
|
||||
ids: Vec<MessageId>,
|
||||
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>>;
|
||||
|
||||
/// Sends a plain text message, optionally replying to `reply_to` and
|
||||
/// attaching `reply_markup`.
|
||||
fn send_message(
|
||||
&self,
|
||||
chat_id: ChatId,
|
||||
text: String,
|
||||
reply_to: Option<MessageId>,
|
||||
reply_markup: Option<InlineKeyboardMarkup>,
|
||||
) -> BoxFuture<'_, Result<Message, RequestError>>;
|
||||
|
||||
/// Sets the chat's "typing / uploading …" indicator (cosmetic).
|
||||
fn send_chat_action(
|
||||
&self,
|
||||
chat_id: ChatId,
|
||||
action: ChatAction,
|
||||
) -> BoxFuture<'_, Result<(), RequestError>>;
|
||||
}
|
||||
|
||||
impl MediaSender for Bot {
|
||||
fn send_media_group(
|
||||
&self,
|
||||
chat_id: ChatId,
|
||||
reply_to: MessageId,
|
||||
items: Vec<InputMedia>,
|
||||
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>> {
|
||||
Box::pin(async move {
|
||||
// Pace media sends per chat (one token per item) so bursts do not
|
||||
// trip Telegram's flood control.
|
||||
crate::rate_limit::limiter_for(chat_id.0)
|
||||
.acquire(items.len() as f64)
|
||||
.await;
|
||||
// `<Bot as Requester>::` disambiguates from this trait's same-named
|
||||
// method (teloxide's API lives in the `Requester` trait).
|
||||
<Bot as Requester>::send_media_group(self, chat_id, items)
|
||||
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply())
|
||||
.await
|
||||
})
|
||||
}
|
||||
|
||||
fn send_animation<'a>(
|
||||
&'a self,
|
||||
chat_id: ChatId,
|
||||
reply_to: MessageId,
|
||||
caption: &'a str,
|
||||
spoiler: bool,
|
||||
file: InputFile,
|
||||
) -> BoxFuture<'a, Result<Message, RequestError>> {
|
||||
Box::pin(async move {
|
||||
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
|
||||
let mut request = <Bot as Requester>::send_animation(self, chat_id, file)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html)
|
||||
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply());
|
||||
if spoiler {
|
||||
request = request.has_spoiler(true);
|
||||
}
|
||||
request.await
|
||||
})
|
||||
}
|
||||
|
||||
fn copy_messages(
|
||||
&self,
|
||||
to: ChatId,
|
||||
from: ChatId,
|
||||
ids: Vec<MessageId>,
|
||||
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>> {
|
||||
Box::pin(async move {
|
||||
// Channel forwards are the burstiest path (batch copies); pace
|
||||
// them per message against the channel's budget.
|
||||
crate::rate_limit::limiter_for(to.0)
|
||||
.acquire(ids.len() as f64)
|
||||
.await;
|
||||
<Bot as Requester>::copy_messages(self, to, from, ids).await
|
||||
})
|
||||
}
|
||||
|
||||
fn send_message(
|
||||
&self,
|
||||
chat_id: ChatId,
|
||||
text: String,
|
||||
reply_to: Option<MessageId>,
|
||||
reply_markup: Option<InlineKeyboardMarkup>,
|
||||
) -> BoxFuture<'_, Result<Message, RequestError>> {
|
||||
Box::pin(async move {
|
||||
let mut request = <Bot as Requester>::send_message(self, chat_id, text);
|
||||
if let Some(reply_to) = reply_to {
|
||||
request = request
|
||||
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply());
|
||||
}
|
||||
if let Some(markup) = reply_markup {
|
||||
request = request.reply_markup(markup);
|
||||
}
|
||||
request.await
|
||||
})
|
||||
}
|
||||
|
||||
fn send_chat_action(
|
||||
&self,
|
||||
chat_id: ChatId,
|
||||
action: ChatAction,
|
||||
) -> BoxFuture<'_, Result<(), RequestError>> {
|
||||
Box::pin(async move {
|
||||
// teloxide's `send_chat_action` returns `Result<True, _>` (its
|
||||
// unit marker type); map the success to `()`.
|
||||
<Bot as Requester>::send_chat_action(self, chat_id, action)
|
||||
.await
|
||||
.map(|_| ())
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Test support: a scripted [`MediaSender`] mock (no Telegram API involved).
|
||||
#[cfg(test)]
|
||||
pub(crate) mod test_support {
|
||||
use super::*;
|
||||
use std::sync::Mutex;
|
||||
|
||||
/// One scripted outcome, consumed front-to-back; the last entry repeats
|
||||
/// for further calls of the same method kind.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum Outcome {
|
||||
GroupOk,
|
||||
GroupErr,
|
||||
AnimationErr,
|
||||
CopyOk,
|
||||
CopyErr,
|
||||
/// An error from `send_message` (replies are fire-and-forget, so an
|
||||
/// error is fine for tests).
|
||||
MessageErr,
|
||||
}
|
||||
|
||||
/// Replays a script and records the method names that were called.
|
||||
pub(crate) struct MockSender {
|
||||
script: Mutex<Vec<Outcome>>,
|
||||
cursor: Mutex<usize>,
|
||||
calls: Mutex<Vec<&'static str>>,
|
||||
/// Builds the error every `*Err` outcome returns (RequestError is not
|
||||
/// cloneable, so the factory recreates it per call).
|
||||
error: Box<dyn Fn() -> RequestError + Send + Sync>,
|
||||
}
|
||||
|
||||
impl MockSender {
|
||||
pub(crate) fn scripted(
|
||||
script: Vec<Outcome>,
|
||||
error: impl Fn() -> RequestError + Send + Sync + 'static,
|
||||
) -> Self {
|
||||
MockSender {
|
||||
script: Mutex::new(script),
|
||||
cursor: Mutex::new(0),
|
||||
calls: Mutex::new(Vec::new()),
|
||||
error: Box::new(error),
|
||||
}
|
||||
}
|
||||
|
||||
/// Method names in call order (e.g. `["send_media_group",
|
||||
/// "send_media_group"]` proves the fallback re-sent).
|
||||
pub(crate) fn calls(&self) -> Vec<&'static str> {
|
||||
self.calls.lock().unwrap().clone()
|
||||
}
|
||||
|
||||
fn next(&self, kind: &'static str) -> Outcome {
|
||||
self.calls.lock().unwrap().push(kind);
|
||||
let script = self.script.lock().unwrap();
|
||||
let mut cursor = self.cursor.lock().unwrap();
|
||||
if script.is_empty() {
|
||||
panic!("mock script exhausted: {kind}");
|
||||
}
|
||||
let idx = (*cursor).min(script.len() - 1);
|
||||
*cursor = idx + 1;
|
||||
script[idx]
|
||||
}
|
||||
|
||||
fn error(&self) -> RequestError {
|
||||
(self.error)()
|
||||
}
|
||||
}
|
||||
|
||||
impl MediaSender for MockSender {
|
||||
fn send_media_group(
|
||||
&self,
|
||||
_chat_id: ChatId,
|
||||
_reply_to: MessageId,
|
||||
_items: Vec<InputMedia>,
|
||||
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>> {
|
||||
Box::pin(async move {
|
||||
match self.next("send_media_group") {
|
||||
Outcome::GroupOk => Ok(Vec::new()),
|
||||
Outcome::GroupErr => Err(self.error()),
|
||||
other => panic!("unexpected outcome {other:?} for send_media_group"),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn send_animation<'a>(
|
||||
&'a self,
|
||||
_chat_id: ChatId,
|
||||
_reply_to: MessageId,
|
||||
_caption: &'a str,
|
||||
_spoiler: bool,
|
||||
_file: InputFile,
|
||||
) -> BoxFuture<'a, Result<Message, RequestError>> {
|
||||
Box::pin(async move {
|
||||
match self.next("send_animation") {
|
||||
Outcome::AnimationErr => Err(self.error()),
|
||||
other => panic!("unexpected outcome {other:?} for send_animation"),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn copy_messages(
|
||||
&self,
|
||||
_to: ChatId,
|
||||
_from: ChatId,
|
||||
_ids: Vec<MessageId>,
|
||||
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>> {
|
||||
Box::pin(async move {
|
||||
match self.next("copy_messages") {
|
||||
Outcome::CopyOk => Ok(vec![MessageId(1)]),
|
||||
Outcome::CopyErr => Err(self.error()),
|
||||
other => panic!("unexpected outcome {other:?} for copy_messages"),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn send_message(
|
||||
&self,
|
||||
_chat_id: ChatId,
|
||||
_text: String,
|
||||
_reply_to: Option<MessageId>,
|
||||
_reply_markup: Option<InlineKeyboardMarkup>,
|
||||
) -> BoxFuture<'_, Result<Message, RequestError>> {
|
||||
Box::pin(async move {
|
||||
match self.next("send_message") {
|
||||
Outcome::MessageErr => Err(self.error()),
|
||||
other => panic!("unexpected outcome {other:?} for send_message"),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn send_chat_action(
|
||||
&self,
|
||||
_chat_id: ChatId,
|
||||
_action: ChatAction,
|
||||
) -> BoxFuture<'_, Result<(), RequestError>> {
|
||||
Box::pin(async move {
|
||||
self.calls.lock().unwrap().push("send_chat_action");
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -67,8 +67,9 @@ impl PixBuf {
|
||||
}
|
||||
|
||||
/// Entry point: detects the format and processes the photo if needed.
|
||||
pub fn prepare_photo(file: NamedTempFile) -> Result<PhotoPrep, String> {
|
||||
let bytes = std::fs::read(file.path()).map_err(|e| format!("prepare read failed: {e}"))?;
|
||||
/// The caller hands in the already-downloaded bytes (they are in memory from
|
||||
/// the download anyway; re-reading the temp file would double the I/O).
|
||||
pub fn prepare_photo(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
|
||||
if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {
|
||||
prepare_png(file, bytes)
|
||||
} else if bytes.starts_with(&[0xFF, 0xD8, 0xFF]) {
|
||||
@@ -216,13 +217,13 @@ fn target_dims(w: u32, h: u32) -> (u32, u32) {
|
||||
/// PNG branch: decode (16→8, palette→RGB; gray/GA stay), flatten RGBA to
|
||||
/// RGB, Lanczos-downscale beyond the dimension cap, encode PNG — a PNG still
|
||||
/// over the upload cap afterwards becomes JPEG.
|
||||
fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
|
||||
let (w, h, _bit_depth, color_type) = parse_png_header(&bytes).ok_or("invalid PNG header")?;
|
||||
fn prepare_png(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
|
||||
let (w, h, _bit_depth, color_type) = parse_png_header(bytes).ok_or("invalid PNG header")?;
|
||||
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
|
||||
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
|
||||
return Ok(PhotoPrep::Upload(file));
|
||||
}
|
||||
log::info!(
|
||||
log::debug!(
|
||||
"photo {w}x{h} ({_bit_depth:?} {color_type:?}, {} bytes) needs processing",
|
||||
bytes.len()
|
||||
);
|
||||
@@ -239,7 +240,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
|
||||
png::ColorType::Indexed => png::Transformations::EXPAND,
|
||||
_ => png::Transformations::STRIP_16,
|
||||
};
|
||||
let mut decoder = png::Decoder::new(std::io::Cursor::new(&bytes));
|
||||
let mut decoder = png::Decoder::new(std::io::Cursor::new(bytes));
|
||||
decoder.set_transformations(transforms);
|
||||
let mut reader = decoder
|
||||
.read_info()
|
||||
@@ -268,7 +269,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
|
||||
let (nw, nh) = target_dims(w, h);
|
||||
pix = resize_pix(pix, w, h, nw, nh)?;
|
||||
(w, h) = (nw, nh);
|
||||
log::info!("downscaled photo to {w}x{h} (Lanczos3)");
|
||||
log::debug!("downscaled photo to {w}x{h} (Lanczos3)");
|
||||
}
|
||||
|
||||
let mut png_bytes = Vec::new();
|
||||
@@ -276,7 +277,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
|
||||
if png_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
|
||||
return Ok(PhotoPrep::Upload(write_temp(&png_bytes, "png")?));
|
||||
}
|
||||
log::info!("PNG still over the upload cap after processing; transcoding to JPEG");
|
||||
log::debug!("PNG still over the upload cap after processing; transcoding to JPEG");
|
||||
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
|
||||
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
|
||||
return Ok(PhotoPrep::Upload(write_temp(&jpeg_bytes, "jpg")?));
|
||||
@@ -286,8 +287,8 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
|
||||
}
|
||||
|
||||
/// JPEG branch: zune-jpeg decode → Lanczos downscale → jpeg-encoder output.
|
||||
fn prepare_jpeg(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
|
||||
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(&bytes));
|
||||
fn prepare_jpeg(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
|
||||
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(bytes));
|
||||
// Decodes to RGB by default. Headers first so dimensions are known before
|
||||
// the (potentially huge) pixel decode.
|
||||
decoder
|
||||
@@ -310,7 +311,7 @@ fn prepare_jpeg(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String
|
||||
let (nw, nh) = target_dims(w, h);
|
||||
pix = resize_pix(pix, w, h, nw, nh)?;
|
||||
(w, h) = (nw, nh);
|
||||
log::info!("downscaled jpeg to {w}x{h} (Lanczos3)");
|
||||
log::debug!("downscaled jpeg to {w}x{h} (Lanczos3)");
|
||||
}
|
||||
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
|
||||
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
|
||||
@@ -423,7 +424,7 @@ mod tests {
|
||||
|
||||
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
|
||||
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
|
||||
prepare_photo(file)
|
||||
prepare_photo(file, &bytes)
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -468,7 +469,7 @@ mod tests {
|
||||
}
|
||||
let mut file = tempfile::Builder::new().suffix(".jpg").tempfile().unwrap();
|
||||
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
|
||||
match prepare_photo(file).unwrap() {
|
||||
match prepare_photo(file, &bytes).unwrap() {
|
||||
PhotoPrep::Upload(file) => {
|
||||
let out = std::fs::read(file.path()).unwrap();
|
||||
assert!(out.starts_with(&[0xFF, 0xD8]), "output must stay jpeg");
|
||||
@@ -516,7 +517,7 @@ mod tests {
|
||||
|
||||
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
|
||||
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
|
||||
match prepare_photo(file).unwrap() {
|
||||
match prepare_photo(file, &bytes).unwrap() {
|
||||
PhotoPrep::Upload(file) => {
|
||||
let out = std::fs::read(file.path()).unwrap();
|
||||
assert!(out.starts_with(&[0xFF, 0xD8]), "must transcode to JPEG");
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
|
||||
use crate::db::now_f64;
|
||||
use parking_lot::Mutex;
|
||||
use rusqlite::{Connection, TransactionBehavior, params};
|
||||
use rusqlite::{TransactionBehavior, params};
|
||||
use serde_json::Value;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
@@ -40,7 +40,7 @@ type Handler = dyn Fn(Value) -> BoxFuture<'static, Result<(), QueueError>> + Sen
|
||||
type DeadLetter = dyn Fn(Value, String) -> BoxFuture<'static, ()> + Send + Sync;
|
||||
|
||||
pub struct PersistentTaskQueue {
|
||||
db_path: String,
|
||||
pool: std::sync::Arc<crate::db::DbPool>,
|
||||
notify: Arc<Notify>,
|
||||
stop: Arc<AtomicBool>,
|
||||
worker: Mutex<Vec<JoinHandle<()>>>,
|
||||
@@ -56,7 +56,7 @@ struct LeasedRow {
|
||||
/// Owned worker state so the spawned loop does not borrow the queue handle.
|
||||
#[derive(Clone)]
|
||||
struct QueueWorker {
|
||||
db_path: String,
|
||||
pool: std::sync::Arc<crate::db::DbPool>,
|
||||
notify: Arc<Notify>,
|
||||
stop: Arc<AtomicBool>,
|
||||
handler: Arc<Handler>,
|
||||
@@ -81,34 +81,12 @@ fn scaled_retry_delay(base: f64, attempts: i32) -> f64 {
|
||||
(base * 2f64.powi(attempts)).min(300.0)
|
||||
}
|
||||
|
||||
fn ensure_schema(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
|
||||
conn.execute_batch(
|
||||
"PRAGMA journal_mode=WAL; \
|
||||
CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
|
||||
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
|
||||
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
|
||||
CREATE INDEX IF NOT EXISTS idx_tasks_pending ON tasks(status, run_after);",
|
||||
)
|
||||
}
|
||||
|
||||
impl PersistentTaskQueue {
|
||||
pub fn new(db_path: &str) -> Self {
|
||||
// Ensure the parent dir and table exist even if only the queue (not
|
||||
// ChatStore) is used — a fresh container without a mounted data dir
|
||||
// must still be able to open the DB.
|
||||
if let Some(parent) = std::path::Path::new(db_path).parent()
|
||||
&& !parent.as_os_str().is_empty()
|
||||
&& let Err(e) = std::fs::create_dir_all(parent)
|
||||
{
|
||||
log::error!("failed to create queue dir: {e}");
|
||||
}
|
||||
if let Ok(conn) = Connection::open(db_path)
|
||||
&& let Err(e) = ensure_schema(&conn)
|
||||
{
|
||||
log::error!("failed to initialize queue schema: {e}");
|
||||
}
|
||||
/// Wraps the shared DB pool; the schema is initialized once by
|
||||
/// [`crate::db::open_store`] (all three stores share the pool).
|
||||
pub fn new(pool: std::sync::Arc<crate::db::DbPool>) -> Self {
|
||||
Self {
|
||||
db_path: db_path.to_string(),
|
||||
pool,
|
||||
notify: Arc::new(Notify::new()),
|
||||
stop: Arc::new(AtomicBool::new(false)),
|
||||
worker: Mutex::new(Vec::new()),
|
||||
@@ -132,7 +110,7 @@ impl PersistentTaskQueue {
|
||||
let mut handles = Vec::with_capacity(QUEUE_WORKERS + 1);
|
||||
for _ in 0..QUEUE_WORKERS {
|
||||
let worker = QueueWorker {
|
||||
db_path: self.db_path.clone(),
|
||||
pool: std::sync::Arc::clone(&self.pool),
|
||||
notify: Arc::clone(&self.notify),
|
||||
stop: Arc::clone(&self.stop),
|
||||
handler: Arc::clone(&handler),
|
||||
@@ -145,7 +123,7 @@ impl PersistentTaskQueue {
|
||||
// the same notify as the workers, so enqueue and stop interrupt the
|
||||
// sleep; the first interval tick fires immediately (harmless extra
|
||||
// recovery at startup).
|
||||
let sweep_db_path = self.db_path.clone();
|
||||
let sweep_pool = std::sync::Arc::clone(&self.pool);
|
||||
let sweep_notify = Arc::clone(&self.notify);
|
||||
let sweep_stop = Arc::clone(&self.stop);
|
||||
handles.push(tokio::spawn(async move {
|
||||
@@ -160,8 +138,7 @@ impl PersistentTaskQueue {
|
||||
if sweep_stop.load(Ordering::Relaxed) {
|
||||
break;
|
||||
}
|
||||
let result =
|
||||
crate::db::with_conn(&sweep_db_path, move |conn| recover_update(conn)).await;
|
||||
let result = sweep_pool.with_conn(move |conn| recover_update(conn)).await;
|
||||
if let Err(e) = result {
|
||||
log::error!("queue sweep failed: {e}");
|
||||
}
|
||||
@@ -189,8 +166,8 @@ impl PersistentTaskQueue {
|
||||
self.counter.fetch_add(1, Ordering::Relaxed)
|
||||
);
|
||||
let payload = payload.to_string();
|
||||
log::info!("enqueued {id} (run_after {run_after:.1})");
|
||||
crate::db::with_conn(&self.db_path, move |conn| {
|
||||
log::debug!("enqueued {id} (run_after {run_after:.1})");
|
||||
self.pool.with_conn(move |conn| {
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
|
||||
VALUES (?1, ?2, ?3, 0, 'pending', 0, ?4)",
|
||||
@@ -212,7 +189,7 @@ impl PersistentTaskQueue {
|
||||
}
|
||||
|
||||
async fn recover_sweep(&self) {
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| recover_update(conn)).await;
|
||||
let result = self.pool.with_conn(move |conn| recover_update(conn)).await;
|
||||
if let Err(e) = result {
|
||||
log::error!("queue recovery failed: {e}");
|
||||
}
|
||||
@@ -267,7 +244,7 @@ impl QueueWorker {
|
||||
/// Leases the oldest due row (sets it `in_progress` with a lock TTL).
|
||||
/// Errors are surfaced so the caller can back off instead of spinning.
|
||||
async fn lease_next(&self) -> Result<Option<LeasedRow>, rusqlite::Error> {
|
||||
crate::db::with_conn(&self.db_path, |conn| {
|
||||
self.pool.with_conn(|conn| {
|
||||
// BEGIN IMMEDIATE: with several workers, a deferred transaction
|
||||
// that read before another worker's lease commit would fail with
|
||||
// SQLITE_BUSY_SNAPSHOT. Taking the write lock up front serializes
|
||||
@@ -309,16 +286,18 @@ impl QueueWorker {
|
||||
}
|
||||
|
||||
async fn earliest_run_after(&self) -> Option<f64> {
|
||||
let result = crate::db::with_conn(&self.db_path, |conn| {
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
|
||||
let mut rows = stmt.query([])?;
|
||||
match rows.next()? {
|
||||
Some(row) => Ok(row.get::<_, Option<f64>>(0)?),
|
||||
None => Ok(None),
|
||||
}
|
||||
})
|
||||
.await;
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(|conn| {
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
|
||||
let mut rows = stmt.query([])?;
|
||||
match rows.next()? {
|
||||
Some(row) => Ok(row.get::<_, Option<f64>>(0)?),
|
||||
None => Ok(None),
|
||||
}
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(v) => v,
|
||||
Err(e) => {
|
||||
@@ -328,6 +307,11 @@ impl QueueWorker {
|
||||
}
|
||||
}
|
||||
|
||||
/// Processes one leased row, keeping the lease alive while the handler
|
||||
/// runs. Without the heartbeat a task longer than [`LOCK_TTL_SECONDS`]
|
||||
/// (slow download, ugoira encode, rate-limited batch forward) would have
|
||||
/// its lease expire mid-run; the expiry sweep would flip the row back to
|
||||
/// `pending` and another worker would process it again — duplicate sends.
|
||||
async fn process(&self, row: LeasedRow) {
|
||||
let payload: Value = match serde_json::from_str(&row.payload) {
|
||||
Ok(value) => value,
|
||||
@@ -338,10 +322,11 @@ impl QueueWorker {
|
||||
return;
|
||||
}
|
||||
};
|
||||
log::info!("processing {} (attempt {})", row.id, row.attempts + 1);
|
||||
match (self.handler)(payload).await {
|
||||
log::debug!("processing {} (attempt {})", row.id, row.attempts + 1);
|
||||
let outcome = self.run_with_lease(&row.id, payload).await;
|
||||
match outcome {
|
||||
Ok(()) => {
|
||||
log::info!("task {} completed", row.id);
|
||||
log::debug!("task {} completed", row.id);
|
||||
self.delete_row(&row.id).await;
|
||||
}
|
||||
Err(QueueError::Retryable {
|
||||
@@ -355,7 +340,7 @@ impl QueueWorker {
|
||||
(self.dead_letter)(payload, message).await;
|
||||
} else {
|
||||
let delay = scaled_retry_delay(delay_seconds, row.attempts);
|
||||
log::info!(
|
||||
log::debug!(
|
||||
"task {} rescheduled in {delay:.1}s (attempt {})",
|
||||
row.id,
|
||||
row.attempts + 1
|
||||
@@ -372,13 +357,51 @@ impl QueueWorker {
|
||||
}
|
||||
}
|
||||
|
||||
/// Drives the handler to completion, refreshing the row's `locked_until`
|
||||
/// every 30 s so the expiry sweep never re-leases a still-running task.
|
||||
/// The heartbeat is part of this future, not a separate spawned task: if
|
||||
/// the worker task dies (panic) the heartbeat dies with it and the sweep
|
||||
/// recovers the row exactly as before.
|
||||
async fn run_with_lease(&self, id: &str, payload: Value) -> Result<(), QueueError> {
|
||||
let fut = (self.handler)(payload);
|
||||
tokio::pin!(fut);
|
||||
let mut interval = tokio::time::interval(Duration::from_secs(30));
|
||||
// The first interval tick fires immediately; skip it (the lease was
|
||||
// just set by lease_next).
|
||||
interval.tick().await;
|
||||
let id_owned = id.to_string();
|
||||
loop {
|
||||
tokio::select! {
|
||||
result = &mut fut => return result,
|
||||
_ = interval.tick() => {
|
||||
let now = now_f64();
|
||||
let id = id_owned.clone();
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(move |conn| {
|
||||
conn.execute(
|
||||
"UPDATE tasks SET locked_until=?1 WHERE id=?2 AND status='in_progress'",
|
||||
params![now + LOCK_TTL_SECONDS, id],
|
||||
)
|
||||
})
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("queue lease heartbeat failed: {e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn delete_row(&self, id: &str) {
|
||||
let id = id.to_string();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(move |conn| {
|
||||
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("queue delete failed: {e}");
|
||||
}
|
||||
@@ -387,7 +410,7 @@ impl QueueWorker {
|
||||
async fn reschedule(&self, id: &str, payload: Value, delay_seconds: f64, attempts: i32) {
|
||||
let id = id.to_string();
|
||||
let payload = payload.to_string();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
let result = self.pool.with_conn(move |conn| {
|
||||
conn.execute(
|
||||
"UPDATE tasks SET payload=?1, run_after=?2, attempts=?3, status='pending', locked_until=0 WHERE id=?4",
|
||||
params![payload, now_f64() + delay_seconds, attempts, id],
|
||||
@@ -421,7 +444,8 @@ mod tests {
|
||||
async fn new_queue() -> (PersistentTaskQueue, tempfile::TempDir) {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("queue.db");
|
||||
let queue = PersistentTaskQueue::new(path.to_str().unwrap());
|
||||
let pool = crate::db::open_store(path.to_str().unwrap()).unwrap();
|
||||
let queue = PersistentTaskQueue::new(pool);
|
||||
(queue, dir)
|
||||
}
|
||||
|
||||
@@ -528,10 +552,11 @@ mod tests {
|
||||
async fn stale_in_progress_row_is_recovered_on_start() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("queue.db");
|
||||
// Insert a stale leased row directly (lease expired).
|
||||
// Insert a stale leased row directly (lease expired). open_store runs
|
||||
// the schema; the queue below shares the same pool.
|
||||
let pool = crate::db::open_store(path.to_str().unwrap()).unwrap();
|
||||
{
|
||||
let conn = Connection::open(&path).unwrap();
|
||||
ensure_schema(&conn).unwrap();
|
||||
let conn = rusqlite::Connection::open(&path).unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
|
||||
VALUES ('task_stale', '{\"s\":1}', 0, 0, 'in_progress', ?1, 0)",
|
||||
@@ -539,7 +564,7 @@ mod tests {
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
let queue = PersistentTaskQueue::new(path.to_str().unwrap());
|
||||
let queue = PersistentTaskQueue::new(pool);
|
||||
let calls = Arc::new(AtomicUsize::new(0));
|
||||
let c = calls.clone();
|
||||
queue
|
||||
@@ -575,8 +600,7 @@ mod tests {
|
||||
// Insert a stale leased row AFTER startup: without a runtime sweep it
|
||||
// would stay `in_progress` forever (only start() used to recover).
|
||||
{
|
||||
let conn = Connection::open(&queue.db_path).unwrap();
|
||||
ensure_schema(&conn).unwrap();
|
||||
let conn = rusqlite::Connection::open(queue.pool.path()).unwrap();
|
||||
conn.execute(
|
||||
"INSERT INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
|
||||
VALUES ('task_stale_runtime', '{\"s\":1}', 0, 0, 'in_progress', ?1, 0)",
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
//! Per-chat token-bucket rate limiting.
|
||||
//!
|
||||
//! Telegram throttles bots that burst past a chat's message budget
|
||||
//! (roughly 20 messages/min for channels/groups); today the bot absorbs
|
||||
//! those 429s with queue retries. This limiter smooths the burst *before*
|
||||
//! it reaches the API: media sends to a chat consume one token per
|
||||
//! message, refilled at [`REFILL_PER_SEC`], so a batch forward paces itself
|
||||
//! instead of tripping flood control. The queue retry stays as the safety
|
||||
//! net for limits this bucket does not model (global per-bot limits etc.).
|
||||
|
||||
use parking_lot::Mutex;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use std::sync::LazyLock;
|
||||
use std::time::Duration;
|
||||
|
||||
/// Burst capacity: how many messages may be sent at once without waiting.
|
||||
const CAPACITY: f64 = 20.0;
|
||||
/// Sustained refill: ~20 messages per minute.
|
||||
const REFILL_PER_SEC: f64 = 20.0 / 60.0;
|
||||
|
||||
struct State {
|
||||
/// Current token balance; may go negative (debt from an acquire larger
|
||||
/// than the capacity, repaid by subsequent refills).
|
||||
tokens: f64,
|
||||
last_refill: tokio::time::Instant,
|
||||
}
|
||||
|
||||
/// A token bucket: at most `CAPACITY` tokens accumulate, refilled at
|
||||
/// `REFILL_PER_SEC`. [`TokenBucket::acquire`] consumes `n` tokens, waiting
|
||||
/// for the deficit (a single acquire may exceed the capacity and goes into
|
||||
/// debt, which the refill repays).
|
||||
pub struct TokenBucket {
|
||||
capacity: f64,
|
||||
refill_per_sec: f64,
|
||||
state: Mutex<State>,
|
||||
}
|
||||
|
||||
impl TokenBucket {
|
||||
fn new(capacity: f64, refill_per_sec: f64) -> Self {
|
||||
TokenBucket {
|
||||
capacity,
|
||||
refill_per_sec,
|
||||
state: Mutex::new(State {
|
||||
tokens: capacity,
|
||||
last_refill: tokio::time::Instant::now(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// Waits until `n` tokens are available, consuming them. The wait is
|
||||
/// bounded: the deficit is committed as debt and repaid over time, so a
|
||||
/// large acquire returns once its share of the refill budget has passed.
|
||||
pub async fn acquire(&self, n: f64) {
|
||||
// The parking_lot guard is confined to this block: only the plain
|
||||
// `wait` duration crosses the await (a guard across an await point
|
||||
// would make the future !Send).
|
||||
let wait = {
|
||||
let mut state = self.state.lock();
|
||||
let now = tokio::time::Instant::now();
|
||||
let elapsed = now
|
||||
.saturating_duration_since(state.last_refill)
|
||||
.as_secs_f64();
|
||||
// Refill up to the capacity; a debt (negative balance) is repaid
|
||||
// before any surplus accumulates.
|
||||
state.tokens = (state.tokens + elapsed * self.refill_per_sec).min(self.capacity);
|
||||
state.last_refill = now;
|
||||
if state.tokens >= n {
|
||||
state.tokens -= n;
|
||||
return;
|
||||
}
|
||||
// Commit the whole consumption now; the caller proceeds once the
|
||||
// deficit's worth of refill time has passed.
|
||||
let debt = n - state.tokens;
|
||||
state.tokens = -debt;
|
||||
debt / self.refill_per_sec
|
||||
};
|
||||
tokio::time::sleep(Duration::from_secs_f64(wait)).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// One limiter per chat, created on first use. Per-chat so one chat's burst
|
||||
/// never throttles another.
|
||||
static LIMITERS: LazyLock<Mutex<HashMap<i64, Arc<TokenBucket>>>> =
|
||||
LazyLock::new(|| Mutex::new(HashMap::new()));
|
||||
|
||||
/// Returns the shared limiter for a chat, creating it on first use.
|
||||
pub fn limiter_for(chat_id: i64) -> Arc<TokenBucket> {
|
||||
LIMITERS
|
||||
.lock()
|
||||
.entry(chat_id)
|
||||
.or_insert_with(|| Arc::new(TokenBucket::new(CAPACITY, REFILL_PER_SEC)))
|
||||
.clone()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn limiter_for_reuses_the_per_chat_bucket() {
|
||||
let a = limiter_for(1);
|
||||
let b = limiter_for(1);
|
||||
let c = limiter_for(2);
|
||||
assert!(Arc::ptr_eq(&a, &b), "same chat → same bucket");
|
||||
assert!(!Arc::ptr_eq(&a, &c), "different chat → different bucket");
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn burst_is_consumed_instantly_then_refill_waits() {
|
||||
let bucket = TokenBucket::new(3.0, 1.0);
|
||||
// A burst within capacity passes without waiting.
|
||||
bucket.acquire(3.0).await;
|
||||
// The bucket is empty now; one token needs 1s of refill.
|
||||
let start = tokio::time::Instant::now();
|
||||
bucket.acquire(1.0).await;
|
||||
assert!(
|
||||
start.elapsed() >= Duration::from_secs(1),
|
||||
"elapsed {:?}",
|
||||
start.elapsed()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn acquire_larger_than_capacity_waits_for_the_deficit() {
|
||||
let bucket = TokenBucket::new(2.0, 1.0);
|
||||
// 5 tokens with a capacity of 2: the 3-token deficit takes 3s.
|
||||
let start = tokio::time::Instant::now();
|
||||
bucket.acquire(5.0).await;
|
||||
assert!(
|
||||
start.elapsed() >= Duration::from_secs(3),
|
||||
"elapsed {:?}",
|
||||
start.elapsed()
|
||||
);
|
||||
}
|
||||
}
|
||||
+607
-244
File diff suppressed because it is too large
Load Diff
@@ -5,7 +5,6 @@ use parking_lot::Mutex;
|
||||
use rusqlite::params;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
|
||||
@@ -38,7 +37,7 @@ pub struct ChatStore {
|
||||
/// Per-chat async locks serializing get→mutate→set so concurrent handler
|
||||
/// tasks (batch-forwards, callbacks) cannot clobber each other's writes.
|
||||
locks: Mutex<HashMap<i64, Arc<tokio::sync::Mutex<()>>>>,
|
||||
db_path: String,
|
||||
pool: Arc<crate::db::DbPool>,
|
||||
}
|
||||
|
||||
pub fn unix_now() -> i64 {
|
||||
@@ -49,26 +48,15 @@ pub fn unix_now() -> i64 {
|
||||
}
|
||||
|
||||
impl ChatStore {
|
||||
/// Creates the parent directory and the `chat_state` table (idempotent).
|
||||
/// The shared `tasks` / `link_cache` tables are owned by `queue.rs` and
|
||||
/// `link_cache.rs` respectively.
|
||||
pub fn open(path: &str) -> rusqlite::Result<Self> {
|
||||
if let Some(parent) = Path::new(path).parent()
|
||||
&& !parent.as_os_str().is_empty()
|
||||
{
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)))?;
|
||||
}
|
||||
let conn = crate::db::open_db(path)?;
|
||||
conn.execute_batch(
|
||||
"CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL);",
|
||||
)?;
|
||||
drop(conn);
|
||||
Ok(ChatStore {
|
||||
/// Wraps the shared DB pool (schema initialized once by
|
||||
/// [`crate::db::open_store`]; the `chat_state` table lives in the merged
|
||||
/// schema alongside `tasks` and `link_cache`).
|
||||
pub fn new(pool: Arc<crate::db::DbPool>) -> Self {
|
||||
ChatStore {
|
||||
cache: Mutex::new(HashMap::new()),
|
||||
locks: Mutex::new(HashMap::new()),
|
||||
db_path: path.to_string(),
|
||||
})
|
||||
pool,
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn get(&self, chat_id: i64) -> ChatData {
|
||||
@@ -76,23 +64,25 @@ impl ChatStore {
|
||||
return data.clone();
|
||||
}
|
||||
let chat_key = chat_id.to_string();
|
||||
let payload = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
// Concurrent handler tasks (batch-forwards) may write chat_state
|
||||
// while this read runs; the shared busy timeout handles the
|
||||
// write-lock collision instead of failing the query.
|
||||
let mut stmt = conn.prepare("SELECT payload FROM chat_state WHERE chat_id = ?1")?;
|
||||
let mut rows = stmt.query(params![chat_key])?;
|
||||
match rows.next()? {
|
||||
Some(row) => Ok(Some(row.get::<_, String>(0)?)),
|
||||
None => Ok(None),
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap_or_else(|e| {
|
||||
log::error!("chat_state read failed: {e}");
|
||||
None
|
||||
})
|
||||
.unwrap_or_default();
|
||||
let payload = self
|
||||
.pool
|
||||
.with_conn(move |conn| {
|
||||
// Concurrent handler tasks (batch-forwards) may write chat_state
|
||||
// while this read runs; the shared busy timeout handles the
|
||||
// write-lock collision instead of failing the query.
|
||||
let mut stmt = conn.prepare("SELECT payload FROM chat_state WHERE chat_id = ?1")?;
|
||||
let mut rows = stmt.query(params![chat_key])?;
|
||||
match rows.next()? {
|
||||
Some(row) => Ok(Some(row.get::<_, String>(0)?)),
|
||||
None => Ok(None),
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap_or_else(|e| {
|
||||
log::error!("chat_state read failed: {e}");
|
||||
None
|
||||
})
|
||||
.unwrap_or_default();
|
||||
let data: ChatData = serde_json::from_str(&payload).unwrap_or_default();
|
||||
self.cache.lock().insert(chat_id, data.clone());
|
||||
data
|
||||
@@ -103,14 +93,16 @@ impl ChatStore {
|
||||
self.cache.lock().insert(chat_id, data.clone());
|
||||
let payload = serde_json::to_string(data).expect("chat state serializes");
|
||||
let chat_id = chat_id.to_string();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
|
||||
params![chat_id, payload],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
let result = self
|
||||
.pool
|
||||
.with_conn(move |conn| {
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
|
||||
params![chat_id, payload],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("chat_state write failed: {e}");
|
||||
}
|
||||
@@ -205,9 +197,8 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn concurrent_updates_do_not_lose_edit_records() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let store = std::sync::Arc::new(
|
||||
ChatStore::open(dir.path().join("s.db").to_str().unwrap()).unwrap(),
|
||||
);
|
||||
let pool = crate::db::open_store(dir.path().join("s.db").to_str().unwrap()).unwrap();
|
||||
let store = std::sync::Arc::new(ChatStore::new(pool));
|
||||
let mut handles = Vec::new();
|
||||
for i in 0..4 {
|
||||
let store = Arc::clone(&store);
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# 架构优化设计:可测试性接缝 + handlers 拆分
|
||||
|
||||
> 状态:**阶段 A、B、C 已实施**(A: `c9e72fd`,B: `50206a9` + `ae69d72`,C:
|
||||
> rate_limit 提交);**D 已延迟**——待下次数据库 schema 变化时实施(见 §5)。
|
||||
> 目标:把仓库最大的测试空白(`handlers.rs`/`send.rs` 的发送与分派逻辑)补上
|
||||
> 可测试接缝,并把 ~1100 行的 handlers 单体拆成模块。
|
||||
|
||||
---
|
||||
|
||||
## 1. 现状与动机
|
||||
|
||||
- `handlers.rs`(~1100 行)混装:命令解析/执行、URL 提取 + 任务通道、inline
|
||||
debounce、callback、edit-before-forward、全部全局静态。
|
||||
- 关键路径零测试:`url_media` 的分派、`dispatch_send` 的失败分类、缓存命中路径、
|
||||
edit-before-forward、转发重试——AGENTS.md 自认 "untested: handlers.rs"。
|
||||
- 根因:`handlers.rs`/`send.rs` 直接依赖 teloxide `Bot`(具体类型)与全局静态
|
||||
(`CHAT_STORE`/`TASK_QUEUE`/`LINK_CACHE`/`CONFIG`),没有注入点。
|
||||
|
||||
## 2. 阶段 A:handlers 拆分(纯组织,零风险,先行)
|
||||
|
||||
把 `handlers.rs` 拆为模块(仅移动代码,不改签名):
|
||||
|
||||
```
|
||||
handlers/
|
||||
mod.rs — 入口:message/inline/callback 分发 + 公共类型(UrlJob、log_key)
|
||||
statics.rs — CHAT_STORE / TASK_QUEUE / LINK_CACHE / DB / CONFIG / URL_JOBS
|
||||
commands.rs — Command enum + execute_command + set_forward_channel_handler
|
||||
urls.rs — extract_urls + start/stop_url_workers + url_media + build_send_task + media_to_payload
|
||||
inline.rs — inline_query_handler + debounce 状态机 + answer_inline_query
|
||||
callback.rs — callback_query_handler + edit_message_handler
|
||||
```
|
||||
|
||||
- `mod.rs` 用 `pub use` 重导出,bot 侧引用 `handlers::xxx` 不变。
|
||||
- 收益:每个模块独立审阅;后续阶段 B 的接缝改动落在明确的模块内。
|
||||
|
||||
## 3. 阶段 B:MediaSender 接缝(核心)
|
||||
|
||||
**动机**:`send.rs` 的所有发送入口(`send_media_group`/`send_animation`/
|
||||
`copy_messages`)都挂在具体 `Bot` 上;测试无法注入失败/成功。
|
||||
|
||||
**设计**:新增 `crates/xmedia-bot/src/media_sender.rs`:
|
||||
|
||||
```rust
|
||||
/// 发送抽象:生产用 teloxide Bot,测试用记录型 mock。
|
||||
/// 方法签名与 teloxide 调用点一一对应,返回 Result 以便注入任意失败。
|
||||
pub trait MediaSender: Send + Sync {
|
||||
fn send_media_group(&self, chat_id: ChatId, items: Vec<InputMedia>)
|
||||
-> BoxFuture<'_, Result<Vec<Message>, RequestError>>;
|
||||
fn send_animation(&self, chat_id: ChatId, file: InputFile, caption: Option<&str>, spoiler: bool, reply_to: i64)
|
||||
-> BoxFuture<'_, Result<Message, RequestError>>;
|
||||
fn copy_messages(&self, to: ChatId, from: ChatId, ids: Vec<MessageId>)
|
||||
-> BoxFuture<'_, Result<Vec<MessageId>, RequestError>>;
|
||||
// 按需扩展:edit_message_caption / delete_message / answer_callback_query …
|
||||
}
|
||||
|
||||
impl MediaSender for Bot { /* 委托现有 teloxide 调用 */ }
|
||||
```
|
||||
|
||||
配套:`ChatStore`/`LinkCache`/`PersistentTaskQueue` 已是具体类型——给 `send.rs`/
|
||||
`url_media` 需要的最小面加 trait(`ChatStoreReader`/`LinkCacheReader` 等),或直接
|
||||
注入具体类型(它们已有内存态,测试用真实 tempdir 即可,见阶段 B-注)。
|
||||
|
||||
**接入点**:
|
||||
- `dispatch_send` / `send_media_sequence` / `send_animation` / `forward_messages` /
|
||||
`post_send_actions` / `notify_failure` 的 `bot: &Bot` 参数改为 `sender: &dyn MediaSender`。
|
||||
- `url_media` 由 `url_media(bot, message, url)` 改为 `url_media(sender, store, queue, cache, message, url)`(或聚合为一个 `AppContext` 结构传引用)。
|
||||
|
||||
**测试策略**(仓库无 mock 框架,手写 mock):
|
||||
- `MockSender` 记录调用序列、按脚本返回 Ok/Err(覆盖:URL 发送成功、media-fetch
|
||||
失败触发兜底、RetryAfter 触发入队、Permanent 触发缓存失效)。
|
||||
- `ChatStore`/`LinkCache` 用真实 tempdir 实例(现有测试已这么做)。
|
||||
- 新增测试:`send_media_sequence` 分批续传、`send_animation` 兜底、`url_media`
|
||||
缓存命中 vs 未命中、`dispatch_send` 三分支。
|
||||
|
||||
**风险**:中。动 `send.rs`/`handlers.rs` 签名(约 15 处调用点),行为不变。
|
||||
**不做**:`main.rs` 的 teloxide 装配不抽象(那是真正的胶水,无测试价值)。
|
||||
|
||||
## 4. 阶段 C:主动限流(已实施)
|
||||
|
||||
批量转发时的突发会触发 Telegram 频道限速,现在靠 `RetryAfter → 队列重试` 被动
|
||||
应对。新增轻量令牌桶(`rate_limit.rs`):
|
||||
|
||||
```rust
|
||||
pub struct TokenBucket { capacity, refill_per_sec, state: Mutex<State> }
|
||||
impl TokenBucket {
|
||||
pub async fn acquire(&self, n: f64); // 按 n 个 token 等待并消费
|
||||
}
|
||||
pub fn limiter_for(chat_id: i64) -> Arc<TokenBucket>; // 每频道一个桶
|
||||
```
|
||||
|
||||
- 默认 `CAPACITY = 20`、`REFILL_PER_SEC = 20/60`(约 20 msg/min);
|
||||
单次 acquire 可超出容量(记为债务,由后续 refill 偿还)。
|
||||
- 挂点:`MediaSender for Bot` 的 `send_media_group`(按 items 数)、
|
||||
`copy_messages`(按 ids 数)、`send_animation`(1 token)前置 `acquire`;
|
||||
MockSender 不受影响(测试不经过限流)。
|
||||
- 收益:减少 429 → 重试 → 死信;队列重试仍是全局限速的安全网。
|
||||
- 风险:低,独立模块;`tokio::time`(paused-clock 可测)。
|
||||
|
||||
## 5. 阶段 D:DB 版本化迁移(**已延迟**)
|
||||
|
||||
> ⚠️ **待办提醒**:本阶段**推迟到下次数据库 schema 变化时实施**(给
|
||||
> `link_cache`/`chat_state`/`tasks` 加列、改结构等)。当前 `schema_init` 是
|
||||
> `CREATE TABLE IF NOT EXISTS`,无版本概念;一旦需要迁移已有线上库,必须先落地
|
||||
> 本方案(`PRAGMA user_version` 迁移链)再改 schema。`db.rs` 的 `schema_init`
|
||||
> 处已留注释指向这里。
|
||||
|
||||
```rust
|
||||
// db.rs
|
||||
const MIGRATIONS: &[&str] = &[
|
||||
// v1: 初始 schema(tasks / chat_state / link_cache)
|
||||
"CREATE TABLE IF NOT EXISTS tasks (...); ...",
|
||||
];
|
||||
pub fn migrate(conn: &Connection) -> rusqlite::Result<()> {
|
||||
let v: i64 = conn.query_row("PRAGMA user_version", [], |r| r.get(0))?;
|
||||
for (i, sql) in MIGRATIONS.iter().enumerate().skip(v as usize) {
|
||||
conn.execute_batch(sql)?;
|
||||
conn.pragma_update(None, "user_version", (i + 1) as i64)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
- 低优先级:schema 未变时无收益;将来加列/改结构时必须有。
|
||||
- `open_store` 改用 `migrate` 替换 `schema_init` 调用。
|
||||
|
||||
## 6. 明确不做
|
||||
|
||||
- **不拆 xmedia-core**:`Task`/队列/发送抽成独立 lib crate 是大工程,除非出现
|
||||
第二个客户端,否则收益不抵成本。
|
||||
- **不引入 DI 框架**:仓库惯例是 LazyLock 静态 + 显式传参,保持。
|
||||
- **不抽象 main.rs 的 teloxide 装配**。
|
||||
|
||||
## 7. 实施记录
|
||||
|
||||
| 阶段 | 提交 | 说明 |
|
||||
|---|---|---|
|
||||
| A | `c9e72fd` | handlers 拆为 `{mod, statics, commands, urls, inline, callback}` |
|
||||
| B | `50206a9` | `media_sender.rs`:`trait MediaSender` + `impl for Bot`(`<Bot as Requester>::` 消歧);send.rs 8 处签名改 `&dyn MediaSender`;`MockSender` 测试覆盖兜底触发与错误分类(+5 测试) |
|
||||
| B | `ae69d72` | `AppContext` 注入 `url_media`(sender/store/queue/cache),url_media 全链路测试(缓存命中/失效/成功/不支持 URL,+3 测试) |
|
||||
| C | rate_limit 提交 | `rate_limit.rs` 令牌桶 + 每频道注册表;`MediaSender for Bot` 的 group/copy/animation 前置 `acquire`(+3 测试) |
|
||||
| D | — | **已延迟**:待下次数据库 schema 变化时实施(见 §5) |
|
||||
|
||||
A、B、C 为核心并已实施;D 在 schema 变更时落地。
|
||||
@@ -0,0 +1,237 @@
|
||||
# 站点适配器重构方案:让新增站点变成"新模块 + 注册一行"
|
||||
|
||||
> 状态:**已实施**(阶段 1-5,提交 `7ca8fd1` / `5e23916` / `bf4e615` / `5679a8c` +
|
||||
> 本文档收尾)。目标:把"加一个新站点"从改 8-9 处收敛到 3 处,并让站点身份、
|
||||
> 重试策略、下载 header 等站点能力归位到站点模块自身。实施过程中的关键偏差
|
||||
> (async 形态)见 §3 的 "async 形态" 段——原生 AFIT 实测不可用于 dyn 分派,
|
||||
> 最终采用手写 `BoxFuture`(`SiteFuture` 别名)。
|
||||
|
||||
---
|
||||
|
||||
## 1. 现状摩擦清单
|
||||
|
||||
以现有三站(twitter / bsky / pixiv)为基线,新增第 4 个站点(代号 `example`)
|
||||
今天需要触碰的位置:
|
||||
|
||||
| # | 位置(当前行号) | 改动 | 必改? |
|
||||
|---|---|---|---|
|
||||
| 1 | 新目录 `crates/x-media/src/site/example/{mod,interface,model}.rs` | 新模块 | 必改 |
|
||||
| 2 | `site/mod.rs:354-365` `fetch_once` | 加一个 `if` 分派分支 | 必改 |
|
||||
| 3 | `site/mod.rs:167-178` `cache_key` | 加一个 `if` 分支 + 约定 key 前缀 `"example:..."` | 必改 |
|
||||
| 4 | `site/mod.rs:53-63` `site_name()` | 加一个 URL `contains` 嗅探分支 | 必改 |
|
||||
| 5 | `handlers.rs:405` `SetFormat` 白名单 | `["twitter","bsky","pixiv"]` 加字符串 | 必改 |
|
||||
| 6 | `config.rs` / `main.rs:74-84` | 仿 pixiv 加启动校验(token、`disable()`) | 视站点 |
|
||||
| 7 | `site/mod.rs:312-326` `fetch_error_is_retryable` | 若重试策略特殊,改中央分类函数 | 视站点 |
|
||||
| 8 | `site/mod.rs:377/391/430` 三个下载函数 | 若媒体有防盗链,加 header(现在是硬编码 pximg 判断) | 视站点 |
|
||||
| 9 | `site/mod.rs:16,180-197` `FetchError` | 若错误类型特殊,加嵌套 variant(仿 `Pixiv(PixivError)`) | 视站点 |
|
||||
|
||||
**根因**:仓库里没有"站点"这个实体。站点的四类能力——URL 识别(PATTERN +
|
||||
cache_key)、抓取、重试策略、下载 header——分别散落在中央 if 链、URL 字符串嗅探、
|
||||
魔法字符串 key 和 bot crate 的白名单里。`AGENTS.md` 现行约定 "no trait, no enum
|
||||
dispatch" 是刻意的简单性选择;本方案的目标是在**不推翻它精神的前提下**收敛摩擦,
|
||||
并在阶段 3 提供完整的 trait 注册表选项。
|
||||
|
||||
## 2. 目标架构
|
||||
|
||||
```
|
||||
crates/x-media/src/site/mod.rs
|
||||
├─ SITES: LazyLock<Vec<Box<dyn Site>>> ← 注册表(唯一的"站点列表")
|
||||
├─ find_site(url) / fetch(url) / cache_key(url) / site_ids()
|
||||
└─ 通用类型:Fetched { site_id, ... } / FetchError(通用类 + Site 变体)
|
||||
│
|
||||
├─ site/twitter/{mod,interface,model}.rs impl Site
|
||||
├─ site/bsky/… impl Site
|
||||
└─ site/pixiv/… impl Site (download_headers: pximg Referer)
|
||||
(validate: token 校验)
|
||||
|
||||
crates/xmedia-bot
|
||||
├─ handlers.rs SetFormat 白名单 ← x_media::site::ids()(不再写死)
|
||||
├─ handlers.rs site 格式查找 ← fetched.site_id(缓存/新鲜两条路径同口径)
|
||||
└─ main.rs 启动校验 ← site::validate_all()(不再特判 pixiv)
|
||||
```
|
||||
|
||||
## 3. 分阶段迁移
|
||||
|
||||
每个阶段是一个独立提交,保持 `cargo fmt` / `cargo clippy -- -D warnings` /
|
||||
`cargo test --workspace` 全绿;行为完全不变,只挪代码、不换语义。
|
||||
|
||||
### 阶段 1:站点身份单一来源(低风险,推荐先做)
|
||||
|
||||
**动机**:同一概念目前有两个来源——缓存命中路径用 `key.split(':').next()`
|
||||
(`handlers.rs:639`),新鲜抓取路径用 `fetched.site_name()`(`handlers.rs:724`);
|
||||
`site_name()` 又是对 `source_url` 的 `contains` 字符串嗅探,还有 `"unknown"`
|
||||
兜底分支。
|
||||
|
||||
**改动**:
|
||||
|
||||
1. `site/mod.rs`:`Fetched` 增加字段 `site_id: &'static str`(由各站点的
|
||||
`impl From<SiteStruct> for Fetched` 填充;`empty_fetched` 同步填)。
|
||||
`Fetched::site_name()` 改为 `return self.site_id`(保留方法名,删除
|
||||
`source_url.contains` 嗅探与 `"unknown"` 分支)。
|
||||
2. `site/mod.rs`:新增 `pub fn site_id_from_key(key: &str) -> &'static str`
|
||||
(解析 `"example:..."` 前缀,未知前缀返回 `"unknown"`),bot 缓存命中路径改用它,
|
||||
与 `fetched.site_id` 口径统一。
|
||||
3. `handlers.rs:405`:`SetFormat` 白名单改为 `x_media::site::ids()`——阶段 1 先实现
|
||||
`ids()` 为 `["twitter","bsky","pixiv"]` 的常量函数(数据源仍集中,行为不变),
|
||||
阶段 3 再改为遍历注册表。
|
||||
4. `twitter/interface.rs:48-60` / `bsky` / `pixiv` 的 `From<SiteStruct> for Fetched`
|
||||
各补 `site_id` 字段。
|
||||
|
||||
**风险**:低。纯增量字段;`site_name()` 语义不变(测试 `pixiv/interface.rs:355`
|
||||
已断言 `"pixiv"`)。
|
||||
**验证**:现有全部单测;`cache_key_normalizes_domain_variants` 等不变。
|
||||
**回滚**:revert 该提交。
|
||||
|
||||
### 阶段 2:站点能力下沉(不引入 trait,静态分派)
|
||||
|
||||
**动机**:把"每个站点自己才知道"的逻辑搬回站点模块,中央只做迭代。这是
|
||||
`AGENTS.md` 现有约定(无 trait)与完整注册表之间的折中,可独立交付。
|
||||
|
||||
**改动**:每个站点模块新增并 `mod.rs` 重新导出:
|
||||
|
||||
```rust
|
||||
// site/twitter/interface.rs(bsky/pixiv 同构)
|
||||
pub fn cache_key(url: &str) -> Option<String>; // 用自身 PATTERN,返回 "twitter:<id>"
|
||||
pub fn is_retryable(err: &FetchError) -> bool; // 默认 Http|Transient;pixiv 覆盖 PixivError 分支
|
||||
pub fn media_headers(url: &str) -> Option<Vec<(&'static str, String)>>;
|
||||
// pixiv: url 含 "pximg.net" → Referer
|
||||
```
|
||||
|
||||
`site/mod.rs` 相应改为迭代三站:
|
||||
|
||||
- `cache_key`:逐个调 `site::cache_key`,不再自己写 key 格式;
|
||||
- `fetch_error_is_retryable`:删除,`fetch()` 重试循环改调 `current_site::is_retryable`
|
||||
(`fetch_once` 已能确定站点,把站点传下去);
|
||||
- `media_size` / `download_media_limited` / `download_media_to_file` 里的
|
||||
`pximg.net → Referer` 硬编码删除,改为遍历 `SITES`(阶段 2 是遍历
|
||||
`[twitter, bsky, pixiv]` 静态列表)取 `media_headers(url)` 合并。
|
||||
|
||||
**注意**:Referer 判定依据是媒体 URL 的 host(`pximg.net`),**不是**站点
|
||||
PATTERN(pixiv 的 PATTERN 只匹配 `pixiv.net/artworks/...`),所以 `media_headers`
|
||||
不能挂在 PATTERN 匹配上,必须按 URL 独立匹配——这正是把它做成独立函数的原因。
|
||||
|
||||
**风险**:中。下载函数签名不变,行为必须逐字节不变;新增单元测试覆盖
|
||||
`media_headers("https://i.pximg.net/...") == Some(Referer)` 与
|
||||
`cache_key` 等价性(对全部既有用例断言新旧结果一致)。
|
||||
**回滚**:revert。
|
||||
|
||||
### 阶段 3:Site trait + SITES 注册表(完整方案,可选)
|
||||
|
||||
**动机**:加站点时 bot crate 与中央分派零改动;站点列表成为唯一注册点。
|
||||
|
||||
**新增**(`site/mod.rs`,按实施后的实际形态):
|
||||
|
||||
```rust
|
||||
/// Boxed, Send future produced by a Site async method. Boxed so the trait
|
||||
/// stays dyn-compatible; Send because URL/queue workers tokio::spawn these.
|
||||
type SiteFuture<'a, T, E = FetchError> =
|
||||
Pin<Box<dyn Future<Output = Result<T, E>> + Send + 'a>>;
|
||||
|
||||
pub trait Site: Send + Sync {
|
||||
fn id(&self) -> &'static str;
|
||||
fn pattern(&self) -> &'static Regex;
|
||||
fn enabled(&self) -> bool { true } // 默认: true
|
||||
fn cache_key(&self, url: &str) -> Option<String>;
|
||||
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched>;
|
||||
fn is_retryable(&self, err: &FetchError) -> bool; // 默认: Http|Transient
|
||||
fn media_headers(&self, url: &str) -> Option<Vec<(&'static str, String)>>; // 默认: None
|
||||
fn validate(&self) -> SiteFuture<'static, (), String>; // 默认: Ok(())
|
||||
}
|
||||
|
||||
static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| vec![
|
||||
Box::new(twitter::TwitterSite), Box::new(bsky::BskySite), Box::new(pixiv::PixivSite),
|
||||
]);
|
||||
```
|
||||
|
||||
- `fetch` → `find_site(url)`(注册表中首个 PATTERN 命中且 `enabled()` 的站点,
|
||||
返回 `&'static dyn Site`)→ `site.fetch_from_url(url).await`;
|
||||
- `cache_key` / `site_ids()` / `site_id_from_key()` / `apply_media_headers()` /
|
||||
`validate_all()` 全部遍历 `SITES`;`validate_all` 返回失败列表,pixiv 的
|
||||
`Site::validate` 失败时自行 `disable()`;
|
||||
- `match_site`/`SiteKind`(阶段 2 的静态分派)与中央 `fetch_error_is_retryable`
|
||||
删除,重试判定走 `site.is_retryable`;
|
||||
- `main.rs` 的 pixiv 特判 → `site::validate_all()` + 通用失败通知;
|
||||
- 保留各站点的 `PATTERN`/`enabled()`/`fetch_from_url()` 顶层导出(兼容既有
|
||||
测试),trait impl 只是薄壳。
|
||||
|
||||
**async 形态**(实施结论):**原生 AFIT 不可行**。
|
||||
|
||||
- 实测(rustc 1.95.0,edition 2024;**1.97.1 复测一致**):trait 里写
|
||||
`async fn` 报 "method is `async`"(非 dyn 兼容);写反糖
|
||||
`-> impl Future<...> + Send + '_` 报 "references an `impl Trait` type in its
|
||||
return type"(同样非 dyn 兼容);纯 RPITIT(无 `+ Send`)也一样。即:
|
||||
**RPITIT/AFIT 目前无法用于 `Vec<Box<dyn Site>>` 注册表**,与早期设计的
|
||||
判断相反。
|
||||
- **为什么**:dyn 分派要求调用方在编译期知道返回值大小以分配空间,而
|
||||
`async fn`/RPITIT 返回不透明的 Future——这是"非定长返回值走 dyn"的普遍问题,
|
||||
与 async 无关。Rust 1.75 稳定的 AFIT 只覆盖**静态分派**,dyn 路径被排除;
|
||||
原生 dyn 支持(AFIDT)是 2026-2027 的已接受项目目标,尚未进入 stable。
|
||||
参见 <https://rust-lang.github.io/rust-project-goals/2026/afidt-box.html>。
|
||||
- **采用 (a) 手写 `Pin<Box<dyn Future + Send + '_>>`**(`SiteFuture` 别名):
|
||||
零新依赖、dyn 兼容、future 保证 Send。签名噪音靠别名缓解;生命周期坑因
|
||||
站点是无状态单元结构体 + `'a` 同时约束 `&self` 与 `url` 而完全可控
|
||||
(future 只借用调用域内的 url)。
|
||||
- **(b) `async-trait`** 仍是可行备选(语法更干净、同样 box),但新增依赖;
|
||||
本仓库采用 (a) 后无需引入。
|
||||
- 若未来 Rust 稳定版落地 AFIDT(调用点 `dyn_box!`),可平滑迁移回原生
|
||||
`async fn`,实现体几乎不动。
|
||||
|
||||
**风险**:中。动中央分派,但每站点行为不变;注册表迭代 + `find_site` 补单测
|
||||
(`fetch`/`cache_key` 对既有 URL 集合的结果与阶段 2 完全一致)。
|
||||
**回滚**:revert。
|
||||
|
||||
### 阶段 4:FetchError 泛化(已实施)
|
||||
|
||||
**改动**:`FetchError` 新增 `Site { site: &'static str, error: Box<dyn std::error::Error + Send + Sync> }`
|
||||
变体(`Display`/`source()` 同步)。**`Pixiv(PixivError)` 变体保留**(未迁移)——
|
||||
它已有完整的 `Display`/`source()`/`is_retryable` 处理,替换纯属 churn。`Site`
|
||||
变体默认永久性(各站点 `is_retryable` 都不匹配它);需要可重试站点错误的站点
|
||||
应自行转换为 `Http`/`Transient` 再返回。
|
||||
|
||||
**风险**:低(纯增量变体)。测试:`site_error_variant_displays_and_sources`。
|
||||
|
||||
### 阶段 5:收尾
|
||||
|
||||
- 更新 `AGENTS.md` 的 "Site adapter convention" 段:写新约定(注册表 + `impl Site` +
|
||||
每站点 `cache_key`/`is_retryable`/`media_headers`),删除 "no trait" 表述;
|
||||
- `examples/fetch.rs` 不变(走 `site::fetch`);
|
||||
- 新增站点 checklist 见 §4。
|
||||
|
||||
## 4. 重构后新增站点 checklist
|
||||
|
||||
```
|
||||
1. crates/x-media/src/site/example/{mod,interface,model}.rs // 新模块
|
||||
2. impl Site for ExampleSite 并注册进 SITES // 注册一行
|
||||
3. (可选)token 读取 + validate() 实现 // 启动校验自动生效
|
||||
── bot crate 零改动 ──
|
||||
```
|
||||
|
||||
对比现状的 8-9 处,bot crate 完全不碰:`SetFormat` 白名单、格式查找口径、
|
||||
缓存 key、启动校验全部自动跟随注册表。
|
||||
|
||||
## 5. 权衡与明确不做的事
|
||||
|
||||
- **不做**:Media 类型扩展(`media.rs` + `MediaItemPayload` + `CachedMediaKind` +
|
||||
send.rs 约 10+ 处 match 的 blast radius)——这是"新增媒体类型"的摩擦,与"新增
|
||||
站点"正交,优先级低,保持现状。
|
||||
- **不做**:DI/全局注入改造(`CHAT_STORE`/`TASK_QUEUE`/`CONFIG` 的 `LazyLock` 静态
|
||||
模式是仓库惯例,与站点扩展无关)。
|
||||
- **不做**:schema 迁移——新站点只产生新的 cache key 前缀与 `message_format` JSON
|
||||
key,`link_cache`/`chat_state` 表结构均无需变化。
|
||||
- **代价**:阶段 3 引入 `dyn Site` 与 boxed future 签名(`SiteFuture`,见 §3);
|
||||
`Send` 约束前移到 trait 边界,站点 impl 的 future 必须 Send(现仅在各
|
||||
`tokio::spawn` 点检查,重构后在 impl 处即报错,提前暴露问题)。
|
||||
若站点数量长期 ≤5 且无新增迹象,阶段 2 的折中方案已够用;本次已按完整方案
|
||||
实施到阶段 4。
|
||||
|
||||
## 6. 提交序列(已按此实施)
|
||||
|
||||
| 阶段 | 提交 | hash |
|
||||
|---|---|---|
|
||||
| 1 | `refactor(site): carry site_id on Fetched; unify cache-key site lookup` | `7ca8fd1` |
|
||||
| 2 | `refactor(site): move cache_key/is_retryable/media_headers into site modules` | `5e23916` |
|
||||
| 3 | `refactor(site): introduce Site trait and SITES registry` | `bf4e615` |
|
||||
| 4 | `refactor(site): genericize FetchError::Site` | `5679a8c` |
|
||||
| 5 | `docs: update site adapter convention in AGENTS.md` | 本文档收尾提交 |
|
||||
|
||||
每阶段独立合入、独立回滚;阶段 2 完成后"加站点"摩擦已收敛,3/4 为深化。
|
||||
Reference in New Issue
Block a user