Compare commits

...
27 Commits
Author SHA1 Message Date
YoursFunny 5830a3f013 chore: bump version to 1.2.1 2026-08-14 17:55:26 +08:00
YoursFunny 183bb7e435 docs: add site registry refactor design 2026-08-14 17:55:10 +08:00
YoursFunny 2a8433a8d2 fix(twitter): map syndication TweetTombstone to NotFound
Deleted tweets answer the syndication endpoint with HTTP 200 and a
TweetTombstone (no `errors`, no `id_str`). The body classifier only
knew the `errors` shape, so tombstones fell through to the
`no id_str -> Sensitive` branch and degraded to an empty result,
making the bot reply "No media found" for a deleted tweet.

- extract parse_syndication_body(); tombstone shape -> NotFound
- propagate NotFound from the TWITTER_AUTH_TOKEN GraphQL fallback
  instead of swallowing it into empty_fetched
- unit tests for all body classes + live regression test on a real
  tombstoned tweet
2026-08-14 12:10:36 +08:00
YoursFunny 6b3e61881d logging: re-level, redact user data at info, and key links by post id
P0 — level rework + redaction:
- info now carries only lifecycle, per-post business results (sent /
  forwarded / copied / template applied), admin actions and anomalies
  (upload fallback, retry enqueue; dead-letter stays error).
- Per-request detail moved to debug: message/command logging, URL
  extraction, fetching/fetched, link-cache hits, media-group batch sends,
  queue processing (enqueue/processing/completed/rescheduled), photo
  processing (downscale/transcode), inline queries, sensitive-tweet note.
- Full user-submitted URLs and message text now appear only at debug; at
  info and above links are printed via the normalized cache key.

P1 — request correlation:
- handlers::log_key() maps a URL to its normalized post key
  (twitter:<id> / pixiv:<id> / bsky:<handle>/<rkey>). The whole lifecycle
  of one link (fetch -> send -> cache -> fallback) now logs [key=...], so
  multi-worker logs can be correlated by grepping the key.

Convention documented in AGENTS.md.
2026-08-13 23:34:06 +08:00
YoursFunny 47935dd7c6 fix(pixiv): stop retrying permanent 4xx API errors
site::fetch retried every PixivError, so a bad/expired token (403) or a
deleted artwork (404) burned all 3 attempts with backoff against pixiv's
API for nothing. Add PixivError::Status(u16) — the app-API calls now
surface the HTTP status — and retry only the transient classes: network
errors, 429 and 5xx. 4xx / Api (token errors) / Json / NoAuth are
returned immediately. The classification is a pure helper
(fetch_error_is_retryable) with unit tests.
2026-08-13 23:17:52 +08:00
YoursFunny 6911e9146e fix(handlers): stop URL workers by closing the job channel
The old stop only set an atomic flag checked between jobs: a worker
blocked in recv() never woke (the channel was never closed), and queued
jobs were neither drained nor abandoned in a defined way despite the
"drains up to 256 jobs" comment. Now stop_url_workers sets the flag,
drops the sender so blocked recv() calls wake with None, and awaits the
worker JoinHandles (each finishes its in-flight job first). main awaits
it inside the existing 30s shutdown timeout.
2026-08-13 23:15:14 +08:00
YoursFunny aa705aef90 chore: bump version to 1.2.0
New in 1.2.0:
- feat: debounced inline queries, bounded graceful shutdown, config
  fail-fast on invalid env values
- fix: local-media tasks (ugoira/bsky MP4) survive queue retries,
  photo-first ordering in mixed media groups, caption truncation to
  Telegram's 1024-char limit
- perf: SQLite connection pool, concurrent upload-fallback downloads,
  ugoira zip streamed to disk, photo processing without re-reading the
  temp file, release profile LTO
- ci: test/clippy gate + layered live/token job
- docs: English README (README.en.md)
2026-08-13 22:52:38 +08:00
YoursFunny 39260a8817 style: rustfmt config.rs and main.rs from the last two features 2026-08-13 22:43:35 +08:00
YoursFunny 6b9640aa48 build: enable thin LTO and single codegen units for release
Smaller/faster production binary (verified: cargo build --release -p
xmedia-bot builds clean with the new profile). panic=abort is
intentionally not set — queue workers and db closures rely on JoinHandle
catching panics, which abort would defeat.
2026-08-13 22:42:42 +08:00
YoursFunny ad59f518ff docs: add English README (README.en.md)
Full English translation of README.md — features, quick start, webhook
deployment (domain + IP-only with acme.sh), env table, command table.
The Chinese README stays the primary one.
2026-08-13 22:37:12 +08:00
YoursFunny e21643063e feat(config): fail fast on misspelled env values
A typo in EDIT_MESSAGE_TTL_SECONDS / WEBHOOK_PORT etc. used to silently
fall back to a default, so the bot ran with different behavior than the
operator intended (or failed much later on a bare .expect). Unparseable
values now log a loud warning naming the variable; invalid webhook
settings still surface as a hard .expect in webhook mode.
2026-08-13 22:26:23 +08:00
YoursFunny 246fc989f0 feat(main): bound graceful shutdown with a 30s timeout
The stop sequence awaited the queue workers, which can be mid-download
(30s client timeout) or mid-ugoira encode (minutes). A stuck worker would
hold shutdown forever; now the process logs and exits after 30s.
2026-08-13 22:26:16 +08:00
YoursFunny 2e2d1b3506 style: rustfmt the DbPool call sites from the connection-pool change
Formatting-only; the pool commit landed before cargo fmt was run.
2026-08-13 22:25:13 +08:00
YoursFunny 1747d321d8 perf(photo): stop re-reading the downloaded temp file
download_to_temp buffered the full bytes, wrote them to a temp file, and
prepare_photo then read the whole file back from disk. The bytes are
already in memory — pass them through (download_to_temp now returns
(file, bytes)) so photo processing never touches the disk for input.
Adds the bytes dependency to xmedia-bot (already in the lock via x-media).
2026-08-13 22:24:58 +08:00
YoursFunny 505990e49e perf(pixiv): stream the ugoira frame zip to disk instead of RAM
download_media_limited buffered the whole frame zip (cap 512 MB) in
memory before extraction, spiking RAM for large ugoira. New
site::download_media_to_file streams chunks straight to a temp file with
the same Content-Length / stream cap checks, and ugoira_video now opens
the zip from disk inside spawn_blocking. Adds FetchError::Io for local
write failures (hand-rolled error pattern preserved).
2026-08-13 22:22:13 +08:00
YoursFunny 95b475ff08 perf(send): prepare upload-fallback items concurrently
The download-and-reupload fallback downloaded each batch item serially,
so a 9-item batch took 9× the slowest download. Items are now prepared
concurrently (bounded to 3 in-flight downloads + photo processing) via a
JoinSet, then the group is uploaded in its original order; the per-item
logic moved into prepare_upload_item. Temp files stay alive until the
group request completes. A failing item still aborts the batch (the
JoinSet drop cancels the remaining prep tasks, as before).
2026-08-13 22:19:39 +08:00
YoursFunny 2297fdc91c fix(site): truncate captions to Telegram's 1024-char limit
A long tweet text or a pixiv artwork with many tags can exceed Telegram's
1024-char caption cap for HTML parse mode, turning an otherwise fine send
into a permanent 400. truncate_caption() cuts at a char boundary (never
splitting a multi-byte rune or an HTML entity like &amp;) and appends an
ellipsis. Applied in caption_with / caption_from_fields, the link-cache
re-send path and the inline-query captions.
2026-08-13 22:15:33 +08:00
YoursFunny 4580b79d4f fix(send): order photos first in mixed media groups
Telegram's sendMediaGroup requires the first item to be a photo when a
group mixes photos and videos; the previous code kept the source-site
order, so a mixed post with a video first (twitter media order is not
guaranteed) would 400 permanently. photos_first() stable-sorts photos
ahead of videos/animations before chunking; within-kind order is kept.
2026-08-13 22:14:07 +08:00
YoursFunny edb32c23b4 perf(db): reuse SQLite connections via a small per-store pool
Every DB operation (queue lease/enqueue, chat_state get/set, link_cache
read/write) used to open a fresh connection — including the busy timeout
and WAL pragma — then close it, on every message, URL job and callback.

Replace with DbPool: a tiny pool (4 connections max, semaphore-bounded
concurrency for backpressure) whose with_conn() method runs the closure on
a pooled connection inside spawn_blocking. Steady-state cost of an
operation is a list pop + semaphore acquire instead of a connection open.
2026-08-13 22:12:34 +08:00
YoursFunny 4a467641aa ci: add test/clippy workflow and gate live/token tests
The docker workflow only builds/pushes; tests were a local responsibility.
Add .github/workflows/ci.yml with two layers:

- test: cargo fmt --check + cargo clippy --workspace --all-targets -D
  warnings + cargo test --workspace (fully offline, no secrets) on every
  push/PR, including forks.
- live: the #[ignore]d live-network tests plus the pixiv token-gated
  tests, run on schedule / manual dispatch / tag pushes only (fork PRs
  cannot read repository secrets), with PIXIV_REFRESH_TOKEN /
  TWITTER_AUTH_TOKEN injected and continue-on-error for flaky sites.

Test gating (documented in AGENTS.md):
- live-network tests now carry #[ignore = "live network: ..."] (twitter 3,
  bsky 2, pixiv bogus-token 1) and run via -- --ignored live.
- pixiv token tests early-return when PIXIV_REFRESH_TOKEN is absent or
  empty (an unset GitHub secret arrives as ""); test_fetch previously
  failed without a token.

Also fixes the three clippy assertions_on_constants warnings in send.rs
(required for -D warnings).
2026-08-13 22:07:32 +08:00
YoursFunny bd43a12dee fix(send): keep local media (ugoira/bsky remux MP4) alive across queue retries
A task whose media is a locally produced file (pixiv ugoira MP4, bsky HLS
remux MP4) references a path inside a tempfile TempDir owned by Fetched.
The retry ran after that TempDir was dropped, so the file was already gone
and the retry always failed permanently ("local media file missing") — and
the upload fallback even tried to GET the local path as a URL.

Keep the temp dirs in a process-wide registry (Fetched::take_keep_alive ->
send::KEEP_ALIVE) that is only released when the task settles (sent or
permanently failed); the upload fallback now uploads local files directly
instead of attempting to download them.

Restart-mid-queue still loses the files (documented behavior in
input_file_for) — only the in-process retry path is fixed here.
2026-08-13 22:05:07 +08:00
YoursFunny c496e41c55 feat(handlers): debounce inline queries to stop fetch storms while typing
Telegram fires an inline query on every keystroke and every prefix of a
pasted URL (status/12, status/123, ...) matches the site patterns, so
typing used to trigger a full 3-attempt fetch per keystroke. Answer only
after the query has been stable for 800ms, dedupe repeats through
Telegram's inline cache (explicit cache_time 300), and let a repeat of a
query that produced no answer retry the fetch.
2026-08-13 21:57:07 +08:00
YoursFunny 68f026c990 docs: note version-bump convention in AGENTS.md 2026-08-10 19:29:21 +08:00
YoursFunny 62d80c8905 docs: sync README and AGENTS.md with 1.1.1 state 2026-08-10 19:16:56 +08:00
YoursFunny 78c9c841c6 chore: bump version to 1.1.1 2026-08-10 18:19:16 +08:00
YoursFunny e49d500d23 deps: switch TLS to rustls, drop libssl from the runtime image
OpenSSL came from two removable defaults: teloxide's `default` feature
(native-tls) and x-media's reqwest default features (default-tls). Switch
both to rustls (webpki-roots) so the binary links no system TLS libs:

- teloxide: default-features=false + rustls + ctrlc_handler (was part of
  the removed default)
- reqwest (x-media): default-features=false + rustls-tls

Verified: openssl-sys/native-tls gone from the tree, cargo check clean,
all 5 live twitter/bsky fetches pass over rustls, full container startup
works. The runtime image now needs no libssl.so.3/libcrypto.so.3 or CA
bundle (ffmpeg only processes local files; all downloads go through
reqwest), saving ~8MB.
2026-08-10 17:07:23 +08:00
YoursFunny 6feabd723b docker: fix 1.1.0 container startup (stub binary + missing libssl)
c40b074 broke the image two ways:
- `cargo clean -p` removes 0 files, so the real sources (host mtimes
  older than the step-1 stub build) were never recompiled and the image
  shipped the 337KB fn-main stub, exiting 0 on start. Restore the
  touch-based rebuild, which forces cargo to see every .rs as newer.
- bookworm-slim does not ship libssl3 despite the old comment; the bot
  links OpenSSL via teloxide/reqwest native-tls, so restore the
  libssl/libcrypto copies from the builder (same Debian release).
2026-08-10 16:37:12 +08:00
23 changed files with 1725 additions and 611 deletions
+64
View File
@@ -0,0 +1,64 @@
name: CI
# Test/lint gate (offline, no secrets) on every push/PR, plus a live-network
# job that exercises the real source sites and the token-gated pixiv tests.
#
# Layering:
# test — fmt + clippy + the full offline unit suite. Runs on every push
# and PR, including forks (it needs no secrets).
# live — the #[ignore]d live-network tests plus the pixiv tests that are
# gated on PIXIV_REFRESH_TOKEN. Runs on schedule / manual dispatch
# / tag pushes only, because pull requests from forks cannot read
# repository secrets. continue-on-error keeps a flaky external site
# from blocking, while the run still records the outcome.
#
# Test gating convention (keep in sync with AGENTS.md "Testing & QA"):
# - pure unit tests: plain #[test] / #[tokio::test], always run.
# - live-network tests: #[ignore = "live network: ..."], only run here.
# - token-gated tests (pixiv): #[tokio::test] with an early return when
# PIXIV_REFRESH_TOKEN is absent or empty (empty = unset CI secret).
on:
push:
branches: [master]
pull_request:
schedule:
# Weekly probe of the live endpoints, so external API changes surface.
- cron: '0 3 * * 1'
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
with:
components: clippy, rustfmt
- uses: Swatinem/rust-cache@v2
- name: Check formatting
run: cargo fmt --check
- name: Lint (deny warnings)
run: cargo clippy --workspace --all-targets -- -D warnings
- name: Run offline tests
run: cargo test --workspace
live:
needs: test
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/v')
runs-on: ubuntu-latest
continue-on-error: true
env:
PIXIV_REFRESH_TOKEN: ${{ secrets.PIXIV_REFRESH_TOKEN }}
TWITTER_AUTH_TOKEN: ${{ secrets.TWITTER_AUTH_TOKEN }}
steps:
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
# Full suite: with the secret present, the pixiv token-gated tests run;
# without it they skip themselves. Live tests stay #[ignore]d here.
- name: Run token-gated tests
run: cargo test --workspace
# The live-network tests, by the "live" name filter (all #[ignore]d).
- name: Run live-network tests
run: cargo test --workspace -- --ignored live
+14 -13
View File
@@ -4,7 +4,7 @@
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README and user-facing strings are in Chinese. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
Two-crate Cargo workspace (both v1.0.3, edition 2024, resolver 3):
Two-crate Cargo workspace (both v1.2.1, edition 2024, resolver 3):
- **`crates/x-media`** — library that fetches and normalizes media from the three sites. Pure, no Telegram knowledge.
- **`crates/xmedia-bot`** — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.
@@ -18,7 +18,7 @@ Telegram update → Dispatcher (polling or axum webhook) → dptree branches
└─ callback_query → "forward" (copy to channel) / "template|<name>" (apply caption template)
```
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)``Fetched` → builds a `Task``send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → single worker leases (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)``Fetched` → builds a `Task``send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
The `x-media` library: `site::fetch(url)` dispatches (in order) twitter → bsky → pixiv via per-site regex `PATTERN` and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
@@ -28,9 +28,9 @@ The `x-media` library: `site::fetch(url)` dispatches (in order) twitter → bsky
|---|---|
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, site struct, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`) |
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, queue worker start, pixiv validation, 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, pixiv validation, 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
| `crates/xmedia-bot/src/handlers.rs` | `Command` enum (teloxide `BotCommands`), message/inline/callback handlers, URL extraction, global statics; per-URL work spawned with a `Semaphore(8)` cap (teloxide's per-chat workers are sequential — batch-forwards need concurrency) |
| `crates/xmedia-bot/src/handlers.rs` | `Command` enum (teloxide `BotCommands`), message/inline/callback handlers, URL extraction, global statics; per-URL work flows through a bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns (teloxide's per-chat workers are sequential — batch-forwards need concurrency) |
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table) |
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections |
@@ -48,19 +48,19 @@ cargo clippy --workspace --all-targets # lint (Clippy is the configured IDE lint
cargo fmt --check # formatting
```
Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb`. Runtime requires **ffmpeg** (built into the image).
Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb`. Runtime requires **ffmpeg** (built into the image). The builder fetches crates.io + ffmpeg; on restricted networks pass proxy build args, e.g. `--build-arg HTTP_PROXY=http://host.docker.internal:10808 --build-arg HTTPS_PROXY=…` (Docker Desktop builds can't reach the host loopback — use `host.docker.internal`).
## Code Conventions & Common Patterns
- **No anyhow/thiserror.** Errors are hand-rolled enums with manual `Display`/`source()`/`From` impls: `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `FetchError` (`Http`/`Json`/`Pixiv`/`NotFound`/`Blocked`), `PixivError`, `Classification`. New errors should follow this pattern.
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers rebuild `Bot::from_env()`.
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers share the process-wide `send::BOT` (`LazyLock<Bot>`, force-initialized in `main` so a missing token fails at startup).
- **Async**: tokio multi-thread runtime (`#[tokio::main]` default). All rusqlite I/O inside `tokio::task::spawn_blocking`. Long loops use `tokio::select!` with `tokio::sync::{watch, Notify}` stop/wake channels. No streams.
- **Blocking sync primitives**: `parking_lot::Mutex` for hot caches, `tokio::sync::Mutex` for async-shared state (pixiv token cache), `AtomicBool` for feature gates.
- **Site adapter convention** (no trait, no enum dispatch — follow the existing convention): each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`; `site/mod.rs` re-exports the site struct and `fetch_once` adds one guarded if-branch. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one branch in `fetch_once`.
- **Serde**: per-site `model.rs` are pure `Deserialize` DTOs mirroring API JSON; site structs in `interface.rs` have private fields, a `caption()` builder, and `impl From<SiteStruct> for Fetched`. Persisted payloads use internally-tagged enums (`#[serde(tag = "kind")]` / `type`).
- **Naming**: module-per-concern, snake_case files, `CamelCase` types, `snake_case` fns. `//!` module docs and `///` docs on non-obvious logic (syndication token, ugoira encoding, `display_text_range`).
- **Retries**: only `x-media::site::fetch` retries (3 attempts, `1 << attempt` backoff, HTTP errors only). Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`).
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`).
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`). Level convention: `info` = lifecycle + per-post business results (`sent`/`forwarded`/`copied`), admin/operator actions and anomalies (fallback, retry enqueue, dead-letter is `error`); `debug` = per-request detail (message/command/URL extraction, `fetching`/`fetched`, batch sends, queue processing, photo processing, inline queries). Full user-submitted URLs and message text only appear at `debug`; at `info` and above links are printed via the normalized cache key (`handlers::log_key`, e.g. `[key=twitter:123...]`) so logs stay short and do not echo user data.
## Important Files
@@ -72,7 +72,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL |
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) |
| `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` |
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime hack, static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint |
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime bump (cargo's freshness is mtime-based and `cargo clean -p` removes 0 files — the touch is what forces the real sources to rebuild while deps stay cached), static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, optional `FFMPEG_SHA256` checksum, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint. Runtime ships **no libssl/libcrypto/CA bundle** — rustls webpki-roots handles all TLS, and the static ffmpeg only processes local files (downloads go through reqwest) |
| `docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim) |
| `docker-compose.yml.example` | Deployment env reference (real `docker-compose.yml` is gitignored). Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) |
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master; **no test step**; buildx gha cache (`cache-from`/`cache-to`, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs |
@@ -82,7 +82,8 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
- **Rust, stable, edition 2024**, workspace resolver 3. No `rust-version`/MSRV pin, no `rust-toolchain.toml` — recent stable is assumed. No nightly features.
- Package manager: **Cargo** (workspace with path dep `x-media``xmedia-bot`). No `[workspace.package]`/shared deps — each crate lists deps independently.
- **Two reqwest versions coexist in the lock** (0.12.28 via teloxide, 0.13.3 in x-media) — don't unify casually.
- **TLS is rustls end-to-end** (no native-tls/openssl in the tree, no libssl in the Docker runtime image): `teloxide` is declared `default-features = false` with `["webhooks-axum", "macros", "rustls", "ctrlc_handler"]` (the removed `default` also carried `native-tls` and `ctrlc_handler` — the latter must stay); x-media's reqwest is `default-features = false` with `["json", "rustls-tls"]` (webpki-roots baked in, so the image ships no CA bundle). One reqwest 0.12.28 in the lock.
- **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag (the tag push triggers the Docker Hub build).
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `data/task_queue.db` is CWD-relative — run from the workspace root, or `/app` in Docker. Mount `./data` and `./cert` volumes.
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `docker-compose.yml`, `/target`, `.idea/`.
@@ -90,10 +91,10 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
## Testing & QA
- **~51 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- **~80 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches.
- Live-network tests exist in `site/twitter/interface.rs` (3), `site/bsky/interface.rs` (2), `site/pixiv/interface.rs`/`api.rs` (env-gated on `PIXIV_REFRESH_TOKEN`/dotenv, skip by early return). Run the full suite with `cargo test --workspace`.
- Live-network tests exist in `site/twitter/interface.rs` (3), `site/bsky/interface.rs` (2), `site/pixiv/interface.rs`/`api.rs`. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""``is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`.
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
- **CI runs no tests** — `.github/workflows/docker.yml` only builds/pushes the image; verification is a local responsibility.
- **CI** — `.github/workflows/ci.yml` runs `cargo fmt --check` + `cargo clippy --workspace --all-targets -- -D warnings` + `cargo test --workspace` (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN` from secrets, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds/pushes the image only.
- Untested and hard to test without a mock seam: `handlers.rs` (depends directly on teloxide `Bot`); `main.rs`, `config.rs`, `state.rs`; `media.rs`, `lib.rs`, all `model.rs`.
- No coverage tracking, no lint gate in CI.
- No coverage tracking.
Generated
+161 -237
View File
@@ -16,7 +16,7 @@ checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
dependencies = [
"cfg-if",
"cipher",
"cpufeatures",
"cpufeatures 0.2.17",
]
[[package]]
@@ -242,6 +242,23 @@ version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
[[package]]
name = "chacha20"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"rand_core 0.10.1",
]
[[package]]
name = "chrono"
version = "0.4.44"
@@ -279,26 +296,6 @@ version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6"
[[package]]
name = "core-foundation"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "core-foundation"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "core-foundation-sys"
version = "0.8.7"
@@ -314,6 +311,15 @@ dependencies = [
"libc",
]
[[package]]
name = "cpufeatures"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
dependencies = [
"libc",
]
[[package]]
name = "crc"
version = "3.4.0"
@@ -495,15 +501,6 @@ version = "1.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
[[package]]
name = "encoding_rs"
version = "0.8.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3"
dependencies = [
"cfg-if",
]
[[package]]
name = "env_logger"
version = "0.10.2"
@@ -598,33 +595,12 @@ dependencies = [
"miniz_oxide",
]
[[package]]
name = "fnv"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
[[package]]
name = "foldhash"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
[[package]]
name = "foreign-types"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1"
dependencies = [
"foreign-types-shared",
]
[[package]]
name = "foreign-types-shared"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b"
[[package]]
name = "form_urlencoded"
version = "1.2.2"
@@ -739,8 +715,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"wasi",
"wasm-bindgen",
]
[[package]]
@@ -764,29 +742,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"r-efi 6.0.0",
"rand_core 0.10.1",
"wasip2",
"wasip3",
]
[[package]]
name = "h2"
version = "0.4.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "171fefbc92fe4a4de27e0698d6a5b392d6a0e333506bc49133760b3bcf948733"
dependencies = [
"atomic-waker",
"bytes",
"fnv",
"futures-core",
"futures-sink",
"http",
"indexmap 2.14.0",
"slab",
"tokio",
"tokio-util",
"tracing",
"wasm-bindgen",
]
[[package]]
@@ -925,7 +887,6 @@ dependencies = [
"bytes",
"futures-channel",
"futures-core",
"h2",
"http",
"http-body",
"httparse",
@@ -950,22 +911,7 @@ dependencies = [
"tokio",
"tokio-rustls",
"tower-service",
]
[[package]]
name = "hyper-tls"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "70206fc6890eaca9fde8a0bf71caa2ddfc9fe045ac9e5c70df101a7dbde866e0"
dependencies = [
"bytes",
"http-body-util",
"hyper",
"hyper-util",
"native-tls",
"tokio",
"tokio-native-tls",
"tower-service",
"webpki-roots",
]
[[package]]
@@ -986,11 +932,9 @@ dependencies = [
"percent-encoding",
"pin-project-lite",
"socket2",
"system-configuration",
"tokio",
"tower-service",
"tracing",
"windows-registry",
]
[[package]]
@@ -1299,6 +1243,12 @@ version = "0.4.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897"
[[package]]
name = "lru-slab"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
[[package]]
name = "lzma-rs"
version = "0.3.0"
@@ -1369,23 +1319,6 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "native-tls"
version = "0.2.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "465500e14ea162429d264d44189adc38b199b62b1c21eea9f69e4b73cb03bbf2"
dependencies = [
"libc",
"log",
"openssl",
"openssl-probe",
"openssl-sys",
"schannel",
"security-framework",
"security-framework-sys",
"tempfile",
]
[[package]]
name = "num-conv"
version = "0.2.1"
@@ -1416,49 +1349,6 @@ version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "openssl"
version = "0.10.79"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf0b434746ee2832f4f0baf10137e1cabb18cbe6912c69e2e33263c45250f542"
dependencies = [
"bitflags",
"cfg-if",
"foreign-types",
"libc",
"openssl-macros",
"openssl-sys",
]
[[package]]
name = "openssl-macros"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "openssl-probe"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
[[package]]
name = "openssl-sys"
version = "0.9.115"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "158fe5b292746440aa6e7a7e690e55aeb72d41505e2804c23c6973ad0e9c9781"
dependencies = [
"cc",
"libc",
"pkg-config",
"vcpkg",
]
[[package]]
name = "parking_lot"
version = "0.12.5"
@@ -1627,6 +1517,62 @@ dependencies = [
"cc",
]
[[package]]
name = "quinn"
version = "0.11.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8"
dependencies = [
"bytes",
"cfg_aliases",
"pin-project-lite",
"quinn-proto",
"quinn-udp",
"rustc-hash",
"rustls",
"socket2",
"thiserror",
"tokio",
"tracing",
"web-time",
]
[[package]]
name = "quinn-proto"
version = "0.11.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
dependencies = [
"bytes",
"getrandom 0.4.2",
"lru-slab",
"rand 0.10.2",
"rand_pcg",
"ring",
"rustc-hash",
"rustls",
"rustls-pki-types",
"slab",
"thiserror",
"tinyvec",
"tracing",
"web-time",
]
[[package]]
name = "quinn-udp"
version = "0.5.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2",
"tracing",
"windows-sys 0.52.0",
]
[[package]]
name = "quote"
version = "1.0.45"
@@ -1656,7 +1602,18 @@ checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a"
dependencies = [
"libc",
"rand_chacha",
"rand_core",
"rand_core 0.6.4",
]
[[package]]
name = "rand"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
dependencies = [
"chacha20",
"getrandom 0.4.2",
"rand_core 0.10.1",
]
[[package]]
@@ -1666,7 +1623,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
dependencies = [
"ppv-lite86",
"rand_core",
"rand_core 0.6.4",
]
[[package]]
@@ -1678,6 +1635,21 @@ dependencies = [
"getrandom 0.2.17",
]
[[package]]
name = "rand_core"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
[[package]]
name = "rand_pcg"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a"
dependencies = [
"rand_core 0.10.1",
]
[[package]]
name = "rc-box"
version = "1.3.0"
@@ -1753,31 +1725,28 @@ checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
dependencies = [
"base64",
"bytes",
"encoding_rs",
"futures-core",
"futures-util",
"h2",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-tls",
"hyper-util",
"js-sys",
"log",
"mime",
"mime_guess",
"native-tls",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls",
"rustls-pki-types",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tokio-native-tls",
"tokio-rustls",
"tokio-util",
"tower",
"tower-http",
@@ -1787,6 +1756,7 @@ dependencies = [
"wasm-bindgen-futures",
"wasm-streams",
"web-sys",
"webpki-roots",
]
[[package]]
@@ -1826,6 +1796,12 @@ dependencies = [
"smallvec",
]
[[package]]
name = "rustc-hash"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
[[package]]
name = "rustix"
version = "1.1.4"
@@ -1846,6 +1822,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b"
dependencies = [
"once_cell",
"ring",
"rustls-pki-types",
"rustls-webpki",
"subtle",
@@ -1858,6 +1835,7 @@ version = "1.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9"
dependencies = [
"web-time",
"zeroize",
]
@@ -1884,15 +1862,6 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "schannel"
version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "schemars"
version = "0.9.0"
@@ -1923,29 +1892,6 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
[[package]]
name = "security-framework"
version = "3.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
dependencies = [
"bitflags",
"core-foundation 0.10.1",
"core-foundation-sys",
"libc",
"security-framework-sys",
]
[[package]]
name = "security-framework-sys"
version = "2.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "semver"
version = "1.0.28"
@@ -2057,7 +2003,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8"
dependencies = [
"cfg-if",
"cpufeatures",
"cpufeatures 0.2.17",
"digest",
]
@@ -2167,27 +2113,6 @@ dependencies = [
"syn",
]
[[package]]
name = "system-configuration"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b"
dependencies = [
"bitflags",
"core-foundation 0.9.4",
"system-configuration-sys",
]
[[package]]
name = "system-configuration-sys"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "take_mut"
version = "0.2.2"
@@ -2216,7 +2141,7 @@ dependencies = [
"log",
"mime",
"pin-project",
"rand",
"rand 0.8.6",
"serde",
"serde_json",
"teloxide-core",
@@ -2399,16 +2324,6 @@ dependencies = [
"syn",
]
[[package]]
name = "tokio-native-tls"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2"
dependencies = [
"native-tls",
"tokio",
]
[[package]]
name = "tokio-rustls"
version = "0.26.4"
@@ -2739,6 +2654,25 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "web-time"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "webpki-roots"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed"
dependencies = [
"rustls-pki-types",
]
[[package]]
name = "winapi-util"
version = "0.1.11"
@@ -2789,17 +2723,6 @@ version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-registry"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720"
dependencies = [
"windows-link",
"windows-result",
"windows-strings",
]
[[package]]
name = "windows-result"
version = "0.4.1"
@@ -3002,13 +2925,13 @@ checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
[[package]]
name = "x-media"
version = "1.1.0"
version = "1.2.1"
dependencies = [
"bytes",
"dotenv",
"html-escape",
"log",
"rand",
"rand 0.8.6",
"regex",
"reqwest",
"serde",
@@ -3021,8 +2944,9 @@ dependencies = [
[[package]]
name = "xmedia-bot"
version = "1.1.0"
version = "1.2.1"
dependencies = [
"bytes",
"dotenv",
"fast_image_resize",
"html-escape",
@@ -3031,7 +2955,7 @@ dependencies = [
"parking_lot",
"png",
"pretty_env_logger",
"rand",
"rand 0.8.6",
"rusqlite",
"serde",
"serde_json",
+6 -1
View File
@@ -2,6 +2,11 @@
members = ["crates/x-media", "crates/xmedia-bot"]
resolver = "3"
# Smaller production binary; debug symbols are not shipped anyway.
# Smaller/faster production binary: strip debug symbols, link-time
# optimization across crates, and one codegen unit per crate (bigger LTO
# wins). panic=abort is intentionally NOT set: queue workers and db
# closures rely on JoinHandle catching panics, which abort would defeat.
[profile.release]
strip = true
lto = "thin"
codegen-units = 1
+12 -9
View File
@@ -42,13 +42,15 @@ RUN wget -q -O /tmp/ffmpeg.zip "$FFMPEG_URL" \
&& rm /tmp/ffmpeg.zip \
&& /usr/local/bin/ffmpeg -version >/dev/null
# 3. Real sources last: only our crates recompile on source changes.
# `cargo clean -p` drops the two crates' artifacts while keeping the
# compiled dependency layer, forcing a deterministic rebuild of the real
# sources. (The previous `touch`-mtimes hack silently shipped the stub
# binary when host files carried future timestamps.)
# 3. Real sources last: only our crates recompile on source changes. Cargo's
# freshness check is mtime-based; the COPY'd host files usually predate the
# step-1 stub build, so cargo would consider the stub up to date and never
# compile the real sources. `touch` makes every .rs newer than the stub
# artifacts, forcing a rebuild of just the two crates while the compiled
# dependency layer stays cached. (`cargo clean -p` does NOT work here — it
# removes 0 files and the stub binary silently ships.)
COPY crates/ ./crates/
RUN cargo clean -p xmedia-bot -p x-media \
RUN find crates -type f -name '*.rs' -exec touch {} + \
&& cargo build --release -p xmedia-bot
# ---------- runtime stage ----------
@@ -62,9 +64,10 @@ LABEL org.opencontainers.image.title="${APP_NAME}"
# Everything is copied in — no apt in the runtime stage. Privilege dropping is
# done by docker-entrypoint.sh with setpriv (util-linux, already in
# bookworm-slim), so no gosu needed. (libssl3/libcrypto are already in
# bookworm-slim; only ca-certificates and ffmpeg need copying.)
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
# bookworm-slim), so no gosu needed. TLS is rustls (webpki-roots baked in,
# see Cargo.toml feature `rustls`/`rustls-tls`), so no system CA bundle or
# libssl are needed; the static ffmpeg only processes local files (all
# downloads go through reqwest).
COPY --from=builder /usr/local/bin/ffmpeg /usr/local/bin/ffmpeg
WORKDIR /app
+123
View File
@@ -0,0 +1,123 @@
# TelegramXMediaBot
A Telegram bot that turns post links from X / Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags.
## Features
- Sending a link in a private chat fetches and sends the images, videos and GIFs automatically; oversized media is split into batches
- Text-only posts report "no media"; unsupported links are silently ignored
- Inline queries (`@bot <link>`)
- Bind a forward channel for automatic forwarding; edit the caption before forwarding and apply custom templates
- Failed sends are retried automatically with persistence; the user is notified after retries are exhausted
- Pixiv ugoira animations are transcoded to MP4; Bluesky videos are remuxed (HLS stream → MP4)
- Photos exceeding Telegram's size/dimension limits are compressed automatically (original format kept, JPEG fallback only when needed)
- Link-result cache: after a successful send the Telegram file ids and caption fields are cached locally, so a repeated link is re-sent from local state — no source-site request, no media file stored (expiry controlled by `LINK_CACHE_TTL_SECONDS`, default 7 days)
## Quick start
```bash
# Required: BotFather token; optional: PIXIV_REFRESH_TOKEN (Pixiv is disabled without it)
export TELOXIDE_TOKEN=<token>
export PIXIV_REFRESH_TOKEN=<token>
cargo run -p xmedia-bot
```
Docker deployment (see `docker-compose.yml.example`):
```bash
docker build -t tgxmb .
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
```
Environment variables: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `BOT_ADMIN`, `EDIT_MESSAGE_TTL_SECONDS`, `LINK_CACHE_TTL_SECONDS`, `RUST_LOG`, `WEBHOOK*`, `TWITTER_AUTH_TOKEN` (optional).
NSFW tweets: the public syndication endpoint does not return sensitive content. Setting `TWITTER_AUTH_TOKEN` (the `auth_token` cookie value of a logged-in x.com session) lets the bot fetch NSFW media in the logged-in state only when it hits a withheld tweet; without it, the bot reports no media.
### Webhook deployment (needs a reverse proxy)
`docker-compose.yml.example` ships an [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) reverse-proxy orchestration. Pick one deployment shape:
**With a domain**
1. Point a DNS A record at the server
2. In compose set `VIRTUAL_HOST` and `WEBHOOK_URL` to the domain, and uncomment `ACME_HOST` (set it to the domain)
3. acme-companion issues and renews certificates automatically — nothing manual
**IP only**
Let's Encrypt can issue certificates for public IPs (available since 2026, validity ~7 days, requires the `shortlived` profile). Use [acme.sh](https://github.com/acmesh-official/acme.sh) to issue and renew automatically, no manual certificates:
1. Add an acme-ip service to compose (issue + daily auto-renewal check):
```yaml
acme-ip:
image: neilpang/acme.sh
container_name: acme-ip
command: daemon
restart: always
volumes:
- certs:/acme.sh
- html:/usr/share/nginx/html
- /var/run/docker.sock:/var/run/docker.sock:ro
networks: [proxy]
```
2. First issuance (replace `<SERVER_IP>` with the server's public IP; IPv6 works too, repeat `-d` for more):
```bash
docker compose exec acme-ip acme.sh --issue --server letsencrypt \
-d <SERVER_IP> --cert-profile shortlived --days 3 \
--webroot /usr/share/nginx/html \
--install-cert --cert-file /acme.sh/<SERVER_IP>.crt \
--key-file /acme.sh/<SERVER_IP>.key \
--reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP"
```
3. In compose set `VIRTUAL_HOST: '<SERVER_IP>'` and `WEBHOOK_URL: 'https://<SERVER_IP>/'`; no `WEBHOOK_CERT` needed. Renewal is handled by the acme.sh daemon (`--days 3` = renew every 3 days, buffer against the 7-day validity), and a successful renewal HUP-notifies nginx-proxy to load the new certificate.
Limitations: certificate validity ~7 days; only http-01/tls-alpn-01 validation (port 80 must be publicly reachable); no DNS-01, private IPs or IP ranges; at most 5 certificates per 168 hours for the same IP set. It is recommended to trial-issue with `--server letsencrypt_test` first, then switch to the production server.
Telegram only accepts ports 443/80/88/8443.
<details>
<summary>Environment variables</summary>
| Variable | Description |
|---|---|
| `TELOXIDE_TOKEN` | Bot token (required) |
| `PIXIV_REFRESH_TOKEN` | Pixiv refresh token; Pixiv is disabled without it |
| `BOT_ADMIN` | Admin chat IDs, comma-separated; receives start/stop notifications |
| `EDIT_MESSAGE_TTL_SECONDS` | Edit-before-forward record expiry in seconds, default 86400 |
| `LINK_CACHE_TTL_SECONDS` | Link-result cache expiry in seconds, default 604800 (7 days) |
| `RUST_LOG` | Log level |
| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW) |
| `LOCAL_USER_ID` | UID the container runs as, default 9001 |
| `VIRTUAL_HOST` | Public domain or IP; nginx-proxy routes by this |
| `VIRTUAL_PORT` | Port the bot listens on inside the container; nginx-proxy's forwarding target |
| `ACME_HOST` | Domain deployment: when set to the domain, acme-companion issues/renews certificates automatically |
| `DEFAULT_HOST` | nginx-proxy routes requests with unknown Host headers to this vhost (needed for IP access) |
| `DEFAULT_EMAIL` | acme-companion certificate notification email |
| `WEBHOOK` | `true` enables webhook mode (polling by default) |
| `WEBHOOK_LISTEN` / `WEBHOOK_PORT` | Listen address/port inside the bot container |
| `WEBHOOK_URL` | Public HTTPS URL (`https://domain/` or `https://IP/`) |
| `WEBHOOK_CERT` | Optional; self-signed certificate path, only used for Telegram-side validation (TLS is terminated by the reverse proxy) |
| `WEBHOOK_SECRET_TOKEN` | Update validation token (`X-Telegram-Bot-Api-Secret-Token`) |
</details>
## Commands
| Command | Description |
|---|---|
| `/start` | Welcome message |
| `/help` | List all commands and usage (this command table) |
| `/set_forward_channel <channel>` | Set the forward channel: `@channel` or channel ID; media messages are forwarded to it automatically afterwards |
| `/remove_forward_channel` | Remove the forward channel |
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or taps a template button to apply one) |
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward") |
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/clear_cache [link]` | Clear the link cache (admin only); with a link only that entry, otherwise everything |
| `/bot_dict` | Show the current chat state (debugging) |
Link processing works only in private chats; commands work in any chat.
## Notes
- State is persisted in `data/task_queue.db`; compose deployments use the bind mount `./data` (keep it a directory for easy backups)
- The runtime needs ffmpeg (built into the Docker image)
- Tests: `cargo test --workspace`
+5 -1
View File
@@ -9,7 +9,8 @@ Telegram 机器人,将 X / Twitter、Pixiv、Bluesky 的帖子链接转换为
- 支持内联查询(`@机器人 <链接>`
- 可绑定转发频道自动转发;支持转发前编辑 caption 与自定义模板
- 发送失败自动重试并持久化,重试耗尽后通知用户
- Pixiv ugoira 动图自动转码为 MP4
- Pixiv ugoira 动图自动转码为 MP4Bluesky 视频自动转码(HLS 流 → MP4)
- 超过 Telegram 尺寸/大小限制的图片自动压缩(保持原格式,必要时转 JPEG)
- 链接结果本地缓存:成功发送后缓存 Telegram file id 与 caption 等,再次收到相同链接直接本地重发,不再请求源站、不保存媒体文件(`LINK_CACHE_TTL_SECONDS` 控制过期,默认 7 天)
## 快速开始
@@ -84,6 +85,7 @@ Telegram 只接受 443/80/88/8443 端口。
| `EDIT_MESSAGE_TTL_SECONDS` | 转发前编辑记录过期秒数,默认 86400 |
| `LINK_CACHE_TTL_SECONDS` | 链接结果缓存过期秒数,默认 604800(7 天) |
| `RUST_LOG` | 日志级别 |
| `TELOXIDE_PROXY` | HTTP 代理(如 `http://127.0.0.1:10808`);同时作用于 Telegram Bot API 与站点抓取请求,网络受限环境(如 GFW)必需 |
| `LOCAL_USER_ID` | 容器内运行用户 UID,默认 9001 |
| `VIRTUAL_HOST` | 对外域名或 IPnginx-proxy 按此路由 |
| `VIRTUAL_PORT` | bot 容器内监听端口,nginx-proxy 的转发目标 |
@@ -93,6 +95,7 @@ Telegram 只接受 443/80/88/8443 端口。
| `WEBHOOK` | `true` 启用 webhook 模式(默认轮询) |
| `WEBHOOK_LISTEN` / `WEBHOOK_PORT` | bot 容器内监听地址/端口 |
| `WEBHOOK_URL` | 对外公网 HTTPS 地址(`https://域名/` 或 `https://IP/` |
| `WEBHOOK_CERT` | 可选;自签名证书路径,仅用于 Telegram 侧验证(TLS 由反向代理终止) |
| `WEBHOOK_SECRET_TOKEN` | 更新校验令牌(`X-Telegram-Bot-Api-Secret-Token` |
</details>
@@ -108,6 +111,7 @@ Telegram 只接受 443/80/88/8443 端口。
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/clear_cache [链接]` | 清空链接缓存(仅管理员);带链接只清该条,否则清空全部 |
| `/bot_dict` | 查看当前聊天状态(调试用) |
链接处理仅限私聊;命令在任意聊天可用。
+2 -2
View File
@@ -1,10 +1,10 @@
[package]
name = "x-media"
version = "1.1.0"
version = "1.2.1"
edition = "2024"
[dependencies]
reqwest = { version = "0.12", features = ["json"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
regex = "1.12"
@@ -416,6 +416,7 @@ mod tests {
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
async fn live_fetch_with_photos() {
let fetched =
fetch_from_url("https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m")
@@ -429,6 +430,7 @@ mod tests {
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
async fn live_fetch_smoke() {
let fetched =
fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
+212 -20
View File
@@ -66,7 +66,8 @@ impl Fetched {
/// HTML-escaped in full, then the (already-escaped) placeholder values
/// are substituted — users can structure text but never inject raw HTML
/// or attributes. An empty/unknown format falls back to the built-in
/// caption.
/// caption. The result is truncated to [`MAX_CAPTION_CHARS`] (Telegram's
/// caption limit for HTML parse mode).
pub fn caption_with(&self, format: &str) -> String {
match (&self.render_data, format.is_empty()) {
(Some(data), false) => caption_from_fields(
@@ -78,7 +79,7 @@ impl Fetched {
&data.title,
&data.tags,
),
_ => self.caption.clone(),
_ => truncate_caption(&self.caption),
}
}
@@ -95,11 +96,47 @@ impl Fetched {
)
})
}
/// Hands over the temp dir keeping locally produced media (ugoira MP4,
/// bsky remux MP4) alive. The bot keeps it while its task may still be
/// retried by the queue, which runs after this [`Fetched`] is dropped and
/// its temp files would otherwise be gone. `None` when no such dir exists.
pub fn take_keep_alive(&mut self) -> Option<tempfile::TempDir> {
self._keep_alive.take()
}
}
/// Telegram's caption length limit (chars) for HTML parse mode; longer
/// captions are rejected with a 400.
pub const MAX_CAPTION_CHARS: usize = 1024;
/// Truncates a caption to at most [`MAX_CAPTION_CHARS`] chars, appending an
/// ellipsis when cut. Backs off to before an unclosed HTML entity (`&amp`
/// without its `;` would be malformed HTML and rejected by Telegram).
pub fn truncate_caption(caption: &str) -> String {
if caption.chars().count() <= MAX_CAPTION_CHARS {
return caption.to_string();
}
// Leave one char for the ellipsis; floor_char_boundary lands on a char
// edge (byte index ≤ MAX-1, so chars ≤ MAX-1).
let mut end = caption.floor_char_boundary(MAX_CAPTION_CHARS - 1);
// Don't split an entity: if the last '&' before `end` has no closing ';'
// inside the kept part, cut before it.
if let Some(amp) = caption[..end].rfind('&')
&& !caption[amp..end].contains(';')
{
end = amp;
}
let mut s = caption[..end].to_string();
s.push('…');
s
}
/// Renders a user-supplied caption format from raw (already-escaped) field
/// values with the same escaping/substitution rules as
/// [`Fetched::caption_with`]. An empty format returns `built_in` unchanged.
/// The result is truncated to [`MAX_CAPTION_CHARS`] (Telegram's caption
/// limit for HTML parse mode).
pub fn caption_from_fields(
format: &str,
built_in: &str,
@@ -110,15 +147,17 @@ pub fn caption_from_fields(
tags: &str,
) -> String {
if format.is_empty() {
return built_in.to_string();
return truncate_caption(built_in);
}
let escaped = html_escape::encode_text(format).into_owned();
escaped
.replace("{url}", url)
.replace("{author}", author)
.replace("{author_url}", author_url)
.replace("{title}", title)
.replace("{tags}", tags)
truncate_caption(
&escaped
.replace("{url}", url)
.replace("{author}", author)
.replace("{author_url}", author_url)
.replace("{title}", title)
.replace("{tags}", tags),
)
}
/// Stable per-post cache key derived from any supported URL, so variant
@@ -152,6 +191,9 @@ pub enum FetchError {
TooLarge,
/// A transient server-side failure (429 / 5xx); [`fetch`] retries these.
Transient(String),
/// A local I/O failure while streaming a download to disk
/// (see [`download_media_to_file`]).
Io(std::io::Error),
}
impl fmt::Display for FetchError {
@@ -165,6 +207,7 @@ impl fmt::Display for FetchError {
FetchError::Sensitive => write!(f, "content withheld (sensitive)"),
FetchError::TooLarge => write!(f, "media too large"),
FetchError::Transient(message) => write!(f, "transient: {message}"),
FetchError::Io(e) => write!(f, "io error: {e}"),
}
}
}
@@ -178,6 +221,7 @@ impl std::error::Error for FetchError {
FetchError::NotFound | FetchError::Blocked | FetchError::Sensitive => None,
FetchError::TooLarge => None,
FetchError::Transient(_) => None,
FetchError::Io(e) => Some(e),
}
}
}
@@ -255,30 +299,53 @@ pub(crate) fn log_once_ffmpeg_missing() {
/// Fetches a post from its URL. Returns `Ok(None)` when no site pattern
/// matches (unsupported links are silently ignored by the bot).
///
/// Transient network failures are retried: 3 total attempts with 1s then 2s
/// delays. Retried classes: bare HTTP errors, [`FetchError::Transient`]
/// (429/5xx from any site), and pixiv errors (its network failures arrive
/// wrapped as `PixivError`). Non-retried: Json/NotFound/Blocked/Sensitive.
/// Transient failures are retried: 3 total attempts with 1s then 2s delays.
/// Retried classes: bare HTTP errors, [`FetchError::Transient`] (429/5xx
/// from any site), pixiv network errors, and pixiv HTTP statuses that are
/// actually transient (429 / 5xx). Permanent classes are returned
/// immediately: Json, NotFound, Blocked, Sensitive, pixiv 4xx statuses
/// (bad/expired token, forbidden, not found) and pixiv API/auth errors.
/// Whether [`fetch`] should retry `err` (3 total attempts, 1s then 2s
/// backoff). Permanent classes — 4xx statuses, invalid tokens, unparseable
/// bodies, not-found/blocked/sensitive — are returned immediately; retrying
/// them only wastes attempts against the source site.
fn fetch_error_is_retryable(err: &FetchError) -> bool {
match err {
FetchError::Http(_) | FetchError::Transient(_) => true,
FetchError::Pixiv(e) => match e {
PixivError::Http(_) => true,
PixivError::Status(code) if *code == 429 || *code >= 500 => true,
// 4xx, invalid token, unparseable body: retrying cannot help.
PixivError::Status(_)
| PixivError::Api(_)
| PixivError::Json(_)
| PixivError::NoAuth => false,
},
_ => false,
}
}
pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
for attempt in 0..3u32 {
match fetch_once(url).await {
Ok(Some(fetched)) => {
log::info!(
"fetched {url}: site {} returned {} media",
// Per-request detail: debug only, keyed by the post id.
log::debug!(
"fetched [key={}]: site {} returned {} media",
cache_key(url).unwrap_or_else(|| "?".into()),
fetched.site_name(),
fetched.media.len()
);
return Ok(Some(fetched));
}
Ok(None) => return Ok(None),
Err(e @ (FetchError::Http(_) | FetchError::Transient(_) | FetchError::Pixiv(_))) => {
if attempt < 2 {
Err(err) => {
if fetch_error_is_retryable(&err) && attempt < 2 {
tokio::time::sleep(Duration::from_secs(1 << attempt)).await;
} else {
return Err(e);
return Err(err);
}
}
Err(other) => return Err(other),
}
}
unreachable!("retry loop always returns")
@@ -345,6 +412,41 @@ pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
download_media_limited(url, u64::MAX).await
}
/// Streams a download to `out`, aborting with [`FetchError::TooLarge`] the
/// moment the body crosses `max_bytes` (or when a declared Content-Length
/// already exceeds it). Unlike [`download_media_limited`] the body is never
/// buffered in memory — used for large files (e.g. the pixiv ugoira frame
/// zip, which can be hundreds of MB) that would otherwise spike RAM.
/// Returns the number of bytes written.
pub async fn download_media_to_file(
url: &str,
max_bytes: u64,
out: &mut std::fs::File,
) -> Result<u64, FetchError> {
use std::io::Write;
let mut request = CLIENT.get(url);
let lower = url.to_ascii_lowercase();
if lower.contains("pximg.net") {
request = request.header("Referer", "https://www.pixiv.net/");
}
let response = request.send().await?.error_for_status()?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let mut response = response;
let mut total: u64 = 0;
while let Some(chunk) = response.chunk().await? {
total += chunk.len() as u64;
if total > max_bytes {
return Err(FetchError::TooLarge);
}
out.write_all(&chunk).map_err(FetchError::Io)?;
}
Ok(total)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -374,6 +476,51 @@ mod tests {
assert_eq!(cache_key("https://example.com/not-a-post"), None);
}
#[test]
fn fetch_error_retryability_classification() {
// Transient: network errors, explicit transient, pixiv 429/5xx.
assert!(fetch_error_is_retryable(&FetchError::Transient(
"429".into()
)));
assert!(fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Status(429)
)));
assert!(fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Status(500)
)));
assert!(fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Status(503)
)));
// Permanent: pixiv 4xx (bad/expired token, forbidden, not found),
// api/auth errors, unparseable bodies, not-found/blocked/sensitive.
assert!(!fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Status(400)
)));
assert!(!fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Status(401)
)));
assert!(!fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Status(403)
)));
assert!(!fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Status(404)
)));
assert!(!fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Api("invalid_grant".into())
)));
assert!(!fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::NoAuth
)));
let json_err = serde_json::from_str::<serde_json::Value>("x").unwrap_err();
assert!(!fetch_error_is_retryable(&FetchError::Pixiv(
PixivError::Json(json_err)
)));
assert!(!fetch_error_is_retryable(&FetchError::NotFound));
assert!(!fetch_error_is_retryable(&FetchError::Blocked));
assert!(!fetch_error_is_retryable(&FetchError::Sensitive));
assert!(!fetch_error_is_retryable(&FetchError::TooLarge));
}
#[test]
fn caption_from_fields_substitutes_and_escapes() {
// The format string is escaped, the field values are substituted
@@ -398,6 +545,45 @@ mod tests {
);
}
#[test]
fn truncate_caption_keeps_short_text() {
assert_eq!(truncate_caption("short"), "short");
// Exactly at the limit: untouched.
let exact = "x".repeat(MAX_CAPTION_CHARS);
assert_eq!(truncate_caption(&exact), exact);
}
#[test]
fn truncate_caption_cuts_long_text_with_ellipsis() {
let long = "x".repeat(MAX_CAPTION_CHARS + 100);
let out = truncate_caption(&long);
assert!(
out.chars().count() <= MAX_CAPTION_CHARS,
"len {}",
out.chars().count()
);
assert!(out.ends_with('…'));
}
#[test]
fn truncate_caption_does_not_split_an_html_entity() {
// An entity crossing the cut must not be left half-open (&amp without ;).
let mut long = "a".repeat(MAX_CAPTION_CHARS - 4);
long.push_str("&amp;bbbb");
let out = truncate_caption(&long);
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
assert!(!out.contains("&amp"), "half entity left: {out:?}");
assert!(!out.ends_with('&'));
}
#[test]
fn truncate_caption_handles_multibyte_boundary() {
// Multi-byte chars near the cut must not panic (char-boundary cut).
let long = "".repeat(MAX_CAPTION_CHARS + 10);
let out = truncate_caption(&long);
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
}
#[tokio::test]
async fn unsupported_url_returns_none() {
let result = fetch("https://example.com/some/article").await;
@@ -414,7 +600,13 @@ mod tests {
async fn download_media_pixiv_original_with_referer() {
// Proves the Referer header is attached for i.pximg.net: a header-less
// GET to a pixiv original URL is rejected with 403.
if std::env::var("PIXIV_REFRESH_TOKEN").is_err() {
// Empty-string check too: an unset CI secret arrives as "" (GitHub
// Actions), which would otherwise run the test tokenless and fail.
if std::env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|s| !s.is_empty())
.is_none()
{
eprintln!("skipping: no PIXIV_REFRESH_TOKEN");
return;
}
+38 -8
View File
@@ -9,7 +9,7 @@ use crate::media::Media;
use crate::site::FetchError;
use std::env;
use std::fmt;
use std::io::{Cursor, Read};
use std::io::Read;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, SystemTime};
@@ -29,6 +29,10 @@ pub enum PixivError {
NoAuth,
Http(reqwest::Error),
Json(serde_json::Error),
/// Non-2xx HTTP status from the app API. The code lets [`crate::site::fetch`]
/// retry only transient classes (429 / 5xx) instead of burning attempts on
/// permanent 4xx (bad token, forbidden, not found).
Status(u16),
Api(String),
}
@@ -38,6 +42,7 @@ impl fmt::Display for PixivError {
PixivError::NoAuth => write!(f, "pixiv: no authentication"),
PixivError::Http(e) => write!(f, "pixiv http error: {e}"),
PixivError::Json(e) => write!(f, "pixiv json error: {e}"),
PixivError::Status(code) => write!(f, "pixiv status {code}"),
PixivError::Api(message) => write!(f, "pixiv api error: {message}"),
}
}
@@ -137,7 +142,7 @@ impl PixivAPI {
.send()
.await?;
if !response.status().is_success() {
return Err(PixivError::Api(format!("status {}", response.status())));
return Err(PixivError::Status(response.status().as_u16()));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
if json.get("error").is_some() {
@@ -190,7 +195,7 @@ impl PixivAPI {
.send()
.await?;
if !response.status().is_success() {
return Err(PixivError::Api(format!("status {}", response.status())));
return Err(PixivError::Status(response.status().as_u16()));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
if json.get("error").is_some() {
@@ -228,7 +233,14 @@ impl PixivAPI {
let Some(zip_url) = zip_url else {
return Ok(None);
};
let zip_bytes = crate::site::download_media_limited(&zip_url, 512 * 1024 * 1024)
// Stream the frame zip to a temp file instead of buffering it in
// memory: ugoira zips can be hundreds of MB, and the old
// download_media_limited path spiked RAM up to the size cap.
let mut zip_file = tempfile::Builder::new()
.suffix(".zip")
.tempfile()
.map_err(|e| PixivError::Api(format!("temp zip failed: {e}")))?;
crate::site::download_media_to_file(&zip_url, 512 * 1024 * 1024, zip_file.as_file_mut())
.await
.map_err(|e| match e {
FetchError::Http(e) => PixivError::Http(e),
@@ -241,9 +253,12 @@ impl PixivAPI {
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
// Extract frames to canonical zero-padded names; pixiv ugoira
// frames are uniformly jpg or png per artwork.
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
.map_err(|e| format!("unzip: {e}"))?;
// frames are uniformly jpg or png per artwork. The zip is read
// from disk; `zip_file` stays alive for the whole extraction.
let mut archive = zip::ZipArchive::new(
std::fs::File::open(zip_file.path()).map_err(|e| e.to_string())?,
)
.map_err(|e| format!("unzip: {e}"))?;
if archive.is_empty() {
return Err("empty frame zip".to_string());
}
@@ -392,16 +407,31 @@ mod tests {
use super::*;
use dotenv::dotenv;
/// Skips when `PIXIV_REFRESH_TOKEN` is absent or empty (CI without the
/// secret must stay green; GitHub Actions exposes an unset secret as an
/// empty string, so `is_err()` alone is not enough).
fn require_pixiv_token() -> bool {
std::env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|s| !s.is_empty())
.is_some()
}
#[tokio::test]
async fn test_fetch() {
dotenv().ok();
if !require_pixiv_token() {
eprintln!("skipping: no PIXIV_REFRESH_TOKEN");
return;
}
let result = fetch(126839080).await;
assert!(result.is_ok());
println!("{:#?}", result);
}
#[tokio::test]
async fn validate_with_bogus_token_fails() {
#[ignore = "live network: requires outbound HTTPS to oauth.secure.pixiv.net"]
async fn live_validate_with_bogus_token_fails() {
dotenv().ok();
// A bogus token must surface as Api error (invalid_grant), not panic.
let client = PixivAPI::new("bogus_token_for_testing".to_string());
+90 -16
View File
@@ -29,13 +29,19 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
if super::auth::enabled() {
match super::auth::fetch(id).await {
Ok(tweet) => Ok(tweet.into()),
// The tweet is genuinely gone (deleted / suspended /
// tombstoned): report it instead of degrading to an
// empty result ("No media found"). Only unexpected
// fallback failures (network, parse) keep the NSFW
// placeholder.
Err(FetchError::NotFound) => Err(FetchError::NotFound),
Err(e) => {
log::warn!("twitter auth fallback failed for {id}: {e}");
Ok(empty_fetched(url))
}
}
} else {
log::info!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
log::debug!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
Ok(empty_fetched(url))
}
}
@@ -59,8 +65,8 @@ fn empty_fetched(url: &str) -> Fetched {
}
}
/// Fetches a tweet from the syndication endpoint. Deleted/blocked tweets
/// surface as `FetchError::NotFound`.
/// Fetches a tweet from the syndication endpoint. Deleted/blocked/tombstoned
/// tweets surface as `FetchError::NotFound`.
pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
let id_num = id.parse::<u64>().map_err(|_| FetchError::NotFound)?;
let response = crate::site::CLIENT
@@ -79,23 +85,33 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
};
}
let text = response.text().await?;
// Deleted tweets answer with {"errors": [...]} instead of a tweet.
if serde_json::from_str::<serde_json::Value>(&text)
.map(|v| v.get("errors").is_some())
.unwrap_or(false)
{
// Deleted/blocked tweets answer with an `errors` array or a
// TweetTombstone (HTTP 200, no `id_str`); NSFW withholding is an empty
// `{}`. Both classes are permanent — classify before parsing the tweet.
parse_syndication_body(&text)?;
Tweet::from_syndication_json(&text).map_err(FetchError::Json)
}
/// Parses and classifies a syndication response body. `Ok` means the body is
/// a real tweet payload; `Err` carries the permanent error class:
/// - `NotFound`: an `errors` array (deleted/blocked) or a `TweetTombstone`
/// (deleted by the author / suspended — HTTP 200, no `errors`, no `id_str`).
/// - `Sensitive`: an empty `{}` (NSFW / age-restricted withholding).
/// - `Json`: an unparseable body.
///
/// The tombstone shape must NOT fall through to `Sensitive`: the bot would
/// otherwise answer "No media found" for a deleted tweet instead of failing.
fn parse_syndication_body(text: &str) -> Result<serde_json::Value, FetchError> {
let body: serde_json::Value = serde_json::from_str(text)?;
let tombstoned = body.get("tombstone").is_some()
|| body.get("__typename").and_then(|t| t.as_str()) == Some("TweetTombstone");
if body.get("errors").is_some() || tombstoned {
return Err(FetchError::NotFound);
}
// NSFW / age-restricted tweets exist but are served as an empty `{}` —
// they surface as FetchError::Sensitive so the caller can retry as a
// logged-in user.
if serde_json::from_str::<serde_json::Value>(&text)
.map(|v| v.get("id_str").is_none())
.unwrap_or(false)
{
if body.get("id_str").is_none() {
return Err(FetchError::Sensitive);
}
Tweet::from_syndication_json(&text).map_err(FetchError::Json)
Ok(body)
}
/// The syndication token: JS `((id / 1e15) * PI).toString(36)` (the
@@ -572,19 +588,64 @@ mod tests {
assert!(token.starts_with("236.v"), "got {token}");
}
#[test]
fn syndication_tombstone_maps_to_not_found() {
// Deleted tweets answer HTTP 200 with a TweetTombstone (no `errors`,
// no `id_str`); it must not fall through to Sensitive, which would
// make the bot reply "No media found" for a deleted tweet.
let raw = serde_json::json!({
"__typename": "TweetTombstone",
"tombstone": {
"text": { "rtl": false, "text": "This Post was deleted by the Post author. Learn more" }
}
});
assert!(matches!(
parse_syndication_body(&raw.to_string()),
Err(FetchError::NotFound)
));
}
#[test]
fn syndication_errors_maps_to_not_found() {
// The classic gone shape: {"errors": [...]}.
let raw = serde_json::json!({ "errors": [{ "message": "Couldn't find Tweet" }] });
assert!(matches!(
parse_syndication_body(&raw.to_string()),
Err(FetchError::NotFound)
));
}
#[test]
fn syndication_empty_object_maps_to_sensitive() {
// NSFW / age-restricted withholding: an empty `{}`.
assert!(matches!(
parse_syndication_body("{}"),
Err(FetchError::Sensitive)
));
}
#[test]
fn syndication_tweet_body_passes() {
let raw = fixture(serde_json::json!([]));
assert!(parse_syndication_body(&raw.to_string()).is_ok());
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_with_photos() {
let fetched = fetch("861627479294746624").await.unwrap();
assert_eq!(fetched.media.len(), 4);
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_text_only() {
let fetched = fetch("1992471125734142256").await.unwrap();
assert!(fetched.media.is_empty());
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_deleted_tweet_is_not_found() {
// Deleted tweet: the syndication endpoint answers with errors.
let result = fetch("0").await;
@@ -593,4 +654,17 @@ mod tests {
"got {result:?}"
);
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_tombstone_deleted_tweet_is_not_found() {
// Regression: a real deleted tweet answering with a TweetTombstone
// (HTTP 200, no errors/id_str) used to surface as Sensitive and
// degrade to an empty result ("No media found").
let result = fetch("2085948045967986859").await;
assert!(
matches!(result, Err(FetchError::NotFound)),
"got {result:?}"
);
}
}
+3 -2
View File
@@ -1,10 +1,10 @@
[package]
name = "xmedia-bot"
version = "1.1.0"
version = "1.2.1"
edition = "2024"
[dependencies]
teloxide = { version = "0.17", features = ["webhooks-axum", "macros"] }
teloxide = { version = "0.17", default-features = false, features = ["webhooks-axum", "macros", "rustls", "ctrlc_handler"] }
tokio = { version = "1.40", features = ["rt-multi-thread", "macros"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
@@ -17,6 +17,7 @@ rusqlite = { version = "0.32", features = ["bundled"] }
rand = "0.8"
tempfile = "3"
parking_lot = "0.12"
bytes = "1"
png = "0.18"
zune-jpeg = "0.5"
fast_image_resize = "6"
+53 -21
View File
@@ -23,32 +23,64 @@ pub struct Config {
impl Config {
pub fn load() -> Config {
let admin_ids = env::var("BOT_ADMIN")
.ok()
.map(|s| {
s.split(',')
.filter_map(|part| part.trim().parse::<i64>().ok())
// Fail-fast helpers: a misspelled value must not silently fall back
// to a default and run with different behavior than the operator
// intended — log a loud warning naming the variable instead.
fn parse_u64(name: &str, default: u64) -> u64 {
match env::var(name) {
Ok(v) => v.parse::<u64>().unwrap_or_else(|_| {
log::warn!("invalid {name}={v:?}; using default {default}");
default
}),
Err(_) => default,
}
}
let admin_ids = match env::var("BOT_ADMIN") {
Ok(s) => {
let (ids, bad): (Vec<_>, Vec<_>) = s
.split(',')
.map(str::trim)
.filter(|part| !part.is_empty())
.partition(|part| part.parse::<i64>().is_ok());
if !bad.is_empty() {
log::warn!("BOT_ADMIN: ignoring non-numeric ids: {bad:?}");
}
ids.into_iter()
.filter_map(|p| p.parse::<i64>().ok())
.collect()
})
.unwrap_or_default();
}
Err(_) => Vec::new(),
};
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(86400));
let link_cache_ttl = env::var("LINK_CACHE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(7 * 24 * 3600));
let edit_message_ttl =
Duration::from_secs(parse_u64("EDIT_MESSAGE_TTL_SECONDS", 24 * 3600));
let link_cache_ttl =
Duration::from_secs(parse_u64("LINK_CACHE_TTL_SECONDS", 7 * 24 * 3600));
let webhook_enabled = env::var("WEBHOOK")
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
// The webhook settings are consumed by `.expect()` in main when
// WEBHOOK=true, so an unparseable value fails fast at startup with a
// clear message; still log here for the WEBHOOK=false case.
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| {
s.parse::<url::Url>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_URL={s:?}");
None
})
});
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| {
s.parse::<IpAddr>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_LISTEN={s:?}");
None
})
});
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| {
s.parse::<u16>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_PORT={s:?}");
None
})
});
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
// value that would otherwise come from `.env`).
let webhook_cert = env::var("WEBHOOK_CERT").ok().filter(|s| !s.is_empty());
+96 -23
View File
@@ -2,16 +2,106 @@
//! (`tasks` in queue.rs, `chat_state` in state.rs, `link_cache` in
//! link_cache.rs).
//!
//! Every operation opens its own short-lived connection with a busy timeout:
//! handler tasks enqueue while workers lease/update rows concurrently, and
//! without the timeout a concurrent write fails immediately with SQLITE_BUSY
//! and the operation is lost. All I/O runs inside `spawn_blocking` via
//! [`with_conn`] — rusqlite connections are not Send-friendly to hold across
//! an await point, and blocking the async executor stalls every handler.
//! All I/O runs inside `spawn_blocking` via [`DbPool::with_conn`] — rusqlite
//! connections are not Send-friendly to hold across an await point, and
//! blocking the async executor stalls every handler. Connections are reused
//! through a small per-store pool instead of opening a fresh connection per
//! operation: WAL lets readers run alongside writer leases, and the pool's
//! semaphore bounds how many DB operations run concurrently, giving natural
//! backpressure on hot paths (every message / URL / callback touches
//! chat_state or the link cache).
use parking_lot::Mutex;
use rusqlite::Connection;
use std::sync::Arc;
use std::time::Duration;
/// Upper bound on pooled (reused) connections and on concurrent DB
/// operations per store. Small on purpose: the queue's `BEGIN IMMEDIATE`
/// leases serialize writes anyway, and WAL readers rarely need more.
const POOL_SIZE: usize = 4;
/// A tiny connection pool for one SQLite file. Connections are checked out
/// on a blocking thread and returned afterwards; `acquire` opens a new
/// connection only when the idle list is empty, so the steady-state cost of
/// an operation is a list pop instead of a fresh open (+ busy timeout + WAL
/// pragma). The semaphore caps the number of concurrent operations, so a
/// burst of handlers queues up instead of opening unbounded connections.
pub struct DbPool {
// Arc so [`DbPool::with_conn`] can hand an owned handle to
// `spawn_blocking` without borrowing across the await point.
inner: Arc<PoolInner>,
}
struct PoolInner {
path: String,
permits: tokio::sync::Semaphore,
idle: Mutex<Vec<Connection>>,
}
impl DbPool {
pub fn new(path: &str) -> Self {
DbPool {
inner: Arc::new(PoolInner {
path: path.to_string(),
permits: tokio::sync::Semaphore::new(POOL_SIZE),
idle: Mutex::new(Vec::new()),
}),
}
}
/// Runs `f` against a pooled connection on a blocking thread, returning
/// the closure's result. Owns the semaphore + `spawn_blocking` +
/// `expect` ceremony shared by every table access; the caller maps
/// errors to its own log line.
pub async fn with_conn<T, F>(&self, f: F) -> rusqlite::Result<T>
where
T: Send + 'static,
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
{
let _permit = self
.inner
.permits
.acquire()
.await
.expect("db pool semaphore closed");
let inner = Arc::clone(&self.inner);
tokio::task::spawn_blocking(move || {
let mut conn = inner.acquire()?;
let result = f(&mut conn);
inner.release(conn);
result
})
.await
.expect("db worker panicked")
}
/// The database file this pool serves (used by tests that need a raw
/// connection, e.g. to seed rows directly).
#[cfg(test)]
pub fn path(&self) -> &str {
&self.inner.path
}
}
impl PoolInner {
/// Reuses an idle connection or opens a fresh one.
fn acquire(&self) -> rusqlite::Result<Connection> {
if let Some(conn) = self.idle.lock().pop() {
return Ok(conn);
}
open_db(&self.path)
}
/// Returns a connection to the pool (dropped when the pool is full).
fn release(&self, conn: Connection) {
let mut idle = self.idle.lock();
if idle.len() < POOL_SIZE {
idle.push(conn);
}
}
}
/// Opens the shared DB with a busy timeout.
pub fn open_db(path: &str) -> rusqlite::Result<Connection> {
let conn = Connection::open(path)?;
@@ -31,20 +121,3 @@ pub fn now_f64() -> f64 {
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
}
/// Runs `f` against a fresh connection on a blocking thread, returning the
/// closure's result. Owns the `spawn_blocking` + `expect` ceremony shared by
/// every table access; the caller maps errors to its own log line.
pub async fn with_conn<T, F>(path: &str, f: F) -> rusqlite::Result<T>
where
T: Send + 'static,
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
{
let path = path.to_string();
tokio::task::spawn_blocking(move || {
let mut conn = open_db(&path)?;
f(&mut conn)
})
.await
.expect("db worker panicked")
}
+155 -24
View File
@@ -26,6 +26,10 @@ static URL_JOBS: LazyLock<parking_lot::Mutex<Option<tokio::sync::mpsc::Sender<Ur
/// Set by main's shutdown sequence; workers stop pulling new jobs.
static URL_STOP: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
/// JoinHandles of the URL workers, awaited by [`stop_url_workers`].
static URL_WORKER_HANDLES: LazyLock<parking_lot::Mutex<Option<Vec<tokio::task::JoinHandle<()>>>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
/// Worker count draining URL jobs; keeps the old 8-permit concurrency cap
/// while bounding how many jobs can be queued at all.
const URL_WORKERS: usize = 8;
@@ -40,9 +44,10 @@ pub async fn start_url_workers() {
let (tx, rx) = tokio::sync::mpsc::channel::<UrlJob>(256);
*URL_JOBS.lock() = Some(tx);
let rx = std::sync::Arc::new(tokio::sync::Mutex::new(rx));
let mut handles = Vec::with_capacity(URL_WORKERS);
for _ in 0..URL_WORKERS {
let rx = std::sync::Arc::clone(&rx);
tokio::spawn(async move {
handles.push(tokio::spawn(async move {
while !URL_STOP.load(std::sync::atomic::Ordering::Relaxed) {
let job = rx.lock().await.recv().await;
match job {
@@ -50,13 +55,28 @@ pub async fn start_url_workers() {
None => break,
}
}
});
}));
}
*URL_WORKER_HANDLES.lock() = Some(handles);
}
/// Stops URL workers (drains up to the 256 queued jobs, then exits).
pub fn stop_url_workers() {
/// Stops the URL workers: sets the stop flag, drops the job channel (so
/// workers blocked in \`recv()\` wake with \`None\` and exit) and awaits the
/// worker tasks. Each worker finishes its in-flight job first; jobs still
/// queued in the channel are abandoned (the old implementation neither
/// drained them nor woke blocked workers — it only set a flag checked
/// between jobs).
pub async fn stop_url_workers() {
URL_STOP.store(true, std::sync::atomic::Ordering::Relaxed);
// Dropping the sender makes every worker's recv() return None.
*URL_JOBS.lock() = None;
// Take the handles first so the lock guard drops before the awaits.
let handles = URL_WORKER_HANDLES.lock().take();
if let Some(handles) = handles {
for handle in handles {
let _ = handle.await;
}
}
}
pub static CHAT_STORE: LazyLock<ChatStore> =
@@ -111,6 +131,14 @@ where
.await
}
/// Log prefix tying the whole lifecycle of one link (fetch → send → cache →
/// forward) together: the normalized cache key (`twitter:123…`, `pixiv:123`,
/// `bsky:handle/rkey`) instead of the raw URL, so logs stay short and do not
/// echo full user-submitted URLs at info level.
pub fn log_key(url: &str) -> String {
x_media::site::cache_key(url).unwrap_or_else(|| "<unsupported>".to_string())
}
/// Extracts URL and text-link entities (text + caption), deduped in order.
pub fn extract_urls(message: &Message) -> Vec<String> {
let mut urls = Vec::new();
@@ -514,14 +542,23 @@ async fn dispatch_send(bot: Bot, message: &Message, task: &Task, url: &str) {
};
match result {
Ok(message_ids) => {
log::info!("sent {} message(s) for {url}", message_ids.len());
log::info!(
"sent {} message(s) for [key={}]",
message_ids.len(),
log_key(url)
);
send::post_send_actions(&bot, task, message_ids).await;
// The task settled: drop any keep-alive temp media.
send::release_keep_alive(task);
}
Err(send::SendError::Retryable {
delay_seconds,
task,
}) => {
log::info!("send for {url} failed, queued for retry in {delay_seconds:.1}s");
log::info!(
"send for [key={}] failed, queued for retry in {delay_seconds:.1}s",
log_key(url)
);
enqueue_retry(task, delay_seconds).await;
let _ = reply(bot, message.clone(), "Send failed. Task queued for retry.").await;
}
@@ -530,6 +567,7 @@ async fn dispatch_send(bot: Bot, message: &Message, task: &Task, url: &str) {
task,
}) => {
send::invalidate_cache(&task).await;
send::release_keep_alive(&task);
log::error!("send for {url} failed permanently: {err_message}");
let _ = reply(bot, message.clone(), format!("Send failed: {err_message}")).await;
}
@@ -566,7 +604,9 @@ fn build_send_task(
chat_id,
reply_to_message_id: message.id.0 as i64,
caption,
media_batches: send::chunk_media_items(items),
// Photos first so a mixed photo+video group starts with a photo
// (Telegram's sendMediaGroup rule); order within each kind is kept.
media_batches: send::chunk_media_items(send::photos_first(items)),
batch_index: 0,
sent_message_ids: vec![],
source_url,
@@ -594,7 +634,7 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
if let Some(key) = x_media::site::cache_key(url)
&& let Some(cached) = LINK_CACHE.get(&key, CONFIG.link_cache_ttl).await
{
log::info!("link cache hit for {url}");
log::debug!("link cache hit for {key}");
let chat_data = CHAT_STORE.get(chat_id).await;
let site = key.split(':').next().unwrap_or("unknown");
let format = chat_data
@@ -603,7 +643,7 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
.cloned()
.unwrap_or_default();
let caption = if format.is_empty() {
cached.caption.clone()
x_media::site::truncate_caption(&cached.caption)
} else {
x_media::site::caption_from_fields(
&format,
@@ -651,11 +691,11 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
return;
}
log::info!("fetching {url}");
log::debug!("fetching {url} [key={}]", log_key(url));
match x_media::site::fetch(url).await {
// Unsupported links are ignored silently (Python parity).
Ok(None) => {
log::info!("no site pattern matches {url}; ignoring");
log::debug!("no site pattern matches {url}; ignoring");
}
// Retries exhausted: notify the user (Rust-only requirement 3).
Err(e) => {
@@ -667,7 +707,7 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
)
.await;
}
Ok(Some(fetched)) => {
Ok(Some(mut fetched)) => {
if fetched.media.is_empty() {
let _ = reply(
bot,
@@ -712,6 +752,13 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
items,
cache_data,
);
// Hand the keep-alive temp dir (ugoira / bsky remux MP4) to the
// retry registry: a queued retry runs after this function returns
// and the fetch's own TempDir is dropped, so without this the
// local file would be gone by the time the retry sends it.
if let Some(dir) = fetched.take_keep_alive() {
send::KEEP_ALIVE.lock().push(dir);
}
dispatch_send(bot, message, &task, url).await;
}
}
@@ -731,7 +778,8 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
&t[..end]
})
.unwrap_or("<no text>");
log::info!(
// Per-request detail: debug only (message text is user data).
log::debug!(
"message from {sender} in {} (private={is_private}): {text_preview}",
message.chat.id
);
@@ -742,14 +790,16 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
if let Some(text) = message.text()
&& let Ok(command) = Command::parse(text, "")
{
log::info!("command from {}: {text_preview}", message.chat.id);
log::debug!("command from {}: {text_preview}", message.chat.id);
execute_command(&bot, &message, command).await?;
return respond(());
}
if is_private {
let urls = extract_urls(&message);
if !urls.is_empty() {
log::info!("extracted {} URL(s): {urls:?}", urls.len());
// Debug only, and echo the normalized keys instead of the raw URLs.
let keys: Vec<String> = urls.iter().map(|u| log_key(u)).collect();
log::debug!("extracted {} URL(s): {keys:?}", urls.len());
}
for url in urls {
// Clone out of the lock: the parking_lot guard is !Send and must
@@ -764,20 +814,96 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
respond(())
}
/// Debounce window for inline queries: Telegram fires an inline query on
/// every keystroke, and each prefix of a pasted URL (e.g. `.../status/12`,
/// `.../status/123`, ...) already matches the site patterns. Without a
/// debounce every keystroke triggers a fetch (3 attempts!) of a half-typed
/// post id. Only answer once the query has been stable for this long.
const INLINE_DEBOUNCE: std::time::Duration = std::time::Duration::from_millis(800);
/// Last seen inline query and whether it was already answered. Guards the
/// debounce timer: a repeat of an answered query is served by Telegram's
/// inline cache (see `cache_time`), not by another fetch.
struct InlineDebounceState {
query: String,
answered: bool,
}
static INLINE_DEBOUNCE_STATE: LazyLock<parking_lot::Mutex<Option<InlineDebounceState>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), RequestError> {
if query.query.is_empty() {
return respond(());
}
// Telegram fires an inline query on every keystroke; only run a fetch
// (3 attempts!) for something that is actually a supported post URL, so
// typing does not hammer the source sites.
// Only run a fetch for something that is actually a supported post URL.
if x_media::site::cache_key(&query.query).is_none() {
return respond(());
}
log::info!("inline query: {}", query.query);
// Debounce: record the query and answer only after it has been stable for
// INLINE_DEBOUNCE (the timer below). An already-answered repeat of the
// same query is left to Telegram's inline cache instead of re-fetching.
{
let mut state = INLINE_DEBOUNCE_STATE.lock();
if let Some(prev) = state.as_ref()
&& prev.query == query.query
&& prev.answered
{
return respond(());
}
*state = Some(InlineDebounceState {
query: query.query.clone(),
answered: false,
});
}
let query_text = query.query.clone();
tokio::spawn(async move {
tokio::time::sleep(INLINE_DEBOUNCE).await;
// Only the last query of a typing burst survives: earlier timers see
// the query changed and give up without answering.
{
let mut state = INLINE_DEBOUNCE_STATE.lock();
let Some(state) = state.as_mut() else {
return;
};
if state.query != query_text || state.answered {
return;
}
// Claim the answer so a repeat of the same query cannot start a
// second fetch; reset below when no answer was produced.
state.answered = true;
}
match answer_inline_query(bot, query).await {
Ok(true) => {}
// No results produced (or nothing to answer): let a repeat of the
// same query retry the fetch.
Ok(false) | Err(_) => {
let mut state = INLINE_DEBOUNCE_STATE.lock();
if let Some(state) = state.as_mut()
&& state.query == query_text
{
state.answered = false;
}
}
}
});
respond(())
}
/// Fetches the post behind an inline query and answers it. The caller has
/// already applied the debounce. Returns `true` when an answer was sent.
async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, RequestError> {
log::debug!(
"inline query: {} [key={}]",
query.query,
log_key(&query.query)
);
match x_media::site::fetch(&query.query).await {
Ok(Some(fetched)) => {
let mut results: Vec<InlineQueryResult> = Vec::new();
// Inline results have the same 1024-char caption limit as regular
// messages; truncate once here for all items.
let caption = x_media::site::truncate_caption(&fetched.caption);
for (i, media) in fetched.media.iter().enumerate() {
let id = format!("{i}");
let Some(url) = url::Url::parse(media.url()).ok() else {
@@ -787,7 +913,7 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
.thumbnail_url()
.and_then(|t| url::Url::parse(t).ok())
.unwrap_or_else(|| url.clone());
let caption = fetched.caption.clone();
let caption = caption.clone();
let result = match media {
Media::Illustration { .. } => {
// Inline photo results have their own (smaller) size
@@ -822,13 +948,18 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
results.push(result);
}
if !results.is_empty() {
bot.answer_inline_query(query.id, results).await?;
// Explicit cache window: repeats of the same query within 5
// minutes are served by Telegram without hitting the bot.
bot.answer_inline_query(query.id, results)
.cache_time(300)
.await?;
return Ok(true);
}
}
Ok(None) => {}
Err(e) => log::error!("inline fetch {}: {e}", query.query),
}
respond(())
Ok(false)
}
pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<(), RequestError> {
@@ -843,7 +974,7 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let chat_data = CHAT_STORE.get(chat_id).await;
let edit = chat_data.edit_message.get(&prompt_message_id).cloned();
let Some(edit) = edit else {
log::info!(
log::debug!(
"callback from {}: no edit record for prompt {prompt_message_id}",
chat_id
);
@@ -919,7 +1050,7 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
}
}
None => {
log::info!("forward callback without a forward channel set");
log::debug!("forward callback without a forward channel set");
bot.answer_callback_query(callback_query_id)
.text("No forward channel set.")
.await?;
+52 -42
View File
@@ -47,7 +47,7 @@ pub struct CachedPost {
/// SQLite-backed cache sharing `data/task_queue.db` with the queue and chat
/// state (same `open_db` pattern: busy timeout, `spawn_blocking` I/O).
pub struct LinkCache {
db_path: String,
pool: crate::db::DbPool,
}
impl LinkCache {
@@ -61,7 +61,7 @@ impl LinkCache {
log::error!("failed to initialize link cache schema: {e}");
}
Self {
db_path: db_path.to_string(),
pool: crate::db::DbPool::new(db_path),
}
}
@@ -70,24 +70,26 @@ impl LinkCache {
pub async fn get(&self, key: &str, ttl: Duration) -> Option<CachedPost> {
let key = key.to_string();
let ttl = ttl.as_secs_f64();
let result = crate::db::with_conn(&self.db_path, move |conn| {
let mut stmt =
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
let mut rows = stmt.query(params![key])?;
let Some(row) = rows.next()? else {
return Ok(None);
};
let payload: String = row.get(0)?;
let created_at: f64 = row.get(1)?;
if now_f64() - created_at > ttl {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
return Ok(None);
}
Ok(Some(serde_json::from_str::<CachedPost>(&payload).map_err(
|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)),
)?))
})
.await;
let result = self
.pool
.with_conn(move |conn| {
let mut stmt =
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
let mut rows = stmt.query(params![key])?;
let Some(row) = rows.next()? else {
return Ok(None);
};
let payload: String = row.get(0)?;
let created_at: f64 = row.get(1)?;
if now_f64() - created_at > ttl {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
return Ok(None);
}
Ok(Some(serde_json::from_str::<CachedPost>(&payload).map_err(
|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)),
)?))
})
.await;
match result {
Ok(v) => v,
Err(e) => {
@@ -100,14 +102,16 @@ impl LinkCache {
pub async fn put(&self, key: &str, post: &CachedPost) {
let key = key.to_string();
let payload = serde_json::to_string(post).expect("cached post serializes");
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"INSERT OR REPLACE INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
params![key, payload, now_f64()],
)?;
Ok(())
})
.await;
Ok(())
})
.await;
if let Err(e) = result {
log::error!("link cache write failed: {e}");
}
@@ -116,11 +120,13 @@ impl LinkCache {
/// Drops an entry (e.g. a cached file id that turned out invalid).
pub async fn remove(&self, key: &str) {
let key = key.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Ok(())
})
.await;
let result = self
.pool
.with_conn(move |conn| {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Ok(())
})
.await;
if let Err(e) = result {
log::error!("link cache delete failed: {e}");
}
@@ -129,13 +135,15 @@ impl LinkCache {
/// Removes expired entries; returns how many were deleted.
pub async fn prune(&self, ttl: Duration) -> usize {
let cutoff = now_f64() - ttl.as_secs_f64();
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"DELETE FROM link_cache WHERE created_at < ?1",
params![cutoff],
)
})
.await;
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"DELETE FROM link_cache WHERE created_at < ?1",
params![cutoff],
)
})
.await;
match result {
Ok(n) => n,
Err(e) => {
@@ -149,11 +157,13 @@ impl LinkCache {
/// `key` is `None`. Returns how many rows were removed.
pub async fn clear(&self, key: Option<&str>) -> usize {
let key = key.map(str::to_string);
let result = crate::db::with_conn(&self.db_path, move |conn| match &key {
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
None => conn.execute("DELETE FROM link_cache", []),
})
.await;
let result = self
.pool
.with_conn(move |conn| match &key {
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
None => conn.execute("DELETE FROM link_cache", []),
})
.await;
match result {
Ok(n) => n,
Err(e) => {
+19 -7
View File
@@ -178,13 +178,25 @@ async fn main() {
.await;
}
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin, drain the queue.
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin,
// drain the queue. Bounded: a worker mid-download (30 s timeout) or a
// long ugoira encode must not hold the shutdown hostage forever.
log::info!("Stopping bot");
let _ = stop_tx.send(true);
handlers::stop_url_workers();
if let Some(admin) = CONFIG.admin_ids.first() {
let _ = bot.send_message(ChatId(*admin), "Shutting down...").await;
const SHUTDOWN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
let shutdown = async {
let _ = stop_tx.send(true);
handlers::stop_url_workers().await;
if let Some(admin) = CONFIG.admin_ids.first() {
let _ = bot.send_message(ChatId(*admin), "Shutting down...").await;
}
TASK_QUEUE.stop().await;
};
if tokio::time::timeout(SHUTDOWN_TIMEOUT, shutdown)
.await
.is_err()
{
log::warn!("graceful shutdown timed out after {SHUTDOWN_TIMEOUT:?}; exiting");
} else {
log::info!("Bot stopped");
}
TASK_QUEUE.stop().await;
log::info!("Bot stopped");
}
+15 -14
View File
@@ -67,8 +67,9 @@ impl PixBuf {
}
/// Entry point: detects the format and processes the photo if needed.
pub fn prepare_photo(file: NamedTempFile) -> Result<PhotoPrep, String> {
let bytes = std::fs::read(file.path()).map_err(|e| format!("prepare read failed: {e}"))?;
/// The caller hands in the already-downloaded bytes (they are in memory from
/// the download anyway; re-reading the temp file would double the I/O).
pub fn prepare_photo(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {
prepare_png(file, bytes)
} else if bytes.starts_with(&[0xFF, 0xD8, 0xFF]) {
@@ -216,13 +217,13 @@ fn target_dims(w: u32, h: u32) -> (u32, u32) {
/// PNG branch: decode (16→8, palette→RGB; gray/GA stay), flatten RGBA to
/// RGB, Lanczos-downscale beyond the dimension cap, encode PNG — a PNG still
/// over the upload cap afterwards becomes JPEG.
fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
let (w, h, _bit_depth, color_type) = parse_png_header(&bytes).ok_or("invalid PNG header")?;
fn prepare_png(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
let (w, h, _bit_depth, color_type) = parse_png_header(bytes).ok_or("invalid PNG header")?;
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
return Ok(PhotoPrep::Upload(file));
}
log::info!(
log::debug!(
"photo {w}x{h} ({_bit_depth:?} {color_type:?}, {} bytes) needs processing",
bytes.len()
);
@@ -239,7 +240,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
png::ColorType::Indexed => png::Transformations::EXPAND,
_ => png::Transformations::STRIP_16,
};
let mut decoder = png::Decoder::new(std::io::Cursor::new(&bytes));
let mut decoder = png::Decoder::new(std::io::Cursor::new(bytes));
decoder.set_transformations(transforms);
let mut reader = decoder
.read_info()
@@ -268,7 +269,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
let (nw, nh) = target_dims(w, h);
pix = resize_pix(pix, w, h, nw, nh)?;
(w, h) = (nw, nh);
log::info!("downscaled photo to {w}x{h} (Lanczos3)");
log::debug!("downscaled photo to {w}x{h} (Lanczos3)");
}
let mut png_bytes = Vec::new();
@@ -276,7 +277,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
if png_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
return Ok(PhotoPrep::Upload(write_temp(&png_bytes, "png")?));
}
log::info!("PNG still over the upload cap after processing; transcoding to JPEG");
log::debug!("PNG still over the upload cap after processing; transcoding to JPEG");
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
return Ok(PhotoPrep::Upload(write_temp(&jpeg_bytes, "jpg")?));
@@ -286,8 +287,8 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
}
/// JPEG branch: zune-jpeg decode → Lanczos downscale → jpeg-encoder output.
fn prepare_jpeg(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(&bytes));
fn prepare_jpeg(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(bytes));
// Decodes to RGB by default. Headers first so dimensions are known before
// the (potentially huge) pixel decode.
decoder
@@ -310,7 +311,7 @@ fn prepare_jpeg(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String
let (nw, nh) = target_dims(w, h);
pix = resize_pix(pix, w, h, nw, nh)?;
(w, h) = (nw, nh);
log::info!("downscaled jpeg to {w}x{h} (Lanczos3)");
log::debug!("downscaled jpeg to {w}x{h} (Lanczos3)");
}
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
@@ -423,7 +424,7 @@ mod tests {
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
prepare_photo(file)
prepare_photo(file, &bytes)
}
#[test]
@@ -468,7 +469,7 @@ mod tests {
}
let mut file = tempfile::Builder::new().suffix(".jpg").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
match prepare_photo(file).unwrap() {
match prepare_photo(file, &bytes).unwrap() {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
assert!(out.starts_with(&[0xFF, 0xD8]), "output must stay jpeg");
@@ -516,7 +517,7 @@ mod tests {
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
match prepare_photo(file).unwrap() {
match prepare_photo(file, &bytes).unwrap() {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
assert!(out.starts_with(&[0xFF, 0xD8]), "must transcode to JPEG");
+34 -31
View File
@@ -40,7 +40,7 @@ type Handler = dyn Fn(Value) -> BoxFuture<'static, Result<(), QueueError>> + Sen
type DeadLetter = dyn Fn(Value, String) -> BoxFuture<'static, ()> + Send + Sync;
pub struct PersistentTaskQueue {
db_path: String,
pool: std::sync::Arc<crate::db::DbPool>,
notify: Arc<Notify>,
stop: Arc<AtomicBool>,
worker: Mutex<Vec<JoinHandle<()>>>,
@@ -56,7 +56,7 @@ struct LeasedRow {
/// Owned worker state so the spawned loop does not borrow the queue handle.
#[derive(Clone)]
struct QueueWorker {
db_path: String,
pool: std::sync::Arc<crate::db::DbPool>,
notify: Arc<Notify>,
stop: Arc<AtomicBool>,
handler: Arc<Handler>,
@@ -108,7 +108,7 @@ impl PersistentTaskQueue {
log::error!("failed to initialize queue schema: {e}");
}
Self {
db_path: db_path.to_string(),
pool: std::sync::Arc::new(crate::db::DbPool::new(db_path)),
notify: Arc::new(Notify::new()),
stop: Arc::new(AtomicBool::new(false)),
worker: Mutex::new(Vec::new()),
@@ -132,7 +132,7 @@ impl PersistentTaskQueue {
let mut handles = Vec::with_capacity(QUEUE_WORKERS + 1);
for _ in 0..QUEUE_WORKERS {
let worker = QueueWorker {
db_path: self.db_path.clone(),
pool: std::sync::Arc::clone(&self.pool),
notify: Arc::clone(&self.notify),
stop: Arc::clone(&self.stop),
handler: Arc::clone(&handler),
@@ -145,7 +145,7 @@ impl PersistentTaskQueue {
// the same notify as the workers, so enqueue and stop interrupt the
// sleep; the first interval tick fires immediately (harmless extra
// recovery at startup).
let sweep_db_path = self.db_path.clone();
let sweep_pool = std::sync::Arc::clone(&self.pool);
let sweep_notify = Arc::clone(&self.notify);
let sweep_stop = Arc::clone(&self.stop);
handles.push(tokio::spawn(async move {
@@ -160,8 +160,7 @@ impl PersistentTaskQueue {
if sweep_stop.load(Ordering::Relaxed) {
break;
}
let result =
crate::db::with_conn(&sweep_db_path, move |conn| recover_update(conn)).await;
let result = sweep_pool.with_conn(move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue sweep failed: {e}");
}
@@ -189,8 +188,8 @@ impl PersistentTaskQueue {
self.counter.fetch_add(1, Ordering::Relaxed)
);
let payload = payload.to_string();
log::info!("enqueued {id} (run_after {run_after:.1})");
crate::db::with_conn(&self.db_path, move |conn| {
log::debug!("enqueued {id} (run_after {run_after:.1})");
self.pool.with_conn(move |conn| {
conn.execute(
"INSERT OR REPLACE INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES (?1, ?2, ?3, 0, 'pending', 0, ?4)",
@@ -212,7 +211,7 @@ impl PersistentTaskQueue {
}
async fn recover_sweep(&self) {
let result = crate::db::with_conn(&self.db_path, move |conn| recover_update(conn)).await;
let result = self.pool.with_conn(move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue recovery failed: {e}");
}
@@ -267,7 +266,7 @@ impl QueueWorker {
/// Leases the oldest due row (sets it `in_progress` with a lock TTL).
/// Errors are surfaced so the caller can back off instead of spinning.
async fn lease_next(&self) -> Result<Option<LeasedRow>, rusqlite::Error> {
crate::db::with_conn(&self.db_path, |conn| {
self.pool.with_conn(|conn| {
// BEGIN IMMEDIATE: with several workers, a deferred transaction
// that read before another worker's lease commit would fail with
// SQLITE_BUSY_SNAPSHOT. Taking the write lock up front serializes
@@ -309,16 +308,18 @@ impl QueueWorker {
}
async fn earliest_run_after(&self) -> Option<f64> {
let result = crate::db::with_conn(&self.db_path, |conn| {
let mut stmt =
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
let mut rows = stmt.query([])?;
match rows.next()? {
Some(row) => Ok(row.get::<_, Option<f64>>(0)?),
None => Ok(None),
}
})
.await;
let result = self
.pool
.with_conn(|conn| {
let mut stmt =
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
let mut rows = stmt.query([])?;
match rows.next()? {
Some(row) => Ok(row.get::<_, Option<f64>>(0)?),
None => Ok(None),
}
})
.await;
match result {
Ok(v) => v,
Err(e) => {
@@ -338,10 +339,10 @@ impl QueueWorker {
return;
}
};
log::info!("processing {} (attempt {})", row.id, row.attempts + 1);
log::debug!("processing {} (attempt {})", row.id, row.attempts + 1);
match (self.handler)(payload).await {
Ok(()) => {
log::info!("task {} completed", row.id);
log::debug!("task {} completed", row.id);
self.delete_row(&row.id).await;
}
Err(QueueError::Retryable {
@@ -355,7 +356,7 @@ impl QueueWorker {
(self.dead_letter)(payload, message).await;
} else {
let delay = scaled_retry_delay(delay_seconds, row.attempts);
log::info!(
log::debug!(
"task {} rescheduled in {delay:.1}s (attempt {})",
row.id,
row.attempts + 1
@@ -374,11 +375,13 @@ impl QueueWorker {
async fn delete_row(&self, id: &str) {
let id = id.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
Ok(())
})
.await;
let result = self
.pool
.with_conn(move |conn| {
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
Ok(())
})
.await;
if let Err(e) = result {
log::error!("queue delete failed: {e}");
}
@@ -387,7 +390,7 @@ impl QueueWorker {
async fn reschedule(&self, id: &str, payload: Value, delay_seconds: f64, attempts: i32) {
let id = id.to_string();
let payload = payload.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
let result = self.pool.with_conn(move |conn| {
conn.execute(
"UPDATE tasks SET payload=?1, run_after=?2, attempts=?3, status='pending', locked_until=0 WHERE id=?4",
params![payload, now_f64() + delay_seconds, attempts, id],
@@ -575,7 +578,7 @@ mod tests {
// Insert a stale leased row AFTER startup: without a runtime sweep it
// would stay `in_progress` forever (only start() used to recover).
{
let conn = Connection::open(&queue.db_path).unwrap();
let conn = Connection::open(queue.pool.path()).unwrap();
ensure_schema(&conn).unwrap();
conn.execute(
"INSERT INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
+323 -113
View File
@@ -3,7 +3,7 @@
//! URL is blocked by hotlink protection; the bot downloads the file itself
//! and uploads it via multipart).
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE};
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE, log_key};
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost};
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
use crate::queue::QueueError;
@@ -147,6 +147,41 @@ impl Task {
fn is_cached_send(&self) -> bool {
self.cache_data().is_some_and(|c| !c.media.is_empty())
}
/// All media payloads of this task (sequence batches flattened plus the
/// lone animation).
fn media_items(&self) -> Vec<&MediaItemPayload> {
match self {
Task::SendMediaSequence { media_batches, .. } => {
media_batches.iter().flatten().collect()
}
Task::SendAnimation { animation, .. } => {
std::slice::from_ref(animation).iter().collect()
}
Task::ForwardMessages { .. } => Vec::new(),
}
}
/// Local file paths referenced by this task's media (ugoira / bsky remux
/// MP4 and the like); empty for URL or Telegram file-id sends.
fn local_media_paths(&self) -> Vec<std::path::PathBuf> {
let mut out = Vec::new();
for item in self.media_items() {
let is_file_id = match item {
MediaItemPayload::Photo { file_id, .. }
| MediaItemPayload::Video { file_id, .. }
| MediaItemPayload::Animation { file_id, .. } => *file_id,
};
if is_file_id {
continue;
}
let media = item_url(item);
if !media.starts_with("http://") && !media.starts_with("https://") {
out.push(std::path::PathBuf::from(media));
}
}
out
}
}
/// Telegram file id of the message's media, matched to the payload kind.
@@ -197,7 +232,7 @@ async fn cache_sent_task(task: &Task, media: Vec<CachedMedia>) {
post.media = media;
if let Some(key) = x_media::site::cache_key(&post.url) {
LINK_CACHE.put(&key, &post).await;
log::info!("cached send for {}", post.url);
log::debug!("cached send for [key={}]", log_key(&post.url));
}
}
@@ -222,11 +257,35 @@ pub async fn invalidate_cache(task: &Task) {
&& let Some(url) = task.source_url()
&& let Some(key) = x_media::site::cache_key(url)
{
log::info!("removing stale link cache entry for {url}");
log::debug!("removing stale link cache entry for [key={}]", log_key(url));
LINK_CACHE.remove(&key).await;
}
}
/// Locally produced media files (ugoira MP4, bsky remux MP4) whose temp dirs
/// must stay alive while their task may be retried by the queue. The fetch
/// pipeline hands ownership here via [`x_media::site::Fetched::take_keep_alive`]
/// before the [`Fetched`] is dropped; a queued retry runs after that drop, so
/// without this the local file would be gone by the time the retry sends it.
/// Entries are removed when the task settles (see [`release_keep_alive`]).
pub static KEEP_ALIVE: LazyLock<parking_lot::Mutex<Vec<tempfile::TempDir>>> =
LazyLock::new(|| parking_lot::Mutex::new(Vec::new()));
/// Drops the keep-alive temp dirs holding media referenced by `task` (matched
/// by path prefix). Called once a task settles — sent or permanently failed —
/// so retry-only temp files do not leak; retryable tasks keep them alive.
pub fn release_keep_alive(task: &Task) {
let paths = task.local_media_paths();
if paths.is_empty() {
return;
}
let mut alive = KEEP_ALIVE.lock();
alive.retain(|dir| {
let dir_path = dir.path();
!paths.iter().any(|p| p.starts_with(dir_path))
});
}
pub const MAX_MEDIA_GROUP: usize = 9;
/// Splits media into batches of at most [`MAX_MEDIA_GROUP`] items.
@@ -237,6 +296,19 @@ pub fn chunk_media_items<T: Clone>(items: Vec<T>) -> Vec<Vec<T>> {
.collect()
}
/// Orders media for a Telegram media group: when photos and videos are
/// mixed, the first item must be a photo (Telegram's sendMediaGroup rule).
/// Stable sort keeps the source order within each kind; a lone animation is
/// untouched (it takes the SendAnimation path before this runs).
pub fn photos_first(items: Vec<MediaItemPayload>) -> Vec<MediaItemPayload> {
let mut items = items;
items.sort_by_key(|item| match item {
MediaItemPayload::Photo { .. } => 0,
MediaItemPayload::Video { .. } | MediaItemPayload::Animation { .. } => 1,
});
items
}
/// Exponential backoff with jitter, capped at 30s.
pub fn retry_delay_seconds(attempts: u32) -> f64 {
let jitter: f64 = rand::thread_rng().gen_range(0.2..0.8);
@@ -484,9 +556,14 @@ enum FallbackError {
/// only if still too big. Anything that cannot be fixed falls back to the
/// item's smaller URL.
///
/// Downloads one media item to a temp file (deleted on drop). Network errors
/// are retryable; size over the upload cap and other download errors are not.
async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, FallbackError> {
/// Downloads one media item to a temp file (deleted on drop), returning the
/// file plus the downloaded bytes (photos keep the bytes for
/// [`photo::prepare_photo`] — re-reading the file would double the I/O).
/// Network errors are retryable; size over the upload cap and other download
/// errors are not.
async fn download_to_temp(
item: &MediaItemPayload,
) -> Result<(NamedTempFile, bytes::Bytes), FallbackError> {
let media_url = match item {
MediaItemPayload::Photo { media, .. }
| MediaItemPayload::Video { media, .. }
@@ -529,7 +606,7 @@ async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, Fall
.map_err(|e| FallbackError::Permanent {
message: format!("temp file write failed: {e}"),
})?;
Ok(file)
Ok((file, bytes))
}
/// Builds the media group item from an uploaded file.
@@ -580,10 +657,138 @@ fn media_from_url(
Ok(media)
}
/// One item prepared for the upload fallback: the ready-to-send media plus
/// the temp file that must stay on disk until the group request completes.
struct PreparedItem {
/// Original position in the batch (concurrent prep completes out of order).
index: usize,
media: InputMedia,
keep_alive: Option<NamedTempFile>,
}
/// Downloads / processes one media item for the upload fallback (see
/// [`send_batch_via_upload`]). Local files are uploaded directly; oversized
/// items fall back to their smaller URL; photos are downscaled/transcoded.
async fn prepare_upload_item(
item: MediaItemPayload,
index: usize,
caption: Option<&str>,
) -> Result<PreparedItem, FallbackError> {
// Locally produced files (ugoira / bsky remux MP4): nothing to download
// or shrink — upload the file directly. The send is a multipart upload,
// so the only remaining failure is an upload-cap error, which is
// permanent (a video cannot be re-encoded here).
let media_url = item_url(&item);
if !media_url.starts_with("http://") && !media_url.starts_with("https://") {
let media = media_from_file(
&item,
std::path::PathBuf::from(media_url),
caption,
item.thumbnail_url(),
)
.map_err(|message| FallbackError::Permanent { message })?;
return Ok(PreparedItem {
index,
media,
keep_alive: None,
});
}
// Size check before downloading/uploading: over the cap, use the
// smaller URL instead of the file. Photos are exempt — they are
// downloaded and processed (downscale / PNG→JPEG) before uploading.
let too_large = match x_media::site::media_size(media_url).await {
Ok(Some(size)) => size > MAX_UPLOAD_BYTES,
_ => false,
};
let too_large = too_large && !matches!(item, MediaItemPayload::Photo { .. });
if too_large {
let url = item
.fallback_url()
.ok_or_else(|| FallbackError::Permanent {
message: "media too large".into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
return Ok(PreparedItem {
index,
media,
keep_alive: None,
});
}
match download_to_temp(&item).await {
Ok((file, bytes)) => {
if matches!(item, MediaItemPayload::Photo { .. }) {
// Telegram rejects photos wider+taller than 10000 px combined
// (PHOTO_INVALID_DIMENSIONS): downscale the downloaded file
// before uploading; photos that cannot be brought within the
// limits degrade to the smaller URL. CPU-heavy work runs off
// the async executor thread.
let prep = tokio::task::spawn_blocking(move || photo::prepare_photo(file, &bytes))
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
})?
.map_err(|message| FallbackError::Permanent { message })?;
match prep {
PhotoPrep::Upload(upload) => {
let path = upload.path().to_path_buf();
let media = media_from_file(&item, path, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: Some(upload),
})
}
PhotoPrep::UseFallback => {
let url = item.fallback_url().ok_or_else(|| FallbackError::Permanent {
message: "photo dimensions exceed Telegram limits and no smaller variant is available"
.into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: None,
})
}
}
} else {
let path = file.path().to_path_buf();
let media = media_from_file(&item, path, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: Some(file),
})
}
}
Err(FallbackError::MediaTooLarge) => {
let url = item
.fallback_url()
.ok_or_else(|| FallbackError::Permanent {
message: "media too large".into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: None,
})
}
Err(e) => Err(e),
}
}
/// Download-and-reupload fallback for one media batch. Files over the upload
/// cap are not downloaded/uploaded; the item falls back to its smaller URL
/// (which Telegram fetches itself). Returns the fallback-error without the
/// task attached; callers wrap it with the updated task state.
/// (which Telegram fetches itself). Items are prepared concurrently (bounded)
/// because the downloads are network-bound; the batch is then uploaded in its
/// original order. Returns the fallback-error without the task attached;
/// callers wrap it with the updated task state.
async fn send_batch_via_upload(
bot: &Bot,
chat_id: i64,
@@ -591,109 +796,57 @@ async fn send_batch_via_upload(
batch: &[MediaItemPayload],
caption: Option<&str>,
) -> Result<Vec<Message>, FallbackError> {
let mut files = Vec::new();
let mut items = Vec::new();
let sem = std::sync::Arc::new(tokio::sync::Semaphore::new(3));
let mut set = tokio::task::JoinSet::new();
for (i, item) in batch.iter().enumerate() {
let item_caption = if i == 0 { caption } else { None };
// Size check before downloading/uploading: over the cap, use the
// smaller URL instead of the file. Photos are exempt — they are
// downloaded and processed (downscale / PNG→JPEG) before uploading.
let too_large = match x_media::site::media_size(item_url(item)).await {
Ok(Some(size)) => size > MAX_UPLOAD_BYTES,
_ => false,
};
let too_large = too_large && !matches!(item, MediaItemPayload::Photo { .. });
let media = if too_large {
match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption, item.thumbnail_url()) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
},
None => {
return Err(FallbackError::Permanent {
message: "media too large".into(),
});
}
}
let item_caption = if i == 0 {
caption.map(str::to_string)
} else {
match download_to_temp(item).await {
Ok(file) => {
// Telegram rejects photos wider+taller than 10000 px
// combined (PHOTO_INVALID_DIMENSIONS): downscale the
// downloaded file before uploading; photos that cannot be
// brought within the limits degrade to the smaller URL.
if matches!(item, MediaItemPayload::Photo { .. }) {
// CPU-heavy (decode/resize/encode): run off the async
// executor thread.
let prep = tokio::task::spawn_blocking(move || photo::prepare_photo(file))
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
})?
.map_err(|message| FallbackError::Permanent { message })?;
match prep {
PhotoPrep::Upload(upload) => {
let path = upload.path().to_path_buf();
files.push(upload);
media_from_file(item, path, item_caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?
}
PhotoPrep::UseFallback => match item.fallback_url() {
Some(url) => match media_from_url(
item,
url,
item_caption,
item.thumbnail_url(),
) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
},
None => {
return Err(FallbackError::Permanent {
message:
"photo dimensions exceed Telegram limits and no smaller variant is available"
.into(),
});
}
},
}
} else {
let path = file.path().to_path_buf();
files.push(file);
media_from_file(item, path, item_caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?
}
}
Err(FallbackError::MediaTooLarge) => match item.fallback_url() {
Some(url) => {
match media_from_url(item, url, item_caption, item.thumbnail_url()) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
}
}
None => {
return Err(FallbackError::Permanent {
message: "media too large".into(),
});
}
},
Err(e) => return Err(e),
None
};
let item = item.clone();
let sem = std::sync::Arc::clone(&sem);
set.spawn(async move {
let _permit = sem.acquire().await.expect("upload semaphore closed");
prepare_upload_item(item, i, item_caption.as_deref()).await
});
}
let mut prepared: Vec<Option<InputMedia>> = (0..batch.len()).map(|_| None).collect();
let mut keep_alive: Vec<NamedTempFile> = Vec::new();
while let Some(joined) = set.join_next().await {
let item = match joined {
Ok(Ok(item)) => item,
// Dropping the JoinSet aborts the remaining prep tasks; their
// temp files are cleaned up on drop (short-circuit like before).
Ok(Err(e)) => return Err(e),
Err(e) => {
return Err(FallbackError::Permanent {
message: format!("upload worker panicked: {e}"),
});
}
};
items.push(media);
let PreparedItem {
index,
media,
keep_alive: file_opt,
} = item;
if let Some(file) = file_opt {
keep_alive.push(file);
}
prepared[index] = Some(media);
}
let items: Vec<InputMedia> = prepared
.into_iter()
.map(|m| m.expect("every upload item was prepared"))
.collect();
// `keep_alive` holds the temp files until the group request completes.
let result = bot
.send_media_group(ChatId(chat_id), items)
.reply_parameters(
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
)
.await;
drop(keep_alive);
match result {
Ok(messages) => Ok(messages),
Err(e) => Err(match classify_request_error(&e) {
@@ -788,7 +941,7 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
.await
{
Ok(messages) => {
log::info!(
log::debug!(
"media group batch {idx}/{} sent ({} item(s))",
media_batches.len(),
batch.len()
@@ -799,7 +952,11 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
log::info!(
"Telegram could not fetch media for batch {idx} ({}), downloading and reuploading",
batch.first().map(item_url).unwrap_or("?")
batch
.first()
.map(item_url)
.map(log_key)
.unwrap_or_else(|| "?".into())
);
match send_batch_via_upload(bot, chat_id, reply_to, batch, caption).await {
Ok(messages) => {
@@ -895,11 +1052,11 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
}
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
log::info!(
"Telegram could not fetch animation URL, downloading and reuploading: {}",
media_url
"Telegram could not fetch animation URL, downloading and reuploading: [key={}]",
log_key(media_url)
);
match download_to_temp(animation).await {
Ok(file) => {
Ok((file, _bytes)) => {
let path = file.path().to_path_buf();
match send_animation_inner(
bot,
@@ -1199,6 +1356,8 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
}
Err(SendError::Permanent { message, task }) => {
invalidate_cache(&task).await;
// The task settles here: drop any keep-alive temp media.
release_keep_alive(&task);
return Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
@@ -1208,6 +1367,7 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
if !resumed {
post_send_actions(&bot, &task, message_ids).await;
}
release_keep_alive(&task);
Ok(())
}
Task::ForwardMessages { .. } => match forward_messages(&bot, &task).await {
@@ -1219,10 +1379,13 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
delay_seconds,
payload: serde_json::to_value(task).expect("task serializes"),
}),
Err(SendError::Permanent { message, task }) => Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
}),
Err(SendError::Permanent { message, task }) => {
release_keep_alive(&task);
Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
})
}
},
}
}
@@ -1258,9 +1421,10 @@ mod tests {
#[test]
fn oversized_photo_boundary() {
// The empirical Telegram limit: sum 10000 passes, 10001 fails.
assert!(crate::photo::PHOTO_MAX_DIMENSION_SUM == 10000);
assert!(6100 + 3900 <= crate::photo::PHOTO_MAX_DIMENSION_SUM);
assert!(6300 + 3730 > crate::photo::PHOTO_MAX_DIMENSION_SUM);
// Const-block asserts so clippy's assertions_on_constants stays quiet.
const { assert!(crate::photo::PHOTO_MAX_DIMENSION_SUM == 10000) };
const { assert!(6100 + 3900 <= crate::photo::PHOTO_MAX_DIMENSION_SUM) };
const { assert!(6300 + 3730 > crate::photo::PHOTO_MAX_DIMENSION_SUM) };
}
#[test]
@@ -1277,6 +1441,52 @@ mod tests {
);
}
#[test]
fn photos_first_orders_photos_before_videos() {
use MediaItemPayload::{Animation, Photo, Video};
let photo = |u: &str| Photo {
media: u.into(),
has_spoiler: false,
fallback_url: None,
file_id: false,
};
let video = |u: &str| Video {
media: u.into(),
has_spoiler: false,
thumbnail: None,
fallback_url: None,
file_id: false,
};
let items = vec![
video("https://v/1.mp4"),
photo("https://p/1.jpg"),
video("https://v/2.mp4"),
photo("https://p/2.jpg"),
];
let ordered = photos_first(items);
// All photos first (stable: p1 before p2), then all videos in order.
let kinds: Vec<&str> = ordered
.iter()
.map(|i| match i {
Photo { media, .. } => media.as_str(),
Video { media, .. } => media.as_str(),
Animation { .. } => unreachable!(),
})
.collect();
assert_eq!(
kinds,
[
"https://p/1.jpg",
"https://p/2.jpg",
"https://v/1.mp4",
"https://v/2.mp4"
]
);
// Already-photos-first input is unchanged.
let items = vec![photo("https://p/1.jpg"), video("https://v/1.mp4")];
assert!(matches!(photos_first(items)[0], Photo { .. }));
}
#[test]
fn retry_delay_seconds_bounds() {
for attempts in 0..10 {
+31 -27
View File
@@ -38,7 +38,7 @@ pub struct ChatStore {
/// Per-chat async locks serializing get→mutate→set so concurrent handler
/// tasks (batch-forwards, callbacks) cannot clobber each other's writes.
locks: Mutex<HashMap<i64, Arc<tokio::sync::Mutex<()>>>>,
db_path: String,
pool: crate::db::DbPool,
}
pub fn unix_now() -> i64 {
@@ -67,7 +67,7 @@ impl ChatStore {
Ok(ChatStore {
cache: Mutex::new(HashMap::new()),
locks: Mutex::new(HashMap::new()),
db_path: path.to_string(),
pool: crate::db::DbPool::new(path),
})
}
@@ -76,23 +76,25 @@ impl ChatStore {
return data.clone();
}
let chat_key = chat_id.to_string();
let payload = crate::db::with_conn(&self.db_path, move |conn| {
// Concurrent handler tasks (batch-forwards) may write chat_state
// while this read runs; the shared busy timeout handles the
// write-lock collision instead of failing the query.
let mut stmt = conn.prepare("SELECT payload FROM chat_state WHERE chat_id = ?1")?;
let mut rows = stmt.query(params![chat_key])?;
match rows.next()? {
Some(row) => Ok(Some(row.get::<_, String>(0)?)),
None => Ok(None),
}
})
.await
.unwrap_or_else(|e| {
log::error!("chat_state read failed: {e}");
None
})
.unwrap_or_default();
let payload = self
.pool
.with_conn(move |conn| {
// Concurrent handler tasks (batch-forwards) may write chat_state
// while this read runs; the shared busy timeout handles the
// write-lock collision instead of failing the query.
let mut stmt = conn.prepare("SELECT payload FROM chat_state WHERE chat_id = ?1")?;
let mut rows = stmt.query(params![chat_key])?;
match rows.next()? {
Some(row) => Ok(Some(row.get::<_, String>(0)?)),
None => Ok(None),
}
})
.await
.unwrap_or_else(|e| {
log::error!("chat_state read failed: {e}");
None
})
.unwrap_or_default();
let data: ChatData = serde_json::from_str(&payload).unwrap_or_default();
self.cache.lock().insert(chat_id, data.clone());
data
@@ -103,14 +105,16 @@ impl ChatStore {
self.cache.lock().insert(chat_id, data.clone());
let payload = serde_json::to_string(data).expect("chat state serializes");
let chat_id = chat_id.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
params![chat_id, payload],
)?;
Ok(())
})
.await;
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
params![chat_id, payload],
)?;
Ok(())
})
.await;
if let Err(e) = result {
log::error!("chat_state write failed: {e}");
}
+215
View File
@@ -0,0 +1,215 @@
# 站点适配器重构方案:让新增站点变成"新模块 + 注册一行"
> 状态:设计稿(未实施)。目标:把"加一个新站点"从改 8-9 处收敛到 3 处,
> 并让站点身份、重试策略、下载 header 等站点能力归位到站点模块自身。
> 本文只改文档,不动代码;每阶段均可独立合入、独立回滚。
---
## 1. 现状摩擦清单
以现有三站(twitter / bsky / pixiv)为基线,新增第 4 个站点(代号 `example`
今天需要触碰的位置:
| # | 位置(当前行号) | 改动 | 必改? |
|---|---|---|---|
| 1 | 新目录 `crates/x-media/src/site/example/{mod,interface,model}.rs` | 新模块 | 必改 |
| 2 | `site/mod.rs:354-365` `fetch_once` | 加一个 `if` 分派分支 | 必改 |
| 3 | `site/mod.rs:167-178` `cache_key` | 加一个 `if` 分支 + 约定 key 前缀 `"example:..."` | 必改 |
| 4 | `site/mod.rs:53-63` `site_name()` | 加一个 URL `contains` 嗅探分支 | 必改 |
| 5 | `handlers.rs:405` `SetFormat` 白名单 | `["twitter","bsky","pixiv"]` 加字符串 | 必改 |
| 6 | `config.rs` / `main.rs:74-84` | 仿 pixiv 加启动校验(token、`disable()` | 视站点 |
| 7 | `site/mod.rs:312-326` `fetch_error_is_retryable` | 若重试策略特殊,改中央分类函数 | 视站点 |
| 8 | `site/mod.rs:377/391/430` 三个下载函数 | 若媒体有防盗链,加 header(现在是硬编码 pximg 判断) | 视站点 |
| 9 | `site/mod.rs:16,180-197` `FetchError` | 若错误类型特殊,加嵌套 variant(仿 `Pixiv(PixivError)` | 视站点 |
**根因**:仓库里没有"站点"这个实体。站点的四类能力——URL 识别(PATTERN +
cache_key)、抓取、重试策略、下载 header——分别散落在中央 if 链、URL 字符串嗅探、
魔法字符串 key 和 bot crate 的白名单里。`AGENTS.md` 现行约定 "no trait, no enum
dispatch" 是刻意的简单性选择;本方案的目标是在**不推翻它精神的前提下**收敛摩擦,
并在阶段 3 提供完整的 trait 注册表选项。
## 2. 目标架构
```
crates/x-media/src/site/mod.rs
├─ SITES: LazyLock<Vec<Box<dyn Site>>> ← 注册表(唯一的"站点列表")
├─ find_site(url) / fetch(url) / cache_key(url) / site_ids()
└─ 通用类型:Fetched { site_id, ... } / FetchError(通用类 + Site 变体)
├─ site/twitter/{mod,interface,model}.rs impl Site
├─ site/bsky/… impl Site
└─ site/pixiv/… impl Site (download_headers: pximg Referer)
(validate: token 校验)
crates/xmedia-bot
├─ handlers.rs SetFormat 白名单 ← x_media::site::ids()(不再写死)
├─ handlers.rs site 格式查找 ← fetched.site_id(缓存/新鲜两条路径同口径)
└─ main.rs 启动校验 ← site::validate_all()(不再特判 pixiv
```
## 3. 分阶段迁移
每个阶段是一个独立提交,保持 `cargo fmt` / `cargo clippy -- -D warnings` /
`cargo test --workspace` 全绿;行为完全不变,只挪代码、不换语义。
### 阶段 1:站点身份单一来源(低风险,推荐先做)
**动机**:同一概念目前有两个来源——缓存命中路径用 `key.split(':').next()`
`handlers.rs:639`),新鲜抓取路径用 `fetched.site_name()``handlers.rs:724`);
`site_name()` 又是对 `source_url``contains` 字符串嗅探,还有 `"unknown"`
兜底分支。
**改动**
1. `site/mod.rs``Fetched` 增加字段 `site_id: &'static str`(由各站点的
`impl From<SiteStruct> for Fetched` 填充;`empty_fetched` 同步填)。
`Fetched::site_name()` 改为 `return self.site_id`(保留方法名,删除
`source_url.contains` 嗅探与 `"unknown"` 分支)。
2. `site/mod.rs`:新增 `pub fn site_id_from_key(key: &str) -> &'static str`
(解析 `"example:..."` 前缀,未知前缀返回 `"unknown"`),bot 缓存命中路径改用它,
`fetched.site_id` 口径统一。
3. `handlers.rs:405``SetFormat` 白名单改为 `x_media::site::ids()`——阶段 1 先实现
`ids()``["twitter","bsky","pixiv"]` 的常量函数(数据源仍集中,行为不变),
阶段 3 再改为遍历注册表。
4. `twitter/interface.rs:48-60` / `bsky` / `pixiv``From<SiteStruct> for Fetched`
各补 `site_id` 字段。
**风险**:低。纯增量字段;`site_name()` 语义不变(测试 `pixiv/interface.rs:355`
已断言 `"pixiv"`)。
**验证**:现有全部单测;`cache_key_normalizes_domain_variants` 等不变。
**回滚**revert 该提交。
### 阶段 2:站点能力下沉(不引入 trait,静态分派)
**动机**:把"每个站点自己才知道"的逻辑搬回站点模块,中央只做迭代。这是
`AGENTS.md` 现有约定(无 trait)与完整注册表之间的折中,可独立交付。
**改动**:每个站点模块新增并 `mod.rs` 重新导出:
```rust
// site/twitter/interface.rsbsky/pixiv 同构)
pub fn cache_key(url: &str) -> Option<String>; // 用自身 PATTERN,返回 "twitter:<id>"
pub fn is_retryable(err: &FetchError) -> bool; // 默认 Http|Transientpixiv 覆盖 PixivError 分支
pub fn media_headers(url: &str) -> Option<Vec<(&'static str, String)>>;
// pixiv: url 含 "pximg.net" → Referer
```
`site/mod.rs` 相应改为迭代三站:
- `cache_key`:逐个调 `site::cache_key`,不再自己写 key 格式;
- `fetch_error_is_retryable`:删除,`fetch()` 重试循环改调 `current_site::is_retryable`
`fetch_once` 已能确定站点,把站点传下去);
- `media_size` / `download_media_limited` / `download_media_to_file` 里的
`pximg.net → Referer` 硬编码删除,改为遍历 `SITES`(阶段 2 是遍历
`[twitter, bsky, pixiv]` 静态列表)取 `media_headers(url)` 合并。
**注意**Referer 判定依据是媒体 URL 的 host(`pximg.net`),**不是**站点
PATTERNpixiv 的 PATTERN 只匹配 `pixiv.net/artworks/...`),所以 `media_headers`
不能挂在 PATTERN 匹配上,必须按 URL 独立匹配——这正是把它做成独立函数的原因。
**风险**:中。下载函数签名不变,行为必须逐字节不变;新增单元测试覆盖
`media_headers("https://i.pximg.net/...") == Some(Referer)`
`cache_key` 等价性(对全部既有用例断言新旧结果一致)。
**回滚**revert。
### 阶段 3Site trait + SITES 注册表(完整方案,可选)
**动机**:加站点时 bot crate 与中央分派零改动;站点列表成为唯一注册点。
**新增**`site/mod.rs`):
```rust
pub trait Site: Send + Sync {
fn id(&self) -> &'static str;
fn pattern(&self) -> &'static Regex;
fn enabled(&self) -> bool;
fn cache_key(&self, url: &str) -> Option<String>; // 默认: id + 捕获组1
fn fetch_from_url(&self, url: &str)
-> Pin<Box<dyn Future<Output = Result<Fetched, FetchError>> + Send>>;
fn is_retryable(&self, err: &FetchError) -> bool; // 默认: Http|Transient
fn media_headers(&self, url: &str) -> Option<Vec<(&'static str, String)>>; // 默认: None
fn validate(&self) -> Option<BoxFuture<'static, Result<(), String>>>; // 默认: None
}
static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| vec![
Box::new(twitter::TwitterSite), Box::new(bsky::BskySite), Box::new(pixiv::PixivSite),
]);
```
- `fetch_once``find_site(url)`(首个 PATTERN 命中且 `enabled()` 的站点)
`site.fetch_from_url(url).await`
- `cache_key` / `site_ids()` / `media_headers` / `validate_all()` 全部遍历 `SITES`
- `fetch_error_is_retryable` 删除,重试判定走 `site.is_retryable`
- `main.rs:74-84` 的 pixiv 特判 → `site::validate_all()`pixiv 的 `validate` 失败时
内部调用现有 `pixiv::disable()`,行为保持);
- 保留各站点的 `PATTERN`/`enabled()`/`fetch_from_url()` 顶层导出(兼容现有
`fetch_once` 及测试),trait 只是包一层薄壳。
**async 形态**:仓库没有 `async-trait` 依赖。两个选择:
(a) 手写 `Pin<Box<dyn Future>>` 返回类型(零新依赖,契合仓库手写风格,签名略丑);
(b) 引入 `async-trait`(可读性好,新增一个依赖)。
建议先 (a),理由:仓库显式偏好手写错误/状态机,且 `BoxFuture` 已有先例
`queue.rs:38``BoxFuture`)。
**风险**:中。动中央分派,但每站点行为不变;注册表迭代 + `find_site` 补单测
`fetch`/`cache_key` 对既有 URL 集合的结果与阶段 2 完全一致)。
**回滚**revert。
### 阶段 4FetchError 泛化(可选,配合阶段 3)
**动机**`FetchError::Pixiv(PixivError)``site/mod.rs:16,184,241-245`)是站点特有
错误嵌进通用枚举;第 4 个站点要么再加变体,要么用泛化变体。
**改动**`FetchError` 增加 `Site { site: &'static str, error: Box<dyn std::error::Error + Send + Sync> }`
`Pixiv(PixivError)` 变体保留但内部迁移到 `Site`(或直接替换并更新
`is_retryable`/`Display`/`source()` 与测试)。重试判定在阶段 3 已归站点,
中央枚举只剩通用类(Http/Json/NotFound/Blocked/Sensitive/TooLarge/Transient/Io)。
**风险**:中。`Display`/`source()`/`From<PixivError>``fetch_error_is_retryable`
测试(`site/mod.rs:480-522`)需同步。
**回滚**revert。
### 阶段 5:收尾
- 更新 `AGENTS.md` 的 "Site adapter convention" 段:写新约定(注册表 + `impl Site` +
每站点 `cache_key`/`is_retryable`/`media_headers`),删除 "no trait" 表述;
- `examples/fetch.rs` 不变(走 `site::fetch`);
- 新增站点 checklist 见 §4。
## 4. 重构后新增站点 checklist
```
1. crates/x-media/src/site/example/{mod,interface,model}.rs // 新模块
2. impl Site for ExampleSite 并注册进 SITES // 注册一行
3. (可选)token 读取 + validate() 实现 // 启动校验自动生效
── bot crate 零改动 ──
```
对比现状的 8-9 处,bot crate 完全不碰:`SetFormat` 白名单、格式查找口径、
缓存 key、启动校验全部自动跟随注册表。
## 5. 权衡与明确不做的事
- **不做**Media 类型扩展(`media.rs` + `MediaItemPayload` + `CachedMediaKind` +
send.rs 约 10+ 处 match 的 blast radius)——这是"新增媒体类型"的摩擦,与"新增
站点"正交,优先级低,保持现状。
- **不做**DI/全局注入改造(`CHAT_STORE`/`TASK_QUEUE`/`CONFIG``LazyLock` 静态
模式是仓库惯例,与站点扩展无关)。
- **不做**:schema 迁移——新站点只产生新的 cache key 前缀与 `message_format` JSON
key`link_cache`/`chat_state` 表结构均无需变化。
- **代价**:阶段 3 引入 `dyn Site` 与(选择 (a) 时)手写 `BoxFuture` 签名;若站点
数量长期 ≤5 且无新增迹象,阶段 2 的折中方案已够用,阶段 3/4 可无限期推迟。
## 6. 建议的提交序列
| 阶段 | 提交消息(建议) |
|---|---|
| 1 | `refactor(site): carry site_id on Fetched; unify cache-key site lookup` |
| 2 | `refactor(site): move cache_key/is_retryable/media_headers into site modules` |
| 3 | `refactor(site): introduce Site trait and SITES registry` |
| 4 | `refactor(site): genericize FetchError::Site` |
| 5 | `docs: update site adapter convention in AGENTS.md` |
每阶段独立合入、独立回滚;阶段 2 完成后即可认为"加站点"摩擦已收敛,
3/4 为可选深化。