Compare commits

..
74 Commits
Author SHA1 Message Date
YoursFunny f6845b1b5c chore: bump version to 1.2.2 2026-08-14 21:45:48 +08:00
YoursFunny fae8dc6f2d fix(twitter): treat empty tombstone as withheld content, not deleted 2026-08-14 21:21:15 +08:00
YoursFunny ac72e414c3 docs: add architecture refactor design 2026-08-14 21:21:15 +08:00
YoursFunny 8b3b2a246b refactor(errors): derive FetchError and PixivError with thiserror 2026-08-14 20:23:14 +08:00
YoursFunny 6f6898c245 refactor(send): share the upload fallback pipeline between group and animation sends 2026-08-14 19:39:58 +08:00
YoursFunny 69698992d5 refactor(send): fold FallbackError into SendError via from_fallback 2026-08-14 19:38:53 +08:00
YoursFunny 1e77bb0478 refactor(db): share one DbPool across stores; merge schema init 2026-08-14 19:37:18 +08:00
YoursFunny 8f2b0a1dcb docs: note AFIT dyn retest on rustc 1.97.1 2026-08-14 19:00:00 +08:00
YoursFunny b65fb967c4 docs: cite official AFIDT goal for the AFIT dyn limitation 2026-08-14 18:45:19 +08:00
YoursFunny a8fd685777 docs: update site adapter convention in AGENTS.md 2026-08-14 18:27:05 +08:00
YoursFunny 5679a8c172 refactor(site): genericize FetchError::Site 2026-08-14 18:26:01 +08:00
YoursFunny bf4e6159b3 refactor(site): introduce Site trait and SITES registry 2026-08-14 18:24:03 +08:00
YoursFunny 5e23916b40 refactor(site): move cache_key/is_retryable/media_headers into site modules 2026-08-14 18:19:39 +08:00
YoursFunny 7ca8fd1da2 refactor(site): carry site_id on Fetched; unify cache-key site lookup 2026-08-14 18:17:22 +08:00
YoursFunny 96c11becb9 docs: prefer native async fn in trait (AFIT) for the site registry 2026-08-14 18:05:06 +08:00
YoursFunny 5830a3f013 chore: bump version to 1.2.1 2026-08-14 17:55:26 +08:00
YoursFunny 183bb7e435 docs: add site registry refactor design 2026-08-14 17:55:10 +08:00
YoursFunny 2a8433a8d2 fix(twitter): map syndication TweetTombstone to NotFound
Deleted tweets answer the syndication endpoint with HTTP 200 and a
TweetTombstone (no `errors`, no `id_str`). The body classifier only
knew the `errors` shape, so tombstones fell through to the
`no id_str -> Sensitive` branch and degraded to an empty result,
making the bot reply "No media found" for a deleted tweet.

- extract parse_syndication_body(); tombstone shape -> NotFound
- propagate NotFound from the TWITTER_AUTH_TOKEN GraphQL fallback
  instead of swallowing it into empty_fetched
- unit tests for all body classes + live regression test on a real
  tombstoned tweet
2026-08-14 12:10:36 +08:00
YoursFunny 6b3e61881d logging: re-level, redact user data at info, and key links by post id
P0 — level rework + redaction:
- info now carries only lifecycle, per-post business results (sent /
  forwarded / copied / template applied), admin actions and anomalies
  (upload fallback, retry enqueue; dead-letter stays error).
- Per-request detail moved to debug: message/command logging, URL
  extraction, fetching/fetched, link-cache hits, media-group batch sends,
  queue processing (enqueue/processing/completed/rescheduled), photo
  processing (downscale/transcode), inline queries, sensitive-tweet note.
- Full user-submitted URLs and message text now appear only at debug; at
  info and above links are printed via the normalized cache key.

P1 — request correlation:
- handlers::log_key() maps a URL to its normalized post key
  (twitter:<id> / pixiv:<id> / bsky:<handle>/<rkey>). The whole lifecycle
  of one link (fetch -> send -> cache -> fallback) now logs [key=...], so
  multi-worker logs can be correlated by grepping the key.

Convention documented in AGENTS.md.
2026-08-13 23:34:06 +08:00
YoursFunny 47935dd7c6 fix(pixiv): stop retrying permanent 4xx API errors
site::fetch retried every PixivError, so a bad/expired token (403) or a
deleted artwork (404) burned all 3 attempts with backoff against pixiv's
API for nothing. Add PixivError::Status(u16) — the app-API calls now
surface the HTTP status — and retry only the transient classes: network
errors, 429 and 5xx. 4xx / Api (token errors) / Json / NoAuth are
returned immediately. The classification is a pure helper
(fetch_error_is_retryable) with unit tests.
2026-08-13 23:17:52 +08:00
YoursFunny 6911e9146e fix(handlers): stop URL workers by closing the job channel
The old stop only set an atomic flag checked between jobs: a worker
blocked in recv() never woke (the channel was never closed), and queued
jobs were neither drained nor abandoned in a defined way despite the
"drains up to 256 jobs" comment. Now stop_url_workers sets the flag,
drops the sender so blocked recv() calls wake with None, and awaits the
worker JoinHandles (each finishes its in-flight job first). main awaits
it inside the existing 30s shutdown timeout.
2026-08-13 23:15:14 +08:00
YoursFunny aa705aef90 chore: bump version to 1.2.0
New in 1.2.0:
- feat: debounced inline queries, bounded graceful shutdown, config
  fail-fast on invalid env values
- fix: local-media tasks (ugoira/bsky MP4) survive queue retries,
  photo-first ordering in mixed media groups, caption truncation to
  Telegram's 1024-char limit
- perf: SQLite connection pool, concurrent upload-fallback downloads,
  ugoira zip streamed to disk, photo processing without re-reading the
  temp file, release profile LTO
- ci: test/clippy gate + layered live/token job
- docs: English README (README.en.md)
2026-08-13 22:52:38 +08:00
YoursFunny 39260a8817 style: rustfmt config.rs and main.rs from the last two features 2026-08-13 22:43:35 +08:00
YoursFunny 6b9640aa48 build: enable thin LTO and single codegen units for release
Smaller/faster production binary (verified: cargo build --release -p
xmedia-bot builds clean with the new profile). panic=abort is
intentionally not set — queue workers and db closures rely on JoinHandle
catching panics, which abort would defeat.
2026-08-13 22:42:42 +08:00
YoursFunny ad59f518ff docs: add English README (README.en.md)
Full English translation of README.md — features, quick start, webhook
deployment (domain + IP-only with acme.sh), env table, command table.
The Chinese README stays the primary one.
2026-08-13 22:37:12 +08:00
YoursFunny e21643063e feat(config): fail fast on misspelled env values
A typo in EDIT_MESSAGE_TTL_SECONDS / WEBHOOK_PORT etc. used to silently
fall back to a default, so the bot ran with different behavior than the
operator intended (or failed much later on a bare .expect). Unparseable
values now log a loud warning naming the variable; invalid webhook
settings still surface as a hard .expect in webhook mode.
2026-08-13 22:26:23 +08:00
YoursFunny 246fc989f0 feat(main): bound graceful shutdown with a 30s timeout
The stop sequence awaited the queue workers, which can be mid-download
(30s client timeout) or mid-ugoira encode (minutes). A stuck worker would
hold shutdown forever; now the process logs and exits after 30s.
2026-08-13 22:26:16 +08:00
YoursFunny 2e2d1b3506 style: rustfmt the DbPool call sites from the connection-pool change
Formatting-only; the pool commit landed before cargo fmt was run.
2026-08-13 22:25:13 +08:00
YoursFunny 1747d321d8 perf(photo): stop re-reading the downloaded temp file
download_to_temp buffered the full bytes, wrote them to a temp file, and
prepare_photo then read the whole file back from disk. The bytes are
already in memory — pass them through (download_to_temp now returns
(file, bytes)) so photo processing never touches the disk for input.
Adds the bytes dependency to xmedia-bot (already in the lock via x-media).
2026-08-13 22:24:58 +08:00
YoursFunny 505990e49e perf(pixiv): stream the ugoira frame zip to disk instead of RAM
download_media_limited buffered the whole frame zip (cap 512 MB) in
memory before extraction, spiking RAM for large ugoira. New
site::download_media_to_file streams chunks straight to a temp file with
the same Content-Length / stream cap checks, and ugoira_video now opens
the zip from disk inside spawn_blocking. Adds FetchError::Io for local
write failures (hand-rolled error pattern preserved).
2026-08-13 22:22:13 +08:00
YoursFunny 95b475ff08 perf(send): prepare upload-fallback items concurrently
The download-and-reupload fallback downloaded each batch item serially,
so a 9-item batch took 9× the slowest download. Items are now prepared
concurrently (bounded to 3 in-flight downloads + photo processing) via a
JoinSet, then the group is uploaded in its original order; the per-item
logic moved into prepare_upload_item. Temp files stay alive until the
group request completes. A failing item still aborts the batch (the
JoinSet drop cancels the remaining prep tasks, as before).
2026-08-13 22:19:39 +08:00
YoursFunny 2297fdc91c fix(site): truncate captions to Telegram's 1024-char limit
A long tweet text or a pixiv artwork with many tags can exceed Telegram's
1024-char caption cap for HTML parse mode, turning an otherwise fine send
into a permanent 400. truncate_caption() cuts at a char boundary (never
splitting a multi-byte rune or an HTML entity like &amp;) and appends an
ellipsis. Applied in caption_with / caption_from_fields, the link-cache
re-send path and the inline-query captions.
2026-08-13 22:15:33 +08:00
YoursFunny 4580b79d4f fix(send): order photos first in mixed media groups
Telegram's sendMediaGroup requires the first item to be a photo when a
group mixes photos and videos; the previous code kept the source-site
order, so a mixed post with a video first (twitter media order is not
guaranteed) would 400 permanently. photos_first() stable-sorts photos
ahead of videos/animations before chunking; within-kind order is kept.
2026-08-13 22:14:07 +08:00
YoursFunny edb32c23b4 perf(db): reuse SQLite connections via a small per-store pool
Every DB operation (queue lease/enqueue, chat_state get/set, link_cache
read/write) used to open a fresh connection — including the busy timeout
and WAL pragma — then close it, on every message, URL job and callback.

Replace with DbPool: a tiny pool (4 connections max, semaphore-bounded
concurrency for backpressure) whose with_conn() method runs the closure on
a pooled connection inside spawn_blocking. Steady-state cost of an
operation is a list pop + semaphore acquire instead of a connection open.
2026-08-13 22:12:34 +08:00
YoursFunny 4a467641aa ci: add test/clippy workflow and gate live/token tests
The docker workflow only builds/pushes; tests were a local responsibility.
Add .github/workflows/ci.yml with two layers:

- test: cargo fmt --check + cargo clippy --workspace --all-targets -D
  warnings + cargo test --workspace (fully offline, no secrets) on every
  push/PR, including forks.
- live: the #[ignore]d live-network tests plus the pixiv token-gated
  tests, run on schedule / manual dispatch / tag pushes only (fork PRs
  cannot read repository secrets), with PIXIV_REFRESH_TOKEN /
  TWITTER_AUTH_TOKEN injected and continue-on-error for flaky sites.

Test gating (documented in AGENTS.md):
- live-network tests now carry #[ignore = "live network: ..."] (twitter 3,
  bsky 2, pixiv bogus-token 1) and run via -- --ignored live.
- pixiv token tests early-return when PIXIV_REFRESH_TOKEN is absent or
  empty (an unset GitHub secret arrives as ""); test_fetch previously
  failed without a token.

Also fixes the three clippy assertions_on_constants warnings in send.rs
(required for -D warnings).
2026-08-13 22:07:32 +08:00
YoursFunny bd43a12dee fix(send): keep local media (ugoira/bsky remux MP4) alive across queue retries
A task whose media is a locally produced file (pixiv ugoira MP4, bsky HLS
remux MP4) references a path inside a tempfile TempDir owned by Fetched.
The retry ran after that TempDir was dropped, so the file was already gone
and the retry always failed permanently ("local media file missing") — and
the upload fallback even tried to GET the local path as a URL.

Keep the temp dirs in a process-wide registry (Fetched::take_keep_alive ->
send::KEEP_ALIVE) that is only released when the task settles (sent or
permanently failed); the upload fallback now uploads local files directly
instead of attempting to download them.

Restart-mid-queue still loses the files (documented behavior in
input_file_for) — only the in-process retry path is fixed here.
2026-08-13 22:05:07 +08:00
YoursFunny c496e41c55 feat(handlers): debounce inline queries to stop fetch storms while typing
Telegram fires an inline query on every keystroke and every prefix of a
pasted URL (status/12, status/123, ...) matches the site patterns, so
typing used to trigger a full 3-attempt fetch per keystroke. Answer only
after the query has been stable for 800ms, dedupe repeats through
Telegram's inline cache (explicit cache_time 300), and let a repeat of a
query that produced no answer retry the fetch.
2026-08-13 21:57:07 +08:00
YoursFunny 68f026c990 docs: note version-bump convention in AGENTS.md 2026-08-10 19:29:21 +08:00
YoursFunny 62d80c8905 docs: sync README and AGENTS.md with 1.1.1 state 2026-08-10 19:16:56 +08:00
YoursFunny 78c9c841c6 chore: bump version to 1.1.1 2026-08-10 18:19:16 +08:00
YoursFunny e49d500d23 deps: switch TLS to rustls, drop libssl from the runtime image
OpenSSL came from two removable defaults: teloxide's `default` feature
(native-tls) and x-media's reqwest default features (default-tls). Switch
both to rustls (webpki-roots) so the binary links no system TLS libs:

- teloxide: default-features=false + rustls + ctrlc_handler (was part of
  the removed default)
- reqwest (x-media): default-features=false + rustls-tls

Verified: openssl-sys/native-tls gone from the tree, cargo check clean,
all 5 live twitter/bsky fetches pass over rustls, full container startup
works. The runtime image now needs no libssl.so.3/libcrypto.so.3 or CA
bundle (ffmpeg only processes local files; all downloads go through
reqwest), saving ~8MB.
2026-08-10 17:07:23 +08:00
YoursFunny 6feabd723b docker: fix 1.1.0 container startup (stub binary + missing libssl)
c40b074 broke the image two ways:
- `cargo clean -p` removes 0 files, so the real sources (host mtimes
  older than the step-1 stub build) were never recompiled and the image
  shipped the 337KB fn-main stub, exiting 0 on start. Restore the
  touch-based rebuild, which forces cargo to see every .rs as newer.
- bookworm-slim does not ship libssl3 despite the old comment; the bot
  links OpenSSL via teloxide/reqwest native-tls, so restore the
  libssl/libcrypto copies from the builder (same Debian release).
2026-08-10 16:37:12 +08:00
YoursFunny ea72516d5c chore: bump version to 1.1.0 2026-08-10 10:32:54 +08:00
YoursFunny 755330e585 chore: clippy and rustfmt cleanup on new code 2026-08-08 20:35:43 +08:00
YoursFunny c40b074b3c docker: fail-closed rebuild, ffmpeg checksum, smaller runtime
- Replace the mtime-touch stub-rebuild hack with cargo clean -p (a
  future-dated host file could silently ship the stub binary)
- Optional FFMPEG_SHA256 build arg verified before extraction
- Drop the redundant libssl3/libcrypto copies and the root
  supplementary group; strip the release binary; add a webhook-mode
  healthcheck to the compose example
2026-08-08 20:34:21 +08:00
YoursFunny 9910da2914 deps: drop unused regex, unify reqwest on 0.12
x-media's reqwest 0.13 dragged in quinn/rustls/aws-lc-rs (cmake C
build) alongside teloxide's 0.12; unifying on 0.12 removes the whole
second TLS/QUIC stack from the build and image. The unused regex dep in
xmedia-bot is gone; x-media keeps url (the bsky HLS remux uses it).
2026-08-08 20:31:03 +08:00
YoursFunny ebc0122264 db: enable WAL and index the pending-task lease query
The lease/earliest_run_after queries full-scanned tasks, and the
rollback journal blocked readers behind worker writes. journal_mode=WAL
(persistent, idempotent) plus idx_tasks_pending(status, run_after)
covers both without a schema migration.
2026-08-08 20:26:07 +08:00
YoursFunny 44cba8abe0 send: reuse one process-wide Bot for queue workers
handle_task and dead_letter_notify built a fresh Bot (env parse + HTTP
client) per queue item. A single LazyLock<Bot> is forced at startup so
a missing TELOXIDE_TOKEN fails fast instead of on the first task.
2026-08-08 20:25:22 +08:00
YoursFunny 99009aae9a db: share one now_f64() instead of four private copies
handlers, queue, send and link_cache each carried the same SystemTime
helper; a single crate::db::now_f64() removes the drift risk.
2026-08-08 20:23:58 +08:00
YoursFunny 72130b9023 caption: escape URLs/handles in HTML captions
Post URLs and author URLs were interpolated raw into <a href> attributes
(and the raw user URL from empty_fetched into caption text), so crafted
links could break the HTML parse and fail the send with a 400. All
attribute interpolations now use encode_double_quoted_attribute; text
stays encode_text.
2026-08-08 20:21:56 +08:00
YoursFunny 734cfc2eb3 handlers: skip inline fetches for non-post queries
Inline queries fire per keystroke and each fetch runs the 3-attempt
retry loop; a user typing any text was pumping requests into
X/Pixiv/BSky and risking rate-limit bans. Queries now pass only if
cache_key recognizes them as a supported post URL.
2026-08-08 20:20:05 +08:00
YoursFunny a8156697fa send: keep video thumbnails in the download-and-reupload fallback
media_from_file/media_from_url never attached the thumbnail, so any
video that tripped the fallback lost its cover frame. Both now take
item.thumbnail_url() and apply it, matching build_media_group.
2026-08-08 20:19:19 +08:00
YoursFunny c093dfe5ac handlers: dedup extracted URLs by normalized post id
Exact-string dedup let https://x.com/u/status/1 and
https://x.com/u/status/1/photo/1 (or the same link in text and caption)
through twice, causing duplicate fetches and sends. Dedup now uses
cache_key, falling back to the raw URL for unsupported links.
2026-08-08 20:17:07 +08:00
YoursFunny ee6f3e4a27 state: evict idle chats from the ChatStore cache
prune_expired only shrank edit_message maps, so the cache kept one
ChatData per chat forever (a leak proportional to chat count). Chats
without live edit records are now dropped from the cache and their
per-chat lock (DB row persists; get() reloads). Lock order kept safe:
prune never holds the cache lock while taking the per-chat locks.
2026-08-08 20:16:26 +08:00
YoursFunny 16ed53fead handlers: replace unbounded per-URL spawn with a bounded job channel
The 8-permit semaphore was acquired inside the spawned task, so a burst
queued unlimited tasks (each cloning Bot+Message) and nothing tracked
them at shutdown — in-flight sends fired after the stop notice. URL work
now flows through a 256-slot mpsc drained by 8 workers started from
main; a full channel backpressures the per-chat handler, and shutdown
sets URL_STOP so workers stop pulling.
2026-08-08 20:15:40 +08:00
YoursFunny 1d9e3629c9 state: serialize per-chat get→mutate→set with ChatStore::update
Concurrent handler tasks (the batch-forward design spawns several per
chat) each snapshotted the same ChatData and last-writer-wins silently
dropped mutations — e.g. a second edit_message record, leaving one
prompt's Forward button dead. All write cycles now run under a per-chat
async lock; read-only callers keep get().
2026-08-08 20:12:51 +08:00
YoursFunny b3d87b4f7d send: fail fast when a retried local media file is gone
A retried ugoira/bsky temp MP4 was already deleted with its TempDir, so
the retry failed at multipart-build time with a confusing Io error and
wasted all three attempts. input_file_for now rejects a missing local
path up front as a clean permanent error.
2026-08-08 20:10:35 +08:00
YoursFunny 98c48b99c0 send: don't repeat post-send actions on queue resumes
A resumed SendMediaSequence (batch_index>0 or already-sent ids) ran
post_send_actions again, opening a second edit prompt and inserting a
second edit_message record for the same messages — both Forward buttons
worked, enabling double forwards. Resumes now skip it.
2026-08-08 20:09:58 +08:00
YoursFunny f40639c799 queue: back off 1s when a lease fails
A lease error (e.g. persistent SQLITE_BUSY) while rows are due made the
worker spin with sleep(0), hammering SQLite and flooding the log.
lease_next now returns the error and the loop sleeps 1s before retrying.
2026-08-08 20:09:04 +08:00
YoursFunny 51cc079a85 queue: use notify_one so wakeups are never lost
notify_waiters drops the notification when every worker is between its
DB reads and registering notified(); a task enqueued in that window sat
until a stale timer fired. notify_one stores a permit, so the next
worker to wait wakes immediately and re-leases. stop() still wakes all
workers with notify_waiters.
2026-08-08 20:08:12 +08:00
YoursFunny aa3083792a queue: wire attempt counts into retry backoff
Every network retry hard-coded retry_delay_seconds(0), so backoff was
flat at 1.2-1.8s regardless of attempt; a multi-minute outage dead-
lettered after three rapid tries. The queue now scales the handler's
delay by 2^attempts (cap 300s) before rescheduling.
2026-08-08 20:07:26 +08:00
YoursFunny 6849006ad7 site: honor TELOXIDE_PROXY for site fetches
The shared HTTP client ignored the proxy the Bot API uses, so on
proxy-required networks (e.g. behind the GFW) every site fetch failed
while the bot itself worked. Explicit proxy overrides reqwest's system
detection; unset keeps direct connections.
2026-08-08 20:05:51 +08:00
YoursFunny 042a04ab6e site: anchor pixiv and bsky URL patterns
Both matched URL substrings anywhere in text, so a link like
https://evil.com/?u=pixiv.net/artworks/1 triggered a real fetch and
cache-key pollution. Prefix ^(?:https?://)? like the twitter pattern.
2026-08-08 20:05:09 +08:00
YoursFunny 9f28af4e6b site: classify HTTP status codes, make transient failures retryable
Twitter (syndication + auth GraphQL) mapped every non-2xx to NotFound,
killing retries on 429/5xx; bsky never checked status; pixiv network
errors arrived wrapped in PixivError and were excluded from the retry
loop. New FetchError::Transient covers 429/5xx from all sites, the
retry loop now also retries Pixiv errors, and 404/410 stay permanent.
2026-08-08 20:04:28 +08:00
YoursFunny d61dba5096 send: stream media downloads with hard size caps
download_to_temp now uses download_media_limited: non-photos abort the
moment the 10 MiB upload cap is crossed mid-stream (no more full-body
buffering before the size check), photos cap at the 512 MiB decode
budget, and the ugoira frame zip gets a 512 MiB cap. MediaTooLarge
routes to the existing smaller-URL fallback.
2026-08-08 20:03:14 +08:00
YoursFunny f6df3e28cb pixiv: drop unneeded mut bindings in ugoira extraction 2026-08-08 20:02:06 +08:00
YoursFunny deb1ef2428 site: add total and connect timeouts to the shared HTTP client 2026-08-08 20:01:39 +08:00
YoursFunny b5e5340edc pixiv: harden ugoira zip extraction, degrade instead of panicking
Sniff the frame extension from magic bytes instead of the entry filename,
cap each frame at 64 MiB (declared size + streamed read), and map a
panicked encode worker to the existing degrade path instead of
expect()-panicking the whole fetch handler.
2026-08-08 20:01:14 +08:00
YoursFunny 425d1505cf handlers: fix /set_forward_channel admin checks
Compare the sender's user id (not the chat id, which only matches in
private chats) and require the bot to actually be an admin with post
rights instead of silently passing when it is missing from the list.
Also stops panicking on get_me network failures.
2026-08-08 19:59:27 +08:00
YoursFunny 6e40f55440 queue: recover expired leases at runtime, supervise workers
Rows left in_progress by a panicked/crashed worker were only recovered at
start(); a runtime sweep (30s interval, woken by the same notify) now
re-queues them once the 120s lock TTL expires. Workers run under a
supervisor that respawns a panicked loop instead of silently shrinking
the pool of 4.
2026-08-08 19:58:28 +08:00
YoursFunny 7998114dc3 bsky: remux HLS video playlists to MP4 via ffmpeg
bsky video embeds expose only an m3u8 playlist, which Telegram cannot
fetch. Download the master/variant playlists and TS segments through the
shared client (proxy-aware, size-capped), then concat-remux locally;
keep the temp dir alive via Fetched._keep_alive like the ugoira path.
Also adds site::download_media_limited (streaming size cap), status
checks on media_size, and moves the ffmpeg probe to site/mod.rs for
pixiv/bsky to share.
2026-08-08 19:56:36 +08:00
YoursFunny 9a96f78177 handlers: fix UTF-8 byte-slice panic in message log preview 2026-08-08 19:52:01 +08:00
YoursFunny b50f794d52 feat: register bot commands with Telegram
Call setMyCommands at startup so clients show the command list in the
/ menu. handlers::register_commands wraps Command::bot_commands()
(teloxide derives it from the #[command(description)] attributes);
a registration failure only warns and does not stop the bot.
2026-08-08 00:08:48 +08:00
YoursFunny d32fa969d6 fix: correct singular 'entry' in clear-cache replies
plural() returned "" for one, rendering '1 entr.'; return "y" so
the suffix composes to '1 entry' / '2 entries'.
2026-08-08 00:07:27 +08:00
31 changed files with 3180 additions and 1335 deletions
+64
View File
@@ -0,0 +1,64 @@
name: CI
# Test/lint gate (offline, no secrets) on every push/PR, plus a live-network
# job that exercises the real source sites and the token-gated pixiv tests.
#
# Layering:
# test — fmt + clippy + the full offline unit suite. Runs on every push
# and PR, including forks (it needs no secrets).
# live — the #[ignore]d live-network tests plus the pixiv tests that are
# gated on PIXIV_REFRESH_TOKEN. Runs on schedule / manual dispatch
# / tag pushes only, because pull requests from forks cannot read
# repository secrets. continue-on-error keeps a flaky external site
# from blocking, while the run still records the outcome.
#
# Test gating convention (keep in sync with AGENTS.md "Testing & QA"):
# - pure unit tests: plain #[test] / #[tokio::test], always run.
# - live-network tests: #[ignore = "live network: ..."], only run here.
# - token-gated tests (pixiv): #[tokio::test] with an early return when
# PIXIV_REFRESH_TOKEN is absent or empty (empty = unset CI secret).
on:
push:
branches: [master]
pull_request:
schedule:
# Weekly probe of the live endpoints, so external API changes surface.
- cron: '0 3 * * 1'
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
with:
components: clippy, rustfmt
- uses: Swatinem/rust-cache@v2
- name: Check formatting
run: cargo fmt --check
- name: Lint (deny warnings)
run: cargo clippy --workspace --all-targets -- -D warnings
- name: Run offline tests
run: cargo test --workspace
live:
needs: test
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/v')
runs-on: ubuntu-latest
continue-on-error: true
env:
PIXIV_REFRESH_TOKEN: ${{ secrets.PIXIV_REFRESH_TOKEN }}
TWITTER_AUTH_TOKEN: ${{ secrets.TWITTER_AUTH_TOKEN }}
steps:
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
# Full suite: with the secret present, the pixiv token-gated tests run;
# without it they skip themselves. Live tests stay #[ignore]d here.
- name: Run token-gated tests
run: cargo test --workspace
# The live-network tests, by the "live" name filter (all #[ignore]d).
- name: Run live-network tests
run: cargo test --workspace -- --ignored live
+18 -17
View File
@@ -4,7 +4,7 @@
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README and user-facing strings are in Chinese. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
Two-crate Cargo workspace (both v1.0.3, edition 2024, resolver 3):
Two-crate Cargo workspace (both v1.2.2, edition 2024, resolver 3):
- **`crates/x-media`** — library that fetches and normalizes media from the three sites. Pure, no Telegram knowledge.
- **`crates/xmedia-bot`** — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.
@@ -18,19 +18,19 @@ Telegram update → Dispatcher (polling or axum webhook) → dptree branches
└─ callback_query → "forward" (copy to channel) / "template|<name>" (apply caption template)
```
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)``Fetched` → builds a `Task``send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → single worker leases (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)``Fetched` → builds a `Task``send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
The `x-media` library: `site::fetch(url)` dispatches (in order) twitter → bsky → pixiv via per-site regex `PATTERN` and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → pixiv) and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, site_id, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
## Key Directories
| Path | Purpose |
|---|---|
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, site struct, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`) |
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, queue worker start, pixiv validation, 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`) |
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
| `crates/xmedia-bot/src/handlers.rs` | `Command` enum (teloxide `BotCommands`), message/inline/callback handlers, URL extraction, global statics; per-URL work spawned with a `Semaphore(8)` cap (teloxide's per-chat workers are sequential — batch-forwards need concurrency) |
| `crates/xmedia-bot/src/handlers.rs` | `Command` enum (teloxide `BotCommands`), message/inline/callback handlers, URL extraction, global statics; per-URL work flows through a bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns (teloxide's per-chat workers are sequential — batch-forwards need concurrency) |
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table) |
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections |
@@ -48,19 +48,19 @@ cargo clippy --workspace --all-targets # lint (Clippy is the configured IDE lint
cargo fmt --check # formatting
```
Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb`. Runtime requires **ffmpeg** (built into the image).
Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb`. Runtime requires **ffmpeg** (built into the image). The builder fetches crates.io + ffmpeg; on restricted networks pass proxy build args, e.g. `--build-arg HTTP_PROXY=http://host.docker.internal:10808 --build-arg HTTPS_PROXY=…` (Docker Desktop builds can't reach the host loopback — use `host.docker.internal`).
## Code Conventions & Common Patterns
- **No anyhow/thiserror.** Errors are hand-rolled enums with manual `Display`/`source()`/`From` impls: `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `FetchError` (`Http`/`Json`/`Pixiv`/`NotFound`/`Blocked`), `PixivError`, `Classification`. New errors should follow this pattern.
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers rebuild `Bot::from_env()`.
- **Errors via `thiserror` derive** (no anyhow): the public, stringified errors — `FetchError` (`Http`/`Json`/`Pixiv`/`Site`/`NotFound`/`Blocked`) and `PixivError` — derive `thiserror::Error` with `#[from]` conversions; `Display`/`source()` come from the derive. The internal control-flow enums — `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `Classification`, `FallbackError` — carry no `Display` and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive `thiserror`, internal flow enums stay plain.
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers share the process-wide `send::BOT` (`LazyLock<Bot>`, force-initialized in `main` so a missing token fails at startup).
- **Async**: tokio multi-thread runtime (`#[tokio::main]` default). All rusqlite I/O inside `tokio::task::spawn_blocking`. Long loops use `tokio::select!` with `tokio::sync::{watch, Notify}` stop/wake channels. No streams.
- **Blocking sync primitives**: `parking_lot::Mutex` for hot caches, `tokio::sync::Mutex` for async-shared state (pixiv token cache), `AtomicBool` for feature gates.
- **Site adapter convention** (no trait, no enum dispatch — follow the existing convention): each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`; `site/mod.rs` re-exports the site struct and `fetch_once` adds one guarded if-branch. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one branch in `fetch_once`.
- **Site adapter convention**: each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`, plus `cache_key`/`is_retryable`/`media_headers`, and a unit struct `<Name>Site` implementing `site::Site`; the central dispatcher (`site/mod.rs`) only iterates the `SITES` registry. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one `Box::new(...)` entry in `SITES` — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return `SiteFuture` (a boxed `Pin<Box<dyn Future + Send>>`) because `async fn` in traits is not dyn-compatible.
- **Serde**: per-site `model.rs` are pure `Deserialize` DTOs mirroring API JSON; site structs in `interface.rs` have private fields, a `caption()` builder, and `impl From<SiteStruct> for Fetched`. Persisted payloads use internally-tagged enums (`#[serde(tag = "kind")]` / `type`).
- **Naming**: module-per-concern, snake_case files, `CamelCase` types, `snake_case` fns. `//!` module docs and `///` docs on non-obvious logic (syndication token, ugoira encoding, `display_text_range`).
- **Retries**: only `x-media::site::fetch` retries (3 attempts, `1 << attempt` backoff, HTTP errors only). Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`).
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`).
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`). Level convention: `info` = lifecycle + per-post business results (`sent`/`forwarded`/`copied`), admin/operator actions and anomalies (fallback, retry enqueue, dead-letter is `error`); `debug` = per-request detail (message/command/URL extraction, `fetching`/`fetched`, batch sends, queue processing, photo processing, inline queries). Full user-submitted URLs and message text only appear at `debug`; at `info` and above links are printed via the normalized cache key (`handlers::log_key`, e.g. `[key=twitter:123...]`) so logs stay short and do not echo user data.
## Important Files
@@ -72,7 +72,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL |
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) |
| `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` |
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime hack, static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint |
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime bump (cargo's freshness is mtime-based and `cargo clean -p` removes 0 files — the touch is what forces the real sources to rebuild while deps stay cached), static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, optional `FFMPEG_SHA256` checksum, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint. Runtime ships **no libssl/libcrypto/CA bundle** — rustls webpki-roots handles all TLS, and the static ffmpeg only processes local files (downloads go through reqwest) |
| `docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim) |
| `docker-compose.yml.example` | Deployment env reference (real `docker-compose.yml` is gitignored). Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) |
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master; **no test step**; buildx gha cache (`cache-from`/`cache-to`, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs |
@@ -82,7 +82,8 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
- **Rust, stable, edition 2024**, workspace resolver 3. No `rust-version`/MSRV pin, no `rust-toolchain.toml` — recent stable is assumed. No nightly features.
- Package manager: **Cargo** (workspace with path dep `x-media``xmedia-bot`). No `[workspace.package]`/shared deps — each crate lists deps independently.
- **Two reqwest versions coexist in the lock** (0.12.28 via teloxide, 0.13.3 in x-media) — don't unify casually.
- **TLS is rustls end-to-end** (no native-tls/openssl in the tree, no libssl in the Docker runtime image): `teloxide` is declared `default-features = false` with `["webhooks-axum", "macros", "rustls", "ctrlc_handler"]` (the removed `default` also carried `native-tls` and `ctrlc_handler` — the latter must stay); x-media's reqwest is `default-features = false` with `["json", "rustls-tls"]` (webpki-roots baked in, so the image ships no CA bundle). One reqwest 0.12.28 in the lock.
- **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag (the tag push triggers the Docker Hub build).
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `data/task_queue.db` is CWD-relative — run from the workspace root, or `/app` in Docker. Mount `./data` and `./cert` volumes.
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `docker-compose.yml`, `/target`, `.idea/`.
@@ -90,10 +91,10 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
## Testing & QA
- **~51 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- **~80 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches.
- Live-network tests exist in `site/twitter/interface.rs` (3), `site/bsky/interface.rs` (2), `site/pixiv/interface.rs`/`api.rs` (env-gated on `PIXIV_REFRESH_TOKEN`/dotenv, skip by early return). Run the full suite with `cargo test --workspace`.
- Live-network tests exist in `site/twitter/interface.rs` (3), `site/bsky/interface.rs` (2), `site/pixiv/interface.rs`/`api.rs`. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""``is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`.
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
- **CI runs no tests** — `.github/workflows/docker.yml` only builds/pushes the image; verification is a local responsibility.
- **CI** — `.github/workflows/ci.yml` runs `cargo fmt --check` + `cargo clippy --workspace --all-targets -- -D warnings` + `cargo test --workspace` (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN` from secrets, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds/pushes the image only.
- Untested and hard to test without a mock seam: `handlers.rs` (depends directly on teloxide `Bot`); `main.rs`, `config.rs`, `state.rs`; `media.rs`, `lib.rs`, all `model.rs`.
- No coverage tracking, no lint gate in CI.
- No coverage tracking.
Generated
+69 -494
View File
@@ -16,7 +16,7 @@ checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
dependencies = [
"cfg-if",
"cipher",
"cpufeatures",
"cpufeatures 0.2.17",
]
[[package]]
@@ -99,28 +99,6 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
[[package]]
name = "aws-lc-rs"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00"
dependencies = [
"aws-lc-sys",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
]
[[package]]
name = "axum"
version = "0.8.9"
@@ -266,9 +244,20 @@ checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
version = "0.2.1"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
[[package]]
name = "chacha20"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"rand_core 0.10.1",
]
[[package]]
name = "chrono"
@@ -292,15 +281,6 @@ dependencies = [
"inout",
]
[[package]]
name = "cmake"
version = "0.1.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
dependencies = [
"cc",
]
[[package]]
name = "colored"
version = "3.1.1"
@@ -310,42 +290,12 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "combine"
version = "4.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd"
dependencies = [
"bytes",
"memchr",
]
[[package]]
name = "constant_time_eq"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6"
[[package]]
name = "core-foundation"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "core-foundation"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "core-foundation-sys"
version = "0.8.7"
@@ -361,6 +311,15 @@ dependencies = [
"libc",
]
[[package]]
name = "cpufeatures"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
dependencies = [
"libc",
]
[[package]]
name = "crc"
version = "3.4.0"
@@ -530,12 +489,6 @@ dependencies = [
"futures",
]
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
[[package]]
name = "dyn-clone"
version = "1.0.20"
@@ -548,15 +501,6 @@ version = "1.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
[[package]]
name = "encoding_rs"
version = "0.8.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3"
dependencies = [
"cfg-if",
]
[[package]]
name = "env_logger"
version = "0.10.2"
@@ -651,33 +595,12 @@ dependencies = [
"miniz_oxide",
]
[[package]]
name = "fnv"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
[[package]]
name = "foldhash"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
[[package]]
name = "foreign-types"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1"
dependencies = [
"foreign-types-shared",
]
[[package]]
name = "foreign-types-shared"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b"
[[package]]
name = "form_urlencoded"
version = "1.2.2"
@@ -687,12 +610,6 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "futures"
version = "0.3.32"
@@ -825,29 +742,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"r-efi 6.0.0",
"rand_core 0.10.1",
"wasip2",
"wasip3",
]
[[package]]
name = "h2"
version = "0.4.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "171fefbc92fe4a4de27e0698d6a5b392d6a0e333506bc49133760b3bcf948733"
dependencies = [
"atomic-waker",
"bytes",
"fnv",
"futures-core",
"futures-sink",
"http",
"indexmap 2.14.0",
"slab",
"tokio",
"tokio-util",
"tracing",
"wasm-bindgen",
]
[[package]]
@@ -986,7 +887,6 @@ dependencies = [
"bytes",
"futures-channel",
"futures-core",
"h2",
"http",
"http-body",
"httparse",
@@ -1011,22 +911,7 @@ dependencies = [
"tokio",
"tokio-rustls",
"tower-service",
]
[[package]]
name = "hyper-tls"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "70206fc6890eaca9fde8a0bf71caa2ddfc9fe045ac9e5c70df101a7dbde866e0"
dependencies = [
"bytes",
"http-body-util",
"hyper",
"hyper-util",
"native-tls",
"tokio",
"tokio-native-tls",
"tower-service",
"webpki-roots",
]
[[package]]
@@ -1047,11 +932,9 @@ dependencies = [
"percent-encoding",
"pin-project-lite",
"socket2",
"system-configuration",
"tokio",
"tower-service",
"tracing",
"windows-registry",
]
[[package]]
@@ -1276,55 +1159,6 @@ version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "jni"
version = "0.22.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498"
dependencies = [
"cfg-if",
"combine",
"jni-macros",
"jni-sys",
"log",
"simd_cesu8",
"thiserror",
"walkdir",
"windows-link",
]
[[package]]
name = "jni-macros"
version = "0.22.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3"
dependencies = [
"proc-macro2",
"quote",
"rustc_version",
"simd_cesu8",
"syn",
]
[[package]]
name = "jni-sys"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2"
dependencies = [
"jni-sys-macros",
]
[[package]]
name = "jni-sys-macros"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264"
dependencies = [
"quote",
"syn",
]
[[package]]
name = "jobserver"
version = "0.1.34"
@@ -1485,23 +1319,6 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "native-tls"
version = "0.2.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "465500e14ea162429d264d44189adc38b199b62b1c21eea9f69e4b73cb03bbf2"
dependencies = [
"libc",
"log",
"openssl",
"openssl-probe",
"openssl-sys",
"schannel",
"security-framework",
"security-framework-sys",
"tempfile",
]
[[package]]
name = "num-conv"
version = "0.2.1"
@@ -1532,49 +1349,6 @@ version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "openssl"
version = "0.10.79"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf0b434746ee2832f4f0baf10137e1cabb18cbe6912c69e2e33263c45250f542"
dependencies = [
"bitflags",
"cfg-if",
"foreign-types",
"libc",
"openssl-macros",
"openssl-sys",
]
[[package]]
name = "openssl-macros"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "openssl-probe"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
[[package]]
name = "openssl-sys"
version = "0.9.115"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "158fe5b292746440aa6e7a7e690e55aeb72d41505e2804c23c6973ad0e9c9781"
dependencies = [
"cc",
"libc",
"pkg-config",
"vcpkg",
]
[[package]]
name = "parking_lot"
version = "0.12.5"
@@ -1745,9 +1519,9 @@ dependencies = [
[[package]]
name = "quinn"
version = "0.11.9"
version = "0.11.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8"
dependencies = [
"bytes",
"cfg_aliases",
@@ -1765,15 +1539,15 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.14"
version = "0.11.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
dependencies = [
"aws-lc-rs",
"bytes",
"getrandom 0.3.4",
"getrandom 0.4.2",
"lru-slab",
"rand 0.9.4",
"rand 0.10.2",
"rand_pcg",
"ring",
"rustc-hash",
"rustls",
@@ -1787,9 +1561,9 @@ dependencies = [
[[package]]
name = "quinn-udp"
version = "0.5.14"
version = "0.5.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
dependencies = [
"cfg_aliases",
"libc",
@@ -1827,18 +1601,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a"
dependencies = [
"libc",
"rand_chacha 0.3.1",
"rand_chacha",
"rand_core 0.6.4",
]
[[package]]
name = "rand"
version = "0.9.4"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea"
checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
dependencies = [
"rand_chacha 0.9.0",
"rand_core 0.9.5",
"chacha20",
"getrandom 0.4.2",
"rand_core 0.10.1",
]
[[package]]
@@ -1851,16 +1626,6 @@ dependencies = [
"rand_core 0.6.4",
]
[[package]]
name = "rand_chacha"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
dependencies = [
"ppv-lite86",
"rand_core 0.9.5",
]
[[package]]
name = "rand_core"
version = "0.6.4"
@@ -1872,11 +1637,17 @@ dependencies = [
[[package]]
name = "rand_core"
version = "0.9.5"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
[[package]]
name = "rand_pcg"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a"
dependencies = [
"getrandom 0.3.4",
"rand_core 0.10.1",
]
[[package]]
@@ -1960,21 +1731,22 @@ dependencies = [
"http-body",
"http-body-util",
"hyper",
"hyper-tls",
"hyper-rustls",
"hyper-util",
"js-sys",
"log",
"mime_guess",
"native-tls",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls",
"rustls-pki-types",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tokio-native-tls",
"tokio-rustls",
"tokio-util",
"tower",
"tower-http",
@@ -1984,47 +1756,7 @@ dependencies = [
"wasm-bindgen-futures",
"wasm-streams",
"web-sys",
]
[[package]]
name = "reqwest"
version = "0.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62e0021ea2c22aed41653bc7e1419abb2c97e038ff2c33d0e1309e49a97deec0"
dependencies = [
"base64",
"bytes",
"encoding_rs",
"futures-core",
"h2",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-util",
"js-sys",
"log",
"mime",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls",
"rustls-pki-types",
"rustls-platform-verifier",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tokio-rustls",
"tower",
"tower-http",
"tower-service",
"url",
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
"webpki-roots",
]
[[package]]
@@ -2066,18 +1798,9 @@ dependencies = [
[[package]]
name = "rustc-hash"
version = "2.1.2"
version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
[[package]]
name = "rustc_version"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
dependencies = [
"semver",
]
checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
[[package]]
name = "rustix"
@@ -2098,26 +1821,14 @@ version = "0.23.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b"
dependencies = [
"aws-lc-rs",
"once_cell",
"ring",
"rustls-pki-types",
"rustls-webpki",
"subtle",
"zeroize",
]
[[package]]
name = "rustls-native-certs"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63"
dependencies = [
"openssl-probe",
"rustls-pki-types",
"schannel",
"security-framework",
]
[[package]]
name = "rustls-pki-types"
version = "1.14.1"
@@ -2128,40 +1839,12 @@ dependencies = [
"zeroize",
]
[[package]]
name = "rustls-platform-verifier"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
dependencies = [
"core-foundation 0.10.1",
"core-foundation-sys",
"jni",
"log",
"once_cell",
"rustls",
"rustls-native-certs",
"rustls-platform-verifier-android",
"rustls-webpki",
"security-framework",
"security-framework-sys",
"webpki-root-certs",
"windows-sys 0.61.2",
]
[[package]]
name = "rustls-platform-verifier-android"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
[[package]]
name = "rustls-webpki"
version = "0.103.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted",
@@ -2179,24 +1862,6 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "same-file"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
dependencies = [
"winapi-util",
]
[[package]]
name = "schannel"
version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "schemars"
version = "0.9.0"
@@ -2227,29 +1892,6 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
[[package]]
name = "security-framework"
version = "3.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
dependencies = [
"bitflags",
"core-foundation 0.10.1",
"core-foundation-sys",
"libc",
"security-framework-sys",
]
[[package]]
name = "security-framework-sys"
version = "2.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "semver"
version = "1.0.28"
@@ -2361,7 +2003,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8"
dependencies = [
"cfg-if",
"cpufeatures",
"cpufeatures 0.2.17",
"digest",
]
@@ -2387,22 +2029,6 @@ version = "0.3.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
[[package]]
name = "simd_cesu8"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94f90157bb87cddf702797c5dadfa0be7d266cdf49e22da2fcaa32eff75b2c33"
dependencies = [
"rustc_version",
"simdutf8",
]
[[package]]
name = "simdutf8"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
[[package]]
name = "slab"
version = "0.4.12"
@@ -2487,27 +2113,6 @@ dependencies = [
"syn",
]
[[package]]
name = "system-configuration"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b"
dependencies = [
"bitflags",
"core-foundation 0.9.4",
"system-configuration-sys",
]
[[package]]
name = "system-configuration-sys"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "take_mut"
version = "0.2.2"
@@ -2567,7 +2172,7 @@ dependencies = [
"once_cell",
"pin-project",
"rc-box",
"reqwest 0.12.28",
"reqwest",
"rgb",
"serde",
"serde_json",
@@ -2719,16 +2324,6 @@ dependencies = [
"syn",
]
[[package]]
name = "tokio-native-tls"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2"
dependencies = [
"native-tls",
"tokio",
]
[[package]]
name = "tokio-rustls"
version = "0.26.4"
@@ -2914,16 +2509,6 @@ version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "walkdir"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
dependencies = [
"same-file",
"winapi-util",
]
[[package]]
name = "want"
version = "0.3.1"
@@ -3080,10 +2665,10 @@ dependencies = [
]
[[package]]
name = "webpki-root-certs"
version = "1.0.7"
name = "webpki-roots"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c"
checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed"
dependencies = [
"rustls-pki-types",
]
@@ -3138,17 +2723,6 @@ version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-registry"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720"
dependencies = [
"windows-link",
"windows-result",
"windows-strings",
]
[[package]]
name = "windows-result"
version = "0.4.1"
@@ -3351,7 +2925,7 @@ checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
[[package]]
name = "x-media"
version = "1.0.8"
version = "1.2.2"
dependencies = [
"bytes",
"dotenv",
@@ -3359,10 +2933,11 @@ dependencies = [
"log",
"rand 0.8.6",
"regex",
"reqwest 0.13.3",
"reqwest",
"serde",
"serde_json",
"tempfile",
"thiserror",
"tokio",
"url",
"zip",
@@ -3370,8 +2945,9 @@ dependencies = [
[[package]]
name = "xmedia-bot"
version = "1.0.8"
version = "1.2.2"
dependencies = [
"bytes",
"dotenv",
"fast_image_resize",
"html-escape",
@@ -3381,7 +2957,6 @@ dependencies = [
"png",
"pretty_env_logger",
"rand 0.8.6",
"regex",
"rusqlite",
"serde",
"serde_json",
+9
View File
@@ -1,3 +1,12 @@
[workspace]
members = ["crates/x-media", "crates/xmedia-bot"]
resolver = "3"
# Smaller/faster production binary: strip debug symbols, link-time
# optimization across crates, and one codegen unit per crate (bigger LTO
# wins). panic=abort is intentionally NOT set: queue workers and db
# closures rely on JoinHandle catching panics, which abort would defeat.
[profile.release]
strip = true
lto = "thin"
codegen-units = 1
+17 -9
View File
@@ -11,6 +11,11 @@ ARG APP_NAME=telegram-twitter-media-bot
# runners. `/redirect/latest/` floats to the newest release build; each build
# also ships a .sha256. Swap `amd64` for `arm64` when building arm64 images.
ARG FFMPEG_URL=https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffmpeg.zip
# Optional sha256 of ffmpeg.zip (pinned releases only): set to verify the
# download. The mirror publishes .sha256 sidecars next to pinned builds, e.g.
# https://ffmpeg.martin-riedl.de/download/linux/amd64/<id>_9.0/ffmpeg.zip.sha256
# (the /redirect/latest/ URL itself has no sidecar — pin the effective URL).
ARG FFMPEG_SHA256=
WORKDIR /build
@@ -30,17 +35,20 @@ RUN mkdir -p crates/x-media/src crates/xmedia-bot/src \
# root. `unzip -t` verifies the archive before extraction so a bad
# download fails loudly here instead of a cryptic later error.
RUN wget -q -O /tmp/ffmpeg.zip "$FFMPEG_URL" \
&& if [ -n "$FFMPEG_SHA256" ]; then echo "$FFMPEG_SHA256 /tmp/ffmpeg.zip" | sha256sum -c -; fi \
&& unzip -tq /tmp/ffmpeg.zip \
&& unzip -q /tmp/ffmpeg.zip -d /usr/local/bin \
&& chmod +x /usr/local/bin/ffmpeg \
&& rm /tmp/ffmpeg.zip \
&& /usr/local/bin/ffmpeg -version >/dev/null
# 3. Real sources last: only our crates recompile on source changes. The
# COPY preserves host mtimes, which predate the stub artifacts from step 1;
# cargo's mtime-based freshness check would otherwise treat the stub build
# as up-to-date and never compile the real sources. `touch` forces cargo to
# see the real files as newer.
# 3. Real sources last: only our crates recompile on source changes. Cargo's
# freshness check is mtime-based; the COPY'd host files usually predate the
# step-1 stub build, so cargo would consider the stub up to date and never
# compile the real sources. `touch` makes every .rs newer than the stub
# artifacts, forcing a rebuild of just the two crates while the compiled
# dependency layer stays cached. (`cargo clean -p` does NOT work here — it
# removes 0 files and the stub binary silently ships.)
COPY crates/ ./crates/
RUN find crates -type f -name '*.rs' -exec touch {} + \
&& cargo build --release -p xmedia-bot
@@ -56,10 +64,10 @@ LABEL org.opencontainers.image.title="${APP_NAME}"
# Everything is copied in — no apt in the runtime stage. Privilege dropping is
# done by docker-entrypoint.sh with setpriv (util-linux, already in
# bookworm-slim), so no gosu needed.
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder /usr/lib/x86_64-linux-gnu/libssl.so.3* /usr/lib/x86_64-linux-gnu/
COPY --from=builder /usr/lib/x86_64-linux-gnu/libcrypto.so.3* /usr/lib/x86_64-linux-gnu/
# bookworm-slim), so no gosu needed. TLS is rustls (webpki-roots baked in,
# see Cargo.toml feature `rustls`/`rustls-tls`), so no system CA bundle or
# libssl are needed; the static ffmpeg only processes local files (all
# downloads go through reqwest).
COPY --from=builder /usr/local/bin/ffmpeg /usr/local/bin/ffmpeg
WORKDIR /app
+123
View File
@@ -0,0 +1,123 @@
# TelegramXMediaBot
A Telegram bot that turns post links from X / Twitter, Pixiv, and Bluesky into media messages (images, video, GIF) with the post's title, author, and tags.
## Features
- Sending a link in a private chat fetches and sends the images, videos and GIFs automatically; oversized media is split into batches
- Text-only posts report "no media"; unsupported links are silently ignored
- Inline queries (`@bot <link>`)
- Bind a forward channel for automatic forwarding; edit the caption before forwarding and apply custom templates
- Failed sends are retried automatically with persistence; the user is notified after retries are exhausted
- Pixiv ugoira animations are transcoded to MP4; Bluesky videos are remuxed (HLS stream → MP4)
- Photos exceeding Telegram's size/dimension limits are compressed automatically (original format kept, JPEG fallback only when needed)
- Link-result cache: after a successful send the Telegram file ids and caption fields are cached locally, so a repeated link is re-sent from local state — no source-site request, no media file stored (expiry controlled by `LINK_CACHE_TTL_SECONDS`, default 7 days)
## Quick start
```bash
# Required: BotFather token; optional: PIXIV_REFRESH_TOKEN (Pixiv is disabled without it)
export TELOXIDE_TOKEN=<token>
export PIXIV_REFRESH_TOKEN=<token>
cargo run -p xmedia-bot
```
Docker deployment (see `docker-compose.yml.example`):
```bash
docker build -t tgxmb .
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
```
Environment variables: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `BOT_ADMIN`, `EDIT_MESSAGE_TTL_SECONDS`, `LINK_CACHE_TTL_SECONDS`, `RUST_LOG`, `WEBHOOK*`, `TWITTER_AUTH_TOKEN` (optional).
NSFW tweets: the public syndication endpoint does not return sensitive content. Setting `TWITTER_AUTH_TOKEN` (the `auth_token` cookie value of a logged-in x.com session) lets the bot fetch NSFW media in the logged-in state only when it hits a withheld tweet; without it, the bot reports no media.
### Webhook deployment (needs a reverse proxy)
`docker-compose.yml.example` ships an [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) reverse-proxy orchestration. Pick one deployment shape:
**With a domain**
1. Point a DNS A record at the server
2. In compose set `VIRTUAL_HOST` and `WEBHOOK_URL` to the domain, and uncomment `ACME_HOST` (set it to the domain)
3. acme-companion issues and renews certificates automatically — nothing manual
**IP only**
Let's Encrypt can issue certificates for public IPs (available since 2026, validity ~7 days, requires the `shortlived` profile). Use [acme.sh](https://github.com/acmesh-official/acme.sh) to issue and renew automatically, no manual certificates:
1. Add an acme-ip service to compose (issue + daily auto-renewal check):
```yaml
acme-ip:
image: neilpang/acme.sh
container_name: acme-ip
command: daemon
restart: always
volumes:
- certs:/acme.sh
- html:/usr/share/nginx/html
- /var/run/docker.sock:/var/run/docker.sock:ro
networks: [proxy]
```
2. First issuance (replace `<SERVER_IP>` with the server's public IP; IPv6 works too, repeat `-d` for more):
```bash
docker compose exec acme-ip acme.sh --issue --server letsencrypt \
-d <SERVER_IP> --cert-profile shortlived --days 3 \
--webroot /usr/share/nginx/html \
--install-cert --cert-file /acme.sh/<SERVER_IP>.crt \
--key-file /acme.sh/<SERVER_IP>.key \
--reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP"
```
3. In compose set `VIRTUAL_HOST: '<SERVER_IP>'` and `WEBHOOK_URL: 'https://<SERVER_IP>/'`; no `WEBHOOK_CERT` needed. Renewal is handled by the acme.sh daemon (`--days 3` = renew every 3 days, buffer against the 7-day validity), and a successful renewal HUP-notifies nginx-proxy to load the new certificate.
Limitations: certificate validity ~7 days; only http-01/tls-alpn-01 validation (port 80 must be publicly reachable); no DNS-01, private IPs or IP ranges; at most 5 certificates per 168 hours for the same IP set. It is recommended to trial-issue with `--server letsencrypt_test` first, then switch to the production server.
Telegram only accepts ports 443/80/88/8443.
<details>
<summary>Environment variables</summary>
| Variable | Description |
|---|---|
| `TELOXIDE_TOKEN` | Bot token (required) |
| `PIXIV_REFRESH_TOKEN` | Pixiv refresh token; Pixiv is disabled without it |
| `BOT_ADMIN` | Admin chat IDs, comma-separated; receives start/stop notifications |
| `EDIT_MESSAGE_TTL_SECONDS` | Edit-before-forward record expiry in seconds, default 86400 |
| `LINK_CACHE_TTL_SECONDS` | Link-result cache expiry in seconds, default 604800 (7 days) |
| `RUST_LOG` | Log level |
| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW) |
| `LOCAL_USER_ID` | UID the container runs as, default 9001 |
| `VIRTUAL_HOST` | Public domain or IP; nginx-proxy routes by this |
| `VIRTUAL_PORT` | Port the bot listens on inside the container; nginx-proxy's forwarding target |
| `ACME_HOST` | Domain deployment: when set to the domain, acme-companion issues/renews certificates automatically |
| `DEFAULT_HOST` | nginx-proxy routes requests with unknown Host headers to this vhost (needed for IP access) |
| `DEFAULT_EMAIL` | acme-companion certificate notification email |
| `WEBHOOK` | `true` enables webhook mode (polling by default) |
| `WEBHOOK_LISTEN` / `WEBHOOK_PORT` | Listen address/port inside the bot container |
| `WEBHOOK_URL` | Public HTTPS URL (`https://domain/` or `https://IP/`) |
| `WEBHOOK_CERT` | Optional; self-signed certificate path, only used for Telegram-side validation (TLS is terminated by the reverse proxy) |
| `WEBHOOK_SECRET_TOKEN` | Update validation token (`X-Telegram-Bot-Api-Secret-Token`) |
</details>
## Commands
| Command | Description |
|---|---|
| `/start` | Welcome message |
| `/help` | List all commands and usage (this command table) |
| `/set_forward_channel <channel>` | Set the forward channel: `@channel` or channel ID; media messages are forwarded to it automatically afterwards |
| `/remove_forward_channel` | Remove the forward channel |
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or taps a template button to apply one) |
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward") |
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/clear_cache [link]` | Clear the link cache (admin only); with a link only that entry, otherwise everything |
| `/bot_dict` | Show the current chat state (debugging) |
Link processing works only in private chats; commands work in any chat.
## Notes
- State is persisted in `data/task_queue.db`; compose deployments use the bind mount `./data` (keep it a directory for easy backups)
- The runtime needs ffmpeg (built into the Docker image)
- Tests: `cargo test --workspace`
+5 -1
View File
@@ -9,7 +9,8 @@ Telegram 机器人,将 X / Twitter、Pixiv、Bluesky 的帖子链接转换为
- 支持内联查询(`@机器人 <链接>`
- 可绑定转发频道自动转发;支持转发前编辑 caption 与自定义模板
- 发送失败自动重试并持久化,重试耗尽后通知用户
- Pixiv ugoira 动图自动转码为 MP4
- Pixiv ugoira 动图自动转码为 MP4Bluesky 视频自动转码(HLS 流 → MP4)
- 超过 Telegram 尺寸/大小限制的图片自动压缩(保持原格式,必要时转 JPEG)
- 链接结果本地缓存:成功发送后缓存 Telegram file id 与 caption 等,再次收到相同链接直接本地重发,不再请求源站、不保存媒体文件(`LINK_CACHE_TTL_SECONDS` 控制过期,默认 7 天)
## 快速开始
@@ -84,6 +85,7 @@ Telegram 只接受 443/80/88/8443 端口。
| `EDIT_MESSAGE_TTL_SECONDS` | 转发前编辑记录过期秒数,默认 86400 |
| `LINK_CACHE_TTL_SECONDS` | 链接结果缓存过期秒数,默认 604800(7 天) |
| `RUST_LOG` | 日志级别 |
| `TELOXIDE_PROXY` | HTTP 代理(如 `http://127.0.0.1:10808`);同时作用于 Telegram Bot API 与站点抓取请求,网络受限环境(如 GFW)必需 |
| `LOCAL_USER_ID` | 容器内运行用户 UID,默认 9001 |
| `VIRTUAL_HOST` | 对外域名或 IPnginx-proxy 按此路由 |
| `VIRTUAL_PORT` | bot 容器内监听端口,nginx-proxy 的转发目标 |
@@ -93,6 +95,7 @@ Telegram 只接受 443/80/88/8443 端口。
| `WEBHOOK` | `true` 启用 webhook 模式(默认轮询) |
| `WEBHOOK_LISTEN` / `WEBHOOK_PORT` | bot 容器内监听地址/端口 |
| `WEBHOOK_URL` | 对外公网 HTTPS 地址(`https://域名/` 或 `https://IP/` |
| `WEBHOOK_CERT` | 可选;自签名证书路径,仅用于 Telegram 侧验证(TLS 由反向代理终止) |
| `WEBHOOK_SECRET_TOKEN` | 更新校验令牌(`X-Telegram-Bot-Api-Secret-Token` |
</details>
@@ -108,6 +111,7 @@ Telegram 只接受 443/80/88/8443 端口。
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/clear_cache [链接]` | 清空链接缓存(仅管理员);带链接只清该条,否则清空全部 |
| `/bot_dict` | 查看当前聊天状态(调试用) |
链接处理仅限私聊;命令在任意聊天可用。
+3 -2
View File
@@ -1,10 +1,10 @@
[package]
name = "x-media"
version = "1.0.8"
version = "1.2.2"
edition = "2024"
[dependencies]
reqwest = { version = "0.13", features = ["json", "query", "form"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
regex = "1.12"
@@ -13,6 +13,7 @@ url = "2.5.2"
bytes = "1"
zip = "2"
tempfile = "3"
thiserror = "2"
rand = "0.8"
log = "0.4"
tokio = { version = "1.40", features = ["time"] }
+214 -8
View File
@@ -1,12 +1,34 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use crate::site::{FetchError, Fetched, Site, SiteFuture};
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
pub static PATTERN: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap());
/// Registry entry for the bluesky adapter (see [`crate::site::Site`]).
pub struct BskySite;
impl Site for BskySite {
fn id(&self) -> &'static str {
"bsky"
}
fn pattern(&self) -> &'static Regex {
&PATTERN
}
fn cache_key(&self, url: &str) -> Option<String> {
cache_key(url)
}
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched> {
Box::pin(async move { fetch_from_url(url).await })
}
}
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"^(?:https?://)?bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap()
});
pub fn enabled() -> bool {
true
@@ -22,7 +44,180 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
.get(2)
.map(|m| m.as_str())
.ok_or(FetchError::NotFound)?;
Ok(fetch(handle, rkey).await?.into())
let post = fetch(handle, rkey).await?;
let mut fetched: Fetched = post.into();
// bsky video embeds expose only an HLS playlist URL, which Telegram
// cannot fetch; remux it to a single MP4 (mirrors the pixiv ugoira
// encode path — the temp file stays alive via `_keep_alive`). On any
// failure the video item is dropped and the post degrades to its text.
let mut media = Vec::with_capacity(fetched.media.len());
for item in fetched.media {
let is_hls = matches!(&item, Media::Video { url, .. }
if url.contains("playlist") || url.ends_with(".m3u8"));
if !is_hls {
media.push(item);
continue;
}
let url = item.url().to_string();
match resolve_bsky_video(&url).await {
Ok(Some((mp4_path, keep_alive))) => {
let thumbnail_url = match &item {
Media::Video { thumbnail_url, .. } => thumbnail_url.clone(),
_ => String::new(),
};
media.push(Media::Video {
title: None,
url: mp4_path.to_string_lossy().into_owned(),
thumbnail_url,
});
fetched._keep_alive = Some(keep_alive);
}
Ok(None) => log::warn!("bsky video remux unavailable for {url}"),
Err(e) => log::warn!("bsky video remux failed for {url}: {e}"),
}
}
fetched.media = media;
Ok(fetched)
}
/// Cache key for a bsky URL: `"bsky:<handle>/<rkey>"`. The prefix is the
/// site id used for caption-format lookup and link-cache keys.
pub fn cache_key(url: &str) -> Option<String> {
PATTERN
.captures(url)
.map(|caps| format!("bsky:{}/{}", &caps[1], &caps[2]))
}
/// Bluesky's fetch-retry policy: transient classes only. Not-found, blocked
/// and parse failures are permanent.
pub fn is_retryable(err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
}
/// bsky media (cdn.bsky.app) needs no extra headers.
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// Downloads an HLS playlist (master or media) and remuxes its segments to a
/// single MP4 via ffmpeg. Returns the MP4 path plus the temp dir that must
/// stay alive until the file is uploaded. `Ok(None)` when ffmpeg is missing.
///
/// Verified live (2026-08): bsky master playlists carry `#EXT-X-STREAM-INF`
/// variant lines (e.g. `720p/video.m3u8?session_id=…`), and the media
/// playlists are VOD MPEG-TS segments (`videoN.ts?…`) without EXT-X-MAP, so
/// a plain `-f concat -c copy` remux is valid.
async fn resolve_bsky_video(
playlist_url: &str,
) -> Result<Option<(std::path::PathBuf, tempfile::TempDir)>, String> {
if !crate::site::ffmpeg_available() {
crate::site::log_once_ffmpeg_missing();
return Ok(None);
}
let master = crate::site::download_media_limited(playlist_url, 1_048_576)
.await
.map_err(|e| format!("bsky video master playlist: {e}"))?;
let master = String::from_utf8_lossy(&master);
// Master playlist: pick the variant with the highest declared bandwidth.
let playlist_url = if master.contains("#EXT-X-STREAM-INF") {
let mut best: Option<(u64, String)> = None;
let mut lines = master.lines();
while let Some(line) = lines.next() {
if !line.starts_with("#EXT-X-STREAM-INF") {
continue;
}
let bandwidth = line
.split_once("BANDWIDTH=")
.and_then(|(_, rest)| rest.split(|c: char| !c.is_ascii_digit()).next())
.and_then(|n| n.parse::<u64>().ok())
.unwrap_or(0);
if let Some(uri) = lines.next().filter(|u| !u.starts_with('#'))
&& bandwidth >= best.as_ref().map(|(b, _)| *b).unwrap_or(0)
{
best = Some((bandwidth, uri.to_string()));
}
}
let Some((_, uri)) = best else {
return Err("bsky video master playlist has no variants".to_string());
};
url::Url::parse(playlist_url)
.and_then(|base| base.join(&uri))
.map_err(|e| format!("bsky video variant URL: {e}"))?
.to_string()
} else {
playlist_url.to_string()
};
let variant = crate::site::download_media_limited(&playlist_url, 1_048_576)
.await
.map_err(|e| format!("bsky video media playlist: {e}"))?;
let variant = String::from_utf8_lossy(&variant);
// Segment URIs: non-#, non-empty lines, resolved relative to the playlist.
let base = url::Url::parse(&playlist_url).map_err(|e| format!("bsky playlist URL: {e}"))?;
let segments: Vec<String> = variant
.lines()
.map(str::trim)
.filter(|l| !l.is_empty() && !l.starts_with('#'))
.map(|l| base.join(l).map(|u| u.to_string()))
.collect::<Result<_, _>>()
.map_err(|e| format!("bsky segment URL: {e}"))?;
if segments.is_empty() {
return Err("bsky video playlist has no segments".to_string());
}
if segments.len() > 500 {
return Err("bsky video has too many segments".to_string());
}
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let mut total: u64 = 0;
let mut list = String::new();
for (i, seg) in segments.iter().enumerate() {
let bytes = crate::site::download_media_limited(seg, 20 * 1024 * 1024)
.await
.map_err(|e| format!("bsky segment {i}: {e}"))?;
total += bytes.len() as u64;
if total > 256 * 1024 * 1024 {
return Err("bsky video exceeds total size cap".to_string());
}
let path = frames_dir.path().join(format!("seg_{i:04}.ts"));
std::fs::write(&path, &bytes).map_err(|e| e.to_string())?;
list.push_str(&format!("file '{}'\n", path.to_string_lossy()));
}
let list_path = frames_dir.path().join("list.txt");
std::fs::write(&list_path, &list).map_err(|e| e.to_string())?;
let output = out_dir.path().join("video.mp4");
let list_str = list_path.to_string_lossy().into_owned();
let output_str = output.to_string_lossy().into_owned();
let status = tokio::task::spawn_blocking(move || {
std::process::Command::new("ffmpeg")
.args([
"-y",
"-f",
"concat",
"-safe",
"0",
"-i",
&list_str,
"-c",
"copy",
"-movflags",
"+faststart",
&output_str,
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
})
.await
.map_err(|e| format!("bsky remux worker panicked: {e}"))?;
match status {
Ok(s) if s.success() => Ok(Some((output, out_dir))),
Ok(s) => Err(format!("ffmpeg exited with {s}")),
Err(e) => Err(format!("ffmpeg spawn failed: {e}")),
}
}
/// Fetches a post thread by handle or DID (`at://` URIs work for both).
@@ -35,8 +230,16 @@ pub async fn fetch(handle: &str, rkey: &str) -> Result<Post, FetchError> {
])
.send()
.await?;
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!("bsky status {status}"))),
};
}
let text = response.text().await?;
Ok(Post::from_json(&text, rkey.to_string())?)
Post::from_json(&text, rkey.to_string())
}
#[derive(Debug)]
@@ -61,8 +264,8 @@ impl Post {
pub fn caption(&self) -> String {
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
url = self.url(),
author_url = self.author_url(),
url = encode_double_quoted_attribute(&self.url()),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
text = encode_text(&self.text),
)
@@ -136,6 +339,7 @@ impl From<Post> for Fetched {
title: post.text.clone(),
media: post.media,
sensitive: post.sensitive,
site_id: "bsky",
render_data,
_keep_alive: None,
}
@@ -253,6 +457,7 @@ mod tests {
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
async fn live_fetch_with_photos() {
let fetched =
fetch_from_url("https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m")
@@ -266,6 +471,7 @@ mod tests {
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
async fn live_fetch_smoke() {
let fetched =
fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
+3 -1
View File
@@ -1,4 +1,6 @@
mod interface;
mod model;
pub use interface::{PATTERN, Post, enabled, fetch_from_url};
pub use interface::{
BskySite, PATTERN, Post, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
};
+410 -113
View File
@@ -4,10 +4,15 @@
//! `PATTERN`, `enabled()` and `fetch_from_url()`; a future site plugs in by
//! adding one guarded entry in [`fetch_once`].
use std::fmt;
use std::future::Future;
use std::pin::Pin;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use regex::Regex;
use thiserror::Error;
pub mod bsky;
pub mod pixiv;
pub mod twitter;
@@ -29,6 +34,10 @@ pub struct Fetched {
pub media: Vec<crate::media::Media>,
/// Spoiler flag for all media of this post.
pub sensitive: bool,
/// Site id (`"twitter"` / `"bsky"` / `"pixiv"`): the single source of
/// truth for site identity — caption-format lookup, cache-key prefix and
/// the SetFormat whitelist all derive from it. Set by the producing site.
pub site_id: &'static str,
/// Raw values (pre-escaped) for user-customizable caption formats.
pub(crate) render_data: Option<RenderData>,
/// Keeps temp files (e.g. an encoded ugoira MP4) alive until the caller
@@ -49,23 +58,18 @@ pub(crate) struct RenderData {
impl Fetched {
/// The site this post came from (used for per-site format overrides).
/// A thin alias over [`Fetched::site_id`] kept for callers that read the
/// site off a fetched post.
pub fn site_name(&self) -> &'static str {
if self.source_url.contains("x.com") || self.source_url.contains("twitter.com") {
"twitter"
} else if self.source_url.contains("bsky.app") {
"bsky"
} else if self.source_url.contains("pixiv.net") {
"pixiv"
} else {
"unknown"
}
self.site_id
}
/// Renders a user-supplied caption format. The format string is
/// HTML-escaped in full, then the (already-escaped) placeholder values
/// are substituted — users can structure text but never inject raw HTML
/// or attributes. An empty/unknown format falls back to the built-in
/// caption.
/// caption. The result is truncated to [`MAX_CAPTION_CHARS`] (Telegram's
/// caption limit for HTML parse mode).
pub fn caption_with(&self, format: &str) -> String {
match (&self.render_data, format.is_empty()) {
(Some(data), false) => caption_from_fields(
@@ -77,7 +81,7 @@ impl Fetched {
&data.title,
&data.tags,
),
_ => self.caption.clone(),
_ => truncate_caption(&self.caption),
}
}
@@ -94,11 +98,47 @@ impl Fetched {
)
})
}
/// Hands over the temp dir keeping locally produced media (ugoira MP4,
/// bsky remux MP4) alive. The bot keeps it while its task may still be
/// retried by the queue, which runs after this [`Fetched`] is dropped and
/// its temp files would otherwise be gone. `None` when no such dir exists.
pub fn take_keep_alive(&mut self) -> Option<tempfile::TempDir> {
self._keep_alive.take()
}
}
/// Telegram's caption length limit (chars) for HTML parse mode; longer
/// captions are rejected with a 400.
pub const MAX_CAPTION_CHARS: usize = 1024;
/// Truncates a caption to at most [`MAX_CAPTION_CHARS`] chars, appending an
/// ellipsis when cut. Backs off to before an unclosed HTML entity (`&amp`
/// without its `;` would be malformed HTML and rejected by Telegram).
pub fn truncate_caption(caption: &str) -> String {
if caption.chars().count() <= MAX_CAPTION_CHARS {
return caption.to_string();
}
// Leave one char for the ellipsis; floor_char_boundary lands on a char
// edge (byte index ≤ MAX-1, so chars ≤ MAX-1).
let mut end = caption.floor_char_boundary(MAX_CAPTION_CHARS - 1);
// Don't split an entity: if the last '&' before `end` has no closing ';'
// inside the kept part, cut before it.
if let Some(amp) = caption[..end].rfind('&')
&& !caption[amp..end].contains(';')
{
end = amp;
}
let mut s = caption[..end].to_string();
s.push('…');
s
}
/// Renders a user-supplied caption format from raw (already-escaped) field
/// values with the same escaping/substitution rules as
/// [`Fetched::caption_with`]. An empty format returns `built_in` unchanged.
/// The result is truncated to [`MAX_CAPTION_CHARS`] (Telegram's caption
/// limit for HTML parse mode).
pub fn caption_from_fields(
format: &str,
built_in: &str,
@@ -109,92 +149,98 @@ pub fn caption_from_fields(
tags: &str,
) -> String {
if format.is_empty() {
return built_in.to_string();
return truncate_caption(built_in);
}
let escaped = html_escape::encode_text(format).into_owned();
escaped
truncate_caption(
&escaped
.replace("{url}", url)
.replace("{author}", author)
.replace("{author_url}", author_url)
.replace("{title}", title)
.replace("{tags}", tags)
.replace("{tags}", tags),
)
}
/// Stable per-post cache key derived from any supported URL, so variant
/// domains (x.com / twitter.com / fxtwitter.com, mobile, `/photo/N`
/// suffixes) map to the same post. Returns `"twitter:<id>"`,
/// `"pixiv:<id>"` or `"bsky:<handle>/<rkey>"`.
/// suffixes) map to the same post. Delegates to each registered site's
/// `cache_key` (dispatch order twitter → bsky → pixiv).
pub fn cache_key(url: &str) -> Option<String> {
if let Some(caps) = twitter::PATTERN.captures(url) {
return Some(format!("twitter:{}", &caps[1]));
}
if let Some(caps) = pixiv::PATTERN.captures(url) {
return Some(format!("pixiv:{}", &caps[1]));
}
if let Some(caps) = bsky::PATTERN.captures(url) {
return Some(format!("bsky:{}/{}", &caps[1], &caps[2]));
}
None
SITES.iter().find_map(|site| site.cache_key(url))
}
#[derive(Debug)]
/// The site id carried by a cache key (`"twitter:123"` → `"twitter"`).
/// Unknown prefixes fall back to `"unknown"`. The bot uses this on the
/// link-cache hit path, where no [`Fetched`] is available — the same value
/// a fresh fetch would read from [`Fetched::site_id`].
pub fn site_id_from_key(key: &str) -> &'static str {
let prefix = key.split(':').next().unwrap_or("");
SITES
.iter()
.map(|site| site.id())
.find(|id| *id == prefix)
.unwrap_or("unknown")
}
#[derive(Debug, Error)]
pub enum FetchError {
Http(reqwest::Error),
Json(serde_json::Error),
Pixiv(PixivError),
#[error("http error: {0}")]
Http(#[from] reqwest::Error),
#[error("json error: {0}")]
Json(#[from] serde_json::Error),
#[error("pixiv error: {0}")]
Pixiv(#[from] PixivError),
/// A site-specific error from a site that keeps its own error type.
/// Permanent by default (sites that need retryable site errors convert
/// them to [`FetchError::Http`] / [`FetchError::Transient`] before
/// returning). Pixiv predates this and keeps the dedicated
/// [`FetchError::Pixiv`] variant.
#[error("{site} error: {error}")]
Site {
site: &'static str,
#[source]
error: Box<dyn std::error::Error + Send + Sync>,
},
#[error("not found")]
NotFound,
#[error("blocked")]
Blocked,
/// The post exists but its content is withheld (twitter NSFW /
/// age-restricted tweets come back as an empty `{}` from syndication).
#[error("content withheld (sensitive)")]
Sensitive,
}
impl fmt::Display for FetchError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
FetchError::Http(e) => write!(f, "http error: {e}"),
FetchError::Json(e) => write!(f, "json error: {e}"),
FetchError::Pixiv(e) => write!(f, "pixiv error: {e}"),
FetchError::NotFound => write!(f, "not found"),
FetchError::Blocked => write!(f, "blocked"),
FetchError::Sensitive => write!(f, "content withheld (sensitive)"),
}
}
}
impl std::error::Error for FetchError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
FetchError::Http(e) => Some(e),
FetchError::Json(e) => Some(e),
FetchError::Pixiv(e) => Some(e),
FetchError::NotFound | FetchError::Blocked | FetchError::Sensitive => None,
}
}
}
impl From<reqwest::Error> for FetchError {
fn from(e: reqwest::Error) -> Self {
FetchError::Http(e)
}
}
impl From<serde_json::Error> for FetchError {
fn from(e: serde_json::Error) -> Self {
FetchError::Json(e)
}
}
impl From<PixivError> for FetchError {
fn from(e: PixivError) -> Self {
FetchError::Pixiv(e)
}
/// A download exceeded the caller's size cap (see [`download_media_limited`]).
#[error("media too large")]
TooLarge,
/// A transient server-side failure (429 / 5xx); [`fetch`] retries these.
#[error("transient: {0}")]
Transient(String),
/// A local I/O failure while streaming a download to disk
/// (see [`download_media_to_file`]).
#[error("io error: {0}")]
Io(std::io::Error),
}
/// Shared HTTP client (browser User-Agent) for twitter/bsky fetches and
/// [`download_media`].
pub(crate) static CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
let builder = reqwest::Client::builder().user_agent("Mozilla/5.0");
let mut builder = reqwest::Client::builder()
.user_agent("Mozilla/5.0")
// reqwest has no total timeout by default; a stalled connection
// would otherwise pin a fetch/handler forever.
.timeout(Duration::from_secs(30))
.connect_timeout(Duration::from_secs(10));
// Route site fetches through the same proxy the Bot API uses, so a
// network that needs TELOXIDE_PROXY (e.g. behind the GFW) does not
// leave site fetches dead while the bot itself works.
if let Some(proxy) = std::env::var("TELOXIDE_PROXY")
.ok()
.filter(|s| !s.is_empty())
&& let Ok(p) = reqwest::Proxy::all(&proxy)
{
builder = builder.proxy(p);
}
// Each `#[tokio::test]` runs on its own runtime; the connection pool is
// bound to the runtime that created it, so cross-runtime reuse of idle
// connections fails with DispatchGone. In test builds every request uses
@@ -204,76 +250,245 @@ pub(crate) static CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
builder.build().expect("failed to build HTTP client")
});
/// Whether a usable `ffmpeg` binary is on PATH (probed once). Shared by the
/// pixiv ugoira encoder and the bsky HLS remuxer.
static FFMPEG_AVAILABLE: LazyLock<bool> = LazyLock::new(|| {
std::process::Command::new("ffmpeg")
.arg("-version")
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
});
static FFMPEG_MISSING_LOGGED: AtomicBool = AtomicBool::new(false);
pub(crate) fn ffmpeg_available() -> bool {
*FFMPEG_AVAILABLE
}
pub(crate) fn log_once_ffmpeg_missing() {
if !FFMPEG_MISSING_LOGGED.swap(true, Ordering::Relaxed) {
log::warn!("ffmpeg not found; ugoira and bsky video posts stay unsupported");
}
}
/// Site adapter: one impl per supported site (twitter / bsky / pixiv),
/// registered in [`SITES`]. All site-specific knowledge — URL pattern,
/// cache-key format, fetch, retry policy, media-host headers, startup
/// validation — lives in the site module; the central dispatcher only
/// iterates the registry.
///
/// Async methods return a boxed future (see [`SiteFuture`]): `async fn` /
/// RPITIT in traits are not dyn-compatible (verified on rustc 1.95), and
/// `+ Send` is required since URL/queue workers spawn these futures. The
/// site structs are stateless unit structs, so the boxed futures never
/// borrow from `self` beyond the call's scope.
pub trait Site: Send + Sync {
/// Stable site id (`"twitter"` / `"bsky"` / `"pixiv"`): caption-format
/// lookup, cache-key prefixes and the SetFormat whitelist derive from it.
fn id(&self) -> &'static str;
/// URL pattern; the dispatcher's first match wins (dispatch order).
fn pattern(&self) -> &'static Regex;
/// Whether the site is usable (env token present, not disabled).
fn enabled(&self) -> bool {
true
}
/// Normalized cache key for a URL of this site (`None` when the URL does
/// not match this site).
fn cache_key(&self, url: &str) -> Option<String>;
/// Fetches and normalizes a post.
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched>;
/// Retry policy for fetch errors: transient classes only.
fn is_retryable(&self, err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
}
/// Extra headers for downloading this site's media (hotlink protection,
/// e.g. pixiv's Referer for pximg.net). Matched on the media URL, not
/// the site pattern.
fn media_headers(&self, _url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// Startup validation (token check etc.); failures are surfaced by
/// [`validate_all`]. The default is a no-op.
fn validate(&self) -> SiteFuture<'static, (), String> {
Box::pin(async { Ok(()) })
}
}
/// A boxed, `Send` future produced by a [`Site`] async method. Boxed so the
/// trait stays dyn-compatible; `Send` because URL/queue workers `tokio::spawn`
/// these futures.
type SiteFuture<'a, T, E = FetchError> = Pin<Box<dyn Future<Output = Result<T, E>> + Send + 'a>>;
/// The one registry of supported sites, in dispatch order (twitter → bsky →
/// pixiv). Adding a site = new module + one `Box::new(...)` entry here; the
/// bot crate never lists sites itself.
static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| {
vec![
Box::new(twitter::TwitterSite),
Box::new(bsky::BskySite),
Box::new(pixiv::PixivSite),
]
});
/// The first enabled site whose pattern matches `url`, in dispatch order.
fn find_site(url: &str) -> Option<&'static dyn Site> {
SITES
.iter()
.find(|site| site.enabled() && site.pattern().is_match(url))
.map(|site| site.as_ref())
}
/// Every supported site id, in dispatch order. The bot's SetFormat whitelist
/// derives from this list.
pub fn site_ids() -> Vec<&'static str> {
SITES.iter().map(|site| site.id()).collect()
}
/// Runs every enabled site's startup validation and returns the failures
/// (site id + message). The caller logs / notifies; failing sites disable
/// themselves (pixiv disables on a bad token).
pub async fn validate_all() -> Vec<(&'static str, String)> {
let mut failures = Vec::new();
for site in SITES.iter() {
if !site.enabled() {
continue;
}
if let Err(e) = site.validate().await {
failures.push((site.id(), e));
}
}
failures
}
/// Fetches a post from its URL. Returns `Ok(None)` when no site pattern
/// matches (unsupported links are silently ignored by the bot).
///
/// Transient network failures are retried: 3 total attempts with 1s then 2s
/// delays. Non-Http errors (Json/NotFound/Blocked/Pixiv) are not retried.
/// Transient failures are retried: 3 total attempts with 1s then 2s delays.
/// What counts as transient is the matched site's own policy (`is_retryable`
/// — e.g. pixiv retries only network errors and 429/5xx). Permanent classes
/// (not-found, blocked, sensitive, parse failures, pixiv 4xx/auth errors)
/// are returned immediately; retrying them only wastes attempts against the
/// source site.
pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
let mut last_http_error = None;
let Some(site) = find_site(url) else {
return Ok(None);
};
for attempt in 0..3u32 {
match fetch_once(url).await {
Ok(Some(fetched)) => {
log::info!(
"fetched {url}: site {} returned {} media",
match site.fetch_from_url(url).await {
Ok(fetched) => {
// Per-request detail: debug only, keyed by the post id.
log::debug!(
"fetched [key={}]: site {} returned {} media",
cache_key(url).unwrap_or_else(|| "?".into()),
fetched.site_name(),
fetched.media.len()
);
return Ok(Some(fetched));
}
Ok(None) => return Ok(None),
Err(FetchError::Http(e)) => {
last_http_error = Some(e);
if attempt < 2 {
Err(err) => {
if site.is_retryable(&err) && attempt < 2 {
tokio::time::sleep(Duration::from_secs(1 << attempt)).await;
} else {
return Err(err);
}
}
Err(other) => return Err(other),
}
}
Err(FetchError::Http(
last_http_error.expect("retry loop always ran 3 attempts"),
))
unreachable!("retry loop always returns")
}
async fn fetch_once(url: &str) -> Result<Option<Fetched>, FetchError> {
if twitter::enabled() && twitter::PATTERN.is_match(url) {
return Ok(Some(twitter::fetch_from_url(url).await?));
/// Applies every site's media-header rule to a download request (pixiv's
/// `Referer` for pximg.net hotlink protection). Sites contribute via their
/// `media_headers(url)` — the central download code carries no per-site logic.
fn apply_media_headers(mut request: reqwest::RequestBuilder, url: &str) -> reqwest::RequestBuilder {
for site in SITES.iter() {
if let Some(headers) = site.media_headers(url) {
for (name, value) in headers {
request = request.header(name, value);
}
if bsky::enabled() && bsky::PATTERN.is_match(url) {
return Ok(Some(bsky::fetch_from_url(url).await?));
}
if pixiv::enabled() && pixiv::PATTERN.is_match(url) {
return Ok(Some(pixiv::fetch_from_url(url).await?));
}
Ok(None)
request
}
/// Downloads media bytes for the bot's upload fallback: when Telegram's own
/// fetch of a media URL is blocked (hotlink protection), the bot downloads
/// the file itself and uploads it via multipart. Site-appropriate headers:
/// pixiv image hosts need the `Referer` header.
/// the file itself and uploads it via multipart. Site-appropriate headers
/// come from each site's `media_headers` (pixiv image hosts need `Referer`).
/// Returns the Content-Length of a media URL, or `None` when the server does
/// not report one. Used to check whether a file fits Telegram's size limits
/// before downloading/uploading it.
pub async fn media_size(url: &str) -> Result<Option<u64>, FetchError> {
let mut request = CLIENT.get(url);
let lower = url.to_ascii_lowercase();
if lower.contains("pximg.net") {
request = request.header("Referer", "https://www.pixiv.net/");
}
let response = request.send().await?;
let response = apply_media_headers(CLIENT.get(url), url)
.send()
.await?
.error_for_status()?;
Ok(response.content_length())
}
pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
let mut request = CLIENT.get(url);
let lower = url.to_ascii_lowercase();
if lower.contains("pximg.net") {
request = request.header("Referer", "https://www.pixiv.net/");
/// Downloads a media file with a hard size cap: the body is streamed and the
/// download aborts with [`FetchError::TooLarge`] the moment the cap is
/// crossed (or when a declared Content-Length already exceeds it). Keeps the
/// bot from buffering arbitrarily large bodies into memory.
pub async fn download_media_limited(url: &str, max_bytes: u64) -> Result<bytes::Bytes, FetchError> {
let response = apply_media_headers(CLIENT.get(url), url)
.send()
.await?
.error_for_status()?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let response = request.send().await?;
Ok(response.bytes().await?)
let mut response = response;
let mut buf = Vec::new();
while let Some(chunk) = response.chunk().await? {
buf.extend_from_slice(&chunk);
if buf.len() as u64 > max_bytes {
return Err(FetchError::TooLarge);
}
}
Ok(bytes::Bytes::from(buf))
}
pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
download_media_limited(url, u64::MAX).await
}
/// Streams a download to `out`, aborting with [`FetchError::TooLarge`] the
/// moment the body crosses `max_bytes` (or when a declared Content-Length
/// already exceeds it). Unlike [`download_media_limited`] the body is never
/// buffered in memory — used for large files (e.g. the pixiv ugoira frame
/// zip, which can be hundreds of MB) that would otherwise spike RAM.
/// Returns the number of bytes written.
pub async fn download_media_to_file(
url: &str,
max_bytes: u64,
out: &mut std::fs::File,
) -> Result<u64, FetchError> {
use std::io::Write;
let response = apply_media_headers(CLIENT.get(url), url)
.send()
.await?
.error_for_status()?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let mut response = response;
let mut total: u64 = 0;
while let Some(chunk) = response.chunk().await? {
total += chunk.len() as u64;
if total > max_bytes {
return Err(FetchError::TooLarge);
}
out.write_all(&chunk).map_err(FetchError::Io)?;
}
Ok(total)
}
#[cfg(test)]
@@ -305,6 +520,43 @@ mod tests {
assert_eq!(cache_key("https://example.com/not-a-post"), None);
}
#[test]
fn site_id_from_key_parses_prefix() {
assert_eq!(site_id_from_key("twitter:123"), "twitter");
assert_eq!(site_id_from_key("pixiv:123"), "pixiv");
assert_eq!(site_id_from_key("bsky:handle.example/3lorem"), "bsky");
assert_eq!(site_id_from_key("unknown:1"), "unknown");
assert_eq!(site_id_from_key("no-colon"), "unknown");
}
#[test]
fn registry_lists_all_sites_in_dispatch_order() {
assert_eq!(site_ids(), vec!["twitter", "bsky", "pixiv"]);
// Enabled sites dispatch; unsupported URLs never match.
assert!(find_site("https://x.com/u/status/1").is_some());
assert!(find_site("https://bsky.app/profile/u/post/3x").is_some());
assert!(find_site("https://example.com/x").is_none());
// Cache keys are pattern-driven, independent of the enabled() gate
// (pixiv is disabled in tests without PIXIV_REFRESH_TOKEN).
assert_eq!(
cache_key("https://www.pixiv.net/artworks/1"),
Some("pixiv:1".into())
);
}
#[test]
fn site_error_variant_displays_and_sources() {
use std::error::Error as _;
let err = FetchError::Site {
site: "example",
error: Box::new(std::io::Error::other("boom")),
};
assert_eq!(err.to_string(), "example error: boom");
assert!(err.source().is_some());
// Permanent by default: no site's is_retryable matches it.
assert!(!twitter::is_retryable(&err));
}
#[test]
fn caption_from_fields_substitutes_and_escapes() {
// The format string is escaped, the field values are substituted
@@ -329,6 +581,45 @@ mod tests {
);
}
#[test]
fn truncate_caption_keeps_short_text() {
assert_eq!(truncate_caption("short"), "short");
// Exactly at the limit: untouched.
let exact = "x".repeat(MAX_CAPTION_CHARS);
assert_eq!(truncate_caption(&exact), exact);
}
#[test]
fn truncate_caption_cuts_long_text_with_ellipsis() {
let long = "x".repeat(MAX_CAPTION_CHARS + 100);
let out = truncate_caption(&long);
assert!(
out.chars().count() <= MAX_CAPTION_CHARS,
"len {}",
out.chars().count()
);
assert!(out.ends_with('…'));
}
#[test]
fn truncate_caption_does_not_split_an_html_entity() {
// An entity crossing the cut must not be left half-open (&amp without ;).
let mut long = "a".repeat(MAX_CAPTION_CHARS - 4);
long.push_str("&amp;bbbb");
let out = truncate_caption(&long);
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
assert!(!out.contains("&amp"), "half entity left: {out:?}");
assert!(!out.ends_with('&'));
}
#[test]
fn truncate_caption_handles_multibyte_boundary() {
// Multi-byte chars near the cut must not panic (char-boundary cut).
let long = "".repeat(MAX_CAPTION_CHARS + 10);
let out = truncate_caption(&long);
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
}
#[tokio::test]
async fn unsupported_url_returns_none() {
let result = fetch("https://example.com/some/article").await;
@@ -345,7 +636,13 @@ mod tests {
async fn download_media_pixiv_original_with_referer() {
// Proves the Referer header is attached for i.pximg.net: a header-less
// GET to a pixiv original URL is rejected with 403.
if std::env::var("PIXIV_REFRESH_TOKEN").is_err() {
// Empty-string check too: an unset CI secret arrives as "" (GitHub
// Actions), which would otherwise run the test tokenless and fail.
if std::env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|s| !s.is_empty())
.is_none()
{
eprintln!("skipping: no PIXIV_REFRESH_TOKEN");
return;
}
+93 -80
View File
@@ -8,11 +8,11 @@ use super::model::{IllustrationModel, TypeModel, UgoiraMetadataModel};
use crate::media::Media;
use crate::site::FetchError;
use std::env;
use std::fmt;
use std::io::{Cursor, Read};
use std::io::Read;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, SystemTime};
use thiserror::Error;
const AUTH_TOKEN_URL: &str = "https://oauth.secure.pixiv.net/auth/token";
const APP_API_URL: &str = "https://app-api.pixiv.net";
@@ -23,48 +23,24 @@ const APP_USER_AGENT: &str = "PixivIOSApp/7.13.3 (iOS 14.6; iPhone13,2)";
/// Token refresh safe margin (seconds).
const TOKEN_REFRESH_SAFE_MARGIN: u64 = 300;
#[derive(Debug)]
#[derive(Debug, Error)]
pub enum PixivError {
/// No refresh token available (PIXIV_REFRESH_TOKEN unset).
#[error("pixiv: no authentication")]
NoAuth,
Http(reqwest::Error),
Json(serde_json::Error),
#[error("pixiv http error: {0}")]
Http(#[from] reqwest::Error),
#[error("pixiv json error: {0}")]
Json(#[from] serde_json::Error),
/// Non-2xx HTTP status from the app API. The code lets [`crate::site::fetch`]
/// retry only transient classes (429 / 5xx) instead of burning attempts on
/// permanent 4xx (bad token, forbidden, not found).
#[error("pixiv status {0}")]
Status(u16),
#[error("pixiv api error: {0}")]
Api(String),
}
impl fmt::Display for PixivError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
PixivError::NoAuth => write!(f, "pixiv: no authentication"),
PixivError::Http(e) => write!(f, "pixiv http error: {e}"),
PixivError::Json(e) => write!(f, "pixiv json error: {e}"),
PixivError::Api(message) => write!(f, "pixiv api error: {message}"),
}
}
}
impl std::error::Error for PixivError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
PixivError::Http(e) => Some(e),
PixivError::Json(e) => Some(e),
_ => None,
}
}
}
impl From<reqwest::Error> for PixivError {
fn from(e: reqwest::Error) -> Self {
PixivError::Http(e)
}
}
impl From<serde_json::Error> for PixivError {
fn from(e: serde_json::Error) -> Self {
PixivError::Json(e)
}
}
/// Native pixiv app-API client.
pub struct PixivAPI {
refresh_token: String,
@@ -136,6 +112,9 @@ impl PixivAPI {
.bearer_auth(access_token)
.send()
.await?;
if !response.status().is_success() {
return Err(PixivError::Status(response.status().as_u16()));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
if json.get("error").is_some() {
let message = json
@@ -186,6 +165,9 @@ impl PixivAPI {
.bearer_auth(access_token)
.send()
.await?;
if !response.status().is_success() {
return Err(PixivError::Status(response.status().as_u16()));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
if json.get("error").is_some() {
let message = json
@@ -207,8 +189,8 @@ impl PixivAPI {
&self,
illust_id: u64,
) -> Result<Option<(String, tempfile::TempDir)>, PixivError> {
if !ffmpeg_available() {
log_once_ffmpeg_missing();
if !crate::site::ffmpeg_available() {
crate::site::log_once_ffmpeg_missing();
return Ok(None);
}
let metadata = self.ugoira_metadata(illust_id).await?;
@@ -222,7 +204,14 @@ impl PixivAPI {
let Some(zip_url) = zip_url else {
return Ok(None);
};
let zip_bytes = crate::site::download_media(&zip_url)
// Stream the frame zip to a temp file instead of buffering it in
// memory: ugoira zips can be hundreds of MB, and the old
// download_media_limited path spiked RAM up to the size cap.
let mut zip_file = tempfile::Builder::new()
.suffix(".zip")
.tempfile()
.map_err(|e| PixivError::Api(format!("temp zip failed: {e}")))?;
crate::site::download_media_to_file(&zip_url, 512 * 1024 * 1024, zip_file.as_file_mut())
.await
.map_err(|e| match e {
FetchError::Http(e) => PixivError::Http(e),
@@ -235,26 +224,54 @@ impl PixivAPI {
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
// Extract frames to canonical zero-padded names; pixiv ugoira
// frames are uniformly jpg or png per artwork.
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
// frames are uniformly jpg or png per artwork. The zip is read
// from disk; `zip_file` stays alive for the whole extraction.
let mut archive = zip::ZipArchive::new(
std::fs::File::open(zip_file.path()).map_err(|e| e.to_string())?,
)
.map_err(|e| format!("unzip: {e}"))?;
// pixiv ugoira frames are uniformly jpg or png per artwork; take
// the extension from the first entry.
let extension = if archive.len() > 0 {
let first_name = archive
.by_index(0)
.map_err(|e| e.to_string())?
.name()
.to_string();
first_name.rsplit('.').next().unwrap_or("jpg").to_string()
if archive.is_empty() {
return Err("empty frame zip".to_string());
}
// Uniform jpg or png per artwork; sniff the first entry's
// magic bytes instead of trusting its filename.
let first = archive.by_index(0).map_err(|e| e.to_string())?;
let mut first_bytes = Vec::new();
first
.take(64 * 1024 * 1024 + 1)
.read_to_end(&mut first_bytes)
.map_err(|e| e.to_string())?;
if first_bytes.len() > 64 * 1024 * 1024 {
return Err("frame exceeds size cap".to_string());
}
let extension = if first_bytes.starts_with(&[0xFF, 0xD8]) {
"jpg"
} else if first_bytes.starts_with(b"\x89PNG") {
"png"
} else {
"jpg".to_string()
"jpg"
};
let mut count = 0usize;
for i in 0..archive.len() {
let mut entry = archive.by_index(i).map_err(|e| e.to_string())?;
{
let path = frames_dir
.path()
.join(format!("img_{count:05}.{extension}"));
std::fs::write(&path, &first_bytes).map_err(|e| e.to_string())?;
count += 1;
}
for i in 1..archive.len() {
let entry = archive.by_index(i).map_err(|e| e.to_string())?;
if entry.size() > 64 * 1024 * 1024 {
return Err(format!("frame {i} exceeds size cap"));
}
let mut bytes = Vec::new();
entry.read_to_end(&mut bytes).map_err(|e| e.to_string())?;
entry
.take(64 * 1024 * 1024 + 1)
.read_to_end(&mut bytes)
.map_err(|e| e.to_string())?;
if bytes.len() > 64 * 1024 * 1024 {
return Err(format!("frame {i} exceeds size cap"));
}
let path = frames_dir
.path()
.join(format!("img_{count:05}.{extension}"));
@@ -304,7 +321,10 @@ impl PixivAPI {
Ok((output.to_string_lossy().into_owned(), out_dir))
})
.await
.expect("ugoira encode worker panicked");
.map_err(|e| {
log::error!("ugoira encode worker panicked for {illust_id}: {e}");
PixivError::Api(format!("ugoira worker failed: {e}"))
})?;
match result {
Ok(pair) => Ok(Some(pair)),
Err(message) => {
@@ -315,28 +335,6 @@ impl PixivAPI {
}
}
static FFMPEG_AVAILABLE: LazyLock<bool> = LazyLock::new(|| {
std::process::Command::new("ffmpeg")
.arg("-version")
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
});
static FFMPEG_MISSING_LOGGED: AtomicBool = AtomicBool::new(false);
fn ffmpeg_available() -> bool {
*FFMPEG_AVAILABLE
}
fn log_once_ffmpeg_missing() {
if !FFMPEG_MISSING_LOGGED.swap(true, Ordering::Relaxed) {
log::warn!("ffmpeg not found; pixiv ugoira posts stay unsupported");
}
}
/// pixiv3-rs replacement: `None` when `PIXIV_REFRESH_TOKEN` is unset.
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> =
LazyLock::new(|| env::var("PIXIV_REFRESH_TOKEN").ok().map(PixivAPI::new));
@@ -380,16 +378,31 @@ mod tests {
use super::*;
use dotenv::dotenv;
/// Skips when `PIXIV_REFRESH_TOKEN` is absent or empty (CI without the
/// secret must stay green; GitHub Actions exposes an unset secret as an
/// empty string, so `is_err()` alone is not enough).
fn require_pixiv_token() -> bool {
std::env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|s| !s.is_empty())
.is_some()
}
#[tokio::test]
async fn test_fetch() {
dotenv().ok();
if !require_pixiv_token() {
eprintln!("skipping: no PIXIV_REFRESH_TOKEN");
return;
}
let result = fetch(126839080).await;
assert!(result.is_ok());
println!("{:#?}", result);
}
#[tokio::test]
async fn validate_with_bogus_token_fails() {
#[ignore = "live network: requires outbound HTTPS to oauth.secure.pixiv.net"]
async fn live_validate_with_bogus_token_fails() {
dotenv().ok();
// A bogus token must surface as Api error (invalid_grant), not panic.
let client = PixivAPI::new("bogus_token_for_testing".to_string());
+127 -5
View File
@@ -1,18 +1,65 @@
use super::model::{IllustrationModel, TypeModel};
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use crate::site::{FetchError, Fetched, PixivError, Site, SiteFuture};
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?:www\.)?pixiv\.net/(?:en/)?(?:(?:i|artworks)/|member_illust\.php\?(?:mode=[a-z_]*&)?illust_id=)(\d+)").unwrap()
Regex::new(r"^(?:https?://)?(?:www\.)?pixiv\.net/(?:en/)?(?:(?:i|artworks)/|member_illust\.php\?(?:mode=[a-z_]*&)?illust_id=)(\d+)").unwrap()
});
pub fn enabled() -> bool {
super::api::enabled()
}
/// Registry entry for the pixiv adapter (see [`crate::site::Site`]).
pub struct PixivSite;
impl Site for PixivSite {
fn id(&self) -> &'static str {
"pixiv"
}
fn pattern(&self) -> &'static Regex {
&PATTERN
}
fn enabled(&self) -> bool {
enabled()
}
fn cache_key(&self, url: &str) -> Option<String> {
cache_key(url)
}
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched> {
Box::pin(async move { fetch_from_url(url).await })
}
fn is_retryable(&self, err: &FetchError) -> bool {
is_retryable(err)
}
fn media_headers(&self, url: &str) -> Option<Vec<(&'static str, String)>> {
media_headers(url)
}
fn validate(&self) -> SiteFuture<'static, (), String> {
Box::pin(async {
match super::api::validate().await {
Ok(()) => Ok(()),
Err(e) => {
// Keep the old behavior: a failed login disables pixiv
// for the rest of this process.
super::api::disable();
Err(format!("{e}"))
}
}
})
}
}
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
let id = PATTERN
.captures(url)
@@ -23,6 +70,43 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
Ok(super::api::fetch(id).await?.into())
}
/// Cache key for a pixiv URL: `"pixiv:<id>"`. The prefix is the site id used
/// for caption-format lookup and link-cache keys.
pub fn cache_key(url: &str) -> Option<String> {
PATTERN
.captures(url)
.map(|caps| format!("pixiv:{}", &caps[1]))
}
/// Pixiv's fetch-retry policy: transient classes only — network errors and
/// HTTP 429/5xx. Permanent 4xx (bad/expired token, forbidden, not found),
/// API/auth errors, unparseable bodies and missing auth are not retried.
pub fn is_retryable(err: &FetchError) -> bool {
match err {
FetchError::Http(_) | FetchError::Transient(_) => true,
FetchError::Pixiv(e) => match e {
PixivError::Http(_) => true,
PixivError::Status(code) if *code == 429 || *code >= 500 => true,
PixivError::Status(_)
| PixivError::Api(_)
| PixivError::Json(_)
| PixivError::NoAuth => false,
},
_ => false,
}
}
/// pximg.net is hotlink-protected: downloads must carry the pixiv Referer.
/// The match is on the media host, not the site PATTERN — pixiv's PATTERN
/// only matches `pixiv.net/artworks/...`, never `i.pximg.net`.
pub fn media_headers(url: &str) -> Option<Vec<(&'static str, String)>> {
if url.to_ascii_lowercase().contains("pximg.net") {
Some(vec![("Referer", "https://www.pixiv.net/".to_string())])
} else {
None
}
}
#[derive(Debug)]
pub struct Illustration {
id: String,
@@ -48,9 +132,9 @@ impl Illustration {
pub fn caption(&self) -> String {
format!(
"<a href=\"{url}\">{title}</a> / <a href=\"{author_url}\">{author}</a>\n{tags}",
url = self.url(),
url = encode_double_quoted_attribute(&self.url()),
title = encode_text(&self.title),
author_url = self.author_url(),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
tags = encode_text(
&self
@@ -142,6 +226,7 @@ impl From<Illustration> for Fetched {
title: illustration.title.clone(),
media: illustration.media,
sensitive: illustration.nsfw,
site_id: "pixiv",
render_data,
_keep_alive: illustration._keep_alive,
}
@@ -232,6 +317,43 @@ mod tests {
}
}
#[test]
fn is_retryable_classifies_transient_and_permanent() {
// Transient: network errors, explicit transient, pixiv 429/5xx.
assert!(is_retryable(&FetchError::Transient("429".into())));
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(429))));
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(500))));
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(503))));
// Permanent: pixiv 4xx (bad/expired token, forbidden, not found),
// api/auth errors, unparseable bodies, not-found/blocked/sensitive.
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(400))));
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(401))));
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(403))));
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(404))));
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Api(
"invalid_grant".into()
))));
assert!(!is_retryable(&FetchError::Pixiv(PixivError::NoAuth)));
let json_err = serde_json::from_str::<serde_json::Value>("x").unwrap_err();
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Json(
json_err
))));
assert!(!is_retryable(&FetchError::NotFound));
assert!(!is_retryable(&FetchError::Blocked));
assert!(!is_retryable(&FetchError::Sensitive));
assert!(!is_retryable(&FetchError::TooLarge));
}
#[test]
fn media_headers_adds_referer_only_for_pximg() {
assert_eq!(
media_headers("https://i.pximg.net/img-original/img/1.png"),
Some(vec![("Referer", "https://www.pixiv.net/".to_string())])
);
assert_eq!(media_headers("https://www.pixiv.net/artworks/1"), None);
assert_eq!(media_headers("https://x.com/u/status/1"), None);
}
#[test]
fn ugoira_yields_empty_media() {
let v = illust_json(
+4 -1
View File
@@ -3,4 +3,7 @@ mod interface;
mod model;
pub use api::{PixivAPI, PixivError, disable, fetch, validate};
pub use interface::{Illustration, PATTERN, enabled, fetch_from_url};
pub use interface::{
Illustration, PATTERN, PixivSite, cache_key, enabled, fetch_from_url, is_retryable,
media_headers,
};
+10 -3
View File
@@ -126,9 +126,16 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
.header("referer", "https://x.com/")
.send()
.await?;
if !response.status().is_success() {
log::warn!("twitter auth fetch {id}: HTTP {}", response.status());
return Err(FetchError::NotFound);
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
log::warn!("twitter auth fetch {id}: HTTP {status}");
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!(
"twitter auth status {status}"
))),
};
}
let text = response.text().await?;
let json: Value = serde_json::from_str(&text)?;
+225 -22
View File
@@ -1,10 +1,31 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use crate::site::{FetchError, Fetched, Site, SiteFuture};
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
/// Registry entry for the twitter adapter (see [`crate::site::Site`]).
pub struct TwitterSite;
impl Site for TwitterSite {
fn id(&self) -> &'static str {
"twitter"
}
fn pattern(&self) -> &'static Regex {
&PATTERN
}
fn cache_key(&self, url: &str) -> Option<String> {
cache_key(url)
}
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched> {
Box::pin(async move { fetch_from_url(url).await })
}
}
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"^(?:https?://)?(?:www\.|mobile\.)?(?:x|twitter|fixvx|vxtwitter|fixupx|fxtwitter)\.com/[^.]+/status/(\d+)").unwrap()
});
@@ -29,13 +50,19 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
if super::auth::enabled() {
match super::auth::fetch(id).await {
Ok(tweet) => Ok(tweet.into()),
// The tweet is genuinely gone (deleted / suspended /
// tombstoned): report it instead of degrading to an
// empty result ("No media found"). Only unexpected
// fallback failures (network, parse) keep the NSFW
// placeholder.
Err(FetchError::NotFound) => Err(FetchError::NotFound),
Err(e) => {
log::warn!("twitter auth fallback failed for {id}: {e}");
Ok(empty_fetched(url))
}
}
} else {
log::info!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
log::debug!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
Ok(empty_fetched(url))
}
}
@@ -43,22 +70,46 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
}
}
/// Cache key for a twitter URL: `"twitter:<id>"`. The prefix is the site id
/// used for caption-format lookup and link-cache keys.
pub fn cache_key(url: &str) -> Option<String> {
PATTERN
.captures(url)
.map(|caps| format!("twitter:{}", &caps[1]))
}
/// Twitter's fetch-retry policy: transient classes only. Not-found, blocked,
/// sensitive (NSFW withholding) and parse failures are permanent — retrying
/// them only wastes attempts against the syndication endpoint.
pub fn is_retryable(err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
}
/// twimg URLs need no extra headers (no hotlink protection).
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// A Fetched with no media for withheld tweets: the bot replies
/// "No media found" and moves on instead of erroring.
fn empty_fetched(url: &str) -> Fetched {
Fetched {
source_url: url.to_string(),
caption: url.to_string(),
// The raw user-supplied URL goes into an HTML caption; escape it so
// crafted links cannot break the parse (Telegram 400).
caption: encode_text(url).into_owned(),
title: String::new(),
media: vec![],
sensitive: true,
site_id: "twitter",
render_data: None,
_keep_alive: None,
}
}
/// Fetches a tweet from the syndication endpoint. Deleted/blocked tweets
/// surface as `FetchError::NotFound`.
/// surface as `FetchError::NotFound`; withheld content (empty tombstone,
/// age-restricted) as `FetchError::Sensitive`.
pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
let id_num = id.parse::<u64>().map_err(|_| FetchError::NotFound)?;
let response = crate::site::CLIENT
@@ -68,27 +119,55 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
))
.send()
.await?;
if !response.status().is_success() {
return Err(FetchError::NotFound);
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!("twitter status {status}"))),
};
}
let text = response.text().await?;
// Deleted tweets answer with {"errors": [...]} instead of a tweet.
if serde_json::from_str::<serde_json::Value>(&text)
.map(|v| v.get("errors").is_some())
.unwrap_or(false)
{
// Classify before parsing the tweet (see [`parse_syndication_body`]).
parse_syndication_body(&text)?;
Tweet::from_syndication_json(&text).map_err(FetchError::Json)
}
/// Parses and classifies a syndication response body. `Ok` means the body is
/// a real tweet payload; `Err` carries the permanent error class:
/// - `NotFound`: an `errors` array (deleted/blocked) or a `TweetTombstone`
/// **with a reason** — "This Post was deleted by the Post author." /
/// "This Post is from a suspended account." (the tweet is gone).
/// - `Sensitive`: content withheld **without a deletion reason** — the empty
/// `{}` shape or an *empty* `TweetTombstone` (`{"__typename":
/// "TweetTombstone","tombstone":{}}`). Live tweets in restricted contexts
/// surface this way; treating them as deleted is a regression (a normal
/// tweet must not report "deleted"). Age-restricted tombstones route here
/// too so the logged-in GraphQL fallback can fetch the real tweet.
/// - `Json`: an unparseable body.
fn parse_syndication_body(text: &str) -> Result<serde_json::Value, FetchError> {
let body: serde_json::Value = serde_json::from_str(text)?;
if body.get("errors").is_some() {
return Err(FetchError::NotFound);
}
// NSFW / age-restricted tweets exist but are served as an empty `{}` —
// they surface as FetchError::Sensitive so the caller can retry as a
// logged-in user.
if serde_json::from_str::<serde_json::Value>(&text)
.map(|v| v.get("id_str").is_none())
.unwrap_or(false)
{
if let Some(tombstone) = body.get("tombstone") {
// Only a tombstone with an explicit reason means the tweet is gone;
// a missing reason (empty `tombstone: {}`) or an age-restricted
// reason means the tweet exists but is withheld.
let reason = tombstone
.get("text")
.and_then(|t| t.get("text"))
.and_then(|t| t.as_str())
.unwrap_or("");
if reason.is_empty() || reason.to_ascii_lowercase().contains("age-restricted") {
return Err(FetchError::Sensitive);
}
Tweet::from_syndication_json(&text).map_err(FetchError::Json)
return Err(FetchError::NotFound);
}
if body.get("id_str").is_none() {
return Err(FetchError::Sensitive);
}
Ok(body)
}
/// The syndication token: JS `((id / 1e15) * PI).toString(36)` (the
@@ -146,8 +225,8 @@ impl Tweet {
pub fn caption(&self) -> String {
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
url = self.url(),
author_url = self.author_url(),
url = encode_double_quoted_attribute(&self.url()),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
text = encode_text(&self.text),
)
@@ -278,6 +357,7 @@ impl From<Tweet> for Fetched {
title: tweet.text.clone(),
media: tweet.media,
sensitive: tweet.sensitive,
site_id: "twitter",
render_data,
_keep_alive: None,
}
@@ -329,6 +409,29 @@ mod tests {
}
}
#[test]
fn cache_key_prefixes_tweet_id() {
assert_eq!(
cache_key("https://x.com/user/status/1234567890"),
Some("twitter:1234567890".into())
);
assert_eq!(cache_key("https://example.com/1"), None);
}
#[test]
fn is_retryable_classifies_transient_and_permanent() {
// Transient: network errors and explicit transient statuses (the
// `Http` arm shares this match arm with `Transient`).
assert!(is_retryable(&FetchError::Transient("429".into())));
// Permanent: gone, blocked, withheld, oversized, unparseable.
assert!(!is_retryable(&FetchError::NotFound));
assert!(!is_retryable(&FetchError::Blocked));
assert!(!is_retryable(&FetchError::Sensitive));
assert!(!is_retryable(&FetchError::TooLarge));
let json_err = serde_json::from_str::<serde_json::Value>("x").unwrap_err();
assert!(!is_retryable(&FetchError::Json(json_err)));
}
#[test]
fn syndication_json_converts_to_fetched() {
let raw = fixture(serde_json::json!([
@@ -565,19 +668,93 @@ mod tests {
assert!(token.starts_with("236.v"), "got {token}");
}
#[test]
fn syndication_tombstone_maps_to_not_found() {
// Deleted tweets answer HTTP 200 with a TweetTombstone carrying a
// reason (no `errors`, no `id_str`); they must not fall through to
// Sensitive, which would make the bot reply "No media found" for a
// deleted tweet.
let raw = serde_json::json!({
"__typename": "TweetTombstone",
"tombstone": {
"text": { "rtl": false, "text": "This Post was deleted by the Post author. Learn more" }
}
});
assert!(matches!(
parse_syndication_body(&raw.to_string()),
Err(FetchError::NotFound)
));
}
#[test]
fn syndication_empty_tombstone_maps_to_sensitive() {
// Regression: live tweets in restricted contexts answer with an
// EMPTY tombstone (`{"__typename":"TweetTombstone","tombstone":{}}`)
// — no deletion reason. They must not be reported as deleted.
let raw = serde_json::json!({ "__typename": "TweetTombstone", "tombstone": {} });
assert!(matches!(
parse_syndication_body(&raw.to_string()),
Err(FetchError::Sensitive)
));
}
#[test]
fn syndication_age_restricted_tombstone_maps_to_sensitive() {
// An age-restricted tombstone withholds a live tweet; route it to
// the logged-in fallback instead of reporting it as gone.
let raw = serde_json::json!({
"__typename": "TweetTombstone",
"tombstone": {
"text": { "rtl": false, "text": "Age-restricted adult content" }
}
});
assert!(matches!(
parse_syndication_body(&raw.to_string()),
Err(FetchError::Sensitive)
));
}
#[test]
fn syndication_errors_maps_to_not_found() {
// The classic gone shape: {"errors": [...]}.
let raw = serde_json::json!({ "errors": [{ "message": "Couldn't find Tweet" }] });
assert!(matches!(
parse_syndication_body(&raw.to_string()),
Err(FetchError::NotFound)
));
}
#[test]
fn syndication_empty_object_maps_to_sensitive() {
// NSFW / age-restricted withholding: an empty `{}`.
assert!(matches!(
parse_syndication_body("{}"),
Err(FetchError::Sensitive)
));
}
#[test]
fn syndication_tweet_body_passes() {
let raw = fixture(serde_json::json!([]));
assert!(parse_syndication_body(&raw.to_string()).is_ok());
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_with_photos() {
let fetched = fetch("861627479294746624").await.unwrap();
assert_eq!(fetched.media.len(), 4);
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_text_only() {
let fetched = fetch("1992471125734142256").await.unwrap();
assert!(fetched.media.is_empty());
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_deleted_tweet_is_not_found() {
// Deleted tweet: the syndication endpoint answers with errors.
let result = fetch("0").await;
@@ -586,4 +763,30 @@ mod tests {
"got {result:?}"
);
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_tombstone_deleted_tweet_is_not_found() {
// Regression: a real deleted tweet answering with a TweetTombstone
// (HTTP 200, no errors/id_str) used to surface as Sensitive and
// degrade to an empty result ("No media found").
let result = fetch("2085948045967986859").await;
assert!(
matches!(result, Err(FetchError::NotFound)),
"got {result:?}"
);
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to cdn.syndication.twimg.com"]
async fn live_fetch_empty_tombstone_is_sensitive() {
// Regression: a LIVE tweet (verified via a third-party API) answers
// syndication with an empty TweetTombstone; it must surface as
// Sensitive (withheld), never as NotFound (deleted).
let result = fetch("2087851366253555752").await;
assert!(
matches!(result, Err(FetchError::Sensitive)),
"got {result:?}"
);
}
}
+3 -1
View File
@@ -2,4 +2,6 @@ mod auth;
mod interface;
mod model;
pub use interface::{PATTERN, Tweet, enabled, fetch_from_url};
pub use interface::{
PATTERN, Tweet, TwitterSite, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
};
+3 -3
View File
@@ -1,10 +1,10 @@
[package]
name = "xmedia-bot"
version = "1.0.8"
version = "1.2.2"
edition = "2024"
[dependencies]
teloxide = { version = "0.17", features = ["webhooks-axum", "macros"] }
teloxide = { version = "0.17", default-features = false, features = ["webhooks-axum", "macros", "rustls", "ctrlc_handler"] }
tokio = { version = "1.40", features = ["rt-multi-thread", "macros"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
@@ -12,12 +12,12 @@ log = "0.4"
pretty_env_logger = "0.5"
dotenv = "0.15"
url = "2.5.2"
regex = "1.12"
html-escape = "0.2"
rusqlite = { version = "0.32", features = ["bundled"] }
rand = "0.8"
tempfile = "3"
parking_lot = "0.12"
bytes = "1"
png = "0.18"
zune-jpeg = "0.5"
fast_image_resize = "6"
+53 -21
View File
@@ -23,32 +23,64 @@ pub struct Config {
impl Config {
pub fn load() -> Config {
let admin_ids = env::var("BOT_ADMIN")
.ok()
.map(|s| {
s.split(',')
.filter_map(|part| part.trim().parse::<i64>().ok())
// Fail-fast helpers: a misspelled value must not silently fall back
// to a default and run with different behavior than the operator
// intended — log a loud warning naming the variable instead.
fn parse_u64(name: &str, default: u64) -> u64 {
match env::var(name) {
Ok(v) => v.parse::<u64>().unwrap_or_else(|_| {
log::warn!("invalid {name}={v:?}; using default {default}");
default
}),
Err(_) => default,
}
}
let admin_ids = match env::var("BOT_ADMIN") {
Ok(s) => {
let (ids, bad): (Vec<_>, Vec<_>) = s
.split(',')
.map(str::trim)
.filter(|part| !part.is_empty())
.partition(|part| part.parse::<i64>().is_ok());
if !bad.is_empty() {
log::warn!("BOT_ADMIN: ignoring non-numeric ids: {bad:?}");
}
ids.into_iter()
.filter_map(|p| p.parse::<i64>().ok())
.collect()
})
.unwrap_or_default();
}
Err(_) => Vec::new(),
};
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(86400));
let link_cache_ttl = env::var("LINK_CACHE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(7 * 24 * 3600));
let edit_message_ttl =
Duration::from_secs(parse_u64("EDIT_MESSAGE_TTL_SECONDS", 24 * 3600));
let link_cache_ttl =
Duration::from_secs(parse_u64("LINK_CACHE_TTL_SECONDS", 7 * 24 * 3600));
let webhook_enabled = env::var("WEBHOOK")
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
// The webhook settings are consumed by `.expect()` in main when
// WEBHOOK=true, so an unparseable value fails fast at startup with a
// clear message; still log here for the WEBHOOK=false case.
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| {
s.parse::<url::Url>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_URL={s:?}");
None
})
});
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| {
s.parse::<IpAddr>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_LISTEN={s:?}");
None
})
});
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| {
s.parse::<u16>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_PORT={s:?}");
None
})
});
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
// value that would otherwise come from `.env`).
let webhook_cert = env::var("WEBHOOK_CERT").ok().filter(|s| !s.is_empty());
+139 -20
View File
@@ -2,36 +2,155 @@
//! (`tasks` in queue.rs, `chat_state` in state.rs, `link_cache` in
//! link_cache.rs).
//!
//! Every operation opens its own short-lived connection with a busy timeout:
//! handler tasks enqueue while workers lease/update rows concurrently, and
//! without the timeout a concurrent write fails immediately with SQLITE_BUSY
//! and the operation is lost. All I/O runs inside `spawn_blocking` via
//! [`with_conn`] — rusqlite connections are not Send-friendly to hold across
//! an await point, and blocking the async executor stalls every handler.
//! All I/O runs inside `spawn_blocking` via [`DbPool::with_conn`] — rusqlite
//! connections are not Send-friendly to hold across an await point, and
//! blocking the async executor stalls every handler. Connections are reused
//! through a small per-store pool instead of opening a fresh connection per
//! operation: WAL lets readers run alongside writer leases, and the pool's
//! semaphore bounds how many DB operations run concurrently, giving natural
//! backpressure on hot paths (every message / URL / callback touches
//! chat_state or the link cache).
use parking_lot::Mutex;
use rusqlite::Connection;
use std::sync::Arc;
use std::time::Duration;
/// Upper bound on pooled (reused) connections and on concurrent DB
/// operations per store. Small on purpose: the queue's `BEGIN IMMEDIATE`
/// leases serialize writes anyway, and WAL readers rarely need more.
const POOL_SIZE: usize = 4;
/// A tiny connection pool for one SQLite file. Connections are checked out
/// on a blocking thread and returned afterwards; `acquire` opens a new
/// connection only when the idle list is empty, so the steady-state cost of
/// an operation is a list pop instead of a fresh open (+ busy timeout + WAL
/// pragma). The semaphore caps the number of concurrent operations, so a
/// burst of handlers queues up instead of opening unbounded connections.
pub struct DbPool {
// Arc so [`DbPool::with_conn`] can hand an owned handle to
// `spawn_blocking` without borrowing across the await point.
inner: Arc<PoolInner>,
}
struct PoolInner {
path: String,
permits: tokio::sync::Semaphore,
idle: Mutex<Vec<Connection>>,
}
impl DbPool {
pub fn new(path: &str) -> Self {
DbPool {
inner: Arc::new(PoolInner {
path: path.to_string(),
permits: tokio::sync::Semaphore::new(POOL_SIZE),
idle: Mutex::new(Vec::new()),
}),
}
}
/// Runs `f` against a pooled connection on a blocking thread, returning
/// the closure's result. Owns the semaphore + `spawn_blocking` +
/// `expect` ceremony shared by every table access; the caller maps
/// errors to its own log line.
pub async fn with_conn<T, F>(&self, f: F) -> rusqlite::Result<T>
where
T: Send + 'static,
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
{
let _permit = self
.inner
.permits
.acquire()
.await
.expect("db pool semaphore closed");
let inner = Arc::clone(&self.inner);
tokio::task::spawn_blocking(move || {
let mut conn = inner.acquire()?;
let result = f(&mut conn);
inner.release(conn);
result
})
.await
.expect("db worker panicked")
}
/// The database file this pool serves (used by tests that need a raw
/// connection, e.g. to seed rows directly).
#[cfg(test)]
pub fn path(&self) -> &str {
&self.inner.path
}
}
impl PoolInner {
/// Reuses an idle connection or opens a fresh one.
fn acquire(&self) -> rusqlite::Result<Connection> {
if let Some(conn) = self.idle.lock().pop() {
return Ok(conn);
}
open_db(&self.path)
}
/// Returns a connection to the pool (dropped when the pool is full).
fn release(&self, conn: Connection) {
let mut idle = self.idle.lock();
if idle.len() < POOL_SIZE {
idle.push(conn);
}
}
}
/// Opens the shared DB with a busy timeout.
pub fn open_db(path: &str) -> rusqlite::Result<Connection> {
let conn = Connection::open(path)?;
conn.busy_timeout(Duration::from_secs(5))?;
// WAL lets readers run alongside writer leases instead of blocking on
// the rollback journal; the mode persists in the DB header, so the
// idempotent pragma here and in ensure_schema only needs to win once.
conn.pragma_update(None, "journal_mode", "WAL")?;
Ok(conn)
}
/// Runs `f` against a fresh connection on a blocking thread, returning the
/// closure's result. Owns the `spawn_blocking` + `expect` ceremony shared by
/// every table access; the caller maps errors to its own log line.
pub async fn with_conn<T, F>(path: &str, f: F) -> rusqlite::Result<T>
where
T: Send + 'static,
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
/// Opens the shared DB file, runs the merged schema for all three tables and
/// returns a pool for it. One call per process in production (the stores
/// share the returned pool); tests call it per tempdir.
pub fn open_store(path: &str) -> rusqlite::Result<Arc<DbPool>> {
if let Some(parent) = std::path::Path::new(path).parent()
&& !parent.as_os_str().is_empty()
{
let path = path.to_string();
tokio::task::spawn_blocking(move || {
let mut conn = open_db(&path)?;
f(&mut conn)
})
.await
.expect("db worker panicked")
std::fs::create_dir_all(parent).map_err(rusqlite_error)?;
}
let conn = open_db(path)?;
schema_init(&conn)?;
Ok(Arc::new(DbPool::new(path)))
}
fn rusqlite_error(e: std::io::Error) -> rusqlite::Error {
rusqlite::Error::ToSqlConversionFailure(Box::new(e))
}
/// Creates the `tasks`, `chat_state` and `link_cache` tables (idempotent).
/// The three stores used to own their own schema; keeping it in one place
/// means one initialization for the whole database file.
pub fn schema_init(conn: &Connection) -> rusqlite::Result<()> {
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
CREATE INDEX IF NOT EXISTS idx_tasks_pending ON tasks(status, run_after); \
CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL); \
CREATE TABLE IF NOT EXISTS link_cache (url TEXT PRIMARY KEY, payload TEXT NOT NULL, \
created_at REAL NOT NULL);",
)
}
/// Unix timestamp in fractional seconds. Shared by the queue, chat store and
/// link cache (previously four private copies).
pub fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
}
+307 -89
View File
@@ -1,10 +1,11 @@
use crate::config::Config;
use crate::db::{self, now_f64};
use crate::link_cache::{CachedMediaKind, CachedPost, LinkCache};
use crate::queue::PersistentTaskQueue;
use crate::send::{self, MediaItemPayload, Task};
use crate::state::{ChatData, ChatStore, unix_now};
use std::collections::HashSet;
use std::sync::LazyLock;
use std::sync::{Arc, LazyLock};
use teloxide::RequestError;
use teloxide::prelude::*;
use teloxide::types::{
@@ -13,24 +14,83 @@ use teloxide::types::{
MessageEntityKind, MessageId, ParseMode, Recipient, ReplyParameters,
};
use teloxide::utils::command::BotCommands;
use tokio::sync::Semaphore;
use x_media::media::Media;
pub static CHAT_STORE: LazyLock<ChatStore> =
LazyLock::new(|| ChatStore::open("data/task_queue.db").expect("failed to open chat store"));
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
LazyLock::new(|| PersistentTaskQueue::new("data/task_queue.db"));
pub static LINK_CACHE: LazyLock<LinkCache> =
LazyLock::new(|| LinkCache::open("data/task_queue.db"));
pub static CONFIG: LazyLock<Config> = LazyLock::new(Config::load);
/// One URL job: bot handle + the message + the extracted URL.
type UrlJob = (Bot, Message, String);
/// Bounded channel of URL jobs drained by [`start_url_workers`]. The bound
/// caps both queued memory and shutdown backlog; a full channel applies
/// backpressure to the per-chat handler instead of spawning unbounded tasks.
static URL_JOBS: LazyLock<parking_lot::Mutex<Option<tokio::sync::mpsc::Sender<UrlJob>>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
/// Set by main's shutdown sequence; workers stop pulling new jobs.
static URL_STOP: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
/// Cap on concurrent per-URL processing. teloxide dispatches updates to a
/// per-chat worker that handles them sequentially, so a batch-forward of many
/// messages would otherwise be processed one at a time (fetch + send each,
/// roughly a second per message). Moving the work into spawned tasks trades
/// per-chat reply ordering for throughput; the semaphore bounds how many run
/// at once so a big burst cannot hammer Telegram's rate limits.
static URL_TASKS: LazyLock<Semaphore> = LazyLock::new(|| Semaphore::new(8));
/// JoinHandles of the URL workers, awaited by [`stop_url_workers`].
static URL_WORKER_HANDLES: LazyLock<parking_lot::Mutex<Option<Vec<tokio::task::JoinHandle<()>>>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
/// Worker count draining URL jobs; keeps the old 8-permit concurrency cap
/// while bounding how many jobs can be queued at all.
const URL_WORKERS: usize = 8;
/// Starts the URL job workers (called once from main after the queue starts).
/// teloxide dispatches updates to a per-chat worker that handles them
/// sequentially, so a batch-forward of many messages would otherwise be
/// processed one at a time (fetch + send each, roughly a second per
/// message); the workers add throughput, and FIFO order preserves per-message
/// URL order.
pub async fn start_url_workers() {
let (tx, rx) = tokio::sync::mpsc::channel::<UrlJob>(256);
*URL_JOBS.lock() = Some(tx);
let rx = std::sync::Arc::new(tokio::sync::Mutex::new(rx));
let mut handles = Vec::with_capacity(URL_WORKERS);
for _ in 0..URL_WORKERS {
let rx = std::sync::Arc::clone(&rx);
handles.push(tokio::spawn(async move {
while !URL_STOP.load(std::sync::atomic::Ordering::Relaxed) {
let job = rx.lock().await.recv().await;
match job {
Some((bot, message, url)) => url_media(bot, &message, &url).await,
None => break,
}
}
}));
}
*URL_WORKER_HANDLES.lock() = Some(handles);
}
/// Stops the URL workers: sets the stop flag, drops the job channel (so
/// workers blocked in \`recv()\` wake with \`None\` and exit) and awaits the
/// worker tasks. Each worker finishes its in-flight job first; jobs still
/// queued in the channel are abandoned (the old implementation neither
/// drained them nor woke blocked workers — it only set a flag checked
/// between jobs).
pub async fn stop_url_workers() {
URL_STOP.store(true, std::sync::atomic::Ordering::Relaxed);
// Dropping the sender makes every worker's recv() return None.
*URL_JOBS.lock() = None;
// Take the handles first so the lock guard drops before the awaits.
let handles = URL_WORKER_HANDLES.lock().take();
if let Some(handles) = handles {
for handle in handles {
let _ = handle.await;
}
}
}
/// One shared SQLite pool for the three stores (chat state, task queue, link
/// cache): a single pool bounds concurrent DB work on `data/task_queue.db`
/// instead of three independent pools competing for the same file. The schema
/// for all three tables is initialized once, here.
static DB: LazyLock<Arc<db::DbPool>> =
LazyLock::new(|| db::open_store("data/task_queue.db").expect("failed to open database"));
pub static CHAT_STORE: LazyLock<ChatStore> = LazyLock::new(|| ChatStore::new(Arc::clone(&DB)));
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
LazyLock::new(|| PersistentTaskQueue::new(Arc::clone(&DB)));
pub static LINK_CACHE: LazyLock<LinkCache> = LazyLock::new(|| LinkCache::new(Arc::clone(&DB)));
pub static CONFIG: LazyLock<Config> = LazyLock::new(Config::load);
#[derive(BotCommands, Clone)]
#[command(
@@ -76,11 +136,12 @@ where
.await
}
fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
/// Log prefix tying the whole lifecycle of one link (fetch → send → cache →
/// forward) together: the normalized cache key (`twitter:123…`, `pixiv:123`,
/// `bsky:handle/rkey`) instead of the raw URL, so logs stay short and do not
/// echo full user-submitted URLs at info level.
pub fn log_key(url: &str) -> String {
x_media::site::cache_key(url).unwrap_or_else(|| "<unsupported>".to_string())
}
/// Extracts URL and text-link entities (text + caption), deduped in order.
@@ -101,7 +162,10 @@ pub fn extract_urls(message: &Message) -> Vec<String> {
}
}
let mut seen = HashSet::new();
urls.retain(|url| seen.insert(url.clone()));
// Dedup by the normalized post id so variant URLs of the same post
// (/status/1 vs /status/1/photo/1) are sent once; unsupported URLs fall
// back to exact-string dedup.
urls.retain(|url| seen.insert(x_media::site::cache_key(url).unwrap_or_else(|| url.clone())));
urls
}
@@ -124,7 +188,7 @@ async fn edit_message_handler(bot: &Bot, message: &Message) -> bool {
};
let link = format!(
"<a href=\"{0}\">{1}</a>",
edit.url,
html_escape::encode_double_quoted_attribute(&edit.url),
html_escape::encode_text(text)
);
let new_text = if edit.template.is_empty() {
@@ -190,19 +254,31 @@ async fn set_forward_channel_handler(
return Err(SetForwardChannelError::NotChannel);
}
let channel_id = chat.id.0;
// The sender must be a channel administrator. Compare against the
// sender's user id, NOT the chat id (they only coincide in private
// chats, so the old check broke group usage).
let Some(sender) = message.from.as_ref() else {
return Err(SetForwardChannelError::NotAdmin);
};
match bot.get_chat_administrators(channel.clone()).await {
Err(e) => {
log::error!("Failed to get channel administrators {}: {}", channel, e);
return Err(SetForwardChannelError::NotBotAdmin(e));
}
Ok(admins) => {
if !admins.iter().any(|admin| admin.user.id == message.chat.id) {
if !admins.iter().any(|admin| admin.user.id == sender.id) {
return Err(SetForwardChannelError::NotAdmin);
}
let bot_id = bot.get_me().await.expect("Failed get bot id").user.id;
if let Some(bot_admin) = admins.iter().find(|admin| admin.user.id == bot_id)
&& !bot_admin.can_post_messages()
{
// The bot itself must be an admin that can post; a missing
// bot entry must not pass silently (copy would fail later).
let bot_id = match bot.get_me().await {
Ok(me) => me.user.id,
Err(e) => return Err(SetForwardChannelError::NotBotAdmin(e)),
};
let bot_ok = admins
.iter()
.any(|admin| admin.user.id == bot_id && admin.can_post_messages());
if !bot_ok {
return Err(SetForwardChannelError::NotBotCanPost);
}
}
@@ -226,9 +302,11 @@ async fn execute_command(
Command::SetForwardChannel(channel) => {
let result = match set_forward_channel_handler(bot, message, channel).await {
Ok(channel_id) => {
let mut chat_data = CHAT_STORE.get(message.chat.id.0).await;
chat_data.forward_channel_id = Some(channel_id);
CHAT_STORE.set(message.chat.id.0, &chat_data).await;
CHAT_STORE
.update(message.chat.id.0, |data| {
data.forward_channel_id = Some(channel_id);
})
.await;
"Add successfully.".to_string()
}
Err(SetForwardChannelError::EmptyParameter) => {
@@ -252,31 +330,34 @@ async fn execute_command(
}
Command::RemoveForwardChannel => {
let chat_id = message.chat.id.0;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let text = if chat_data.forward_channel_id.is_some() {
chat_data.forward_channel_id = None;
CHAT_STORE.set(chat_id, &chat_data).await;
let text = CHAT_STORE
.update(chat_id, |data| {
if data.forward_channel_id.is_some() {
data.forward_channel_id = None;
"Remove successfully.".to_string()
} else {
"No channel to remove.".to_string()
};
}
})
.await;
reply(bot.clone(), message.clone(), text).await?;
}
Command::EditBeforeForward => {
let chat_id = message.chat.id.0;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let text = if chat_data.forward_channel_id.is_none() {
let text = CHAT_STORE
.update(chat_id, |data| {
if data.forward_channel_id.is_none() {
"Please enable forward channel first.".to_string()
} else if chat_data.edit_before_forward {
chat_data.edit_before_forward = false;
chat_data.edit_message.clear();
CHAT_STORE.set(chat_id, &chat_data).await;
} else if data.edit_before_forward {
data.edit_before_forward = false;
data.edit_message.clear();
"Disable edit before forward.".to_string()
} else {
chat_data.edit_before_forward = true;
CHAT_STORE.set(chat_id, &chat_data).await;
data.edit_before_forward = true;
"Enable edit before forward.".to_string()
};
}
})
.await;
reply(bot.clone(), message.clone(), text).await?;
}
Command::SetTemplate(name) => {
@@ -290,11 +371,14 @@ async fn execute_command(
} else if name.is_empty() {
"Please provide a name for the template.".to_string()
} else {
let mut chat_data = CHAT_STORE.get(chat_id).await;
chat_data
.template
.insert(name, html_escape::encode_text(reply_text).into_owned());
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.template.insert(
name,
html_escape::encode_text(reply_text).into_owned(),
);
})
.await;
"Template set.".to_string()
}
}
@@ -323,7 +407,7 @@ async fn execute_command(
return Ok(());
}
};
if !["twitter", "bsky", "pixiv"].contains(&site) {
if !x_media::site::site_ids().contains(&site) {
reply(
bot.clone(),
message.clone(),
@@ -332,9 +416,11 @@ async fn execute_command(
.await?;
return Ok(());
}
let mut chat_data = CHAT_STORE.get(chat_id).await;
chat_data.message_format.insert(site.to_string(), format);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.message_format.insert(site.to_string(), format);
})
.await;
reply(bot.clone(), message.clone(), "Format set.").await?;
}
Command::ClearCache(arg) => {
@@ -388,9 +474,18 @@ async fn execute_command(
Ok(())
}
/// `""` for one, `"ies"` for anything else — "1 entry" / "2 entries".
/// `"y"` for one, `"ies"` for anything else — "1 entry" / "2 entries".
fn plural(n: usize) -> &'static str {
if n == 1 { "" } else { "ies" }
if n == 1 { "y" } else { "ies" }
}
/// Registers the bot's command list with Telegram so clients show it in the
/// `/` menu (Bot API `setMyCommands`).
pub async fn register_commands(bot: &Bot) -> Result<(), RequestError> {
let commands = Command::bot_commands();
bot.set_my_commands(commands.clone()).await?;
log::info!("registered {} commands", commands.len());
Ok(())
}
/// For locally produced media (encoded ugoira MP4) the thumbnail URL is a
@@ -452,14 +547,23 @@ async fn dispatch_send(bot: Bot, message: &Message, task: &Task, url: &str) {
};
match result {
Ok(message_ids) => {
log::info!("sent {} message(s) for {url}", message_ids.len());
log::info!(
"sent {} message(s) for [key={}]",
message_ids.len(),
log_key(url)
);
send::post_send_actions(&bot, task, message_ids).await;
// The task settled: drop any keep-alive temp media.
send::release_keep_alive(task);
}
Err(send::SendError::Retryable {
delay_seconds,
task,
}) => {
log::info!("send for {url} failed, queued for retry in {delay_seconds:.1}s");
log::info!(
"send for [key={}] failed, queued for retry in {delay_seconds:.1}s",
log_key(url)
);
enqueue_retry(task, delay_seconds).await;
let _ = reply(bot, message.clone(), "Send failed. Task queued for retry.").await;
}
@@ -468,6 +572,7 @@ async fn dispatch_send(bot: Bot, message: &Message, task: &Task, url: &str) {
task,
}) => {
send::invalidate_cache(&task).await;
send::release_keep_alive(&task);
log::error!("send for {url} failed permanently: {err_message}");
let _ = reply(bot, message.clone(), format!("Send failed: {err_message}")).await;
}
@@ -504,7 +609,9 @@ fn build_send_task(
chat_id,
reply_to_message_id: message.id.0 as i64,
caption,
media_batches: send::chunk_media_items(items),
// Photos first so a mixed photo+video group starts with a photo
// (Telegram's sendMediaGroup rule); order within each kind is kept.
media_batches: send::chunk_media_items(send::photos_first(items)),
batch_index: 0,
sent_message_ids: vec![],
source_url,
@@ -532,16 +639,18 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
if let Some(key) = x_media::site::cache_key(url)
&& let Some(cached) = LINK_CACHE.get(&key, CONFIG.link_cache_ttl).await
{
log::info!("link cache hit for {url}");
log::debug!("link cache hit for {key}");
let chat_data = CHAT_STORE.get(chat_id).await;
let site = key.split(':').next().unwrap_or("unknown");
// Cache keys are prefixed with the site id ("twitter:…"), matching
// the value a fresh fetch would read from Fetched::site_id.
let site = x_media::site::site_id_from_key(&key);
let format = chat_data
.message_format
.get(site)
.cloned()
.unwrap_or_default();
let caption = if format.is_empty() {
cached.caption.clone()
x_media::site::truncate_caption(&cached.caption)
} else {
x_media::site::caption_from_fields(
&format,
@@ -589,11 +698,11 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
return;
}
log::info!("fetching {url}");
log::debug!("fetching {url} [key={}]", log_key(url));
match x_media::site::fetch(url).await {
// Unsupported links are ignored silently (Python parity).
Ok(None) => {
log::info!("no site pattern matches {url}; ignoring");
log::debug!("no site pattern matches {url}; ignoring");
}
// Retries exhausted: notify the user (Rust-only requirement 3).
Err(e) => {
@@ -605,7 +714,7 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
)
.await;
}
Ok(Some(fetched)) => {
Ok(Some(mut fetched)) => {
if fetched.media.is_empty() {
let _ = reply(
bot,
@@ -650,6 +759,13 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
items,
cache_data,
);
// Hand the keep-alive temp dir (ugoira / bsky remux MP4) to the
// retry registry: a queued retry runs after this function returns
// and the fetch's own TempDir is dropped, so without this the
// local file would be gone by the time the retry sends it.
if let Some(dir) = fetched.take_keep_alive() {
send::KEEP_ALIVE.lock().push(dir);
}
dispatch_send(bot, message, &task, url).await;
}
}
@@ -664,9 +780,13 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
.unwrap_or_else(|| "unknown".to_string());
let text_preview = message
.text()
.map(|t| if t.len() > 120 { &t[..120] } else { t })
.map(|t| {
let end = t.floor_char_boundary(120.min(t.len()));
&t[..end]
})
.unwrap_or("<no text>");
log::info!(
// Per-request detail: debug only (message text is user data).
log::debug!(
"message from {sender} in {} (private={is_private}): {text_preview}",
message.chat.id
);
@@ -677,36 +797,120 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
if let Some(text) = message.text()
&& let Ok(command) = Command::parse(text, "")
{
log::info!("command from {}: {text_preview}", message.chat.id);
log::debug!("command from {}: {text_preview}", message.chat.id);
execute_command(&bot, &message, command).await?;
return respond(());
}
if is_private {
let urls = extract_urls(&message);
if !urls.is_empty() {
log::info!("extracted {} URL(s): {urls:?}", urls.len());
// Debug only, and echo the normalized keys instead of the raw URLs.
let keys: Vec<String> = urls.iter().map(|u| log_key(u)).collect();
log::debug!("extracted {} URL(s): {keys:?}", urls.len());
}
for url in urls {
let bot = bot.clone();
let message = message.clone();
tokio::spawn(async move {
// Held for the whole task; the semaphore is never closed.
let _permit = URL_TASKS.acquire().await.expect("URL semaphore closed");
url_media(bot, &message, &url).await;
});
// Clone out of the lock: the parking_lot guard is !Send and must
// not be held across the await below.
let Some(tx) = URL_JOBS.lock().clone() else {
log::warn!("url workers not started; dropping link");
break;
};
let _ = tx.send((bot.clone(), message.clone(), url)).await;
}
}
respond(())
}
/// Debounce window for inline queries: Telegram fires an inline query on
/// every keystroke, and each prefix of a pasted URL (e.g. `.../status/12`,
/// `.../status/123`, ...) already matches the site patterns. Without a
/// debounce every keystroke triggers a fetch (3 attempts!) of a half-typed
/// post id. Only answer once the query has been stable for this long.
const INLINE_DEBOUNCE: std::time::Duration = std::time::Duration::from_millis(800);
/// Last seen inline query and whether it was already answered. Guards the
/// debounce timer: a repeat of an answered query is served by Telegram's
/// inline cache (see `cache_time`), not by another fetch.
struct InlineDebounceState {
query: String,
answered: bool,
}
static INLINE_DEBOUNCE_STATE: LazyLock<parking_lot::Mutex<Option<InlineDebounceState>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), RequestError> {
if query.query.is_empty() {
return respond(());
}
log::info!("inline query: {}", query.query);
// Only run a fetch for something that is actually a supported post URL.
if x_media::site::cache_key(&query.query).is_none() {
return respond(());
}
// Debounce: record the query and answer only after it has been stable for
// INLINE_DEBOUNCE (the timer below). An already-answered repeat of the
// same query is left to Telegram's inline cache instead of re-fetching.
{
let mut state = INLINE_DEBOUNCE_STATE.lock();
if let Some(prev) = state.as_ref()
&& prev.query == query.query
&& prev.answered
{
return respond(());
}
*state = Some(InlineDebounceState {
query: query.query.clone(),
answered: false,
});
}
let query_text = query.query.clone();
tokio::spawn(async move {
tokio::time::sleep(INLINE_DEBOUNCE).await;
// Only the last query of a typing burst survives: earlier timers see
// the query changed and give up without answering.
{
let mut state = INLINE_DEBOUNCE_STATE.lock();
let Some(state) = state.as_mut() else {
return;
};
if state.query != query_text || state.answered {
return;
}
// Claim the answer so a repeat of the same query cannot start a
// second fetch; reset below when no answer was produced.
state.answered = true;
}
match answer_inline_query(bot, query).await {
Ok(true) => {}
// No results produced (or nothing to answer): let a repeat of the
// same query retry the fetch.
Ok(false) | Err(_) => {
let mut state = INLINE_DEBOUNCE_STATE.lock();
if let Some(state) = state.as_mut()
&& state.query == query_text
{
state.answered = false;
}
}
}
});
respond(())
}
/// Fetches the post behind an inline query and answers it. The caller has
/// already applied the debounce. Returns `true` when an answer was sent.
async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, RequestError> {
log::debug!(
"inline query: {} [key={}]",
query.query,
log_key(&query.query)
);
match x_media::site::fetch(&query.query).await {
Ok(Some(fetched)) => {
let mut results: Vec<InlineQueryResult> = Vec::new();
// Inline results have the same 1024-char caption limit as regular
// messages; truncate once here for all items.
let caption = x_media::site::truncate_caption(&fetched.caption);
for (i, media) in fetched.media.iter().enumerate() {
let id = format!("{i}");
let Some(url) = url::Url::parse(media.url()).ok() else {
@@ -716,7 +920,7 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
.thumbnail_url()
.and_then(|t| url::Url::parse(t).ok())
.unwrap_or_else(|| url.clone());
let caption = fetched.caption.clone();
let caption = caption.clone();
let result = match media {
Media::Illustration { .. } => {
// Inline photo results have their own (smaller) size
@@ -751,13 +955,18 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
results.push(result);
}
if !results.is_empty() {
bot.answer_inline_query(query.id, results).await?;
// Explicit cache window: repeats of the same query within 5
// minutes are served by Telegram without hitting the bot.
bot.answer_inline_query(query.id, results)
.cache_time(300)
.await?;
return Ok(true);
}
}
Ok(None) => {}
Err(e) => log::error!("inline fetch {}: {e}", query.query),
}
respond(())
Ok(false)
}
pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<(), RequestError> {
@@ -769,10 +978,10 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let chat_id = message.chat().id.0;
let prompt_message_id = message.id().0 as i64;
let ttl_secs = CONFIG.edit_message_ttl.as_secs() as i64;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let chat_data = CHAT_STORE.get(chat_id).await;
let edit = chat_data.edit_message.get(&prompt_message_id).cloned();
let Some(edit) = edit else {
log::info!(
log::debug!(
"callback from {}: no edit record for prompt {prompt_message_id}",
chat_id
);
@@ -783,8 +992,11 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
};
// Lazy expiry: a stale record (past the TTL, not yet swept) is dropped.
if edit.created_at + ttl_secs <= unix_now() {
chat_data.edit_message.remove(&prompt_message_id);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
bot.answer_callback_query(callback_query_id)
.text("Expired")
.await?;
@@ -820,8 +1032,11 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let _ = bot
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
.await;
chat_data.edit_message.remove(&prompt_message_id);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
}
Err(send::SendError::Retryable {
delay_seconds,
@@ -842,7 +1057,7 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
}
}
None => {
log::info!("forward callback without a forward channel set");
log::debug!("forward callback without a forward channel set");
bot.answer_callback_query(callback_query_id)
.text("No forward channel set.")
.await?;
@@ -860,10 +1075,13 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
.caption(template_html)
.parse_mode(ParseMode::Html)
.await;
if let Some(entry) = chat_data.edit_message.get_mut(&prompt_message_id) {
CHAT_STORE
.update(chat_id, |data| {
if let Some(entry) = data.edit_message.get_mut(&prompt_message_id) {
entry.template = name.to_string();
}
CHAT_STORE.set(chat_id, &chat_data).await;
})
.await;
log::info!("template '{name}' applied to prompt {prompt_message_id}");
}
bot.answer_callback_query(callback_query_id).await?;
+38 -33
View File
@@ -8,8 +8,10 @@
//! [`Config::link_cache_ttl`]; a stale entry is dropped lazily on read and
//! by the periodic prune in `main`.
use rusqlite::{Connection, params};
use crate::db::now_f64;
use rusqlite::params;
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use std::time::Duration;
#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)]
@@ -44,24 +46,16 @@ pub struct CachedPost {
}
/// SQLite-backed cache sharing `data/task_queue.db` with the queue and chat
/// state (same `open_db` pattern: busy timeout, `spawn_blocking` I/O).
/// state (same shared pool, see [`crate::db::open_store`]).
pub struct LinkCache {
db_path: String,
pool: Arc<crate::db::DbPool>,
}
impl LinkCache {
pub fn open(db_path: &str) -> Self {
if let Ok(conn) = Connection::open(db_path)
&& let Err(e) = conn.execute_batch(
"CREATE TABLE IF NOT EXISTS link_cache (url TEXT PRIMARY KEY, \
payload TEXT NOT NULL, created_at REAL NOT NULL);",
)
{
log::error!("failed to initialize link cache schema: {e}");
}
Self {
db_path: db_path.to_string(),
}
/// Wraps the shared DB pool (the `link_cache` table lives in the merged
/// schema alongside `tasks` and `chat_state`).
pub fn new(pool: Arc<crate::db::DbPool>) -> Self {
LinkCache { pool }
}
/// Returns the cached post if present and not expired; a stale entry is
@@ -69,7 +63,9 @@ impl LinkCache {
pub async fn get(&self, key: &str, ttl: Duration) -> Option<CachedPost> {
let key = key.to_string();
let ttl = ttl.as_secs_f64();
let result = crate::db::with_conn(&self.db_path, move |conn| {
let result = self
.pool
.with_conn(move |conn| {
let mut stmt =
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
let mut rows = stmt.query(params![key])?;
@@ -99,7 +95,9 @@ impl LinkCache {
pub async fn put(&self, key: &str, post: &CachedPost) {
let key = key.to_string();
let payload = serde_json::to_string(post).expect("cached post serializes");
let result = crate::db::with_conn(&self.db_path, move |conn| {
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"INSERT OR REPLACE INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
params![key, payload, now_f64()],
@@ -115,7 +113,9 @@ impl LinkCache {
/// Drops an entry (e.g. a cached file id that turned out invalid).
pub async fn remove(&self, key: &str) {
let key = key.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
let result = self
.pool
.with_conn(move |conn| {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Ok(())
})
@@ -128,7 +128,9 @@ impl LinkCache {
/// Removes expired entries; returns how many were deleted.
pub async fn prune(&self, ttl: Duration) -> usize {
let cutoff = now_f64() - ttl.as_secs_f64();
let result = crate::db::with_conn(&self.db_path, move |conn| {
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"DELETE FROM link_cache WHERE created_at < ?1",
params![cutoff],
@@ -148,7 +150,9 @@ impl LinkCache {
/// `key` is `None`. Returns how many rows were removed.
pub async fn clear(&self, key: Option<&str>) -> usize {
let key = key.map(str::to_string);
let result = crate::db::with_conn(&self.db_path, move |conn| match &key {
let result = self
.pool
.with_conn(move |conn| match &key {
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
None => conn.execute("DELETE FROM link_cache", []),
})
@@ -163,13 +167,6 @@ impl LinkCache {
}
}
fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -193,7 +190,9 @@ mod tests {
#[tokio::test]
async fn put_get_roundtrip() {
let dir = tempfile::tempdir().unwrap();
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
cache.put("twitter:1", &entry()).await;
let got = cache.get("twitter:1", Duration::from_secs(3600)).await;
assert!(got.is_some());
@@ -205,11 +204,13 @@ mod tests {
#[tokio::test]
async fn expired_entry_removed_on_read() {
let dir = tempfile::tempdir().unwrap();
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
cache.put("twitter:1", &entry()).await;
// Force the row into the past so a 1s TTL expires it.
{
let conn = Connection::open(dir.path().join("c.db")).unwrap();
let conn = rusqlite::Connection::open(dir.path().join("c.db")).unwrap();
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
.unwrap();
}
@@ -230,7 +231,9 @@ mod tests {
#[tokio::test]
async fn remove_and_prune() {
let dir = tempfile::tempdir().unwrap();
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
cache.put("twitter:1", &entry()).await;
cache.put("pixiv:2", &entry()).await;
cache.remove("twitter:1").await;
@@ -247,7 +250,7 @@ mod tests {
.is_some()
);
{
let conn = Connection::open(dir.path().join("c.db")).unwrap();
let conn = rusqlite::Connection::open(dir.path().join("c.db")).unwrap();
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
.unwrap();
}
@@ -263,7 +266,9 @@ mod tests {
#[tokio::test]
async fn clear_one_entry_or_all() {
let dir = tempfile::tempdir().unwrap();
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
cache.put("twitter:1", &entry()).await;
cache.put("pixiv:2", &entry()).await;
// By key: only the matching row is removed.
+35 -11
View File
@@ -43,6 +43,14 @@ async fn main() {
log::info!("Starting bot");
let bot = Bot::from_env();
// Force the queue workers' shared Bot to initialize now so a missing
// token fails at startup, not on the first queued task.
let _ = &*send::BOT;
// Register the command list with Telegram (client `/` menu).
if let Err(e) = handlers::register_commands(&bot).await {
log::warn!("failed to register commands: {e}");
}
log::info!(
"config: {} admin(s), edit-message TTL {}s",
@@ -57,20 +65,23 @@ async fn main() {
.await;
log::info!("task queue worker started");
// Pixiv login validation (user request): a failed login notifies the
// admin and disables pixiv for this process.
if site::pixiv::enabled() {
match site::pixiv::validate().await {
Ok(()) => log::info!("pixiv login validated"),
Err(e) => {
log::error!("pixiv login failed: {e}");
// URL job workers: bounded channel + fixed pool for per-URL work.
handlers::start_url_workers().await;
log::info!("url workers started");
// Site login validation (user request): a failed login notifies the
// admin and the site disables itself for this process (pixiv).
let failures = site::validate_all().await;
if failures.is_empty() {
log::info!("site logins validated");
} else {
for (site_id, message) in &failures {
log::error!("{site_id} login failed: {message}");
if let Some(admin) = CONFIG.admin_ids.first() {
let _ = bot
.send_message(ChatId(*admin), format!("Pixiv login failed: {e}"))
.send_message(ChatId(*admin), format!("{site_id} login failed: {message}"))
.await;
}
site::pixiv::disable();
}
}
}
@@ -166,12 +177,25 @@ async fn main() {
.await;
}
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin, drain the queue.
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin,
// drain the queue. Bounded: a worker mid-download (30 s timeout) or a
// long ugoira encode must not hold the shutdown hostage forever.
log::info!("Stopping bot");
const SHUTDOWN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
let shutdown = async {
let _ = stop_tx.send(true);
handlers::stop_url_workers().await;
if let Some(admin) = CONFIG.admin_ids.first() {
let _ = bot.send_message(ChatId(*admin), "Shutting down...").await;
}
TASK_QUEUE.stop().await;
};
if tokio::time::timeout(SHUTDOWN_TIMEOUT, shutdown)
.await
.is_err()
{
log::warn!("graceful shutdown timed out after {SHUTDOWN_TIMEOUT:?}; exiting");
} else {
log::info!("Bot stopped");
}
}
+18 -16
View File
@@ -26,8 +26,9 @@ pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900;
/// to a smaller media URL instead.
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024;
/// Decode budget (bytes): a larger intermediate buffer is not worth the peak
/// memory; the photo degrades to the smaller URL instead.
const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
/// memory; the photo degrades to the smaller URL instead. Also the cap for
/// downloading photos in the send fallback (they must be downloaded whole).
pub(crate) const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
/// JPEG output quality (1-100).
const JPEG_QUALITY: u8 = 90;
@@ -66,8 +67,9 @@ impl PixBuf {
}
/// Entry point: detects the format and processes the photo if needed.
pub fn prepare_photo(file: NamedTempFile) -> Result<PhotoPrep, String> {
let bytes = std::fs::read(file.path()).map_err(|e| format!("prepare read failed: {e}"))?;
/// The caller hands in the already-downloaded bytes (they are in memory from
/// the download anyway; re-reading the temp file would double the I/O).
pub fn prepare_photo(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {
prepare_png(file, bytes)
} else if bytes.starts_with(&[0xFF, 0xD8, 0xFF]) {
@@ -215,13 +217,13 @@ fn target_dims(w: u32, h: u32) -> (u32, u32) {
/// PNG branch: decode (16→8, palette→RGB; gray/GA stay), flatten RGBA to
/// RGB, Lanczos-downscale beyond the dimension cap, encode PNG — a PNG still
/// over the upload cap afterwards becomes JPEG.
fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
let (w, h, _bit_depth, color_type) = parse_png_header(&bytes).ok_or("invalid PNG header")?;
fn prepare_png(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
let (w, h, _bit_depth, color_type) = parse_png_header(bytes).ok_or("invalid PNG header")?;
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
return Ok(PhotoPrep::Upload(file));
}
log::info!(
log::debug!(
"photo {w}x{h} ({_bit_depth:?} {color_type:?}, {} bytes) needs processing",
bytes.len()
);
@@ -238,7 +240,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
png::ColorType::Indexed => png::Transformations::EXPAND,
_ => png::Transformations::STRIP_16,
};
let mut decoder = png::Decoder::new(std::io::Cursor::new(&bytes));
let mut decoder = png::Decoder::new(std::io::Cursor::new(bytes));
decoder.set_transformations(transforms);
let mut reader = decoder
.read_info()
@@ -267,7 +269,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
let (nw, nh) = target_dims(w, h);
pix = resize_pix(pix, w, h, nw, nh)?;
(w, h) = (nw, nh);
log::info!("downscaled photo to {w}x{h} (Lanczos3)");
log::debug!("downscaled photo to {w}x{h} (Lanczos3)");
}
let mut png_bytes = Vec::new();
@@ -275,7 +277,7 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
if png_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
return Ok(PhotoPrep::Upload(write_temp(&png_bytes, "png")?));
}
log::info!("PNG still over the upload cap after processing; transcoding to JPEG");
log::debug!("PNG still over the upload cap after processing; transcoding to JPEG");
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
return Ok(PhotoPrep::Upload(write_temp(&jpeg_bytes, "jpg")?));
@@ -285,8 +287,8 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
}
/// JPEG branch: zune-jpeg decode → Lanczos downscale → jpeg-encoder output.
fn prepare_jpeg(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(&bytes));
fn prepare_jpeg(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(bytes));
// Decodes to RGB by default. Headers first so dimensions are known before
// the (potentially huge) pixel decode.
decoder
@@ -309,7 +311,7 @@ fn prepare_jpeg(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String
let (nw, nh) = target_dims(w, h);
pix = resize_pix(pix, w, h, nw, nh)?;
(w, h) = (nw, nh);
log::info!("downscaled jpeg to {w}x{h} (Lanczos3)");
log::debug!("downscaled jpeg to {w}x{h} (Lanczos3)");
}
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
@@ -422,7 +424,7 @@ mod tests {
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
prepare_photo(file)
prepare_photo(file, &bytes)
}
#[test]
@@ -467,7 +469,7 @@ mod tests {
}
let mut file = tempfile::Builder::new().suffix(".jpg").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
match prepare_photo(file).unwrap() {
match prepare_photo(file, &bytes).unwrap() {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
assert!(out.starts_with(&[0xFF, 0xD8]), "output must stay jpeg");
@@ -515,7 +517,7 @@ mod tests {
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
match prepare_photo(file).unwrap() {
match prepare_photo(file, &bytes).unwrap() {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
assert!(out.starts_with(&[0xFF, 0xD8]), "must transcode to JPEG");
+158 -73
View File
@@ -5,13 +5,14 @@
//! flow. The Python dict-mutation hack (attempts inside the payload) is
//! replaced by dedicated columns.
use crate::db::now_f64;
use parking_lot::Mutex;
use rusqlite::{Connection, TransactionBehavior, params};
use rusqlite::{TransactionBehavior, params};
use serde_json::Value;
use std::pin::Pin;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use std::time::Duration;
use tokio::sync::Notify;
use tokio::task::JoinHandle;
@@ -39,7 +40,7 @@ type Handler = dyn Fn(Value) -> BoxFuture<'static, Result<(), QueueError>> + Sen
type DeadLetter = dyn Fn(Value, String) -> BoxFuture<'static, ()> + Send + Sync;
pub struct PersistentTaskQueue {
db_path: String,
pool: std::sync::Arc<crate::db::DbPool>,
notify: Arc<Notify>,
stop: Arc<AtomicBool>,
worker: Mutex<Vec<JoinHandle<()>>>,
@@ -53,47 +54,39 @@ struct LeasedRow {
}
/// Owned worker state so the spawned loop does not borrow the queue handle.
#[derive(Clone)]
struct QueueWorker {
db_path: String,
pool: std::sync::Arc<crate::db::DbPool>,
notify: Arc<Notify>,
stop: Arc<AtomicBool>,
handler: Arc<Handler>,
dead_letter: Arc<DeadLetter>,
}
fn now_f64() -> f64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
/// Resets rows left `in_progress` with an expired lock TTL back to `pending`
/// so they can be leased again (crash/panic recovery).
fn recover_update(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
conn.execute(
"UPDATE tasks SET status='pending', locked_until=0 WHERE status='in_progress' AND locked_until < ?1",
params![now_f64()],
)?;
Ok(())
}
fn ensure_schema(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
locked_until REAL NOT NULL, created_at REAL NOT NULL);",
)
/// Base delay × 2^attempts (attempts = retries already done), capped at 300s.
/// Applied at the queue layer so the attempt count actually reaches the
/// backoff computation; Telegram `RetryAfter` delays get the same treatment
/// (conservatively larger wait, no API change needed).
fn scaled_retry_delay(base: f64, attempts: i32) -> f64 {
(base * 2f64.powi(attempts)).min(300.0)
}
impl PersistentTaskQueue {
pub fn new(db_path: &str) -> Self {
// Ensure the parent dir and table exist even if only the queue (not
// ChatStore) is used — a fresh container without a mounted data dir
// must still be able to open the DB.
if let Some(parent) = std::path::Path::new(db_path).parent()
&& !parent.as_os_str().is_empty()
&& let Err(e) = std::fs::create_dir_all(parent)
{
log::error!("failed to create queue dir: {e}");
}
if let Ok(conn) = Connection::open(db_path)
&& let Err(e) = ensure_schema(&conn)
{
log::error!("failed to initialize queue schema: {e}");
}
/// Wraps the shared DB pool; the schema is initialized once by
/// [`crate::db::open_store`] (all three stores share the pool).
pub fn new(pool: std::sync::Arc<crate::db::DbPool>) -> Self {
Self {
db_path: db_path.to_string(),
pool,
notify: Arc::new(Notify::new()),
stop: Arc::new(AtomicBool::new(false)),
worker: Mutex::new(Vec::new()),
@@ -114,17 +107,43 @@ impl PersistentTaskQueue {
let dead_letter: Arc<DeadLetter> =
Arc::new(move |payload, message| Box::pin(dead_letter(payload, message)));
self.recover_stale().await;
let mut handles = Vec::with_capacity(QUEUE_WORKERS);
let mut handles = Vec::with_capacity(QUEUE_WORKERS + 1);
for _ in 0..QUEUE_WORKERS {
let worker = QueueWorker {
db_path: self.db_path.clone(),
pool: std::sync::Arc::clone(&self.pool),
notify: Arc::clone(&self.notify),
stop: Arc::clone(&self.stop),
handler: Arc::clone(&handler),
dead_letter: Arc::clone(&dead_letter),
};
handles.push(tokio::spawn(worker.run_loop()));
handles.push(tokio::spawn(worker.run_loop_supervised()));
}
// Periodic lease-expiry sweep: recovers rows a crashed/panicked
// worker left `in_progress` (the lock TTL bounds the wait). Woken by
// the same notify as the workers, so enqueue and stop interrupt the
// sleep; the first interval tick fires immediately (harmless extra
// recovery at startup).
let sweep_pool = std::sync::Arc::clone(&self.pool);
let sweep_notify = Arc::clone(&self.notify);
let sweep_stop = Arc::clone(&self.stop);
handles.push(tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(30));
loop {
let notified = sweep_notify.notified();
tokio::pin!(notified);
tokio::select! {
_ = &mut notified => {}
_ = interval.tick() => {}
}
if sweep_stop.load(Ordering::Relaxed) {
break;
}
let result = sweep_pool.with_conn(move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue sweep failed: {e}");
}
}
}));
*self.worker.lock() = handles;
}
@@ -147,8 +166,8 @@ impl PersistentTaskQueue {
self.counter.fetch_add(1, Ordering::Relaxed)
);
let payload = payload.to_string();
log::info!("enqueued {id} (run_after {run_after:.1})");
crate::db::with_conn(&self.db_path, move |conn| {
log::debug!("enqueued {id} (run_after {run_after:.1})");
self.pool.with_conn(move |conn| {
conn.execute(
"INSERT OR REPLACE INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES (?1, ?2, ?3, 0, 'pending', 0, ?4)",
@@ -157,21 +176,20 @@ impl PersistentTaskQueue {
Ok(())
})
.await?;
// Wake every sleeping worker: with several workers the one that finds
// nothing due must not starve the newly inserted row.
self.notify.notify_waiters();
// `notify_one` stores a permit when no worker is registered, so a
// notification fired between a worker's DB reads and its `notified()`
// registration is not lost (notify_waiters would drop it). The
// awakened worker re-leases and finds the new row.
self.notify.notify_one();
Ok(())
}
async fn recover_stale(&self) {
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"UPDATE tasks SET status='pending', locked_until=0 WHERE status='in_progress' AND locked_until < ?1",
params![now_f64()],
)?;
Ok(())
})
.await;
self.recover_sweep().await;
}
async fn recover_sweep(&self) {
let result = self.pool.with_conn(move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue recovery failed: {e}");
}
@@ -179,11 +197,24 @@ impl PersistentTaskQueue {
}
impl QueueWorker {
/// Supervised worker: the inner loop runs in its own task so a panic
/// (e.g. inside a handler or a DB closure) kills only that task; the
/// supervisor respawns it until stop is set. The row a dead worker had
/// leased is recovered by the periodic sweep once its lock TTL expires.
async fn run_loop_supervised(self) {
while !self.stop.load(Ordering::Relaxed) {
let worker = self.clone();
if let Err(e) = tokio::spawn(async move { worker.run_loop().await }).await {
log::error!("queue worker panicked, restarting: {e}");
}
}
}
async fn run_loop(self) {
while !self.stop.load(Ordering::Relaxed) {
match self.lease_next().await {
Some(row) => self.process(row).await,
None => {
Ok(Some(row)) => self.process(row).await,
Ok(None) => {
let wait_until = self.earliest_run_after().await;
let notified = self.notify.notified();
tokio::pin!(notified);
@@ -200,13 +231,20 @@ impl QueueWorker {
}
}
}
// A lease failure while rows are due would otherwise loop
// with sleep(0) and hammer SQLite; back off briefly.
Err(e) => {
log::error!("queue lease failed: {e}");
tokio::time::sleep(Duration::from_secs(1)).await;
}
}
}
}
/// Leases the oldest due row (sets it `in_progress` with a lock TTL).
async fn lease_next(&self) -> Option<LeasedRow> {
let result = crate::db::with_conn(&self.db_path, |conn| {
/// Errors are surfaced so the caller can back off instead of spinning.
async fn lease_next(&self) -> Result<Option<LeasedRow>, rusqlite::Error> {
self.pool.with_conn(|conn| {
// BEGIN IMMEDIATE: with several workers, a deferred transaction
// that read before another worker's lease commit would fail with
// SQLITE_BUSY_SNAPSHOT. Taking the write lock up front serializes
@@ -244,18 +282,13 @@ impl QueueWorker {
attempts,
}))
})
.await;
match result {
Ok(row) => row,
Err(e) => {
log::error!("queue lease failed: {e}");
None
}
}
.await
}
async fn earliest_run_after(&self) -> Option<f64> {
let result = crate::db::with_conn(&self.db_path, |conn| {
let result = self
.pool
.with_conn(|conn| {
let mut stmt =
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
let mut rows = stmt.query([])?;
@@ -284,10 +317,10 @@ impl QueueWorker {
return;
}
};
log::info!("processing {} (attempt {})", row.id, row.attempts + 1);
log::debug!("processing {} (attempt {})", row.id, row.attempts + 1);
match (self.handler)(payload).await {
Ok(()) => {
log::info!("task {} completed", row.id);
log::debug!("task {} completed", row.id);
self.delete_row(&row.id).await;
}
Err(QueueError::Retryable {
@@ -300,12 +333,13 @@ impl QueueWorker {
self.delete_row(&row.id).await;
(self.dead_letter)(payload, message).await;
} else {
log::info!(
"task {} rescheduled in {delay_seconds:.1}s (attempt {})",
let delay = scaled_retry_delay(delay_seconds, row.attempts);
log::debug!(
"task {} rescheduled in {delay:.1}s (attempt {})",
row.id,
row.attempts + 1
);
self.reschedule(&row.id, payload, delay_seconds, row.attempts + 1)
self.reschedule(&row.id, payload, delay, row.attempts + 1)
.await;
}
}
@@ -319,7 +353,9 @@ impl QueueWorker {
async fn delete_row(&self, id: &str) {
let id = id.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
let result = self
.pool
.with_conn(move |conn| {
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
Ok(())
})
@@ -332,7 +368,7 @@ impl QueueWorker {
async fn reschedule(&self, id: &str, payload: Value, delay_seconds: f64, attempts: i32) {
let id = id.to_string();
let payload = payload.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
let result = self.pool.with_conn(move |conn| {
conn.execute(
"UPDATE tasks SET payload=?1, run_after=?2, attempts=?3, status='pending', locked_until=0 WHERE id=?4",
params![payload, now_f64() + delay_seconds, attempts, id],
@@ -343,7 +379,8 @@ impl QueueWorker {
if let Err(e) = result {
log::error!("queue reschedule failed: {e}");
}
self.notify.notify_waiters();
// Same permit semantics as enqueue: never lose the wakeup.
self.notify.notify_one();
}
}
@@ -352,10 +389,21 @@ mod tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering};
#[test]
fn scaled_retry_delay_scales_and_caps() {
assert_eq!(scaled_retry_delay(1.0, 0), 1.0);
assert_eq!(scaled_retry_delay(1.0, 1), 2.0);
assert_eq!(scaled_retry_delay(1.0, 2), 4.0);
assert_eq!(scaled_retry_delay(1.5, 1), 3.0);
assert_eq!(scaled_retry_delay(1.0, 10), 300.0, "capped at 300s");
assert_eq!(scaled_retry_delay(300.0, 0), 300.0);
}
async fn new_queue() -> (PersistentTaskQueue, tempfile::TempDir) {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("queue.db");
let queue = PersistentTaskQueue::new(path.to_str().unwrap());
let pool = crate::db::open_store(path.to_str().unwrap()).unwrap();
let queue = PersistentTaskQueue::new(pool);
(queue, dir)
}
@@ -462,10 +510,11 @@ mod tests {
async fn stale_in_progress_row_is_recovered_on_start() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("queue.db");
// Insert a stale leased row directly (lease expired).
// Insert a stale leased row directly (lease expired). open_store runs
// the schema; the queue below shares the same pool.
let pool = crate::db::open_store(path.to_str().unwrap()).unwrap();
{
let conn = Connection::open(&path).unwrap();
ensure_schema(&conn).unwrap();
let conn = rusqlite::Connection::open(&path).unwrap();
conn.execute(
"INSERT INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES ('task_stale', '{\"s\":1}', 0, 0, 'in_progress', ?1, 0)",
@@ -473,7 +522,7 @@ mod tests {
)
.unwrap();
}
let queue = PersistentTaskQueue::new(path.to_str().unwrap());
let queue = PersistentTaskQueue::new(pool);
let calls = Arc::new(AtomicUsize::new(0));
let c = calls.clone();
queue
@@ -490,4 +539,40 @@ mod tests {
assert_eq!(calls.load(AtomicOrdering::SeqCst), 1);
queue.stop().await;
}
#[tokio::test]
async fn runtime_sweep_recovers_expired_lease() {
let (queue, _dir) = new_queue().await;
let calls = Arc::new(AtomicUsize::new(0));
let c = calls.clone();
queue
.start(
move |payload| {
assert_eq!(payload["s"], 1);
c.fetch_add(1, AtomicOrdering::SeqCst);
async { Ok(()) }
},
|_payload, _message| async {},
)
.await;
// Insert a stale leased row AFTER startup: without a runtime sweep it
// would stay `in_progress` forever (only start() used to recover).
{
let conn = rusqlite::Connection::open(queue.pool.path()).unwrap();
conn.execute(
"INSERT INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES ('task_stale_runtime', '{\"s\":1}', 0, 0, 'in_progress', ?1, 0)",
params![now_f64() - 1000.0],
)
.unwrap();
}
queue.recover_sweep().await;
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(
calls.load(AtomicOrdering::SeqCst),
1,
"expired lease must be recovered and processed exactly once"
);
queue.stop().await;
}
}
+425 -161
View File
@@ -3,7 +3,7 @@
//! URL is blocked by hotlink protection; the bot downloads the file itself
//! and uploads it via multipart).
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE};
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE, log_key};
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost};
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
use crate::queue::QueueError;
@@ -11,6 +11,7 @@ use crate::state::{EditMessage, unix_now};
use rand::Rng;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::LazyLock;
use teloxide::prelude::*;
use teloxide::types::{
ChatId, InlineKeyboardButton, InlineKeyboardMarkup, InputFile, InputMedia, InputMediaAnimation,
@@ -20,6 +21,11 @@ use teloxide::{ApiError, RequestError};
use tempfile::NamedTempFile;
use x_media::site::FetchError;
/// One process-wide Bot for queue workers. Building a fresh Bot (and its HTTP
/// client) per queue task was pure waste; forced at startup in main so a
/// missing token fails fast instead of on the first task.
pub static BOT: LazyLock<Bot> = LazyLock::new(Bot::from_env);
#[derive(Serialize, Deserialize, Clone, Debug)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum MediaItemPayload {
@@ -61,6 +67,15 @@ impl MediaItemPayload {
MediaItemPayload::Animation { .. } => None,
}
}
/// The cover-frame URL for videos (used by the upload fallback, which
/// otherwise drops the thumbnail the URL-send path applies).
fn thumbnail_url(&self) -> Option<&str> {
match self {
MediaItemPayload::Video { thumbnail, .. } => thumbnail.as_deref(),
MediaItemPayload::Photo { .. } | MediaItemPayload::Animation { .. } => None,
}
}
}
#[derive(Serialize, Deserialize, Clone, Debug)]
@@ -132,6 +147,41 @@ impl Task {
fn is_cached_send(&self) -> bool {
self.cache_data().is_some_and(|c| !c.media.is_empty())
}
/// All media payloads of this task (sequence batches flattened plus the
/// lone animation).
fn media_items(&self) -> Vec<&MediaItemPayload> {
match self {
Task::SendMediaSequence { media_batches, .. } => {
media_batches.iter().flatten().collect()
}
Task::SendAnimation { animation, .. } => {
std::slice::from_ref(animation).iter().collect()
}
Task::ForwardMessages { .. } => Vec::new(),
}
}
/// Local file paths referenced by this task's media (ugoira / bsky remux
/// MP4 and the like); empty for URL or Telegram file-id sends.
fn local_media_paths(&self) -> Vec<std::path::PathBuf> {
let mut out = Vec::new();
for item in self.media_items() {
let is_file_id = match item {
MediaItemPayload::Photo { file_id, .. }
| MediaItemPayload::Video { file_id, .. }
| MediaItemPayload::Animation { file_id, .. } => *file_id,
};
if is_file_id {
continue;
}
let media = item_url(item);
if !media.starts_with("http://") && !media.starts_with("https://") {
out.push(std::path::PathBuf::from(media));
}
}
out
}
}
/// Telegram file id of the message's media, matched to the payload kind.
@@ -182,7 +232,7 @@ async fn cache_sent_task(task: &Task, media: Vec<CachedMedia>) {
post.media = media;
if let Some(key) = x_media::site::cache_key(&post.url) {
LINK_CACHE.put(&key, &post).await;
log::info!("cached send for {}", post.url);
log::debug!("cached send for [key={}]", log_key(&post.url));
}
}
@@ -207,11 +257,35 @@ pub async fn invalidate_cache(task: &Task) {
&& let Some(url) = task.source_url()
&& let Some(key) = x_media::site::cache_key(url)
{
log::info!("removing stale link cache entry for {url}");
log::debug!("removing stale link cache entry for [key={}]", log_key(url));
LINK_CACHE.remove(&key).await;
}
}
/// Locally produced media files (ugoira MP4, bsky remux MP4) whose temp dirs
/// must stay alive while their task may be retried by the queue. The fetch
/// pipeline hands ownership here via [`x_media::site::Fetched::take_keep_alive`]
/// before the [`Fetched`] is dropped; a queued retry runs after that drop, so
/// without this the local file would be gone by the time the retry sends it.
/// Entries are removed when the task settles (see [`release_keep_alive`]).
pub static KEEP_ALIVE: LazyLock<parking_lot::Mutex<Vec<tempfile::TempDir>>> =
LazyLock::new(|| parking_lot::Mutex::new(Vec::new()));
/// Drops the keep-alive temp dirs holding media referenced by `task` (matched
/// by path prefix). Called once a task settles — sent or permanently failed —
/// so retry-only temp files do not leak; retryable tasks keep them alive.
pub fn release_keep_alive(task: &Task) {
let paths = task.local_media_paths();
if paths.is_empty() {
return;
}
let mut alive = KEEP_ALIVE.lock();
alive.retain(|dir| {
let dir_path = dir.path();
!paths.iter().any(|p| p.starts_with(dir_path))
});
}
pub const MAX_MEDIA_GROUP: usize = 9;
/// Splits media into batches of at most [`MAX_MEDIA_GROUP`] items.
@@ -222,6 +296,19 @@ pub fn chunk_media_items<T: Clone>(items: Vec<T>) -> Vec<Vec<T>> {
.collect()
}
/// Orders media for a Telegram media group: when photos and videos are
/// mixed, the first item must be a photo (Telegram's sendMediaGroup rule).
/// Stable sort keeps the source order within each kind; a lone animation is
/// untouched (it takes the SendAnimation path before this runs).
pub fn photos_first(items: Vec<MediaItemPayload>) -> Vec<MediaItemPayload> {
let mut items = items;
items.sort_by_key(|item| match item {
MediaItemPayload::Photo { .. } => 0,
MediaItemPayload::Video { .. } | MediaItemPayload::Animation { .. } => 1,
});
items
}
/// Exponential backoff with jitter, capped at 30s.
pub fn retry_delay_seconds(attempts: u32) -> f64 {
let jitter: f64 = rand::thread_rng().gen_range(0.2..0.8);
@@ -312,6 +399,23 @@ fn classify_to_send_error(e: &RequestError, task: Task) -> SendError {
}
}
impl SendError {
/// Attaches the (updated) task to a task-free [`FallbackError`] from the
/// download/upload pipeline. [`FallbackError::MediaTooLarge`] never
/// escapes the pipeline (it is handled by falling back to the smaller
/// URL), so it is unreachable here.
fn from_fallback(f: FallbackError, task: Task) -> SendError {
match f {
FallbackError::Retryable { delay_seconds } => SendError::Retryable {
delay_seconds,
task,
},
FallbackError::Permanent { message } => SendError::Permanent { message, task },
FallbackError::MediaTooLarge => unreachable!("handled inside the upload fallback"),
}
}
}
fn parse_media_url(s: &str) -> Result<url::Url, String> {
url::Url::parse(s).map_err(|e| format!("invalid media URL: {e}"))
}
@@ -329,6 +433,11 @@ fn item_url(item: &MediaItemPayload) -> &str {
fn input_file_for(media: &str) -> Result<InputFile, String> {
if media.starts_with("http://") || media.starts_with("https://") {
Ok(InputFile::url(parse_media_url(media)?))
} else if !std::path::Path::new(media).exists() {
// A retried task may reference a temp file the original send's
// TempDir already cleaned up; fail fast and permanent instead of
// burning retries on a file that can never come back.
Err(format!("local media file missing: {media}"))
} else {
Ok(InputFile::file(media))
}
@@ -464,32 +573,43 @@ enum FallbackError {
/// only if still too big. Anything that cannot be fixed falls back to the
/// item's smaller URL.
///
/// Downloads one media item to a temp file (deleted on drop). Network errors
/// are retryable; size over the upload cap and other download errors are not.
async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, FallbackError> {
/// Downloads one media item to a temp file (deleted on drop), returning the
/// file plus the downloaded bytes (photos keep the bytes for
/// [`photo::prepare_photo`] — re-reading the file would double the I/O).
/// Network errors are retryable; size over the upload cap and other download
/// errors are not.
async fn download_to_temp(
item: &MediaItemPayload,
) -> Result<(NamedTempFile, bytes::Bytes), FallbackError> {
let media_url = match item {
MediaItemPayload::Photo { media, .. }
| MediaItemPayload::Video { media, .. }
| MediaItemPayload::Animation { media, .. } => media,
};
let bytes = match x_media::site::download_media(media_url).await {
// Photos are downloaded even over the upload cap so `prepare_photo` can
// downscale / transcode them (cap = decode budget); videos/animations
// abort as soon as the upload cap is crossed mid-stream.
let limit = if matches!(item, MediaItemPayload::Photo { .. }) {
photo::MAX_DECODE_BYTES
} else {
MAX_UPLOAD_BYTES + 1
};
let bytes = match x_media::site::download_media_limited(media_url, limit).await {
Ok(bytes) => bytes,
Err(FetchError::Http(_)) => {
return Err(FallbackError::Retryable {
delay_seconds: retry_delay_seconds(0),
});
}
Err(FetchError::TooLarge) => {
return Err(FallbackError::MediaTooLarge);
}
Err(e) => {
return Err(FallbackError::Permanent {
message: format!("download failed: {e}"),
});
}
};
// Photos are downloaded even over the cap so `prepare_photo` can
// downscale / transcode them; only videos/animations short-circuit.
if !matches!(item, MediaItemPayload::Photo { .. }) && bytes.len() as u64 > MAX_UPLOAD_BYTES {
return Err(FallbackError::MediaTooLarge);
}
let ext = sniff_ext(&bytes);
let mut file = tempfile::Builder::new()
.suffix(&format!(".{ext}"))
@@ -503,7 +623,7 @@ async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, Fall
.map_err(|e| FallbackError::Permanent {
message: format!("temp file write failed: {e}"),
})?;
Ok(file)
Ok((file, bytes))
}
/// Builds the media group item from an uploaded file.
@@ -511,8 +631,9 @@ fn media_from_file(
item: &MediaItemPayload,
path: std::path::PathBuf,
caption: Option<&str>,
) -> InputMedia {
match item {
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(InputFile::file(path), caption, *has_spoiler)
}
@@ -522,7 +643,11 @@ fn media_from_file(
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(InputFile::file(path), caption, *has_spoiler)
}
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
}
/// Builds the media group item from a (smaller) URL.
@@ -530,8 +655,9 @@ fn media_from_url(
item: &MediaItemPayload,
url: &str,
caption: Option<&str>,
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
Ok(match item {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(input_file_for(url)?, caption, *has_spoiler)
}
@@ -541,57 +667,80 @@ fn media_from_url(
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(input_file_for(url)?, caption, *has_spoiler)
}
})
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
}
/// Download-and-reupload fallback for one media batch. Files over the upload
/// cap are not downloaded/uploaded; the item falls back to its smaller URL
/// (which Telegram fetches itself). Returns the fallback-error without the
/// task attached; callers wrap it with the updated task state.
async fn send_batch_via_upload(
bot: &Bot,
chat_id: i64,
reply_to: i64,
batch: &[MediaItemPayload],
/// One item prepared for the upload fallback: the ready-to-send media plus
/// the temp file that must stay on disk until the group request completes.
struct PreparedItem {
/// Original position in the batch (concurrent prep completes out of order).
index: usize,
media: InputMedia,
keep_alive: Option<NamedTempFile>,
}
/// Downloads / processes one media item for the upload fallback (see
/// [`send_batch_via_upload`]). Local files are uploaded directly; oversized
/// items fall back to their smaller URL; photos are downscaled/transcoded.
async fn prepare_upload_item(
item: MediaItemPayload,
index: usize,
caption: Option<&str>,
) -> Result<Vec<Message>, FallbackError> {
let mut files = Vec::new();
let mut items = Vec::new();
for (i, item) in batch.iter().enumerate() {
let item_caption = if i == 0 { caption } else { None };
) -> Result<PreparedItem, FallbackError> {
// Locally produced files (ugoira / bsky remux MP4): nothing to download
// or shrink — upload the file directly. The send is a multipart upload,
// so the only remaining failure is an upload-cap error, which is
// permanent (a video cannot be re-encoded here).
let media_url = item_url(&item);
if !media_url.starts_with("http://") && !media_url.starts_with("https://") {
let media = media_from_file(
&item,
std::path::PathBuf::from(media_url),
caption,
item.thumbnail_url(),
)
.map_err(|message| FallbackError::Permanent { message })?;
return Ok(PreparedItem {
index,
media,
keep_alive: None,
});
}
// Size check before downloading/uploading: over the cap, use the
// smaller URL instead of the file. Photos are exempt — they are
// downloaded and processed (downscale / PNG→JPEG) before uploading.
let too_large = match x_media::site::media_size(item_url(item)).await {
let too_large = match x_media::site::media_size(media_url).await {
Ok(Some(size)) => size > MAX_UPLOAD_BYTES,
_ => false,
};
let too_large = too_large && !matches!(item, MediaItemPayload::Photo { .. });
let media = if too_large {
match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
},
None => {
return Err(FallbackError::Permanent {
if too_large {
let url = item
.fallback_url()
.ok_or_else(|| FallbackError::Permanent {
message: "media too large".into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
return Ok(PreparedItem {
index,
media,
keep_alive: None,
});
}
}
} else {
match download_to_temp(item).await {
Ok(file) => {
// Telegram rejects photos wider+taller than 10000 px
// combined (PHOTO_INVALID_DIMENSIONS): downscale the
// downloaded file before uploading; photos that cannot be
// brought within the limits degrade to the smaller URL.
match download_to_temp(&item).await {
Ok((file, bytes)) => {
if matches!(item, MediaItemPayload::Photo { .. }) {
// CPU-heavy (decode/resize/encode): run off the async
// executor thread.
let prep = tokio::task::spawn_blocking(move || photo::prepare_photo(file))
// Telegram rejects photos wider+taller than 10000 px combined
// (PHOTO_INVALID_DIMENSIONS): downscale the downloaded file
// before uploading; photos that cannot be brought within the
// limits degrade to the smaller URL. CPU-heavy work runs off
// the async executor thread.
let prep = tokio::task::spawn_blocking(move || photo::prepare_photo(file, &bytes))
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
@@ -600,64 +749,134 @@ async fn send_batch_via_upload(
match prep {
PhotoPrep::Upload(upload) => {
let path = upload.path().to_path_buf();
files.push(upload);
media_from_file(item, path, item_caption)
let media = media_from_file(&item, path, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: Some(upload),
})
}
PhotoPrep::UseFallback => match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
},
None => {
return Err(FallbackError::Permanent {
message:
"photo dimensions exceed Telegram limits and no smaller variant is available"
PhotoPrep::UseFallback => {
let url = item.fallback_url().ok_or_else(|| FallbackError::Permanent {
message: "photo dimensions exceed Telegram limits and no smaller variant is available"
.into(),
});
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: None,
})
}
},
}
} else {
let path = file.path().to_path_buf();
files.push(file);
media_from_file(item, path, item_caption)
let media = media_from_file(&item, path, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: Some(file),
})
}
}
Err(FallbackError::MediaTooLarge) => match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
},
None => {
return Err(FallbackError::Permanent {
Err(FallbackError::MediaTooLarge) => {
let url = item
.fallback_url()
.ok_or_else(|| FallbackError::Permanent {
message: "media too large".into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: None,
})
}
Err(e) => Err(e),
}
}
/// Download-and-reupload fallback for one media batch. Files over the upload
/// cap are not downloaded/uploaded; the item falls back to its smaller URL
/// (which Telegram fetches itself). Items are prepared concurrently (bounded)
/// because the downloads are network-bound; the batch is then uploaded in its
/// original order. Returns the fallback-error without the task attached;
/// callers wrap it with the updated task state.
async fn send_batch_via_upload(
bot: &Bot,
chat_id: i64,
reply_to: i64,
batch: &[MediaItemPayload],
caption: Option<&str>,
task: Task,
) -> Result<Vec<Message>, SendError> {
let sem = std::sync::Arc::new(tokio::sync::Semaphore::new(3));
let mut set = tokio::task::JoinSet::new();
for (i, item) in batch.iter().enumerate() {
let item_caption = if i == 0 {
caption.map(str::to_string)
} else {
None
};
let item = item.clone();
let sem = std::sync::Arc::clone(&sem);
set.spawn(async move {
let _permit = sem.acquire().await.expect("upload semaphore closed");
prepare_upload_item(item, i, item_caption.as_deref()).await
});
}
},
Err(e) => return Err(e),
let mut prepared: Vec<Option<InputMedia>> = (0..batch.len()).map(|_| None).collect();
let mut keep_alive: Vec<NamedTempFile> = Vec::new();
while let Some(joined) = set.join_next().await {
let item = match joined {
Ok(Ok(item)) => item,
// Dropping the JoinSet aborts the remaining prep tasks; their
// temp files are cleaned up on drop (short-circuit like before).
Ok(Err(e)) => return Err(SendError::from_fallback(e, task.clone())),
Err(e) => {
return Err(SendError::Permanent {
message: format!("upload worker panicked: {e}"),
task,
});
}
};
items.push(media);
let PreparedItem {
index,
media,
keep_alive: file_opt,
} = item;
if let Some(file) = file_opt {
keep_alive.push(file);
}
prepared[index] = Some(media);
}
let items: Vec<InputMedia> = prepared
.into_iter()
.map(|m| m.expect("every upload item was prepared"))
.collect();
// `keep_alive` holds the temp files until the group request completes.
let result = bot
.send_media_group(ChatId(chat_id), items)
.reply_parameters(
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
)
.await;
drop(keep_alive);
match result {
Ok(messages) => Ok(messages),
Err(e) => Err(match classify_request_error(&e) {
Classification::Retryable { delay_seconds } => {
FallbackError::Retryable { delay_seconds }
}
Classification::Permanent { message } => FallbackError::Permanent { message },
Classification::MediaFetchFailure => FallbackError::Permanent {
Classification::Retryable { delay_seconds } => SendError::Retryable {
delay_seconds,
task: task.clone(),
},
Classification::Permanent { message } => SendError::Permanent { message, task },
Classification::MediaFetchFailure => SendError::Permanent {
message: "upload failed".into(),
task,
},
}),
}
@@ -743,7 +962,7 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
.await
{
Ok(messages) => {
log::info!(
log::debug!(
"media group batch {idx}/{} sent ({} item(s))",
media_batches.len(),
batch.len()
@@ -754,26 +973,27 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
log::info!(
"Telegram could not fetch media for batch {idx} ({}), downloading and reuploading",
batch.first().map(item_url).unwrap_or("?")
batch
.first()
.map(item_url)
.map(log_key)
.unwrap_or_else(|| "?".into())
);
match send_batch_via_upload(bot, chat_id, reply_to, batch, caption).await {
match send_batch_via_upload(
bot,
chat_id,
reply_to,
batch,
caption,
updated_sequence_task(task, idx, sent.clone()),
)
.await
{
Ok(messages) => {
collect_file_ids(&messages, batch, &mut cached_media);
sent.extend(messages.into_iter().map(|m| m.id.0 as i64));
}
Err(FallbackError::Retryable { delay_seconds }) => {
return Err(SendError::Retryable {
delay_seconds,
task: updated_sequence_task(task, idx, sent),
});
}
Err(FallbackError::Permanent { message }) => {
return Err(SendError::Permanent {
message,
task: updated_sequence_task(task, idx, sent),
});
}
Err(FallbackError::MediaTooLarge) => unreachable!("handled inside upload"),
Err(e) => return Err(e),
}
}
Err(e) => {
@@ -850,19 +1070,31 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
}
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
log::info!(
"Telegram could not fetch animation URL, downloading and reuploading: {}",
media_url
"Telegram could not fetch animation URL, downloading and reuploading: [key={}]",
log_key(media_url)
);
match download_to_temp(animation).await {
Ok(file) => {
let path = file.path().to_path_buf();
// Single-item local preparation — the same pipeline the media
// group fallback uses (download with the upload-cap check,
// downscale/transcode photos, smaller-URL fallback). Animations
// have no smaller variant, so an oversized file surfaces as a
// permanent error here.
match prepare_upload_item(animation.clone(), 0, None).await {
Ok(prepared) => {
let PreparedItem {
media, keep_alive, ..
} = prepared;
let InputMedia::Animation(animation) = media else {
unreachable!("an Animation payload prepares to InputMedia::Animation")
};
// Hold the temp file until the request completes.
let _keep_alive = keep_alive;
match send_animation_inner(
bot,
chat_id,
reply_to,
caption,
has_spoiler,
InputFile::file(path),
animation.media,
)
.await
{
@@ -874,46 +1106,7 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
Err(e) => Err(classify_to_send_error(&e, task.clone())),
}
}
// Over the upload cap: fall back to the smaller URL.
Err(FallbackError::MediaTooLarge) => match animation.fallback_url() {
Some(url) => match input_file_for(url) {
Ok(file) => {
match send_animation_inner(
bot,
chat_id,
reply_to,
caption,
has_spoiler,
file,
)
.await
{
Ok(message) => {
let id = message.id.0 as i64;
cache_animation_send(task, &message).await;
Ok(vec![id])
}
Err(e) => Err(classify_to_send_error(&e, task.clone())),
}
}
Err(message) => Err(SendError::Permanent {
message,
task: task.clone(),
}),
},
None => Err(SendError::Permanent {
message: "media too large".into(),
task: task.clone(),
}),
},
Err(FallbackError::Retryable { delay_seconds }) => Err(SendError::Retryable {
delay_seconds,
task: task.clone(),
}),
Err(FallbackError::Permanent { message }) => Err(SendError::Permanent {
message,
task: task.clone(),
}),
Err(e) => Err(SendError::from_fallback(e, task.clone())),
}
}
Err(e) => Err(classify_to_send_error(&e, task.clone())),
@@ -1037,8 +1230,7 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
};
if edit_before_forward {
let mut chat_data = CHAT_STORE.get(chat_id).await;
let keyboard = build_edit_markup(&chat_data.template);
let keyboard = build_edit_markup(&CHAT_STORE.get(chat_id).await.template);
match bot
.send_message(ChatId(chat_id), "Reply to edit message.")
.reply_markup(keyboard)
@@ -1053,8 +1245,12 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
prompt.id.0,
message_ids.len()
);
chat_data.edit_message.insert(
prompt.id.0 as i64,
let prompt_id = prompt.id.0 as i64;
let source_url = source_url.clone();
CHAT_STORE
.update(chat_id, move |data| {
data.edit_message.insert(
prompt_id,
EditMessage {
url: source_url,
chat_id,
@@ -1063,7 +1259,8 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
created_at: unix_now(),
},
);
CHAT_STORE.set(chat_id, &chat_data).await;
})
.await;
}
Err(e) => log::error!("failed to send edit prompt: {e}"),
}
@@ -1122,7 +1319,19 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
});
}
};
let bot = Bot::from_env();
let bot = BOT.clone();
// A resumed multi-batch send already ran post_send_actions (edit prompt /
// forward) when it first started; running them again on the resume would
// open a duplicate edit prompt and double-forward. SendAnimation is
// atomic (always a fresh run), so only SendMediaSequence can resume.
let resumed = matches!(
&task,
Task::SendMediaSequence {
batch_index,
sent_message_ids,
..
} if *batch_index > 0 || !sent_message_ids.is_empty()
);
match task {
Task::SendMediaSequence { .. } | Task::SendAnimation { .. } => {
let message_ids = match send_media_or_animation(&bot, &task).await {
@@ -1138,13 +1347,18 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
}
Err(SendError::Permanent { message, task }) => {
invalidate_cache(&task).await;
// The task settles here: drop any keep-alive temp media.
release_keep_alive(&task);
return Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
});
}
};
if !resumed {
post_send_actions(&bot, &task, message_ids).await;
}
release_keep_alive(&task);
Ok(())
}
Task::ForwardMessages { .. } => match forward_messages(&bot, &task).await {
@@ -1156,10 +1370,13 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
delay_seconds,
payload: serde_json::to_value(task).expect("task serializes"),
}),
Err(SendError::Permanent { message, task }) => Err(QueueError::Permanent {
Err(SendError::Permanent { message, task }) => {
release_keep_alive(&task);
Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
}),
})
}
},
}
}
@@ -1177,7 +1394,7 @@ pub async fn dead_letter_notify(payload: serde_json::Value, message: String) {
let notify_chat_id = payload.get("notify_chat_id").and_then(|v| v.as_i64());
let notify_message_id = payload.get("notify_message_id").and_then(|v| v.as_i64());
if notify_chat_id.is_some() {
let bot = Bot::from_env();
let bot = BOT.clone();
notify_failure(
&bot,
notify_chat_id,
@@ -1195,9 +1412,10 @@ mod tests {
#[test]
fn oversized_photo_boundary() {
// The empirical Telegram limit: sum 10000 passes, 10001 fails.
assert!(crate::photo::PHOTO_MAX_DIMENSION_SUM == 10000);
assert!(6100 + 3900 <= crate::photo::PHOTO_MAX_DIMENSION_SUM);
assert!(6300 + 3730 > crate::photo::PHOTO_MAX_DIMENSION_SUM);
// Const-block asserts so clippy's assertions_on_constants stays quiet.
const { assert!(crate::photo::PHOTO_MAX_DIMENSION_SUM == 10000) };
const { assert!(6100 + 3900 <= crate::photo::PHOTO_MAX_DIMENSION_SUM) };
const { assert!(6300 + 3730 > crate::photo::PHOTO_MAX_DIMENSION_SUM) };
}
#[test]
@@ -1214,6 +1432,52 @@ mod tests {
);
}
#[test]
fn photos_first_orders_photos_before_videos() {
use MediaItemPayload::{Animation, Photo, Video};
let photo = |u: &str| Photo {
media: u.into(),
has_spoiler: false,
fallback_url: None,
file_id: false,
};
let video = |u: &str| Video {
media: u.into(),
has_spoiler: false,
thumbnail: None,
fallback_url: None,
file_id: false,
};
let items = vec![
video("https://v/1.mp4"),
photo("https://p/1.jpg"),
video("https://v/2.mp4"),
photo("https://p/2.jpg"),
];
let ordered = photos_first(items);
// All photos first (stable: p1 before p2), then all videos in order.
let kinds: Vec<&str> = ordered
.iter()
.map(|i| match i {
Photo { media, .. } => media.as_str(),
Video { media, .. } => media.as_str(),
Animation { .. } => unreachable!(),
})
.collect();
assert_eq!(
kinds,
[
"https://p/1.jpg",
"https://p/2.jpg",
"https://v/1.mp4",
"https://v/2.mp4"
]
);
// Already-photos-first input is unchanged.
let items = vec![photo("https://p/1.jpg"), video("https://v/1.mp4")];
assert!(matches!(photos_first(items)[0], Photo { .. }));
}
#[test]
fn retry_delay_seconds_bounds() {
for attempts in 0..10 {
+100 -22
View File
@@ -5,7 +5,7 @@ use parking_lot::Mutex;
use rusqlite::params;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::path::Path;
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
#[derive(Serialize, Deserialize, Default, Clone, Debug)]
@@ -34,7 +34,10 @@ pub struct EditMessage {
pub struct ChatStore {
/// In-memory cache; the DB is the source of truth on first access.
cache: Mutex<HashMap<i64, ChatData>>,
db_path: String,
/// Per-chat async locks serializing get→mutate→set so concurrent handler
/// tasks (batch-forwards, callbacks) cannot clobber each other's writes.
locks: Mutex<HashMap<i64, Arc<tokio::sync::Mutex<()>>>>,
pool: Arc<crate::db::DbPool>,
}
pub fn unix_now() -> i64 {
@@ -45,25 +48,15 @@ pub fn unix_now() -> i64 {
}
impl ChatStore {
/// Creates the parent directory and the `chat_state` table (idempotent).
/// The shared `tasks` / `link_cache` tables are owned by `queue.rs` and
/// `link_cache.rs` respectively.
pub fn open(path: &str) -> rusqlite::Result<Self> {
if let Some(parent) = Path::new(path).parent()
&& !parent.as_os_str().is_empty()
{
std::fs::create_dir_all(parent)
.map_err(|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)))?;
}
let conn = crate::db::open_db(path)?;
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL);",
)?;
drop(conn);
Ok(ChatStore {
/// Wraps the shared DB pool (schema initialized once by
/// [`crate::db::open_store`]; the `chat_state` table lives in the merged
/// schema alongside `tasks` and `link_cache`).
pub fn new(pool: Arc<crate::db::DbPool>) -> Self {
ChatStore {
cache: Mutex::new(HashMap::new()),
db_path: path.to_string(),
})
locks: Mutex::new(HashMap::new()),
pool,
}
}
pub async fn get(&self, chat_id: i64) -> ChatData {
@@ -71,7 +64,9 @@ impl ChatStore {
return data.clone();
}
let chat_key = chat_id.to_string();
let payload = crate::db::with_conn(&self.db_path, move |conn| {
let payload = self
.pool
.with_conn(move |conn| {
// Concurrent handler tasks (batch-forwards) may write chat_state
// while this read runs; the shared busy timeout handles the
// write-lock collision instead of failing the query.
@@ -98,7 +93,9 @@ impl ChatStore {
self.cache.lock().insert(chat_id, data.clone());
let payload = serde_json::to_string(data).expect("chat state serializes");
let chat_id = chat_id.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
params![chat_id, payload],
@@ -111,6 +108,26 @@ impl ChatStore {
}
}
/// Serializes a get→mutate→set cycle per chat: concurrent handler tasks
/// (the batch-forward design spawns several per chat) each snapshot the
/// same `ChatData` and last-writer-wins would silently drop mutations,
/// e.g. a second `edit_message` record. The per-chat lock makes the
/// cycle atomic. Returns the closure's result.
pub async fn update<R>(&self, chat_id: i64, f: impl FnOnce(&mut ChatData) -> R) -> R {
let lock = {
let mut locks = self.locks.lock();
locks
.entry(chat_id)
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
};
let _guard = lock.lock().await;
let mut data = self.get(chat_id).await;
let r = f(&mut data);
self.set(chat_id, &data).await;
r
}
/// Removes edit-before-forward records whose `created_at + ttl` is in the
/// past. Returns the removed `(chat_id, prompt_message_id)` pairs so the
/// caller can clear the prompt's buttons.
@@ -118,6 +135,10 @@ impl ChatStore {
let now = unix_now();
let ttl_secs = ttl.as_secs() as i64;
let mut removed = Vec::new();
// Chats with no live edit records: evicted from the cache (and their
// per-chat lock) so the cache stays bounded to active prompts. The DB
// keeps the row; the next get() reloads it.
let mut evicted_chats = Vec::new();
let changed: Vec<(i64, ChatData)> = {
let mut cache = self.cache.lock();
let mut out = Vec::new();
@@ -134,15 +155,31 @@ impl ChatStore {
}
}
if kept.len() != data.edit_message.len() {
// Persist the pruned row (removes expired records from
// the DB too, not just the cache).
data.edit_message = kept;
out.push((*chat_id, data.clone()));
}
if data.edit_message.is_empty() {
evicted_chats.push(*chat_id);
}
}
// Lock order: update() takes the per-chat lock before the cache
// lock, so prune must not hold the cache lock while taking locks.
drop(cache);
out
};
for (chat_id, data) in changed {
self.set(chat_id, &data).await;
}
if !evicted_chats.is_empty() {
let mut cache = self.cache.lock();
let mut locks = self.locks.lock();
for chat_id in &evicted_chats {
cache.remove(chat_id);
locks.remove(chat_id);
}
}
if !removed.is_empty() {
log::info!(
"pruned {} expired edit-before-forward record(s)",
@@ -152,3 +189,44 @@ impl ChatStore {
removed
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn concurrent_updates_do_not_lose_edit_records() {
let dir = tempfile::tempdir().unwrap();
let pool = crate::db::open_store(dir.path().join("s.db").to_str().unwrap()).unwrap();
let store = std::sync::Arc::new(ChatStore::new(pool));
let mut handles = Vec::new();
for i in 0..4 {
let store = Arc::clone(&store);
handles.push(tokio::spawn(async move {
store
.update(1001, |data| {
data.edit_message.insert(
i,
EditMessage {
url: format!("https://x.com/u/status/{i}"),
chat_id: 1001,
forward_message_ids: vec![i],
template: String::new(),
created_at: 0,
},
);
})
.await;
}));
}
for h in handles {
h.await.unwrap();
}
let data = store.get(1001).await;
assert_eq!(
data.edit_message.len(),
4,
"concurrent get→mutate→set must not drop records"
);
}
}
+8
View File
@@ -55,6 +55,14 @@ services:
depends_on:
- nginx-proxy
container_name: tgxmb
# Webhook mode only: the bot listens on WEBHOOK_PORT; nginx-proxy shows
# 502s while this is down, so surface it to the orchestrator.
healthcheck:
test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/8443'"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
volumes:
certs:
-1
View File
@@ -16,7 +16,6 @@ then
else
usermod -u ${USER_ID} -o user > /dev/null 2>&1 || true
fi
usermod -a -G root user > /dev/null 2>&1 || true
# Bind-mounted volumes may not support chown; a failure here must not kill
# the container either.
chown -R `id -u user`:`id -u user` /app > /dev/null 2>&1 || true
+134
View File
@@ -0,0 +1,134 @@
# 架构优化设计:可测试性接缝 + handlers 拆分
> 状态:设计稿(未实施)。目标:把仓库最大的测试空白(`handlers.rs`/`send.rs` 的
> 发送与分派逻辑)补上可测试接缝,并把 ~1100 行的 handlers 单体拆成模块。
> 每个阶段独立提交、独立回滚;全程 fmt / clippy / test 全绿,行为不变。
---
## 1. 现状与动机
- `handlers.rs`(~1100 行)混装:命令解析/执行、URL 提取 + 任务通道、inline
debounce、callback、edit-before-forward、全部全局静态。
- 关键路径零测试:`url_media` 的分派、`dispatch_send` 的失败分类、缓存命中路径、
edit-before-forward、转发重试——AGENTS.md 自认 "untested: handlers.rs"。
- 根因:`handlers.rs`/`send.rs` 直接依赖 teloxide `Bot`(具体类型)与全局静态
`CHAT_STORE`/`TASK_QUEUE`/`LINK_CACHE`/`CONFIG`),没有注入点。
## 2. 阶段 A:handlers 拆分(纯组织,零风险,先行)
`handlers.rs` 拆为模块(仅移动代码,不改签名):
```
handlers/
mod.rs — 入口:message/inline/callback 分发 + 公共类型(UrlJob、log_key
statics.rs — CHAT_STORE / TASK_QUEUE / LINK_CACHE / DB / CONFIG / URL_JOBS
commands.rs — Command enum + execute_command + set_forward_channel_handler
urls.rs — extract_urls + start/stop_url_workers + url_media + build_send_task + media_to_payload
inline.rs — inline_query_handler + debounce 状态机 + answer_inline_query
callback.rs — callback_query_handler + edit_message_handler
```
- `mod.rs``pub use` 重导出,bot 侧引用 `handlers::xxx` 不变。
- 收益:每个模块独立审阅;后续阶段 B 的接缝改动落在明确的模块内。
## 3. 阶段 BMediaSender 接缝(核心)
**动机**`send.rs` 的所有发送入口(`send_media_group`/`send_animation`/
`copy_messages`)都挂在具体 `Bot` 上;测试无法注入失败/成功。
**设计**:新增 `crates/xmedia-bot/src/media_sender.rs`
```rust
/// 发送抽象:生产用 teloxide Bot,测试用记录型 mock。
/// 方法签名与 teloxide 调用点一一对应,返回 Result 以便注入任意失败。
pub trait MediaSender: Send + Sync {
fn send_media_group(&self, chat_id: ChatId, items: Vec<InputMedia>)
-> BoxFuture<'_, Result<Vec<Message>, RequestError>>;
fn send_animation(&self, chat_id: ChatId, file: InputFile, caption: Option<&str>, spoiler: bool, reply_to: i64)
-> BoxFuture<'_, Result<Message, RequestError>>;
fn copy_messages(&self, to: ChatId, from: ChatId, ids: Vec<MessageId>)
-> BoxFuture<'_, Result<Vec<MessageId>, RequestError>>;
// 按需扩展:edit_message_caption / delete_message / answer_callback_query …
}
impl MediaSender for Bot { /* 委托现有 teloxide 调用 */ }
```
配套:`ChatStore`/`LinkCache`/`PersistentTaskQueue` 已是具体类型——给 `send.rs`/
`url_media` 需要的最小面加 trait`ChatStoreReader`/`LinkCacheReader` 等),或直接
注入具体类型(它们已有内存态,测试用真实 tempdir 即可,见阶段 B-注)。
**接入点**
- `dispatch_send` / `send_media_sequence` / `send_animation` / `forward_messages` /
`post_send_actions` / `notify_failure``bot: &Bot` 参数改为 `sender: &dyn MediaSender`
- `url_media``url_media(bot, message, url)` 改为 `url_media(sender, store, queue, cache, message, url)`(或聚合为一个 `AppContext` 结构传引用)。
**测试策略**(仓库无 mock 框架,手写 mock):
- `MockSender` 记录调用序列、按脚本返回 Ok/Err(覆盖:URL 发送成功、media-fetch
失败触发兜底、RetryAfter 触发入队、Permanent 触发缓存失效)。
- `ChatStore`/`LinkCache` 用真实 tempdir 实例(现有测试已这么做)。
- 新增测试:`send_media_sequence` 分批续传、`send_animation` 兜底、`url_media`
缓存命中 vs 未命中、`dispatch_send` 三分支。
**风险**:中。动 `send.rs`/`handlers.rs` 签名(约 15 处调用点),行为不变。
**不做**`main.rs` 的 teloxide 装配不抽象(那是真正的胶水,无测试价值)。
## 4. 阶段 C(可选):主动限流
批量转发时的突发会触发 Telegram 频道限速,现在靠 `RetryAfter → 队列重试` 被动
应对。新增轻量令牌桶(`rate_limit.rs`~50 行):
```rust
pub struct TokenBucket { /* capacity, refill_rate, state */ }
impl TokenBucket {
pub async fn acquire(&self, n: u64) -> Duration; // 等待时长(或 Notify 唤醒)
}
```
- 按频道粒度(`HashMap<ChatId, Arc<TokenBucket>>`),在 `send_media_group`/
`copy_messages` 前置 `acquire`
- 收益:减少 429 → 重试 → 死信;风险低,独立模块。
- 不做的理由(若选不做):当前重试链路已能自愈,容量可按需再加。
## 5. 阶段 D(可选):DB 版本化迁移
`schema_init``CREATE TABLE IF NOT EXISTS`,无版本概念。改为:
```rust
// db.rs
const MIGRATIONS: &[&str] = &[
// v1: 初始 schematasks / chat_state / link_cache
"CREATE TABLE IF NOT EXISTS tasks (...); ...",
];
pub fn migrate(conn: &Connection) -> rusqlite::Result<()> {
let v: i64 = conn.query_row("PRAGMA user_version", [], |r| r.get(0))?;
for (i, sql) in MIGRATIONS.iter().enumerate().skip(v as usize) {
conn.execute_batch(sql)?;
conn.pragma_update(None, "user_version", (i + 1) as i64)?;
}
Ok(())
}
```
- 低优先级:schema 未变时无收益;将来加列/改结构时必须有。
- `open_store` 改用 `migrate` 替换 `schema_init` 调用。
## 6. 明确不做
- **不拆 xmedia-core**`Task`/队列/发送抽成独立 lib crate 是大工程,除非出现
第二个客户端,否则收益不抵成本。
- **不引入 DI 框架**:仓库惯例是 LazyLock 静态 + 显式传参,保持。
- **不抽象 main.rs 的 teloxide 装配**。
## 7. 提交序列
| 阶段 | 提交消息(建议) |
|---|---|
| A | `refactor(handlers): split monolithic handlers.rs into modules` |
| B | `refactor(send): introduce MediaSender seam for testable send paths` |
| B+ | `test(send): cover fallback and classification via MockSender` |
| C | `feat(send): add per-chat token bucket rate limiting` |
| D | `refactor(db): versioned schema migrations` |
每阶段独立合入;A、B 为核心,C、D 可选。
+237
View File
@@ -0,0 +1,237 @@
# 站点适配器重构方案:让新增站点变成"新模块 + 注册一行"
> 状态:**已实施**(阶段 1-5,提交 `7ca8fd1` / `5e23916` / `bf4e615` / `5679a8c` +
> 本文档收尾)。目标:把"加一个新站点"从改 8-9 处收敛到 3 处,并让站点身份、
> 重试策略、下载 header 等站点能力归位到站点模块自身。实施过程中的关键偏差
> async 形态)见 §3 的 "async 形态" 段——原生 AFIT 实测不可用于 dyn 分派,
> 最终采用手写 `BoxFuture``SiteFuture` 别名)。
---
## 1. 现状摩擦清单
以现有三站(twitter / bsky / pixiv)为基线,新增第 4 个站点(代号 `example`
今天需要触碰的位置:
| # | 位置(当前行号) | 改动 | 必改? |
|---|---|---|---|
| 1 | 新目录 `crates/x-media/src/site/example/{mod,interface,model}.rs` | 新模块 | 必改 |
| 2 | `site/mod.rs:354-365` `fetch_once` | 加一个 `if` 分派分支 | 必改 |
| 3 | `site/mod.rs:167-178` `cache_key` | 加一个 `if` 分支 + 约定 key 前缀 `"example:..."` | 必改 |
| 4 | `site/mod.rs:53-63` `site_name()` | 加一个 URL `contains` 嗅探分支 | 必改 |
| 5 | `handlers.rs:405` `SetFormat` 白名单 | `["twitter","bsky","pixiv"]` 加字符串 | 必改 |
| 6 | `config.rs` / `main.rs:74-84` | 仿 pixiv 加启动校验(token、`disable()` | 视站点 |
| 7 | `site/mod.rs:312-326` `fetch_error_is_retryable` | 若重试策略特殊,改中央分类函数 | 视站点 |
| 8 | `site/mod.rs:377/391/430` 三个下载函数 | 若媒体有防盗链,加 header(现在是硬编码 pximg 判断) | 视站点 |
| 9 | `site/mod.rs:16,180-197` `FetchError` | 若错误类型特殊,加嵌套 variant(仿 `Pixiv(PixivError)` | 视站点 |
**根因**:仓库里没有"站点"这个实体。站点的四类能力——URL 识别(PATTERN +
cache_key)、抓取、重试策略、下载 header——分别散落在中央 if 链、URL 字符串嗅探、
魔法字符串 key 和 bot crate 的白名单里。`AGENTS.md` 现行约定 "no trait, no enum
dispatch" 是刻意的简单性选择;本方案的目标是在**不推翻它精神的前提下**收敛摩擦,
并在阶段 3 提供完整的 trait 注册表选项。
## 2. 目标架构
```
crates/x-media/src/site/mod.rs
├─ SITES: LazyLock<Vec<Box<dyn Site>>> ← 注册表(唯一的"站点列表")
├─ find_site(url) / fetch(url) / cache_key(url) / site_ids()
└─ 通用类型:Fetched { site_id, ... } / FetchError(通用类 + Site 变体)
├─ site/twitter/{mod,interface,model}.rs impl Site
├─ site/bsky/… impl Site
└─ site/pixiv/… impl Site (download_headers: pximg Referer)
(validate: token 校验)
crates/xmedia-bot
├─ handlers.rs SetFormat 白名单 ← x_media::site::ids()(不再写死)
├─ handlers.rs site 格式查找 ← fetched.site_id(缓存/新鲜两条路径同口径)
└─ main.rs 启动校验 ← site::validate_all()(不再特判 pixiv
```
## 3. 分阶段迁移
每个阶段是一个独立提交,保持 `cargo fmt` / `cargo clippy -- -D warnings` /
`cargo test --workspace` 全绿;行为完全不变,只挪代码、不换语义。
### 阶段 1:站点身份单一来源(低风险,推荐先做)
**动机**:同一概念目前有两个来源——缓存命中路径用 `key.split(':').next()`
`handlers.rs:639`),新鲜抓取路径用 `fetched.site_name()``handlers.rs:724`);
`site_name()` 又是对 `source_url``contains` 字符串嗅探,还有 `"unknown"`
兜底分支。
**改动**
1. `site/mod.rs``Fetched` 增加字段 `site_id: &'static str`(由各站点的
`impl From<SiteStruct> for Fetched` 填充;`empty_fetched` 同步填)。
`Fetched::site_name()` 改为 `return self.site_id`(保留方法名,删除
`source_url.contains` 嗅探与 `"unknown"` 分支)。
2. `site/mod.rs`:新增 `pub fn site_id_from_key(key: &str) -> &'static str`
(解析 `"example:..."` 前缀,未知前缀返回 `"unknown"`),bot 缓存命中路径改用它,
`fetched.site_id` 口径统一。
3. `handlers.rs:405``SetFormat` 白名单改为 `x_media::site::ids()`——阶段 1 先实现
`ids()``["twitter","bsky","pixiv"]` 的常量函数(数据源仍集中,行为不变),
阶段 3 再改为遍历注册表。
4. `twitter/interface.rs:48-60` / `bsky` / `pixiv``From<SiteStruct> for Fetched`
各补 `site_id` 字段。
**风险**:低。纯增量字段;`site_name()` 语义不变(测试 `pixiv/interface.rs:355`
已断言 `"pixiv"`)。
**验证**:现有全部单测;`cache_key_normalizes_domain_variants` 等不变。
**回滚**revert 该提交。
### 阶段 2:站点能力下沉(不引入 trait,静态分派)
**动机**:把"每个站点自己才知道"的逻辑搬回站点模块,中央只做迭代。这是
`AGENTS.md` 现有约定(无 trait)与完整注册表之间的折中,可独立交付。
**改动**:每个站点模块新增并 `mod.rs` 重新导出:
```rust
// site/twitter/interface.rsbsky/pixiv 同构)
pub fn cache_key(url: &str) -> Option<String>; // 用自身 PATTERN,返回 "twitter:<id>"
pub fn is_retryable(err: &FetchError) -> bool; // 默认 Http|Transientpixiv 覆盖 PixivError 分支
pub fn media_headers(url: &str) -> Option<Vec<(&'static str, String)>>;
// pixiv: url 含 "pximg.net" → Referer
```
`site/mod.rs` 相应改为迭代三站:
- `cache_key`:逐个调 `site::cache_key`,不再自己写 key 格式;
- `fetch_error_is_retryable`:删除,`fetch()` 重试循环改调 `current_site::is_retryable`
`fetch_once` 已能确定站点,把站点传下去);
- `media_size` / `download_media_limited` / `download_media_to_file` 里的
`pximg.net → Referer` 硬编码删除,改为遍历 `SITES`(阶段 2 是遍历
`[twitter, bsky, pixiv]` 静态列表)取 `media_headers(url)` 合并。
**注意**Referer 判定依据是媒体 URL 的 host(`pximg.net`),**不是**站点
PATTERNpixiv 的 PATTERN 只匹配 `pixiv.net/artworks/...`),所以 `media_headers`
不能挂在 PATTERN 匹配上,必须按 URL 独立匹配——这正是把它做成独立函数的原因。
**风险**:中。下载函数签名不变,行为必须逐字节不变;新增单元测试覆盖
`media_headers("https://i.pximg.net/...") == Some(Referer)`
`cache_key` 等价性(对全部既有用例断言新旧结果一致)。
**回滚**revert。
### 阶段 3Site trait + SITES 注册表(完整方案,可选)
**动机**:加站点时 bot crate 与中央分派零改动;站点列表成为唯一注册点。
**新增**`site/mod.rs`,按实施后的实际形态):
```rust
/// Boxed, Send future produced by a Site async method. Boxed so the trait
/// stays dyn-compatible; Send because URL/queue workers tokio::spawn these.
type SiteFuture<'a, T, E = FetchError> =
Pin<Box<dyn Future<Output = Result<T, E>> + Send + 'a>>;
pub trait Site: Send + Sync {
fn id(&self) -> &'static str;
fn pattern(&self) -> &'static Regex;
fn enabled(&self) -> bool { true } // 默认: true
fn cache_key(&self, url: &str) -> Option<String>;
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched>;
fn is_retryable(&self, err: &FetchError) -> bool; // 默认: Http|Transient
fn media_headers(&self, url: &str) -> Option<Vec<(&'static str, String)>>; // 默认: None
fn validate(&self) -> SiteFuture<'static, (), String>; // 默认: Ok(())
}
static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| vec![
Box::new(twitter::TwitterSite), Box::new(bsky::BskySite), Box::new(pixiv::PixivSite),
]);
```
- `fetch``find_site(url)`(注册表中首个 PATTERN 命中且 `enabled()` 的站点,
返回 `&'static dyn Site`)→ `site.fetch_from_url(url).await`
- `cache_key` / `site_ids()` / `site_id_from_key()` / `apply_media_headers()` /
`validate_all()` 全部遍历 `SITES``validate_all` 返回失败列表,pixiv 的
`Site::validate` 失败时自行 `disable()`
- `match_site`/`SiteKind`(阶段 2 的静态分派)与中央 `fetch_error_is_retryable`
删除,重试判定走 `site.is_retryable`
- `main.rs` 的 pixiv 特判 → `site::validate_all()` + 通用失败通知;
- 保留各站点的 `PATTERN`/`enabled()`/`fetch_from_url()` 顶层导出(兼容既有
测试),trait impl 只是薄壳。
**async 形态**(实施结论):**原生 AFIT 不可行**。
- 实测(rustc 1.95.0edition 2024**1.97.1 复测一致**):trait 里写
`async fn` 报 "method is `async`"(非 dyn 兼容);写反糖
`-> impl Future<...> + Send + '_` 报 "references an `impl Trait` type in its
return type"(同样非 dyn 兼容);纯 RPITIT(无 `+ Send`)也一样。即:
**RPITIT/AFIT 目前无法用于 `Vec<Box<dyn Site>>` 注册表**,与早期设计的
判断相反。
- **为什么**:dyn 分派要求调用方在编译期知道返回值大小以分配空间,而
`async fn`/RPITIT 返回不透明的 Future——这是"非定长返回值走 dyn"的普遍问题,
与 async 无关。Rust 1.75 稳定的 AFIT 只覆盖**静态分派**,dyn 路径被排除;
原生 dyn 支持(AFIDT)是 2026-2027 的已接受项目目标,尚未进入 stable。
参见 <https://rust-lang.github.io/rust-project-goals/2026/afidt-box.html>。
- **采用 (a) 手写 `Pin<Box<dyn Future + Send + '_>>`**`SiteFuture` 别名):
零新依赖、dyn 兼容、future 保证 Send。签名噪音靠别名缓解;生命周期坑因
站点是无状态单元结构体 + `'a` 同时约束 `&self``url` 而完全可控
(future 只借用调用域内的 url)。
- **(b) `async-trait`** 仍是可行备选(语法更干净、同样 box),但新增依赖;
本仓库采用 (a) 后无需引入。
- 若未来 Rust 稳定版落地 AFIDT(调用点 `dyn_box!`),可平滑迁移回原生
`async fn`,实现体几乎不动。
**风险**:中。动中央分派,但每站点行为不变;注册表迭代 + `find_site` 补单测
`fetch`/`cache_key` 对既有 URL 集合的结果与阶段 2 完全一致)。
**回滚**revert。
### 阶段 4FetchError 泛化(已实施)
**改动**`FetchError` 新增 `Site { site: &'static str, error: Box<dyn std::error::Error + Send + Sync> }`
变体(`Display`/`source()` 同步)。**`Pixiv(PixivError)` 变体保留**(未迁移)——
它已有完整的 `Display`/`source()`/`is_retryable` 处理,替换纯属 churn。`Site`
变体默认永久性(各站点 `is_retryable` 都不匹配它);需要可重试站点错误的站点
应自行转换为 `Http`/`Transient` 再返回。
**风险**:低(纯增量变体)。测试:`site_error_variant_displays_and_sources`
### 阶段 5:收尾
- 更新 `AGENTS.md` 的 "Site adapter convention" 段:写新约定(注册表 + `impl Site` +
每站点 `cache_key`/`is_retryable`/`media_headers`),删除 "no trait" 表述;
- `examples/fetch.rs` 不变(走 `site::fetch`);
- 新增站点 checklist 见 §4。
## 4. 重构后新增站点 checklist
```
1. crates/x-media/src/site/example/{mod,interface,model}.rs // 新模块
2. impl Site for ExampleSite 并注册进 SITES // 注册一行
3. (可选)token 读取 + validate() 实现 // 启动校验自动生效
── bot crate 零改动 ──
```
对比现状的 8-9 处,bot crate 完全不碰:`SetFormat` 白名单、格式查找口径、
缓存 key、启动校验全部自动跟随注册表。
## 5. 权衡与明确不做的事
- **不做**Media 类型扩展(`media.rs` + `MediaItemPayload` + `CachedMediaKind` +
send.rs 约 10+ 处 match 的 blast radius)——这是"新增媒体类型"的摩擦,与"新增
站点"正交,优先级低,保持现状。
- **不做**DI/全局注入改造(`CHAT_STORE`/`TASK_QUEUE`/`CONFIG``LazyLock` 静态
模式是仓库惯例,与站点扩展无关)。
- **不做**:schema 迁移——新站点只产生新的 cache key 前缀与 `message_format` JSON
key`link_cache`/`chat_state` 表结构均无需变化。
- **代价**:阶段 3 引入 `dyn Site` 与 boxed future 签名(`SiteFuture`,见 §3);
`Send` 约束前移到 trait 边界,站点 impl 的 future 必须 Send(现仅在各
`tokio::spawn` 点检查,重构后在 impl 处即报错,提前暴露问题)。
若站点数量长期 ≤5 且无新增迹象,阶段 2 的折中方案已够用;本次已按完整方案
实施到阶段 4。
## 6. 提交序列(已按此实施)
| 阶段 | 提交 | hash |
|---|---|---|
| 1 | `refactor(site): carry site_id on Fetched; unify cache-key site lookup` | `7ca8fd1` |
| 2 | `refactor(site): move cache_key/is_retryable/media_headers into site modules` | `5e23916` |
| 3 | `refactor(site): introduce Site trait and SITES registry` | `bf4e615` |
| 4 | `refactor(site): genericize FetchError::Site` | `5679a8c` |
| 5 | `docs: update site adapter convention in AGENTS.md` | 本文档收尾提交 |
每阶段独立合入、独立回滚;阶段 2 完成后"加站点"摩擦已收敛,3/4 为深化。