Compare commits

...
47 Commits
Author SHA1 Message Date
YoursFunny ea72516d5c chore: bump version to 1.1.0 2026-08-10 10:32:54 +08:00
YoursFunny 755330e585 chore: clippy and rustfmt cleanup on new code 2026-08-08 20:35:43 +08:00
YoursFunny c40b074b3c docker: fail-closed rebuild, ffmpeg checksum, smaller runtime
- Replace the mtime-touch stub-rebuild hack with cargo clean -p (a
  future-dated host file could silently ship the stub binary)
- Optional FFMPEG_SHA256 build arg verified before extraction
- Drop the redundant libssl3/libcrypto copies and the root
  supplementary group; strip the release binary; add a webhook-mode
  healthcheck to the compose example
2026-08-08 20:34:21 +08:00
YoursFunny 9910da2914 deps: drop unused regex, unify reqwest on 0.12
x-media's reqwest 0.13 dragged in quinn/rustls/aws-lc-rs (cmake C
build) alongside teloxide's 0.12; unifying on 0.12 removes the whole
second TLS/QUIC stack from the build and image. The unused regex dep in
xmedia-bot is gone; x-media keeps url (the bsky HLS remux uses it).
2026-08-08 20:31:03 +08:00
YoursFunny ebc0122264 db: enable WAL and index the pending-task lease query
The lease/earliest_run_after queries full-scanned tasks, and the
rollback journal blocked readers behind worker writes. journal_mode=WAL
(persistent, idempotent) plus idx_tasks_pending(status, run_after)
covers both without a schema migration.
2026-08-08 20:26:07 +08:00
YoursFunny 44cba8abe0 send: reuse one process-wide Bot for queue workers
handle_task and dead_letter_notify built a fresh Bot (env parse + HTTP
client) per queue item. A single LazyLock<Bot> is forced at startup so
a missing TELOXIDE_TOKEN fails fast instead of on the first task.
2026-08-08 20:25:22 +08:00
YoursFunny 99009aae9a db: share one now_f64() instead of four private copies
handlers, queue, send and link_cache each carried the same SystemTime
helper; a single crate::db::now_f64() removes the drift risk.
2026-08-08 20:23:58 +08:00
YoursFunny 72130b9023 caption: escape URLs/handles in HTML captions
Post URLs and author URLs were interpolated raw into <a href> attributes
(and the raw user URL from empty_fetched into caption text), so crafted
links could break the HTML parse and fail the send with a 400. All
attribute interpolations now use encode_double_quoted_attribute; text
stays encode_text.
2026-08-08 20:21:56 +08:00
YoursFunny 734cfc2eb3 handlers: skip inline fetches for non-post queries
Inline queries fire per keystroke and each fetch runs the 3-attempt
retry loop; a user typing any text was pumping requests into
X/Pixiv/BSky and risking rate-limit bans. Queries now pass only if
cache_key recognizes them as a supported post URL.
2026-08-08 20:20:05 +08:00
YoursFunny a8156697fa send: keep video thumbnails in the download-and-reupload fallback
media_from_file/media_from_url never attached the thumbnail, so any
video that tripped the fallback lost its cover frame. Both now take
item.thumbnail_url() and apply it, matching build_media_group.
2026-08-08 20:19:19 +08:00
YoursFunny c093dfe5ac handlers: dedup extracted URLs by normalized post id
Exact-string dedup let https://x.com/u/status/1 and
https://x.com/u/status/1/photo/1 (or the same link in text and caption)
through twice, causing duplicate fetches and sends. Dedup now uses
cache_key, falling back to the raw URL for unsupported links.
2026-08-08 20:17:07 +08:00
YoursFunny ee6f3e4a27 state: evict idle chats from the ChatStore cache
prune_expired only shrank edit_message maps, so the cache kept one
ChatData per chat forever (a leak proportional to chat count). Chats
without live edit records are now dropped from the cache and their
per-chat lock (DB row persists; get() reloads). Lock order kept safe:
prune never holds the cache lock while taking the per-chat locks.
2026-08-08 20:16:26 +08:00
YoursFunny 16ed53fead handlers: replace unbounded per-URL spawn with a bounded job channel
The 8-permit semaphore was acquired inside the spawned task, so a burst
queued unlimited tasks (each cloning Bot+Message) and nothing tracked
them at shutdown — in-flight sends fired after the stop notice. URL work
now flows through a 256-slot mpsc drained by 8 workers started from
main; a full channel backpressures the per-chat handler, and shutdown
sets URL_STOP so workers stop pulling.
2026-08-08 20:15:40 +08:00
YoursFunny 1d9e3629c9 state: serialize per-chat get→mutate→set with ChatStore::update
Concurrent handler tasks (the batch-forward design spawns several per
chat) each snapshotted the same ChatData and last-writer-wins silently
dropped mutations — e.g. a second edit_message record, leaving one
prompt's Forward button dead. All write cycles now run under a per-chat
async lock; read-only callers keep get().
2026-08-08 20:12:51 +08:00
YoursFunny b3d87b4f7d send: fail fast when a retried local media file is gone
A retried ugoira/bsky temp MP4 was already deleted with its TempDir, so
the retry failed at multipart-build time with a confusing Io error and
wasted all three attempts. input_file_for now rejects a missing local
path up front as a clean permanent error.
2026-08-08 20:10:35 +08:00
YoursFunny 98c48b99c0 send: don't repeat post-send actions on queue resumes
A resumed SendMediaSequence (batch_index>0 or already-sent ids) ran
post_send_actions again, opening a second edit prompt and inserting a
second edit_message record for the same messages — both Forward buttons
worked, enabling double forwards. Resumes now skip it.
2026-08-08 20:09:58 +08:00
YoursFunny f40639c799 queue: back off 1s when a lease fails
A lease error (e.g. persistent SQLITE_BUSY) while rows are due made the
worker spin with sleep(0), hammering SQLite and flooding the log.
lease_next now returns the error and the loop sleeps 1s before retrying.
2026-08-08 20:09:04 +08:00
YoursFunny 51cc079a85 queue: use notify_one so wakeups are never lost
notify_waiters drops the notification when every worker is between its
DB reads and registering notified(); a task enqueued in that window sat
until a stale timer fired. notify_one stores a permit, so the next
worker to wait wakes immediately and re-leases. stop() still wakes all
workers with notify_waiters.
2026-08-08 20:08:12 +08:00
YoursFunny aa3083792a queue: wire attempt counts into retry backoff
Every network retry hard-coded retry_delay_seconds(0), so backoff was
flat at 1.2-1.8s regardless of attempt; a multi-minute outage dead-
lettered after three rapid tries. The queue now scales the handler's
delay by 2^attempts (cap 300s) before rescheduling.
2026-08-08 20:07:26 +08:00
YoursFunny 6849006ad7 site: honor TELOXIDE_PROXY for site fetches
The shared HTTP client ignored the proxy the Bot API uses, so on
proxy-required networks (e.g. behind the GFW) every site fetch failed
while the bot itself worked. Explicit proxy overrides reqwest's system
detection; unset keeps direct connections.
2026-08-08 20:05:51 +08:00
YoursFunny 042a04ab6e site: anchor pixiv and bsky URL patterns
Both matched URL substrings anywhere in text, so a link like
https://evil.com/?u=pixiv.net/artworks/1 triggered a real fetch and
cache-key pollution. Prefix ^(?:https?://)? like the twitter pattern.
2026-08-08 20:05:09 +08:00
YoursFunny 9f28af4e6b site: classify HTTP status codes, make transient failures retryable
Twitter (syndication + auth GraphQL) mapped every non-2xx to NotFound,
killing retries on 429/5xx; bsky never checked status; pixiv network
errors arrived wrapped in PixivError and were excluded from the retry
loop. New FetchError::Transient covers 429/5xx from all sites, the
retry loop now also retries Pixiv errors, and 404/410 stay permanent.
2026-08-08 20:04:28 +08:00
YoursFunny d61dba5096 send: stream media downloads with hard size caps
download_to_temp now uses download_media_limited: non-photos abort the
moment the 10 MiB upload cap is crossed mid-stream (no more full-body
buffering before the size check), photos cap at the 512 MiB decode
budget, and the ugoira frame zip gets a 512 MiB cap. MediaTooLarge
routes to the existing smaller-URL fallback.
2026-08-08 20:03:14 +08:00
YoursFunny f6df3e28cb pixiv: drop unneeded mut bindings in ugoira extraction 2026-08-08 20:02:06 +08:00
YoursFunny deb1ef2428 site: add total and connect timeouts to the shared HTTP client 2026-08-08 20:01:39 +08:00
YoursFunny b5e5340edc pixiv: harden ugoira zip extraction, degrade instead of panicking
Sniff the frame extension from magic bytes instead of the entry filename,
cap each frame at 64 MiB (declared size + streamed read), and map a
panicked encode worker to the existing degrade path instead of
expect()-panicking the whole fetch handler.
2026-08-08 20:01:14 +08:00
YoursFunny 425d1505cf handlers: fix /set_forward_channel admin checks
Compare the sender's user id (not the chat id, which only matches in
private chats) and require the bot to actually be an admin with post
rights instead of silently passing when it is missing from the list.
Also stops panicking on get_me network failures.
2026-08-08 19:59:27 +08:00
YoursFunny 6e40f55440 queue: recover expired leases at runtime, supervise workers
Rows left in_progress by a panicked/crashed worker were only recovered at
start(); a runtime sweep (30s interval, woken by the same notify) now
re-queues them once the 120s lock TTL expires. Workers run under a
supervisor that respawns a panicked loop instead of silently shrinking
the pool of 4.
2026-08-08 19:58:28 +08:00
YoursFunny 7998114dc3 bsky: remux HLS video playlists to MP4 via ffmpeg
bsky video embeds expose only an m3u8 playlist, which Telegram cannot
fetch. Download the master/variant playlists and TS segments through the
shared client (proxy-aware, size-capped), then concat-remux locally;
keep the temp dir alive via Fetched._keep_alive like the ugoira path.
Also adds site::download_media_limited (streaming size cap), status
checks on media_size, and moves the ffmpeg probe to site/mod.rs for
pixiv/bsky to share.
2026-08-08 19:56:36 +08:00
YoursFunny 9a96f78177 handlers: fix UTF-8 byte-slice panic in message log preview 2026-08-08 19:52:01 +08:00
YoursFunny b50f794d52 feat: register bot commands with Telegram
Call setMyCommands at startup so clients show the command list in the
/ menu. handlers::register_commands wraps Command::bot_commands()
(teloxide derives it from the #[command(description)] attributes);
a registration failure only warns and does not stop the bot.
2026-08-08 00:08:48 +08:00
YoursFunny d32fa969d6 fix: correct singular 'entry' in clear-cache replies
plural() returned "" for one, rendering '1 entr.'; return "y" so
the suffix composes to '1 entry' / '2 entries'.
2026-08-08 00:07:27 +08:00
YoursFunny 020e2d01a3 chore: bump version to 1.0.8 2026-08-07 16:23:40 +08:00
YoursFunny 3d6f8548c3 style: cargo fmt across the workspace
Apply rustfmt to the 11 files that had drifted (86 hunks): x-media
site modules (bsky/pixiv/twitter) and xmedia-bot (config/main/
photo/send). Formatting only - no semantic changes; full test suite
still green.
2026-08-07 16:12:01 +08:00
YoursFunny 063e910473 feat: add admin-only /clear_cache command
/clear_cache with no argument wipes the whole link_cache table;
with a post URL it removes that single entry (normalized via
site::cache_key so fxtwitter/mobile/photo variants collide with
the write-side key). Non-admins get 'Admin only.'. LinkCache gains
clear(Option<&str>) -> usize reporting removed rows.
2026-08-07 16:10:25 +08:00
YoursFunny b0ced34b4c refactor: share sqlite open/with_conn helpers in db.rs
Converge the duplicated open_db (open + busy_timeout) and the
spawn_blocking + expect ceremony that every table access repeated
into one db.rs module. ChatStore no longer creates the tasks table
(schema ownership: queue.rs owns tasks, state.rs chat_state,
link_cache.rs link_cache). No schema or behavior change - all
CREATE TABLE statements are byte-identical, IF NOT EXISTS stays
idempotent, so existing data/task_queue.db files need no migration.
2026-08-07 16:00:12 +08:00
YoursFunny 4060a88031 fix: expand twitter short links like FxEmbed linkFixer
Replace display_text_range slicing with FxEmbed-style content matching:
expand mapped t.co links to their real URLs (dropping internal
x.com/i/web/status pages), then strip every leftover t.co short link
(appended media link, unmapped links).

The old code cut by display_text_range, whose index unit differs per
endpoint (UTF-16 on the syndication endpoint, code points in the
GraphQL fallback), so slicing by either unit left a partial
"https://t." caption tail on the other path. Content matching is
unit-agnostic and also keeps user-posted/quote links at the end of
the text that the trailing cut previously dropped.
2026-08-07 10:25:54 +08:00
YoursFunny fb43441c56 feat: add command descriptions and document commands in README
All bot commands now carry English descriptions, shown in the Telegram
command menu and by /help (which prints Command::descriptions()). The
README command table explains each command's arguments and behavior:
forward channel (@channel or ID), edit-before-forward flow, template
[] placeholder semantics and per-site caption format placeholders.
2026-08-06 21:21:28 +08:00
YoursFunny bf628dc999 chore: drop label value in compose example 2026-08-06 20:45:35 +08:00
YoursFunny de22aa9b4d chore: leave DEFAULT_EMAIL blank in compose example 2026-08-06 20:03:56 +08:00
YoursFunny 65a9554173 bump version to 1.0.7 2026-08-06 18:49:08 +08:00
YoursFunny e755785147 deploy: use named volumes for certs/html/acme, keep data bind mount
certs, html and acme are shared across nginx-proxy, acme-companion and
the acme-ip service, so they become named volumes (dropping the nginx-
prefix). The vhost.d mount is removed: acme-companion only needs it with
ACME_HTTP_CHALLENGE_LOCATION=true or letsencrypt_user_data, and users
who customize nginx config can mount their own. ./data stays a bind
mount so the SQLite state stays directly backup-able. README updated.
2026-08-06 17:22:05 +08:00
YoursFunny 5d5b6d56e7 chore: strip compose comments, document env vars in README
compose files now carry a single comment: the commented ACME_HOST line
(uncomment to enable domain cert issuance). Everything else moved to the
README env table and the deployment sections, which explain ACME_HOST,
DEFAULT_HOST and every other variable.
2026-08-06 17:07:50 +08:00
YoursFunny d0fdf1c5da deploy: default to ACME-issued certs, drop manual cert setup
Domain deployment now uses ACME_HOST (acme-companion auto issue/renew)
instead of the legacy LETSENCRYPT_HOST name. IP-only deployments can use
acme.sh to obtain Let's Encrypt IP certificates (shortlived profile,
~7-day validity, http-01 only) instead of self-signed manual certs:
documented the acme-ip service, first-issue command with install-cert
into nginx-certs and nginx reload via docker socket.

Removed the manual WEBHOOK_CERT flow and the ./cert volume from the
compose example.
2026-08-06 16:48:52 +08:00
YoursFunny 1db4ecfafa chore: use modern nginx-proxy label for acme-companion
acme-companion prefers com.github.nginx-proxy.nginx over the legacy
jrcs.letsencrypt_nginx_proxy_companion namespace (functions.sh checks
the new label first, falls back to the old one).
2026-08-06 11:31:01 +08:00
YoursFunny 47039bbe2d fix: make docker entrypoint restart-safe
docker compose restart reuses the container, so the overlay fs keeps the
user created on first boot; a second useradd fails with exit code 9 and
set -e kills the container on every restart (crash loop, bot never comes
back). Create the user only if missing, align UID otherwise, and never
let chown failure on bind-mounted volumes kill the container.

Also comment out nginx-proxy's empty environment block in the compose
example: an empty mapping fails validation on newer compose versions
(must be a mapping).
2026-08-06 11:10:52 +08:00
YoursFunny bc954e6e0b fix: handle oversized photos Telegram rejects with pure Rust processing 2026-08-06 10:17:29 +08:00
26 changed files with 2446 additions and 1121 deletions
+2 -1
View File
@@ -68,7 +68,8 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
|---|---|
| `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) |
| `crates/xmedia-bot/src/handlers.rs` | `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `data/task_queue.db` **relative to CWD**); command dispatch; URL extraction; retry enqueue |
| `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`, `MAX_UPLOAD_BYTES = 10 MiB`; fallback chain; `classify_request_error` |
| `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`; fallback chain; `classify_request_error`; download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`) |
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL |
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) |
| `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` |
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime hack, static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint |
Generated
+88 -364
View File
@@ -99,28 +99,6 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
[[package]]
name = "aws-lc-rs"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00"
dependencies = [
"aws-lc-sys",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
]
[[package]]
name = "axum"
version = "0.8.9"
@@ -264,12 +242,6 @@ version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrono"
version = "0.4.44"
@@ -292,15 +264,6 @@ dependencies = [
"inout",
]
[[package]]
name = "cmake"
version = "0.1.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
dependencies = [
"cc",
]
[[package]]
name = "colored"
version = "3.1.1"
@@ -310,16 +273,6 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "combine"
version = "4.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd"
dependencies = [
"bytes",
"memchr",
]
[[package]]
name = "constant_time_eq"
version = "0.3.1"
@@ -505,6 +458,15 @@ dependencies = [
"syn",
]
[[package]]
name = "document-features"
version = "0.2.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d4b8a88685455ed29a21542a33abd9cb6510b6b129abadabdcef0f4c55bc8f61"
dependencies = [
"litrs",
]
[[package]]
name = "dotenv"
version = "0.15.0"
@@ -521,12 +483,6 @@ dependencies = [
"futures",
]
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
[[package]]
name = "dyn-clone"
version = "1.0.20"
@@ -599,12 +555,33 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
[[package]]
name = "fast_image_resize"
version = "6.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9c50201dc184ba6553da1695aac20a042efffbe2d84542cee31917c86c3ab1e"
dependencies = [
"cfg-if",
"document-features",
"num-traits",
"thiserror",
]
[[package]]
name = "fastrand"
version = "2.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6"
[[package]]
name = "fdeflate"
version = "0.3.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e6853b52649d4ac5c0bd02320cddc5ba956bdb407c4b75a2c6b75bf51500f8c"
dependencies = [
"simd-adler32",
]
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
@@ -657,12 +634,6 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "futures"
version = "0.3.32"
@@ -768,10 +739,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"wasi",
"wasm-bindgen",
]
[[package]]
@@ -1246,55 +1215,6 @@ version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "jni"
version = "0.22.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498"
dependencies = [
"cfg-if",
"combine",
"jni-macros",
"jni-sys",
"log",
"simd_cesu8",
"thiserror",
"walkdir",
"windows-link",
]
[[package]]
name = "jni-macros"
version = "0.22.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3"
dependencies = [
"proc-macro2",
"quote",
"rustc_version",
"simd_cesu8",
"syn",
]
[[package]]
name = "jni-sys"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2"
dependencies = [
"jni-sys-macros",
]
[[package]]
name = "jni-sys-macros"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264"
dependencies = [
"quote",
"syn",
]
[[package]]
name = "jobserver"
version = "0.1.34"
@@ -1305,6 +1225,12 @@ dependencies = [
"libc",
]
[[package]]
name = "jpeg-encoder"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a0370574b86f7eca156b9f298392b5e69a23f8c86f3f865add60bbc2e79467a6"
[[package]]
name = "js-sys"
version = "0.3.98"
@@ -1352,6 +1278,12 @@ version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0"
[[package]]
name = "litrs"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11d3d7f243d5c5a8b9bb5d6dd2b1602c0cb0b9db1621bafc7ed66e35ff9fe092"
[[package]]
name = "lock_api"
version = "0.4.14"
@@ -1367,12 +1299,6 @@ version = "0.4.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897"
[[package]]
name = "lru-slab"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
[[package]]
name = "lzma-rs"
version = "0.3.0"
@@ -1604,6 +1530,19 @@ version = "0.3.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
[[package]]
name = "png"
version = "0.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60769b8b31b2a9f263dae2776c37b1b28ae246943cf719eb6946a1db05128a61"
dependencies = [
"bitflags",
"crc32fast",
"fdeflate",
"flate2",
"miniz_oxide",
]
[[package]]
name = "potential_utf"
version = "0.1.5"
@@ -1688,62 +1627,6 @@ dependencies = [
"cc",
]
[[package]]
name = "quinn"
version = "0.11.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
dependencies = [
"bytes",
"cfg_aliases",
"pin-project-lite",
"quinn-proto",
"quinn-udp",
"rustc-hash",
"rustls",
"socket2",
"thiserror",
"tokio",
"tracing",
"web-time",
]
[[package]]
name = "quinn-proto"
version = "0.11.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
dependencies = [
"aws-lc-rs",
"bytes",
"getrandom 0.3.4",
"lru-slab",
"rand 0.9.4",
"ring",
"rustc-hash",
"rustls",
"rustls-pki-types",
"slab",
"thiserror",
"tinyvec",
"tracing",
"web-time",
]
[[package]]
name = "quinn-udp"
version = "0.5.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2",
"tracing",
"windows-sys 0.52.0",
]
[[package]]
name = "quote"
version = "1.0.45"
@@ -1772,18 +1655,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a"
dependencies = [
"libc",
"rand_chacha 0.3.1",
"rand_core 0.6.4",
]
[[package]]
name = "rand"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea"
dependencies = [
"rand_chacha 0.9.0",
"rand_core 0.9.5",
"rand_chacha",
"rand_core",
]
[[package]]
@@ -1793,17 +1666,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
dependencies = [
"ppv-lite86",
"rand_core 0.6.4",
]
[[package]]
name = "rand_chacha"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
dependencies = [
"ppv-lite86",
"rand_core 0.9.5",
"rand_core",
]
[[package]]
@@ -1815,15 +1678,6 @@ dependencies = [
"getrandom 0.2.17",
]
[[package]]
name = "rand_core"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
dependencies = [
"getrandom 0.3.4",
]
[[package]]
name = "rc-box"
version = "1.3.0"
@@ -1899,16 +1753,20 @@ checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
dependencies = [
"base64",
"bytes",
"encoding_rs",
"futures-core",
"futures-util",
"h2",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-tls",
"hyper-util",
"js-sys",
"log",
"mime",
"mime_guess",
"native-tls",
"percent-encoding",
@@ -1931,47 +1789,6 @@ dependencies = [
"web-sys",
]
[[package]]
name = "reqwest"
version = "0.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62e0021ea2c22aed41653bc7e1419abb2c97e038ff2c33d0e1309e49a97deec0"
dependencies = [
"base64",
"bytes",
"encoding_rs",
"futures-core",
"h2",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-util",
"js-sys",
"log",
"mime",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls",
"rustls-pki-types",
"rustls-platform-verifier",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tokio-rustls",
"tower",
"tower-http",
"tower-service",
"url",
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
]
[[package]]
name = "rgb"
version = "0.8.53"
@@ -2009,21 +1826,6 @@ dependencies = [
"smallvec",
]
[[package]]
name = "rustc-hash"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
[[package]]
name = "rustc_version"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
dependencies = [
"semver",
]
[[package]]
name = "rustix"
version = "1.1.4"
@@ -2043,7 +1845,6 @@ version = "0.23.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b"
dependencies = [
"aws-lc-rs",
"once_cell",
"rustls-pki-types",
"rustls-webpki",
@@ -2051,62 +1852,21 @@ dependencies = [
"zeroize",
]
[[package]]
name = "rustls-native-certs"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63"
dependencies = [
"openssl-probe",
"rustls-pki-types",
"schannel",
"security-framework",
]
[[package]]
name = "rustls-pki-types"
version = "1.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9"
dependencies = [
"web-time",
"zeroize",
]
[[package]]
name = "rustls-platform-verifier"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
dependencies = [
"core-foundation 0.10.1",
"core-foundation-sys",
"jni",
"log",
"once_cell",
"rustls",
"rustls-native-certs",
"rustls-platform-verifier-android",
"rustls-webpki",
"security-framework",
"security-framework-sys",
"webpki-root-certs",
"windows-sys 0.61.2",
]
[[package]]
name = "rustls-platform-verifier-android"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
[[package]]
name = "rustls-webpki"
version = "0.103.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted",
@@ -2124,15 +1884,6 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "same-file"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
dependencies = [
"winapi-util",
]
[[package]]
name = "schannel"
version = "0.1.29"
@@ -2332,22 +2083,6 @@ version = "0.3.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
[[package]]
name = "simd_cesu8"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94f90157bb87cddf702797c5dadfa0be7d266cdf49e22da2fcaa32eff75b2c33"
dependencies = [
"rustc_version",
"simdutf8",
]
[[package]]
name = "simdutf8"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
[[package]]
name = "slab"
version = "0.4.12"
@@ -2481,7 +2216,7 @@ dependencies = [
"log",
"mime",
"pin-project",
"rand 0.8.6",
"rand",
"serde",
"serde_json",
"teloxide-core",
@@ -2512,7 +2247,7 @@ dependencies = [
"once_cell",
"pin-project",
"rc-box",
"reqwest 0.12.28",
"reqwest",
"rgb",
"serde",
"serde_json",
@@ -2859,16 +2594,6 @@ version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "walkdir"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
dependencies = [
"same-file",
"winapi-util",
]
[[package]]
name = "want"
version = "0.3.1"
@@ -3014,25 +2739,6 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "web-time"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "webpki-root-certs"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c"
dependencies = [
"rustls-pki-types",
]
[[package]]
name = "winapi-util"
version = "0.1.11"
@@ -3296,15 +3002,15 @@ checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
[[package]]
name = "x-media"
version = "1.0.6"
version = "1.1.0"
dependencies = [
"bytes",
"dotenv",
"html-escape",
"log",
"rand 0.8.6",
"rand",
"regex",
"reqwest 0.13.3",
"reqwest",
"serde",
"serde_json",
"tempfile",
@@ -3315,15 +3021,17 @@ dependencies = [
[[package]]
name = "xmedia-bot"
version = "1.0.6"
version = "1.1.0"
dependencies = [
"dotenv",
"fast_image_resize",
"html-escape",
"jpeg-encoder",
"log",
"parking_lot",
"png",
"pretty_env_logger",
"rand 0.8.6",
"regex",
"rand",
"rusqlite",
"serde",
"serde_json",
@@ -3332,6 +3040,7 @@ dependencies = [
"tokio",
"url",
"x-media",
"zune-jpeg",
]
[[package]]
@@ -3535,3 +3244,18 @@ dependencies = [
"cc",
"pkg-config",
]
[[package]]
name = "zune-core"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb8a0807f7c01457d0379ba880ba6322660448ddebc890ce29bb64da71fb40f9"
[[package]]
name = "zune-jpeg"
version = "0.5.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "27bc9d5b815bc103f142aa054f561d9187d191692ec7c2d1e2b4737f8dbd7296"
dependencies = [
"zune-core",
]
+4
View File
@@ -1,3 +1,7 @@
[workspace]
members = ["crates/x-media", "crates/xmedia-bot"]
resolver = "3"
# Smaller production binary; debug symbols are not shipped anyway.
[profile.release]
strip = true
+14 -9
View File
@@ -11,6 +11,11 @@ ARG APP_NAME=telegram-twitter-media-bot
# runners. `/redirect/latest/` floats to the newest release build; each build
# also ships a .sha256. Swap `amd64` for `arm64` when building arm64 images.
ARG FFMPEG_URL=https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffmpeg.zip
# Optional sha256 of ffmpeg.zip (pinned releases only): set to verify the
# download. The mirror publishes .sha256 sidecars next to pinned builds, e.g.
# https://ffmpeg.martin-riedl.de/download/linux/amd64/<id>_9.0/ffmpeg.zip.sha256
# (the /redirect/latest/ URL itself has no sidecar — pin the effective URL).
ARG FFMPEG_SHA256=
WORKDIR /build
@@ -30,19 +35,20 @@ RUN mkdir -p crates/x-media/src crates/xmedia-bot/src \
# root. `unzip -t` verifies the archive before extraction so a bad
# download fails loudly here instead of a cryptic later error.
RUN wget -q -O /tmp/ffmpeg.zip "$FFMPEG_URL" \
&& if [ -n "$FFMPEG_SHA256" ]; then echo "$FFMPEG_SHA256 /tmp/ffmpeg.zip" | sha256sum -c -; fi \
&& unzip -tq /tmp/ffmpeg.zip \
&& unzip -q /tmp/ffmpeg.zip -d /usr/local/bin \
&& chmod +x /usr/local/bin/ffmpeg \
&& rm /tmp/ffmpeg.zip \
&& /usr/local/bin/ffmpeg -version >/dev/null
# 3. Real sources last: only our crates recompile on source changes. The
# COPY preserves host mtimes, which predate the stub artifacts from step 1;
# cargo's mtime-based freshness check would otherwise treat the stub build
# as up-to-date and never compile the real sources. `touch` forces cargo to
# see the real files as newer.
# 3. Real sources last: only our crates recompile on source changes.
# `cargo clean -p` drops the two crates' artifacts while keeping the
# compiled dependency layer, forcing a deterministic rebuild of the real
# sources. (The previous `touch`-mtimes hack silently shipped the stub
# binary when host files carried future timestamps.)
COPY crates/ ./crates/
RUN find crates -type f -name '*.rs' -exec touch {} + \
RUN cargo clean -p xmedia-bot -p x-media \
&& cargo build --release -p xmedia-bot
# ---------- runtime stage ----------
@@ -56,10 +62,9 @@ LABEL org.opencontainers.image.title="${APP_NAME}"
# Everything is copied in — no apt in the runtime stage. Privilege dropping is
# done by docker-entrypoint.sh with setpriv (util-linux, already in
# bookworm-slim), so no gosu needed.
# bookworm-slim), so no gosu needed. (libssl3/libcrypto are already in
# bookworm-slim; only ca-certificates and ffmpeg need copying.)
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder /usr/lib/x86_64-linux-gnu/libssl.so.3* /usr/lib/x86_64-linux-gnu/
COPY --from=builder /usr/lib/x86_64-linux-gnu/libcrypto.so.3* /usr/lib/x86_64-linux-gnu/
COPY --from=builder /usr/local/bin/ffmpeg /usr/local/bin/ffmpeg
WORKDIR /app
+39 -17
View File
@@ -39,16 +39,37 @@ NSFW 推文:公开的 syndication 接口不返回敏感内容。设置 `TWITTE
**有域名**
1. DNS A 记录指向服务器
2. compose 里设 `VIRTUAL_HOST``LETSENCRYPT_HOST` 为域名,`WEBHOOK_URL` 设为 `https://域名/`
3. 证书自动签发与续期,无需手动处理
2. compose 里设 `VIRTUAL_HOST``WEBHOOK_URL` 为域名,并取消注释 `ACME_HOST`设为域名)
3. acme-companion 自动签发与续期证书,无需手动处理
**只有 IP**
1. 生成自签证书(PEM 格式,见第 3 步)
`openssl req -x509 -newkey rsa:2048 -nodes -days 365 -keyout nginx-certs/default.key -out nginx-certs/default.crt`
2. compose 里 nginx-proxy 设 `DEFAULT_HOST`bot 设 `WEBHOOK_CERT: './cert/cert.pem'`(须与代理所服务的为同一张证书)
3. 证书必须是 PEM 编码(ASCII BASE64,以 `-----BEGIN CERTIFICATE-----` 开头)—— Telegram 只接受该格式;若现有证书是 DER 二进制,转换:
`openssl x509 -in cert.der -inform DER -out cert.pem -outform PEM`
(私钥同理:`openssl rsa -in key.der -inform DER -out key.pem -outform PEM`
Let's Encrypt 支持为公网 IP 签发证书(2026 年起可用,有效期约 7 天,须 `shortlived` profile)。用 [acme.sh](https://github.com/acmesh-official/acme.sh) 自动签发与续期,无需手动证书
1. compose 里增加 acme-ip 服务(签发 + 每日检查自动续期):
```yaml
acme-ip:
image: neilpang/acme.sh
container_name: acme-ip
command: daemon
restart: always
volumes:
- certs:/acme.sh
- html:/usr/share/nginx/html
- /var/run/docker.sock:/var/run/docker.sock:ro
networks: [proxy]
```
2. 首次签发(把 `<SERVER_IP>` 换成服务器公网 IP,IPv6 同样支持,多个 `-d` 可并列):
```bash
docker compose exec acme-ip acme.sh --issue --server letsencrypt \
-d <SERVER_IP> --cert-profile shortlived --days 3 \
--webroot /usr/share/nginx/html \
--install-cert --cert-file /acme.sh/<SERVER_IP>.crt \
--key-file /acme.sh/<SERVER_IP>.key \
--reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP"
```
3. compose 里设 `VIRTUAL_HOST: '<SERVER_IP>'`、`WEBHOOK_URL: 'https://<SERVER_IP>/'`,无需 `WEBHOOK_CERT`。续期由 acme.sh daemon 自动完成(`--days 3` = 每 3 天续一次,证书 7 天有效有缓冲),续期成功后自动 HUP 通知 nginx-proxy 加载新证书。
限制:证书约 7 天有效;验证仅支持 http-01/tls-alpn-0180 端口必须公网可达);不支持 DNS-01、私有 IP 与 IP 段;同一 IP 集合每 168 小时限签发 5 张。建议先用 `--server letsencrypt_test` 试签,成功后再切正式服务器。
Telegram 只接受 443/80/88/8443 端口。
@@ -66,13 +87,12 @@ Telegram 只接受 443/80/88/8443 端口。
| `LOCAL_USER_ID` | 容器内运行用户 UID,默认 9001 |
| `VIRTUAL_HOST` | 对外域名或 IPnginx-proxy 按此路由 |
| `VIRTUAL_PORT` | bot 容器内监听端口,nginx-proxy 的转发目标 |
| `LETSENCRYPT_HOST` | 设为域名时由 acme-companion 自动签发/续期证书 |
| `ACME_HOST` | 域名部署:设为域名时由 acme-companion 自动签发/续期证书 |
| `DEFAULT_HOST` | nginx-proxy 将未知 Host 的请求路由到该 vhost(IP 访问时需要) |
| `DEFAULT_EMAIL` | acme-companion 证书通知邮箱 |
| `WEBHOOK` | `true` 启用 webhook 模式(默认轮询) |
| `WEBHOOK_LISTEN` / `WEBHOOK_PORT` | bot 容器内监听地址/端口 |
| `WEBHOOK_URL` | 对外公网 HTTPS 地址(`https://域名/` |
| `WEBHOOK_CERT` | 自签证书路径(仅 IP 路径需要,须为 PEM 且与代理所服务的一致) |
| `WEBHOOK_URL` | 对外公网 HTTPS 地址(`https://域名/` 或 `https://IP/` |
| `WEBHOOK_SECRET_TOKEN` | 更新校验令牌(`X-Telegram-Bot-Api-Secret-Token` |
</details>
@@ -81,17 +101,19 @@ Telegram 只接受 443/80/88/8443 端口。
| 命令 | 说明 |
|---|---|
| `/set_forward_channel <频道>` | 设置转发频道 |
| `/start` | 欢迎语 |
| `/help` | 查看全部命令及用法(即本文档的命令表) |
| `/set_forward_channel <频道>` | 设置转发频道,参数为 `@频道名` 或频道 ID;设置后发送的媒体消息会自动转发到该频道 |
| `/remove_forward_channel` | 取消转发频道 |
| `/edit_before_forward` | 开关转发前编辑 |
| `/set_template <名称>` | 回复的消息(含 `[]`)保存为模板 |
| `/set_format <站点> <格式>` | 自定义 caption 格式(占位符 `{url}` `{title}` `{tags}` 等) |
| `/bot_dict` | 查看聊天状态 |
| `/edit_before_forward` | 开关转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/bot_dict` | 查看当前聊天状态(调试用) |
链接处理仅限私聊;命令在任意聊天可用。
## 备注
- 数据持久化于 `data/task_queue.db`,容器部署需挂载该目录
- 数据持久化于 `data/task_queue.db`compose 部署使用 bind mount `./data`(保持目录形式便于备份)
- 运行环境需安装 ffmpeg(Docker 镜像已内置)
- 测试:`cargo test --workspace`
+2 -2
View File
@@ -1,10 +1,10 @@
[package]
name = "x-media"
version = "1.0.6"
version = "1.1.0"
edition = "2024"
[dependencies]
reqwest = { version = "0.13", features = ["json", "query", "form"] }
reqwest = { version = "0.12", features = ["json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
regex = "1.12"
+188 -17
View File
@@ -1,12 +1,12 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap()
Regex::new(r"^(?:https?://)?bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap()
});
pub fn enabled() -> bool {
@@ -15,9 +15,169 @@ pub fn enabled() -> bool {
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
let caps = PATTERN.captures(url).ok_or(FetchError::NotFound)?;
let handle = caps.get(1).map(|m| m.as_str()).ok_or(FetchError::NotFound)?;
let rkey = caps.get(2).map(|m| m.as_str()).ok_or(FetchError::NotFound)?;
Ok(fetch(handle, rkey).await?.into())
let handle = caps
.get(1)
.map(|m| m.as_str())
.ok_or(FetchError::NotFound)?;
let rkey = caps
.get(2)
.map(|m| m.as_str())
.ok_or(FetchError::NotFound)?;
let post = fetch(handle, rkey).await?;
let mut fetched: Fetched = post.into();
// bsky video embeds expose only an HLS playlist URL, which Telegram
// cannot fetch; remux it to a single MP4 (mirrors the pixiv ugoira
// encode path — the temp file stays alive via `_keep_alive`). On any
// failure the video item is dropped and the post degrades to its text.
let mut media = Vec::with_capacity(fetched.media.len());
for item in fetched.media {
let is_hls = matches!(&item, Media::Video { url, .. }
if url.contains("playlist") || url.ends_with(".m3u8"));
if !is_hls {
media.push(item);
continue;
}
let url = item.url().to_string();
match resolve_bsky_video(&url).await {
Ok(Some((mp4_path, keep_alive))) => {
let thumbnail_url = match &item {
Media::Video { thumbnail_url, .. } => thumbnail_url.clone(),
_ => String::new(),
};
media.push(Media::Video {
title: None,
url: mp4_path.to_string_lossy().into_owned(),
thumbnail_url,
});
fetched._keep_alive = Some(keep_alive);
}
Ok(None) => log::warn!("bsky video remux unavailable for {url}"),
Err(e) => log::warn!("bsky video remux failed for {url}: {e}"),
}
}
fetched.media = media;
Ok(fetched)
}
/// Downloads an HLS playlist (master or media) and remuxes its segments to a
/// single MP4 via ffmpeg. Returns the MP4 path plus the temp dir that must
/// stay alive until the file is uploaded. `Ok(None)` when ffmpeg is missing.
///
/// Verified live (2026-08): bsky master playlists carry `#EXT-X-STREAM-INF`
/// variant lines (e.g. `720p/video.m3u8?session_id=…`), and the media
/// playlists are VOD MPEG-TS segments (`videoN.ts?…`) without EXT-X-MAP, so
/// a plain `-f concat -c copy` remux is valid.
async fn resolve_bsky_video(
playlist_url: &str,
) -> Result<Option<(std::path::PathBuf, tempfile::TempDir)>, String> {
if !crate::site::ffmpeg_available() {
crate::site::log_once_ffmpeg_missing();
return Ok(None);
}
let master = crate::site::download_media_limited(playlist_url, 1_048_576)
.await
.map_err(|e| format!("bsky video master playlist: {e}"))?;
let master = String::from_utf8_lossy(&master);
// Master playlist: pick the variant with the highest declared bandwidth.
let playlist_url = if master.contains("#EXT-X-STREAM-INF") {
let mut best: Option<(u64, String)> = None;
let mut lines = master.lines();
while let Some(line) = lines.next() {
if !line.starts_with("#EXT-X-STREAM-INF") {
continue;
}
let bandwidth = line
.split_once("BANDWIDTH=")
.and_then(|(_, rest)| rest.split(|c: char| !c.is_ascii_digit()).next())
.and_then(|n| n.parse::<u64>().ok())
.unwrap_or(0);
if let Some(uri) = lines.next().filter(|u| !u.starts_with('#'))
&& bandwidth >= best.as_ref().map(|(b, _)| *b).unwrap_or(0)
{
best = Some((bandwidth, uri.to_string()));
}
}
let Some((_, uri)) = best else {
return Err("bsky video master playlist has no variants".to_string());
};
url::Url::parse(playlist_url)
.and_then(|base| base.join(&uri))
.map_err(|e| format!("bsky video variant URL: {e}"))?
.to_string()
} else {
playlist_url.to_string()
};
let variant = crate::site::download_media_limited(&playlist_url, 1_048_576)
.await
.map_err(|e| format!("bsky video media playlist: {e}"))?;
let variant = String::from_utf8_lossy(&variant);
// Segment URIs: non-#, non-empty lines, resolved relative to the playlist.
let base = url::Url::parse(&playlist_url).map_err(|e| format!("bsky playlist URL: {e}"))?;
let segments: Vec<String> = variant
.lines()
.map(str::trim)
.filter(|l| !l.is_empty() && !l.starts_with('#'))
.map(|l| base.join(l).map(|u| u.to_string()))
.collect::<Result<_, _>>()
.map_err(|e| format!("bsky segment URL: {e}"))?;
if segments.is_empty() {
return Err("bsky video playlist has no segments".to_string());
}
if segments.len() > 500 {
return Err("bsky video has too many segments".to_string());
}
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let mut total: u64 = 0;
let mut list = String::new();
for (i, seg) in segments.iter().enumerate() {
let bytes = crate::site::download_media_limited(seg, 20 * 1024 * 1024)
.await
.map_err(|e| format!("bsky segment {i}: {e}"))?;
total += bytes.len() as u64;
if total > 256 * 1024 * 1024 {
return Err("bsky video exceeds total size cap".to_string());
}
let path = frames_dir.path().join(format!("seg_{i:04}.ts"));
std::fs::write(&path, &bytes).map_err(|e| e.to_string())?;
list.push_str(&format!("file '{}'\n", path.to_string_lossy()));
}
let list_path = frames_dir.path().join("list.txt");
std::fs::write(&list_path, &list).map_err(|e| e.to_string())?;
let output = out_dir.path().join("video.mp4");
let list_str = list_path.to_string_lossy().into_owned();
let output_str = output.to_string_lossy().into_owned();
let status = tokio::task::spawn_blocking(move || {
std::process::Command::new("ffmpeg")
.args([
"-y",
"-f",
"concat",
"-safe",
"0",
"-i",
&list_str,
"-c",
"copy",
"-movflags",
"+faststart",
&output_str,
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
})
.await
.map_err(|e| format!("bsky remux worker panicked: {e}"))?;
match status {
Ok(s) if s.success() => Ok(Some((output, out_dir))),
Ok(s) => Err(format!("ffmpeg exited with {s}")),
Err(e) => Err(format!("ffmpeg spawn failed: {e}")),
}
}
/// Fetches a post thread by handle or DID (`at://` URIs work for both).
@@ -30,8 +190,16 @@ pub async fn fetch(handle: &str, rkey: &str) -> Result<Post, FetchError> {
])
.send()
.await?;
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!("bsky status {status}"))),
};
}
let text = response.text().await?;
Ok(Post::from_json(&text, rkey.to_string())?)
Post::from_json(&text, rkey.to_string())
}
#[derive(Debug)]
@@ -56,8 +224,8 @@ impl Post {
pub fn caption(&self) -> String {
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
url = self.url(),
author_url = self.author_url(),
url = encode_double_quoted_attribute(&self.url()),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
text = encode_text(&self.text),
)
@@ -197,7 +365,10 @@ mod tests {
}));
let post = Post::from_json(&raw.to_string(), "3xxxx".into()).unwrap();
let fetched: Fetched = post.into();
assert_eq!(fetched.source_url, "https://bsky.app/profile/user.bsky.social/post/3xxxx");
assert_eq!(
fetched.source_url,
"https://bsky.app/profile/user.bsky.social/post/3xxxx"
);
assert_eq!(fetched.title, "hello <world>");
assert_eq!(fetched.media.len(), 1);
assert!(!fetched.sensitive);
@@ -246,11 +417,10 @@ mod tests {
#[tokio::test]
async fn live_fetch_with_photos() {
let fetched = fetch_from_url(
"https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m",
)
.await
.unwrap();
let fetched =
fetch_from_url("https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m")
.await
.unwrap();
assert_eq!(
fetched.source_url,
"https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m"
@@ -260,9 +430,10 @@ mod tests {
#[tokio::test]
async fn live_fetch_smoke() {
let fetched = fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
.await
.unwrap();
let fetched =
fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
.await
.unwrap();
assert_eq!(
fetched.source_url,
"https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224"
+81 -13
View File
@@ -6,6 +6,7 @@
use std::fmt;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
pub mod bsky;
@@ -147,6 +148,10 @@ pub enum FetchError {
/// The post exists but its content is withheld (twitter NSFW /
/// age-restricted tweets come back as an empty `{}` from syndication).
Sensitive,
/// A download exceeded the caller's size cap (see [`download_media_limited`]).
TooLarge,
/// A transient server-side failure (429 / 5xx); [`fetch`] retries these.
Transient(String),
}
impl fmt::Display for FetchError {
@@ -158,6 +163,8 @@ impl fmt::Display for FetchError {
FetchError::NotFound => write!(f, "not found"),
FetchError::Blocked => write!(f, "blocked"),
FetchError::Sensitive => write!(f, "content withheld (sensitive)"),
FetchError::TooLarge => write!(f, "media too large"),
FetchError::Transient(message) => write!(f, "transient: {message}"),
}
}
}
@@ -169,6 +176,8 @@ impl std::error::Error for FetchError {
FetchError::Json(e) => Some(e),
FetchError::Pixiv(e) => Some(e),
FetchError::NotFound | FetchError::Blocked | FetchError::Sensitive => None,
FetchError::TooLarge => None,
FetchError::Transient(_) => None,
}
}
}
@@ -179,7 +188,6 @@ impl From<reqwest::Error> for FetchError {
}
}
impl From<serde_json::Error> for FetchError {
fn from(e: serde_json::Error) -> Self {
FetchError::Json(e)
@@ -195,7 +203,22 @@ impl From<PixivError> for FetchError {
/// Shared HTTP client (browser User-Agent) for twitter/bsky fetches and
/// [`download_media`].
pub(crate) static CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
let builder = reqwest::Client::builder().user_agent("Mozilla/5.0");
let mut builder = reqwest::Client::builder()
.user_agent("Mozilla/5.0")
// reqwest has no total timeout by default; a stalled connection
// would otherwise pin a fetch/handler forever.
.timeout(Duration::from_secs(30))
.connect_timeout(Duration::from_secs(10));
// Route site fetches through the same proxy the Bot API uses, so a
// network that needs TELOXIDE_PROXY (e.g. behind the GFW) does not
// leave site fetches dead while the bot itself works.
if let Some(proxy) = std::env::var("TELOXIDE_PROXY")
.ok()
.filter(|s| !s.is_empty())
&& let Ok(p) = reqwest::Proxy::all(&proxy)
{
builder = builder.proxy(p);
}
// Each `#[tokio::test]` runs on its own runtime; the connection pool is
// bound to the runtime that created it, so cross-runtime reuse of idle
// connections fails with DispatchGone. In test builds every request uses
@@ -205,13 +228,38 @@ pub(crate) static CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
builder.build().expect("failed to build HTTP client")
});
/// Whether a usable `ffmpeg` binary is on PATH (probed once). Shared by the
/// pixiv ugoira encoder and the bsky HLS remuxer.
static FFMPEG_AVAILABLE: LazyLock<bool> = LazyLock::new(|| {
std::process::Command::new("ffmpeg")
.arg("-version")
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
});
static FFMPEG_MISSING_LOGGED: AtomicBool = AtomicBool::new(false);
pub(crate) fn ffmpeg_available() -> bool {
*FFMPEG_AVAILABLE
}
pub(crate) fn log_once_ffmpeg_missing() {
if !FFMPEG_MISSING_LOGGED.swap(true, Ordering::Relaxed) {
log::warn!("ffmpeg not found; ugoira and bsky video posts stay unsupported");
}
}
/// Fetches a post from its URL. Returns `Ok(None)` when no site pattern
/// matches (unsupported links are silently ignored by the bot).
///
/// Transient network failures are retried: 3 total attempts with 1s then 2s
/// delays. Non-Http errors (Json/NotFound/Blocked/Pixiv) are not retried.
/// delays. Retried classes: bare HTTP errors, [`FetchError::Transient`]
/// (429/5xx from any site), and pixiv errors (its network failures arrive
/// wrapped as `PixivError`). Non-retried: Json/NotFound/Blocked/Sensitive.
pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
let mut last_http_error = None;
for attempt in 0..3u32 {
match fetch_once(url).await {
Ok(Some(fetched)) => {
@@ -223,18 +271,17 @@ pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
return Ok(Some(fetched));
}
Ok(None) => return Ok(None),
Err(FetchError::Http(e)) => {
last_http_error = Some(e);
Err(e @ (FetchError::Http(_) | FetchError::Transient(_) | FetchError::Pixiv(_))) => {
if attempt < 2 {
tokio::time::sleep(Duration::from_secs(1 << attempt)).await;
} else {
return Err(e);
}
}
Err(other) => return Err(other),
}
}
Err(FetchError::Http(
last_http_error.expect("retry loop always ran 3 attempts"),
))
unreachable!("retry loop always returns")
}
async fn fetch_once(url: &str) -> Result<Option<Fetched>, FetchError> {
@@ -263,18 +310,39 @@ pub async fn media_size(url: &str) -> Result<Option<u64>, FetchError> {
if lower.contains("pximg.net") {
request = request.header("Referer", "https://www.pixiv.net/");
}
let response = request.send().await?;
let response = request.send().await?.error_for_status()?;
Ok(response.content_length())
}
pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
/// Downloads a media file with a hard size cap: the body is streamed and the
/// download aborts with [`FetchError::TooLarge`] the moment the cap is
/// crossed (or when a declared Content-Length already exceeds it). Keeps the
/// bot from buffering arbitrarily large bodies into memory.
pub async fn download_media_limited(url: &str, max_bytes: u64) -> Result<bytes::Bytes, FetchError> {
let mut request = CLIENT.get(url);
let lower = url.to_ascii_lowercase();
if lower.contains("pximg.net") {
request = request.header("Referer", "https://www.pixiv.net/");
}
let response = request.send().await?;
Ok(response.bytes().await?)
let response = request.send().await?.error_for_status()?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let mut response = response;
let mut buf = Vec::new();
while let Some(chunk) = response.chunk().await? {
buf.extend_from_slice(&chunk);
if buf.len() as u64 > max_bytes {
return Err(FetchError::TooLarge);
}
}
Ok(bytes::Bytes::from(buf))
}
pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
download_media_limited(url, u64::MAX).await
}
#[cfg(test)]
+124 -107
View File
@@ -10,8 +10,8 @@ use crate::site::FetchError;
use std::env;
use std::fmt;
use std::io::{Cursor, Read};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::LazyLock;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, SystemTime};
const AUTH_TOKEN_URL: &str = "https://oauth.secure.pixiv.net/auth/token";
@@ -127,13 +127,18 @@ impl PixivAPI {
pub async fn illust_detail(&self, illust_id: u64) -> Result<IllustrationModel, PixivError> {
let access_token = self.get_access_token().await?;
let response = crate::site::CLIENT
.get(format!("{APP_API_URL}/v1/illust/detail?illust_id={illust_id}"))
.get(format!(
"{APP_API_URL}/v1/illust/detail?illust_id={illust_id}"
))
.header("app-os", "ios")
.header("app-os-version", "14.6")
.header("User-Agent", APP_USER_AGENT)
.bearer_auth(access_token)
.send()
.await?;
if !response.status().is_success() {
return Err(PixivError::Api(format!("status {}", response.status())));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
if json.get("error").is_some() {
let message = json
@@ -175,13 +180,18 @@ impl PixivAPI {
pub async fn ugoira_metadata(&self, illust_id: u64) -> Result<UgoiraMetadataModel, PixivError> {
let access_token = self.get_access_token().await?;
let response = crate::site::CLIENT
.get(format!("{APP_API_URL}/v1/ugoira/metadata?illust_id={illust_id}"))
.get(format!(
"{APP_API_URL}/v1/ugoira/metadata?illust_id={illust_id}"
))
.header("app-os", "ios")
.header("app-os-version", "14.6")
.header("User-Agent", APP_USER_AGENT)
.bearer_auth(access_token)
.send()
.await?;
if !response.status().is_success() {
return Err(PixivError::Api(format!("status {}", response.status())));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
if json.get("error").is_some() {
let message = json
@@ -203,8 +213,8 @@ impl PixivAPI {
&self,
illust_id: u64,
) -> Result<Option<(String, tempfile::TempDir)>, PixivError> {
if !ffmpeg_available() {
log_once_ffmpeg_missing();
if !crate::site::ffmpeg_available() {
crate::site::log_once_ffmpeg_missing();
return Ok(None);
}
let metadata = self.ugoira_metadata(illust_id).await?;
@@ -218,87 +228,117 @@ impl PixivAPI {
let Some(zip_url) = zip_url else {
return Ok(None);
};
let zip_bytes = crate::site::download_media(&zip_url).await.map_err(|e| match e {
FetchError::Http(e) => PixivError::Http(e),
other => PixivError::Api(format!("frame zip download failed: {other}")),
})?;
let zip_bytes = crate::site::download_media_limited(&zip_url, 512 * 1024 * 1024)
.await
.map_err(|e| match e {
FetchError::Http(e) => PixivError::Http(e),
other => PixivError::Api(format!("frame zip download failed: {other}")),
})?;
let frame_delays = metadata.frames.iter().map(|f| f.delay).collect::<Vec<_>>();
let result = tokio::task::spawn_blocking(
move || -> Result<(String, tempfile::TempDir), String> {
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let result =
tokio::task::spawn_blocking(move || -> Result<(String, tempfile::TempDir), String> {
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
// Extract frames to canonical zero-padded names; pixiv ugoira
// frames are uniformly jpg or png per artwork.
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
.map_err(|e| format!("unzip: {e}"))?;
// pixiv ugoira frames are uniformly jpg or png per artwork; take
// the extension from the first entry.
let extension = if archive.len() > 0 {
let first_name = archive
.by_index(0)
.map_err(|e| e.to_string())?
.name()
.to_string();
first_name
.rsplit('.')
.next()
.unwrap_or("jpg")
.to_string()
} else {
"jpg".to_string()
};
let mut count = 0usize;
for i in 0..archive.len() {
let mut entry = archive.by_index(i).map_err(|e| e.to_string())?;
let mut bytes = Vec::new();
entry.read_to_end(&mut bytes).map_err(|e| e.to_string())?;
let path = frames_dir.path().join(format!("img_{count:05}.{extension}"));
std::fs::write(&path, bytes).map_err(|e| e.to_string())?;
count += 1;
}
if count == 0 {
return Err("empty frame zip".to_string());
}
// Extract frames to canonical zero-padded names; pixiv ugoira
// frames are uniformly jpg or png per artwork.
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
.map_err(|e| format!("unzip: {e}"))?;
if archive.is_empty() {
return Err("empty frame zip".to_string());
}
// Uniform jpg or png per artwork; sniff the first entry's
// magic bytes instead of trusting its filename.
let first = archive.by_index(0).map_err(|e| e.to_string())?;
let mut first_bytes = Vec::new();
first
.take(64 * 1024 * 1024 + 1)
.read_to_end(&mut first_bytes)
.map_err(|e| e.to_string())?;
if first_bytes.len() > 64 * 1024 * 1024 {
return Err("frame exceeds size cap".to_string());
}
let extension = if first_bytes.starts_with(&[0xFF, 0xD8]) {
"jpg"
} else if first_bytes.starts_with(b"\x89PNG") {
"png"
} else {
"jpg"
};
let mut count = 0usize;
{
let path = frames_dir
.path()
.join(format!("img_{count:05}.{extension}"));
std::fs::write(&path, &first_bytes).map_err(|e| e.to_string())?;
count += 1;
}
for i in 1..archive.len() {
let entry = archive.by_index(i).map_err(|e| e.to_string())?;
if entry.size() > 64 * 1024 * 1024 {
return Err(format!("frame {i} exceeds size cap"));
}
let mut bytes = Vec::new();
entry
.take(64 * 1024 * 1024 + 1)
.read_to_end(&mut bytes)
.map_err(|e| e.to_string())?;
if bytes.len() > 64 * 1024 * 1024 {
return Err(format!("frame {i} exceeds size cap"));
}
let path = frames_dir
.path()
.join(format!("img_{count:05}.{extension}"));
std::fs::write(&path, bytes).map_err(|e| e.to_string())?;
count += 1;
}
if count == 0 {
return Err("empty frame zip".to_string());
}
// Constant rate from the median frame delay (ms).
let mut delays = frame_delays;
delays.sort_unstable();
let median = delays[delays.len() / 2].max(1);
let framerate = 1000.0 / median as f64;
// Constant rate from the median frame delay (ms).
let mut delays = frame_delays;
delays.sort_unstable();
let median = delays[delays.len() / 2].max(1);
let framerate = 1000.0 / median as f64;
let output = out_dir.path().join("ugoira.mp4");
let status = std::process::Command::new("ffmpeg")
.args([
"-y",
"-framerate",
&framerate.to_string(),
"-i",
&frames_dir.path().join(format!("img_%05d.{extension}")).to_string_lossy(),
// libx264 needs even dimensions; pixiv ugoira frames can
// be odd-sized (e.g. 277x405).
"-vf",
"scale=trunc(iw/2)*2:trunc(ih/2)*2",
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
&output.to_string_lossy(),
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map_err(|e| format!("ffmpeg spawn failed: {e}"))?;
if !status.success() {
return Err(format!("ffmpeg exited with {status}"));
}
Ok((output.to_string_lossy().into_owned(), out_dir))
},
)
.await
.expect("ugoira encode worker panicked");
let output = out_dir.path().join("ugoira.mp4");
let status = std::process::Command::new("ffmpeg")
.args([
"-y",
"-framerate",
&framerate.to_string(),
"-i",
&frames_dir
.path()
.join(format!("img_%05d.{extension}"))
.to_string_lossy(),
// libx264 needs even dimensions; pixiv ugoira frames can
// be odd-sized (e.g. 277x405).
"-vf",
"scale=trunc(iw/2)*2:trunc(ih/2)*2",
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
&output.to_string_lossy(),
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map_err(|e| format!("ffmpeg spawn failed: {e}"))?;
if !status.success() {
return Err(format!("ffmpeg exited with {status}"));
}
Ok((output.to_string_lossy().into_owned(), out_dir))
})
.await
.map_err(|e| {
log::error!("ugoira encode worker panicked for {illust_id}: {e}");
PixivError::Api(format!("ugoira worker failed: {e}"))
})?;
match result {
Ok(pair) => Ok(Some(pair)),
Err(message) => {
@@ -309,32 +349,9 @@ impl PixivAPI {
}
}
static FFMPEG_AVAILABLE: LazyLock<bool> = LazyLock::new(|| {
std::process::Command::new("ffmpeg")
.arg("-version")
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
});
static FFMPEG_MISSING_LOGGED: AtomicBool = AtomicBool::new(false);
fn ffmpeg_available() -> bool {
*FFMPEG_AVAILABLE
}
fn log_once_ffmpeg_missing() {
if !FFMPEG_MISSING_LOGGED.swap(true, Ordering::Relaxed) {
log::warn!("ffmpeg not found; pixiv ugoira posts stay unsupported");
}
}
/// pixiv3-rs replacement: `None` when `PIXIV_REFRESH_TOKEN` is unset.
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> = LazyLock::new(|| {
env::var("PIXIV_REFRESH_TOKEN").ok().map(PixivAPI::new)
});
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> =
LazyLock::new(|| env::var("PIXIV_REFRESH_TOKEN").ok().map(PixivAPI::new));
/// Set at startup when the login validation fails; pixiv stays disabled until
/// the next process start.
+74 -24
View File
@@ -1,12 +1,12 @@
use super::model::{IllustrationModel, TypeModel};
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?:www\.)?pixiv\.net/(?:en/)?(?:(?:i|artworks)/|member_illust\.php\?(?:mode=[a-z_]*&)?illust_id=)(\d+)").unwrap()
Regex::new(r"^(?:https?://)?(?:www\.)?pixiv\.net/(?:en/)?(?:(?:i|artworks)/|member_illust\.php\?(?:mode=[a-z_]*&)?illust_id=)(\d+)").unwrap()
});
pub fn enabled() -> bool {
@@ -48,9 +48,9 @@ impl Illustration {
pub fn caption(&self) -> String {
format!(
"<a href=\"{url}\">{title}</a> / <a href=\"{author_url}\">{author}</a>\n{tags}",
url = self.url(),
url = encode_double_quoted_attribute(&self.url()),
title = encode_text(&self.title),
author_url = self.author_url(),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
tags = encode_text(
&self
@@ -82,12 +82,15 @@ impl Illustration {
// keeps media empty when encoding fails or ffmpeg is missing.
} else if model.page_count > 1 {
media.extend(model.meta_pages.iter().filter_map(|page| {
page.image_urls.original.clone().map(|original| Media::Illustration {
title: None,
url: original,
thumbnail_url: Some(page.image_urls.medium.clone()),
fallback_url: Some(page.image_urls.large.clone()),
})
page.image_urls
.original
.clone()
.map(|original| Media::Illustration {
title: None,
url: original,
thumbnail_url: Some(page.image_urls.medium.clone()),
fallback_url: Some(page.image_urls.large.clone()),
})
}));
} else if let Some(original) = model
.meta_single_page
@@ -147,8 +150,8 @@ impl From<Illustration> for Fetched {
#[cfg(test)]
mod tests {
use super::*;
use super::super::model::IllustrationModel;
use super::*;
fn illust_json(
type_: &str,
@@ -203,8 +206,14 @@ mod tests {
("https://pixiv.net/artworks/123456", "123456"),
("https://www.pixiv.net/en/artworks/123456", "123456"),
("https://www.pixiv.net/i/123456", "123456"),
("https://www.pixiv.net/member_illust.php?mode=medium&illust_id=123456", "123456"),
("https://www.pixiv.net/en/member_illust.php?illust_id=123456", "123456"),
(
"https://www.pixiv.net/member_illust.php?mode=medium&illust_id=123456",
"123456",
),
(
"https://www.pixiv.net/en/member_illust.php?illust_id=123456",
"123456",
),
];
for (url, id) in cases {
let caps = PATTERN.captures(url).unwrap_or_else(|| panic!("{url}"));
@@ -225,7 +234,14 @@ mod tests {
#[test]
fn ugoira_yields_empty_media() {
let v = illust_json("ugoira", 1, Some("https://i.pximg.net/orig.jpg"), None, vec![], 0);
let v = illust_json(
"ugoira",
1,
Some("https://i.pximg.net/orig.jpg"),
None,
vec![],
0,
);
let illustration = parse(v);
let fetched: Fetched = illustration.into();
assert!(fetched.media.is_empty());
@@ -296,7 +312,12 @@ mod tests {
let fetched: Fetched = parse(v).into();
assert_eq!(fetched.media.len(), 1);
match &fetched.media[0] {
Media::Illustration { url, thumbnail_url, fallback_url, .. } => {
Media::Illustration {
url,
thumbnail_url,
fallback_url,
..
} => {
assert_eq!(url, "https://i.pximg.net/p2.jpg");
assert_eq!(thumbnail_url.as_deref(), Some("m2.jpg"));
assert_eq!(fallback_url.as_deref(), Some("l2.jpg"));
@@ -307,7 +328,14 @@ mod tests {
#[test]
fn caption_with_escapes_format_and_substitutes() {
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 0);
let v = illust_json(
"illust",
1,
Some("https://i.pximg.net/o.jpg"),
None,
vec![],
0,
);
let fetched: Fetched = parse(v).into();
// Format string is escaped in full, then placeholders substituted.
let out = fetched.caption_with("{title} by {author} <script> {tags}");
@@ -330,7 +358,14 @@ mod tests {
#[test]
fn ai_work_gets_leading_ai_tag() {
// illust_ai_type == 2 is the only AI marker.
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 2);
let v = illust_json(
"illust",
1,
Some("https://i.pximg.net/o.jpg"),
None,
vec![],
2,
);
let fetched: Fetched = parse(v).into();
assert!(
fetched.caption.contains("#AI #tag1 #tag2"),
@@ -338,14 +373,25 @@ mod tests {
fetched.caption
);
// The {tags} placeholder reflects the tag array too.
assert!(fetched.caption_with("{tags}").starts_with("#AI "), "got: {}", fetched.caption_with("{tags}"));
assert!(
fetched.caption_with("{tags}").starts_with("#AI "),
"got: {}",
fetched.caption_with("{tags}")
);
}
#[test]
fn non_ai_work_has_no_ai_tag() {
// 1 = explicitly not AI, 0 = undefined: neither gets the #AI tag.
for ai_type in [0, 1] {
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], ai_type);
let v = illust_json(
"illust",
1,
Some("https://i.pximg.net/o.jpg"),
None,
vec![],
ai_type,
);
let fetched: Fetched = parse(v).into();
assert!(
!fetched.caption.contains("#AI"),
@@ -357,7 +403,14 @@ mod tests {
#[test]
fn caption_escapes_and_links() {
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 0);
let v = illust_json(
"illust",
1,
Some("https://i.pximg.net/o.jpg"),
None,
vec![],
0,
);
let fetched: Fetched = parse(v).into();
assert!(
fetched
@@ -367,9 +420,6 @@ mod tests {
fetched.caption
);
assert!(fetched.caption.contains("#tag1 #tag2"));
assert_eq!(
fetched.source_url,
"https://www.pixiv.net/artworks/123"
);
assert_eq!(fetched.source_url, "https://www.pixiv.net/artworks/123");
}
}
+1 -1
View File
@@ -3,4 +3,4 @@ mod interface;
mod model;
pub use api::{PixivAPI, PixivError, disable, fetch, validate};
pub use interface::{PATTERN, Illustration, enabled, fetch_from_url};
pub use interface::{Illustration, PATTERN, enabled, fetch_from_url};
+30 -20
View File
@@ -23,7 +23,7 @@
use std::sync::LazyLock;
use serde_json::{json, Value};
use serde_json::{Value, json};
use crate::site::FetchError;
@@ -40,8 +40,7 @@ static AUTH_TOKEN: LazyLock<Option<String>> = LazyLock::new(|| {
});
/// Public "logged in" client token used by the x.com web app.
const LOGGED_IN_BEARER: &str =
"Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA";
const LOGGED_IN_BEARER: &str = "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA";
/// `TweetDetail` query id (from nazurin; still valid as of 2026-08,
/// corroborated by the current FxEmbed build — see module caveats).
@@ -103,9 +102,7 @@ pub fn enabled() -> bool {
/// Fetches a tweet as the logged-in user via the private GraphQL API.
/// Returns the syndication-shaped [`Tweet`] (media included for NSFW posts).
pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
let token = AUTH_TOKEN
.as_deref()
.ok_or(FetchError::Sensitive)?;
let token = AUTH_TOKEN.as_deref().ok_or(FetchError::Sensitive)?;
// 16 random bytes as 32 hex chars: X rejects ct0 values of any other
// length with 403 code 353 ("matching csrf cookie and header").
let ct0: String = (0..16)
@@ -129,18 +126,26 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
.header("referer", "https://x.com/")
.send()
.await?;
if !response.status().is_success() {
log::warn!("twitter auth fetch {id}: HTTP {}", response.status());
return Err(FetchError::NotFound);
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
log::warn!("twitter auth fetch {id}: HTTP {status}");
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!(
"twitter auth status {status}"
))),
};
}
let text = response.text().await?;
let json: Value = serde_json::from_str(&text)?;
let result = parse_tweet_result(&json, id)?;
let syndication_shape = to_syndication_shape(&result)
.ok_or_else(|| FetchError::Json(serde_json::Error::io(std::io::Error::new(
let syndication_shape = to_syndication_shape(&result).ok_or_else(|| {
FetchError::Json(serde_json::Error::io(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"missing tweet fields in GraphQL response",
))))?;
)))
})?;
Tweet::from_syndication_json(&syndication_shape.to_string()).map_err(FetchError::Json)
}
@@ -228,7 +233,6 @@ fn to_syndication_shape(tweet: &Value) -> Option<Value> {
"screen_name": user.get("screen_name"),
},
"possibly_sensitive": legacy.get("possibly_sensitive"),
"display_text_range": legacy.get("display_text_range"),
"entities": legacy.get("entities"),
"mediaDetails": legacy.pointer("/extended_entities/media"),
}))
@@ -251,12 +255,12 @@ mod tests {
"legacy": {
"id_str": "2083868672721039569",
"full_text": "nsfw content https://t.co/abc123",
"display_text_range": [0, 12],
"possibly_sensitive": true,
"entities": {
"urls": [
{ "url": "https://t.co/abc123", "expanded_url": "https://example.com/x" }
]
// The appended media link lives in extended_entities.media,
// not entities.urls, so it has no expansion mapping and the
// content-based strip removes it.
"urls": []
},
"extended_entities": {
"media": [
@@ -318,8 +322,11 @@ mod tests {
}
other => panic!("expected video, got {other:?}"),
}
assert_eq!(fetched.source_url, "https://x.com/nsfw_author/status/2083868672721039569");
// display_text_range cuts the trailing t.co link.
assert_eq!(
fetched.source_url,
"https://x.com/nsfw_author/status/2083868672721039569"
);
// The appended media short link (no URL-entity mapping) is stripped.
assert_eq!(fetched.title, "nsfw content");
}
@@ -331,7 +338,10 @@ mod tests {
let json = conversation(rt);
let result = parse_tweet_result(&json, "2083868672721039569").unwrap();
assert!(result.pointer("/legacy/retweeted_status_result").is_none());
assert_eq!(result.pointer("/legacy/id_str").unwrap(), "2083868672721039569");
assert_eq!(
result.pointer("/legacy/id_str").unwrap(),
"2083868672721039569"
);
}
#[test]
+117 -73
View File
@@ -1,7 +1,7 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
@@ -35,9 +35,7 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
}
}
} else {
log::info!(
"tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media"
);
log::info!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
Ok(empty_fetched(url))
}
}
@@ -50,7 +48,9 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
fn empty_fetched(url: &str) -> Fetched {
Fetched {
source_url: url.to_string(),
caption: url.to_string(),
// The raw user-supplied URL goes into an HTML caption; escape it so
// crafted links cannot break the parse (Telegram 400).
caption: encode_text(url).into_owned(),
title: String::new(),
media: vec![],
sensitive: true,
@@ -70,8 +70,13 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
))
.send()
.await?;
if !response.status().is_success() {
return Err(FetchError::NotFound);
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!("twitter status {status}"))),
};
}
let text = response.text().await?;
// Deleted tweets answer with {"errors": [...]} instead of a tweet.
@@ -148,8 +153,8 @@ impl Tweet {
pub fn caption(&self) -> String {
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
url = self.url(),
author_url = self.author_url(),
url = encode_double_quoted_attribute(&self.url()),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
text = encode_text(&self.text),
)
@@ -158,12 +163,10 @@ impl Tweet {
pub fn from_syndication_json(raw_json: &str) -> Result<Self, serde_json::Error> {
let json: model::SyndicationTweet = serde_json::from_str(raw_json)?;
let id = json.id_str;
// Strip the appended media short link first, then expand the remaining
// t.co short links (the user's own URLs) to their real destinations.
let text = expand_links(
&strip_trailing_short_links(&json.text, json.display_text_range),
&json.entities.urls,
);
// Expand the user's t.co short links to their real destinations and
// strip the appended media short link, mirroring FxEmbed's linkFixer
// (no display_text_range arithmetic — see expand_links).
let text = expand_links(&json.text, &json.entities.urls);
// `name` is the display name, `screen_name` the handle (Python's
// vxtwitter mapping: author = display name, author_id = handle).
let author = json.user.name;
@@ -204,51 +207,48 @@ impl Tweet {
}
}
/// The raw syndication `text` ends with the appended media short link
/// (" https://t.co/wmI8McgXul"). `display_text_range` marks the visible text;
/// a regex strips any remaining trailing t.co link when the range is absent
/// or a tweet ends in a URL short link.
///
/// X reports these indices in Unicode **code points**, not UTF-16 units
/// (verified against GraphQL responses containing emoji: cutting an emoji
/// tweet by UTF-16 units silently drops the character after the emoji).
fn strip_trailing_short_links(text: &str, display_text_range: Option<[usize; 2]>) -> String {
let mut out = match display_text_range {
Some([start, end]) if start < end => {
text.chars().skip(start).take(end - start).collect()
}
_ => text.to_string(),
};
while TRAILING_TCO.is_match(&out) {
out = TRAILING_TCO.replace(&out, "").into_owned();
}
out
}
/// Trailing Twitter short link, optionally preceded by whitespace.
static TRAILING_TCO: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"\s*https?://t\.co/[A-Za-z0-9]+$").unwrap()
});
/// Replaces every t.co short link that has an entity mapping with its
/// expanded URL. Short links without a mapping stay untouched.
/// Mirrors FxEmbed's `linkFixer` (link-fixer.ts): expand every t.co short
/// link that has an entity mapping to its real destination, drop internal
/// `x.com/i/web/status/…` plumbing links, then strip any remaining t.co
/// short link (the appended media link and other unmapped short links).
/// Pure content matching — no `display_text_range` arithmetic, so the
/// endpoint's inconsistent index units (UTF-16 vs code points, see the
/// deleted `strip_trailing_short_links`) never matter.
fn expand_links(text: &str, urls: &[model::SyndicationEntityUrl]) -> String {
let mut out = text.to_string();
for entity in urls {
if let Some(expanded) = &entity.expanded_url {
out = out.replace(&entity.url, expanded);
}
let Some(expanded) = &entity.expanded_url else {
continue;
};
let replacement = if WEB_STATUS_URL.is_match(expanded) {
""
} else {
expanded
};
out = out.replace(&entity.url, replacement);
}
out
TCO_LINK.replace_all(&out, "").into_owned()
}
/// Internal x.com page links (reply / quote plumbing) expand to
/// `x.com/i/web/status/<id>`; FxEmbed drops them — the tweet's own content
/// already carries the information.
static WEB_STATUS_URL: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^https://(?:x\.com|twitter\.com)/i/web/status/\w+").unwrap());
/// A t.co short link, optionally preceded by a space. Any leftover
/// occurrence (unmapped — e.g. the appended media link) is removed,
/// mirroring FxEmbed. Real short-link codes are 10 alphanumerics; the
/// length-agnostic class keeps fixtures and hypothetical odd lengths safe.
static TCO_LINK: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r" ?https?://t\.co/[A-Za-z0-9]+").unwrap());
/// pbs.twimg.com serves a reduced default size without size params; `name=orig`
/// returns the original file (fxtwitter used to hand out the original
/// directly, the syndication API does not). Non-twimg URLs pass through
/// unchanged.
fn original_twimg_url(url: &str) -> String {
if url.starts_with("https://pbs.twimg.com/")
&& (url.ends_with(".jpg") || url.ends_with(".png"))
if url.starts_with("https://pbs.twimg.com/") && (url.ends_with(".jpg") || url.ends_with(".png"))
{
format!("{url}?name=orig")
} else {
@@ -363,24 +363,23 @@ mod tests {
match &fetched.media[0] {
Media::Illustration { url, .. } => {
// Photo URL is rewritten to request the original file.
assert_eq!(
url,
"https://pbs.twimg.com/media/photo.jpg?name=orig"
);
assert_eq!(url, "https://pbs.twimg.com/media/photo.jpg?name=orig");
}
other => panic!("expected illustration, got {other:?}"),
}
match &fetched.media[1] {
Media::Video { url, thumbnail_url, .. } => {
Media::Video {
url, thumbnail_url, ..
} => {
assert_eq!(url, "https://video.twimg.com/v.mp4");
assert_eq!(thumbnail_url, "https://pbs.twimg.com/thumb.jpg");
}
other => panic!("expected video, got {other:?}"),
}
assert!(
fetched
.caption
.contains("<a href=\"https://x.com/author_handle\">Display Name</a>: a &amp; b &lt;c&gt;"),
fetched.caption.contains(
"<a href=\"https://x.com/author_handle\">Display Name</a>: a &amp; b &lt;c&gt;"
),
"caption: {}",
fetched.caption
);
@@ -411,13 +410,12 @@ mod tests {
#[test]
fn syndication_text_strips_trailing_media_short_link() {
// Real syndication shape: the media short link sits after the visible
// text, and display_text_range marks where it begins.
// Real syndication shape: the appended media short link sits after the
// visible text; the unmapped t.co link is stripped by content.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": "hello world https://t.co/abc123",
"display_text_range": [0, 11],
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
@@ -427,8 +425,31 @@ mod tests {
}
#[test]
fn syndication_text_strips_trailing_short_link_without_range() {
// No display_text_range: the regex fallback removes the trailing link.
fn syndication_text_strips_trailing_link_regardless_of_index_units() {
// Real tweet 2084567054481571919: the visible text is 30 code points
// but 41 UTF-16 units, and the two endpoints historically reported
// display_text_range in different units (UTF-16 on syndication, code
// points on GraphQL). The FxEmbed-style content-based strip ignores
// the range entirely, so the appended media link is removed for any
// response shape.
let text = "妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero https://t.co/XnIi83EkEB";
let visible = "妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero";
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "2084567054481571919",
"text": text,
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, visible, "left a partial link");
assert!(!tweet.caption().contains("t.co"));
}
#[test]
fn syndication_text_strips_trailing_short_link_without_entities() {
// No URL entities at all: the leftover t.co link is stripped by the
// content regex.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
@@ -449,7 +470,6 @@ mod tests {
"__typename": "Tweet",
"id_str": "1",
"text": "Test Tweet with @mentionThis $twtr https://t.co/RzmrQ6wAzD #hashtag https://t.co/9r69akA484",
"display_text_range": [0, 67],
"user": { "name": "N", "screen_name": "h" },
"entities": {
"urls": [{
@@ -469,25 +489,47 @@ mod tests {
}
#[test]
fn syndication_text_keeps_unmapped_short_links() {
// No entity mapping for the embedded link: it stays as-is. Only the
// trailing media link is stripped.
fn syndication_text_strips_unmapped_short_links() {
// FxEmbed parity: short links without an entity mapping (appended
// media link, embedded unmapped links) are stripped, not kept.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": "check https://t.co/abc123 #tag https://t.co/def456",
"display_text_range": [0, 30],
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, "check https://t.co/abc123 #tag");
assert_eq!(tweet.text, "check #tag");
}
#[test]
fn syndication_text_utf16_display_range_keeps_multibyte() {
// display_text_range is in UTF-16 units; a Japanese text must not be
// sliced by UTF-8 bytes.
fn syndication_text_drops_internal_web_status_links() {
// FxEmbed parity: a mapped link expanding to an internal
// x.com/i/web/status/... page (reply/quote plumbing) is removed
// instead of being shown.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": "see https://t.co/xyz1234567 for context",
"user": { "name": "N", "screen_name": "h" },
"entities": {
"urls": [{
"url": "https://t.co/xyz1234567",
"expanded_url": "https://x.com/i/web/status/9876543210",
"display_url": "x.com/i/web/status/9876543210"
}]
},
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, "see for context");
assert!(!tweet.caption().contains("t.co"));
}
#[test]
fn syndication_text_keeps_multibyte_text() {
// Text-only tweet: no short links, the multibyte text is untouched.
let text = "コミティア落ちたので、明日は行きません。🙏ごめんなさい";
let units: Vec<u16> = text.encode_utf16().collect();
assert_eq!(units.len(), 28);
@@ -495,7 +537,6 @@ mod tests {
"__typename": "Tweet",
"id_str": "1",
"text": text,
"display_text_range": [0, 28],
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
@@ -547,6 +588,9 @@ mod tests {
async fn live_fetch_deleted_tweet_is_not_found() {
// Deleted tweet: the syndication endpoint answers with errors.
let result = fetch("0").await;
assert!(matches!(result, Err(FetchError::NotFound)), "got {result:?}");
assert!(
matches!(result, Err(FetchError::NotFound)),
"got {result:?}"
);
}
}
-4
View File
@@ -9,10 +9,6 @@ pub struct SyndicationTweet {
pub user: SyndicationUser,
#[serde(default)]
pub possibly_sensitive: Option<bool>,
/// Visible-text span; the raw `text` field has the appended media short
/// link after it. Indices are Unicode code points (not UTF-16 units).
#[serde(default, rename = "display_text_range")]
pub display_text_range: Option<[usize; 2]>,
#[serde(default)]
pub entities: SyndicationEntities,
#[serde(default, rename = "mediaDetails")]
+5 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "xmedia-bot"
version = "1.0.6"
version = "1.1.0"
edition = "2024"
[dependencies]
@@ -12,10 +12,13 @@ log = "0.4"
pretty_env_logger = "0.5"
dotenv = "0.15"
url = "2.5.2"
regex = "1.12"
html-escape = "0.2"
rusqlite = { version = "0.32", features = ["bundled"] }
rand = "0.8"
tempfile = "3"
parking_lot = "0.12"
png = "0.18"
zune-jpeg = "0.5"
fast_image_resize = "6"
jpeg-encoder = "0.7"
x-media = { path = "../x-media" }
+28 -30
View File
@@ -24,39 +24,37 @@ pub struct Config {
impl Config {
pub fn load() -> Config {
let admin_ids = env::var("BOT_ADMIN")
.ok()
.map(|s| {
s.split(',')
.filter_map(|part| part.trim().parse::<i64>().ok())
.collect()
})
.unwrap_or_default();
.ok()
.map(|s| {
s.split(',')
.filter_map(|part| part.trim().parse::<i64>().ok())
.collect()
})
.unwrap_or_default();
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(86400));
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(86400));
let link_cache_ttl = env::var("LINK_CACHE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(7 * 24 * 3600));
let link_cache_ttl = env::var("LINK_CACHE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(7 * 24 * 3600));
let webhook_enabled = env::var("WEBHOOK")
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
// value that would otherwise come from `.env`).
let webhook_cert = env::var("WEBHOOK_CERT")
.ok()
.filter(|s| !s.is_empty());
let webhook_secret_token = env::var("WEBHOOK_SECRET_TOKEN")
.ok()
.filter(|s| !s.is_empty());
let webhook_enabled = env::var("WEBHOOK")
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
// value that would otherwise come from `.env`).
let webhook_cert = env::var("WEBHOOK_CERT").ok().filter(|s| !s.is_empty());
let webhook_secret_token = env::var("WEBHOOK_SECRET_TOKEN")
.ok()
.filter(|s| !s.is_empty());
Config {
admin_ids,
+50
View File
@@ -0,0 +1,50 @@
//! Shared SQLite plumbing for the three tables in `data/task_queue.db`
//! (`tasks` in queue.rs, `chat_state` in state.rs, `link_cache` in
//! link_cache.rs).
//!
//! Every operation opens its own short-lived connection with a busy timeout:
//! handler tasks enqueue while workers lease/update rows concurrently, and
//! without the timeout a concurrent write fails immediately with SQLITE_BUSY
//! and the operation is lost. All I/O runs inside `spawn_blocking` via
//! [`with_conn`] — rusqlite connections are not Send-friendly to hold across
//! an await point, and blocking the async executor stalls every handler.
use rusqlite::Connection;
use std::time::Duration;
/// Opens the shared DB with a busy timeout.
pub fn open_db(path: &str) -> rusqlite::Result<Connection> {
let conn = Connection::open(path)?;
conn.busy_timeout(Duration::from_secs(5))?;
// WAL lets readers run alongside writer leases instead of blocking on
// the rollback journal; the mode persists in the DB header, so the
// idempotent pragma here and in ensure_schema only needs to win once.
conn.pragma_update(None, "journal_mode", "WAL")?;
Ok(conn)
}
/// Unix timestamp in fractional seconds. Shared by the queue, chat store and
/// link cache (previously four private copies).
pub fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
}
/// Runs `f` against a fresh connection on a blocking thread, returning the
/// closure's result. Owns the `spawn_blocking` + `expect` ceremony shared by
/// every table access; the caller maps errors to its own log line.
pub async fn with_conn<T, F>(path: &str, f: F) -> rusqlite::Result<T>
where
T: Send + 'static,
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
{
let path = path.to_string();
tokio::task::spawn_blocking(move || {
let mut conn = open_db(&path)?;
f(&mut conn)
})
.await
.expect("db worker panicked")
}
+276 -101
View File
@@ -1,57 +1,105 @@
use crate::config::Config;
use crate::db::now_f64;
use crate::link_cache::{CachedMediaKind, CachedPost, LinkCache};
use crate::queue::PersistentTaskQueue;
use crate::send::{self, MediaItemPayload, Task};
use crate::state::{ChatData, ChatStore, unix_now};
use std::collections::HashSet;
use std::sync::LazyLock;
use teloxide::RequestError;
use teloxide::prelude::*;
use tokio::sync::Semaphore;
use teloxide::types::{
CallbackQuery, ChatAction, ChatId, ChatKind, InlineQuery, InlineQueryResult,
InlineQueryResultMpeg4Gif, InlineQueryResultPhoto, InlineQueryResultVideo, Message,
MessageEntityKind, MessageId, ParseMode, Recipient, ReplyParameters,
};
use teloxide::utils::command::BotCommands;
use teloxide::RequestError;
use x_media::media::Media;
pub static CHAT_STORE: LazyLock<ChatStore> = LazyLock::new(|| {
ChatStore::open("data/task_queue.db").expect("failed to open chat store")
});
/// One URL job: bot handle + the message + the extracted URL.
type UrlJob = (Bot, Message, String);
/// Bounded channel of URL jobs drained by [`start_url_workers`]. The bound
/// caps both queued memory and shutdown backlog; a full channel applies
/// backpressure to the per-chat handler instead of spawning unbounded tasks.
static URL_JOBS: LazyLock<parking_lot::Mutex<Option<tokio::sync::mpsc::Sender<UrlJob>>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
/// Set by main's shutdown sequence; workers stop pulling new jobs.
static URL_STOP: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
/// Worker count draining URL jobs; keeps the old 8-permit concurrency cap
/// while bounding how many jobs can be queued at all.
const URL_WORKERS: usize = 8;
/// Starts the URL job workers (called once from main after the queue starts).
/// teloxide dispatches updates to a per-chat worker that handles them
/// sequentially, so a batch-forward of many messages would otherwise be
/// processed one at a time (fetch + send each, roughly a second per
/// message); the workers add throughput, and FIFO order preserves per-message
/// URL order.
pub async fn start_url_workers() {
let (tx, rx) = tokio::sync::mpsc::channel::<UrlJob>(256);
*URL_JOBS.lock() = Some(tx);
let rx = std::sync::Arc::new(tokio::sync::Mutex::new(rx));
for _ in 0..URL_WORKERS {
let rx = std::sync::Arc::clone(&rx);
tokio::spawn(async move {
while !URL_STOP.load(std::sync::atomic::Ordering::Relaxed) {
let job = rx.lock().await.recv().await;
match job {
Some((bot, message, url)) => url_media(bot, &message, &url).await,
None => break,
}
}
});
}
}
/// Stops URL workers (drains up to the 256 queued jobs, then exits).
pub fn stop_url_workers() {
URL_STOP.store(true, std::sync::atomic::Ordering::Relaxed);
}
pub static CHAT_STORE: LazyLock<ChatStore> =
LazyLock::new(|| ChatStore::open("data/task_queue.db").expect("failed to open chat store"));
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
LazyLock::new(|| PersistentTaskQueue::new("data/task_queue.db"));
pub static LINK_CACHE: LazyLock<LinkCache> =
LazyLock::new(|| LinkCache::open("data/task_queue.db"));
pub static CONFIG: LazyLock<Config> = LazyLock::new(Config::load);
/// Cap on concurrent per-URL processing. teloxide dispatches updates to a
/// per-chat worker that handles them sequentially, so a batch-forward of many
/// messages would otherwise be processed one at a time (fetch + send each,
/// roughly a second per message). Moving the work into spawned tasks trades
/// per-chat reply ordering for throughput; the semaphore bounds how many run
/// at once so a big burst cannot hammer Telegram's rate limits.
static URL_TASKS: LazyLock<Semaphore> = LazyLock::new(|| Semaphore::new(8));
#[derive(BotCommands, Clone)]
#[command(rename_rule = "snake_case", description = "")]
#[command(
rename_rule = "snake_case",
description = "Turn X/Pixiv/Bluesky links into media messages"
)]
enum Command {
#[command(description = "")]
#[command(description = "Get started")]
Start,
#[command(description = "")]
#[command(description = "Show command help")]
Help,
#[command(description = "", parse_with = "split")]
#[command(
description = "Set forward channel (@channel or ID)",
parse_with = "split"
)]
SetForwardChannel(String),
#[command(description = "")]
#[command(description = "Remove forward channel")]
RemoveForwardChannel,
#[command(description = "")]
#[command(description = "Toggle edit-before-forward")]
EditBeforeForward,
#[command(description = "", parse_with = "split")]
#[command(
description = "Reply with [] to save as template",
parse_with = "split"
)]
SetTemplate(String),
#[command(description = "")]
#[command(description = "Show chat state (debug)")]
BotDict,
#[command(description = "", parse_with = "split")]
#[command(description = "Set site caption format", parse_with = "split")]
SetFormat(String),
#[command(
description = "Clear link cache (admin; optional URL, else all)",
parse_with = "split"
)]
ClearCache(String),
}
async fn reply<T>(bot: Bot, message: Message, text: T) -> Result<Message, RequestError>
@@ -63,13 +111,6 @@ where
.await
}
fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
}
/// Extracts URL and text-link entities (text + caption), deduped in order.
pub fn extract_urls(message: &Message) -> Vec<String> {
let mut urls = Vec::new();
@@ -88,7 +129,10 @@ pub fn extract_urls(message: &Message) -> Vec<String> {
}
}
let mut seen = HashSet::new();
urls.retain(|url| seen.insert(url.clone()));
// Dedup by the normalized post id so variant URLs of the same post
// (/status/1 vs /status/1/photo/1) are sent once; unsupported URLs fall
// back to exact-string dedup.
urls.retain(|url| seen.insert(x_media::site::cache_key(url).unwrap_or_else(|| url.clone())));
urls
}
@@ -111,7 +155,7 @@ async fn edit_message_handler(bot: &Bot, message: &Message) -> bool {
};
let link = format!(
"<a href=\"{0}\">{1}</a>",
edit.url,
html_escape::encode_double_quoted_attribute(&edit.url),
html_escape::encode_text(text)
);
let new_text = if edit.template.is_empty() {
@@ -177,19 +221,31 @@ async fn set_forward_channel_handler(
return Err(SetForwardChannelError::NotChannel);
}
let channel_id = chat.id.0;
// The sender must be a channel administrator. Compare against the
// sender's user id, NOT the chat id (they only coincide in private
// chats, so the old check broke group usage).
let Some(sender) = message.from.as_ref() else {
return Err(SetForwardChannelError::NotAdmin);
};
match bot.get_chat_administrators(channel.clone()).await {
Err(e) => {
log::error!("Failed to get channel administrators {}: {}", channel, e);
return Err(SetForwardChannelError::NotBotAdmin(e));
}
Ok(admins) => {
if !admins.iter().any(|admin| admin.user.id == message.chat.id) {
if !admins.iter().any(|admin| admin.user.id == sender.id) {
return Err(SetForwardChannelError::NotAdmin);
}
let bot_id = bot.get_me().await.expect("Failed get bot id").user.id;
if let Some(bot_admin) = admins.iter().find(|admin| admin.user.id == bot_id)
&& !bot_admin.can_post_messages()
{
// The bot itself must be an admin that can post; a missing
// bot entry must not pass silently (copy would fail later).
let bot_id = match bot.get_me().await {
Ok(me) => me.user.id,
Err(e) => return Err(SetForwardChannelError::NotBotAdmin(e)),
};
let bot_ok = admins
.iter()
.any(|admin| admin.user.id == bot_id && admin.can_post_messages());
if !bot_ok {
return Err(SetForwardChannelError::NotBotCanPost);
}
}
@@ -197,7 +253,11 @@ async fn set_forward_channel_handler(
Ok(channel_id)
}
async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Result<(), RequestError> {
async fn execute_command(
bot: &Bot,
message: &Message,
command: Command,
) -> Result<(), RequestError> {
match command {
Command::Start => {
bot.send_message(message.chat.id, "Hello!").await?;
@@ -209,13 +269,16 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
Command::SetForwardChannel(channel) => {
let result = match set_forward_channel_handler(bot, message, channel).await {
Ok(channel_id) => {
let mut chat_data = CHAT_STORE.get(message.chat.id.0).await;
chat_data.forward_channel_id = Some(channel_id);
CHAT_STORE.set(message.chat.id.0, &chat_data).await;
CHAT_STORE
.update(message.chat.id.0, |data| {
data.forward_channel_id = Some(channel_id);
})
.await;
"Add successfully.".to_string()
}
Err(SetForwardChannelError::EmptyParameter) => {
"Receive empty parameter.\nYou should enter a channel id or username".to_string()
"Receive empty parameter.\nYou should enter a channel id or username"
.to_string()
}
Err(SetForwardChannelError::NotChannel) => {
"Given id / username is not a channel".to_string()
@@ -234,31 +297,34 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
}
Command::RemoveForwardChannel => {
let chat_id = message.chat.id.0;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let text = if chat_data.forward_channel_id.is_some() {
chat_data.forward_channel_id = None;
CHAT_STORE.set(chat_id, &chat_data).await;
"Remove successfully.".to_string()
} else {
"No channel to remove.".to_string()
};
let text = CHAT_STORE
.update(chat_id, |data| {
if data.forward_channel_id.is_some() {
data.forward_channel_id = None;
"Remove successfully.".to_string()
} else {
"No channel to remove.".to_string()
}
})
.await;
reply(bot.clone(), message.clone(), text).await?;
}
Command::EditBeforeForward => {
let chat_id = message.chat.id.0;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let text = if chat_data.forward_channel_id.is_none() {
"Please enable forward channel first.".to_string()
} else if chat_data.edit_before_forward {
chat_data.edit_before_forward = false;
chat_data.edit_message.clear();
CHAT_STORE.set(chat_id, &chat_data).await;
"Disable edit before forward.".to_string()
} else {
chat_data.edit_before_forward = true;
CHAT_STORE.set(chat_id, &chat_data).await;
"Enable edit before forward.".to_string()
};
let text = CHAT_STORE
.update(chat_id, |data| {
if data.forward_channel_id.is_none() {
"Please enable forward channel first.".to_string()
} else if data.edit_before_forward {
data.edit_before_forward = false;
data.edit_message.clear();
"Disable edit before forward.".to_string()
} else {
data.edit_before_forward = true;
"Enable edit before forward.".to_string()
}
})
.await;
reply(bot.clone(), message.clone(), text).await?;
}
Command::SetTemplate(name) => {
@@ -272,11 +338,14 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
} else if name.is_empty() {
"Please provide a name for the template.".to_string()
} else {
let mut chat_data = CHAT_STORE.get(chat_id).await;
chat_data
.template
.insert(name, html_escape::encode_text(reply_text).into_owned());
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.template.insert(
name,
html_escape::encode_text(reply_text).into_owned(),
);
})
.await;
"Template set.".to_string()
}
}
@@ -292,7 +361,9 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
Command::SetFormat(arg) => {
let chat_id = message.chat.id.0;
let (site, format) = match arg.split_once(char::is_whitespace) {
Some((site, format)) if !format.trim().is_empty() => (site.trim(), format.trim().to_string()),
Some((site, format)) if !format.trim().is_empty() => {
(site.trim(), format.trim().to_string())
}
_ => {
reply(
bot.clone(),
@@ -312,15 +383,78 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
.await?;
return Ok(());
}
let mut chat_data = CHAT_STORE.get(chat_id).await;
chat_data.message_format.insert(site.to_string(), format);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.message_format.insert(site.to_string(), format);
})
.await;
reply(bot.clone(), message.clone(), "Format set.").await?;
}
Command::ClearCache(arg) => {
let sender_id = message
.from
.as_ref()
.map(|user| user.id.0 as i64)
.unwrap_or(-1);
if !CONFIG.admin_ids.contains(&sender_id) {
reply(bot.clone(), message.clone(), "Admin only.").await?;
return Ok(());
}
let arg = arg.trim();
if arg.is_empty() {
let removed = LINK_CACHE.clear(None).await;
log::info!("cache cleared by {sender_id}: {removed} entries");
reply(
bot.clone(),
message.clone(),
format!("Cleared {removed} cached entr{}.", plural(removed)),
)
.await?;
} else {
let key = match x_media::site::cache_key(arg) {
Some(key) => key,
None => {
reply(
bot.clone(),
message.clone(),
"Unrecognized link. Use a twitter/x, pixiv or bsky post URL.",
)
.await?;
return Ok(());
}
};
let removed = LINK_CACHE.clear(Some(&key)).await;
log::info!("cache entry cleared by {sender_id}: {key} ({removed} rows)");
reply(
bot.clone(),
message.clone(),
format!(
"Cleared cache for {arg} ({} entr{}).",
removed,
plural(removed)
),
)
.await?;
}
}
}
Ok(())
}
/// `"y"` for one, `"ies"` for anything else — "1 entry" / "2 entries".
fn plural(n: usize) -> &'static str {
if n == 1 { "y" } else { "ies" }
}
/// Registers the bot's command list with Telegram so clients show it in the
/// `/` menu (Bot API `setMyCommands`).
pub async fn register_commands(bot: &Bot) -> Result<(), RequestError> {
let commands = Command::bot_commands();
bot.set_my_commands(commands.clone()).await?;
log::info!("registered {} commands", commands.len());
Ok(())
}
/// For locally produced media (encoded ugoira MP4) the thumbnail URL is a
/// hotlink-protected remote URL Telegram may not fetch; let Telegram generate
/// its own thumbnail instead.
@@ -383,7 +517,10 @@ async fn dispatch_send(bot: Bot, message: &Message, task: &Task, url: &str) {
log::info!("sent {} message(s) for {url}", message_ids.len());
send::post_send_actions(&bot, task, message_ids).await;
}
Err(send::SendError::Retryable { delay_seconds, task }) => {
Err(send::SendError::Retryable {
delay_seconds,
task,
}) => {
log::info!("send for {url} failed, queued for retry in {delay_seconds:.1}s");
enqueue_retry(task, delay_seconds).await;
let _ = reply(bot, message.clone(), "Send failed. Task queued for retry.").await;
@@ -444,7 +581,10 @@ fn build_send_task(
async fn url_media(bot: Bot, message: &Message, url: &str) {
let chat_id = message.chat.id.0;
if let Err(e) = bot.send_chat_action(ChatId(chat_id), ChatAction::Typing).await {
if let Err(e) = bot
.send_chat_action(ChatId(chat_id), ChatAction::Typing)
.await
{
log::error!("send_chat_action failed: {e}");
}
@@ -520,7 +660,12 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
// Retries exhausted: notify the user (Rust-only requirement 3).
Err(e) => {
log::error!("fetch {url}: {e}");
let _ = reply(bot, message.clone(), "Failed to fetch media from this link.").await;
let _ = reply(
bot,
message.clone(),
"Failed to fetch media from this link.",
)
.await;
}
Ok(Some(fetched)) => {
if fetched.media.is_empty() {
@@ -542,8 +687,9 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
let caption = fetched.caption_with(&format);
// Raw render data for the link cache; the send fills in the
// Telegram file ids and persists the entry.
let cache_data = fetched.render_fields().map(|(author, author_url, title, tags)| {
CachedPost {
let cache_data = fetched
.render_fields()
.map(|(author, author_url, title, tags)| CachedPost {
url: fetched.source_url.clone(),
caption: fetched.caption.clone(),
title: title.to_string(),
@@ -552,8 +698,7 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
tags: tags.to_string(),
sensitive: fetched.sensitive,
media: vec![],
}
});
});
let items: Vec<MediaItemPayload> = fetched
.media
.iter()
@@ -581,9 +726,15 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
.unwrap_or_else(|| "unknown".to_string());
let text_preview = message
.text()
.map(|t| if t.len() > 120 { &t[..120] } else { t })
.map(|t| {
let end = t.floor_char_boundary(120.min(t.len()));
&t[..end]
})
.unwrap_or("<no text>");
log::info!("message from {sender} in {} (private={is_private}): {text_preview}", message.chat.id);
log::info!(
"message from {sender} in {} (private={is_private}): {text_preview}",
message.chat.id
);
// URL/edit flows only run in private chats; commands run in any chat.
if is_private && edit_message_handler(&bot, &message).await {
return respond(());
@@ -601,13 +752,13 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
log::info!("extracted {} URL(s): {urls:?}", urls.len());
}
for url in urls {
let bot = bot.clone();
let message = message.clone();
tokio::spawn(async move {
// Held for the whole task; the semaphore is never closed.
let _permit = URL_TASKS.acquire().await.expect("URL semaphore closed");
url_media(bot, &message, &url).await;
});
// Clone out of the lock: the parking_lot guard is !Send and must
// not be held across the await below.
let Some(tx) = URL_JOBS.lock().clone() else {
log::warn!("url workers not started; dropping link");
break;
};
let _ = tx.send((bot.clone(), message.clone(), url)).await;
}
}
respond(())
@@ -617,6 +768,12 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
if query.query.is_empty() {
return respond(());
}
// Telegram fires an inline query on every keystroke; only run a fetch
// (3 attempts!) for something that is actually a supported post URL, so
// typing does not hammer the source sites.
if x_media::site::cache_key(&query.query).is_none() {
return respond(());
}
log::info!("inline query: {}", query.query);
match x_media::site::fetch(&query.query).await {
Ok(Some(fetched)) => {
@@ -653,8 +810,8 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
thumbnail,
fetched.title.clone(),
)
.caption(caption)
.parse_mode(ParseMode::Html),
.caption(caption)
.parse_mode(ParseMode::Html),
),
Media::Animated { .. } => InlineQueryResult::Mpeg4Gif(
InlineQueryResultMpeg4Gif::new(id, url, thumbnail)
@@ -683,10 +840,13 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let chat_id = message.chat().id.0;
let prompt_message_id = message.id().0 as i64;
let ttl_secs = CONFIG.edit_message_ttl.as_secs() as i64;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let chat_data = CHAT_STORE.get(chat_id).await;
let edit = chat_data.edit_message.get(&prompt_message_id).cloned();
let Some(edit) = edit else {
log::info!("callback from {}: no edit record for prompt {prompt_message_id}", chat_id);
log::info!(
"callback from {}: no edit record for prompt {prompt_message_id}",
chat_id
);
bot.answer_callback_query(callback_query_id)
.text("Expired")
.await?;
@@ -694,8 +854,11 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
};
// Lazy expiry: a stale record (past the TTL, not yet swept) is dropped.
if edit.created_at + ttl_secs <= unix_now() {
chat_data.edit_message.remove(&prompt_message_id);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
bot.answer_callback_query(callback_query_id)
.text("Expired")
.await?;
@@ -705,7 +868,10 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let Some(data) = data else {
return respond(());
};
log::info!("callback from {} on prompt {prompt_message_id}: {data}", chat_id);
log::info!(
"callback from {} on prompt {prompt_message_id}: {data}",
chat_id
);
if data == "forward" {
match chat_data.forward_channel_id {
Some(channel_id) => {
@@ -728,10 +894,16 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let _ = bot
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
.await;
chat_data.edit_message.remove(&prompt_message_id);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
}
Err(send::SendError::Retryable { delay_seconds, task }) => {
Err(send::SendError::Retryable {
delay_seconds,
task,
}) => {
log::info!("forward queued for retry in {delay_seconds:.1}s");
enqueue_retry(task, delay_seconds).await;
bot.answer_callback_query(callback_query_id)
@@ -765,10 +937,13 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
.caption(template_html)
.parse_mode(ParseMode::Html)
.await;
if let Some(entry) = chat_data.edit_message.get_mut(&prompt_message_id) {
entry.template = name.to_string();
}
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
if let Some(entry) = data.edit_message.get_mut(&prompt_message_id) {
entry.template = name.to_string();
}
})
.await;
log::info!("template '{name}' applied to prompt {prompt_message_id}");
}
bot.answer_callback_query(callback_query_id).await?;
+113 -56
View File
@@ -8,7 +8,8 @@
//! [`Config::link_cache_ttl`]; a stale entry is dropped lazily on read and
//! by the periodic prune in `main`.
use rusqlite::{params, Connection};
use crate::db::now_f64;
use rusqlite::{Connection, params};
use serde::{Deserialize, Serialize};
use std::time::Duration;
@@ -49,12 +50,6 @@ pub struct LinkCache {
db_path: String,
}
fn open_db(path: &str) -> rusqlite::Result<Connection> {
let conn = Connection::open(path)?;
conn.busy_timeout(Duration::from_secs(5))?;
Ok(conn)
}
impl LinkCache {
pub fn open(db_path: &str) -> Self {
if let Ok(conn) = Connection::open(db_path)
@@ -73,11 +68,9 @@ impl LinkCache {
/// Returns the cached post if present and not expired; a stale entry is
/// removed on the spot.
pub async fn get(&self, key: &str, ttl: Duration) -> Option<CachedPost> {
let db_path = self.db_path.clone();
let key = key.to_string();
let ttl = ttl.as_secs_f64();
tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<CachedPost>> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
let mut stmt =
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
let mut rows = stmt.query(params![key])?;
@@ -90,74 +83,85 @@ impl LinkCache {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
return Ok(None);
}
serde_json::from_str(&payload).map(Some).map_err(|e| {
rusqlite::Error::ToSqlConversionFailure(Box::new(e))
})
})
.await
.expect("link cache read worker panicked")
.unwrap_or_else(|e| {
log::error!("link cache read failed: {e}");
None
Ok(Some(serde_json::from_str::<CachedPost>(&payload).map_err(
|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)),
)?))
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::error!("link cache read failed: {e}");
None
}
}
}
pub async fn put(&self, key: &str, post: &CachedPost) {
let db_path = self.db_path.clone();
let key = key.to_string();
let payload = serde_json::to_string(post).expect("cached post serializes");
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"INSERT OR REPLACE INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
params![key, payload, now_f64()],
)?;
Ok(())
})
.await
.expect("link cache write worker panicked")
.unwrap_or_else(|e| log::error!("link cache write failed: {e}"));
.await;
if let Err(e) = result {
log::error!("link cache write failed: {e}");
}
}
/// Drops an entry (e.g. a cached file id that turned out invalid).
pub async fn remove(&self, key: &str) {
let db_path = self.db_path.clone();
let key = key.to_string();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Ok(())
})
.await
.expect("link cache delete worker panicked")
.unwrap_or_else(|e| log::error!("link cache delete failed: {e}"));
.await;
if let Err(e) = result {
log::error!("link cache delete failed: {e}");
}
}
/// Removes expired entries; returns how many were deleted.
pub async fn prune(&self, ttl: Duration) -> usize {
let db_path = self.db_path.clone();
let cutoff = now_f64() - ttl.as_secs_f64();
tokio::task::spawn_blocking(move || -> rusqlite::Result<usize> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"DELETE FROM link_cache WHERE created_at < ?1",
params![cutoff],
)
})
.await
.expect("link cache prune worker panicked")
.unwrap_or_else(|e| {
log::error!("link cache prune failed: {e}");
0
})
.await;
match result {
Ok(n) => n,
Err(e) => {
log::error!("link cache prune failed: {e}");
0
}
}
}
}
fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
/// Deletes one entry (by normalized cache key) or the whole cache when
/// `key` is `None`. Returns how many rows were removed.
pub async fn clear(&self, key: Option<&str>) -> usize {
let key = key.map(str::to_string);
let result = crate::db::with_conn(&self.db_path, move |conn| match &key {
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
None => conn.execute("DELETE FROM link_cache", []),
})
.await;
match result {
Ok(n) => n,
Err(e) => {
log::error!("link cache clear failed: {e}");
0
}
}
}
}
#[cfg(test)]
@@ -200,14 +204,21 @@ mod tests {
// Force the row into the past so a 1s TTL expires it.
{
let conn = Connection::open(dir.path().join("c.db")).unwrap();
conn.execute(
"UPDATE link_cache SET created_at = created_at - 100",
[],
)
.unwrap();
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
.unwrap();
}
assert!(cache.get("twitter:1", Duration::from_secs(1)).await.is_none());
assert!(cache.get("twitter:1", Duration::from_secs(3600)).await.is_none());
assert!(
cache
.get("twitter:1", Duration::from_secs(1))
.await
.is_none()
);
assert!(
cache
.get("twitter:1", Duration::from_secs(3600))
.await
.is_none()
);
}
#[tokio::test]
@@ -217,14 +228,60 @@ mod tests {
cache.put("twitter:1", &entry()).await;
cache.put("pixiv:2", &entry()).await;
cache.remove("twitter:1").await;
assert!(cache.get("twitter:1", Duration::from_secs(3600)).await.is_none());
assert!(cache.get("pixiv:2", Duration::from_secs(3600)).await.is_some());
assert!(
cache
.get("twitter:1", Duration::from_secs(3600))
.await
.is_none()
);
assert!(
cache
.get("pixiv:2", Duration::from_secs(3600))
.await
.is_some()
);
{
let conn = Connection::open(dir.path().join("c.db")).unwrap();
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
.unwrap();
}
assert_eq!(cache.prune(Duration::from_secs(1)).await, 1);
assert!(cache.get("pixiv:2", Duration::from_secs(3600)).await.is_none());
assert!(
cache
.get("pixiv:2", Duration::from_secs(3600))
.await
.is_none()
);
}
#[tokio::test]
async fn clear_one_entry_or_all() {
let dir = tempfile::tempdir().unwrap();
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
cache.put("twitter:1", &entry()).await;
cache.put("pixiv:2", &entry()).await;
// By key: only the matching row is removed.
assert_eq!(cache.clear(Some("twitter:1")).await, 1);
assert!(
cache
.get("twitter:1", Duration::from_secs(3600))
.await
.is_none()
);
assert!(
cache
.get("pixiv:2", Duration::from_secs(3600))
.await
.is_some()
);
// Whole cache: nothing left; removing an absent key deletes 0 rows.
assert_eq!(cache.clear(None).await, 1);
assert!(
cache
.get("pixiv:2", Duration::from_secs(3600))
.await
.is_none()
);
assert_eq!(cache.clear(None).await, 0);
}
}
+26 -8
View File
@@ -1,15 +1,17 @@
use dotenv::dotenv;
use teloxide::dptree::endpoint;
use teloxide::prelude::*;
use teloxide::stop::StopToken;
use teloxide::types::{ChatId, InputFile, MessageId};
use teloxide::update_listeners::{self, webhooks, UpdateListener};
use teloxide::prelude::*;
use teloxide::update_listeners::{self, UpdateListener, webhooks};
use tokio::sync::watch;
use x_media::site;
mod config;
mod db;
mod handlers;
mod link_cache;
mod photo;
mod queue;
mod send;
mod state;
@@ -41,6 +43,14 @@ async fn main() {
log::info!("Starting bot");
let bot = Bot::from_env();
// Force the queue workers' shared Bot to initialize now so a missing
// token fails at startup, not on the first queued task.
let _ = &*send::BOT;
// Register the command list with Telegram (client `/` menu).
if let Err(e) = handlers::register_commands(&bot).await {
log::warn!("failed to register commands: {e}");
}
log::info!(
"config: {} admin(s), edit-message TTL {}s",
@@ -55,6 +65,10 @@ async fn main() {
.await;
log::info!("task queue worker started");
// URL job workers: bounded channel + fixed pool for per-URL work.
handlers::start_url_workers().await;
log::info!("url workers started");
// Pixiv login validation (user request): a failed login notifies the
// admin and disables pixiv for this process.
if site::pixiv::enabled() {
@@ -73,7 +87,10 @@ async fn main() {
}
// Edit-expiry sweep: clears the prompt's buttons once the record expires.
log::info!("edit-expiry sweep: every 300s, ttl {}", CONFIG.edit_message_ttl.as_secs());
log::info!(
"edit-expiry sweep: every 300s, ttl {}",
CONFIG.edit_message_ttl.as_secs()
);
let (stop_tx, stop_rx) = watch::channel(false);
{
let bot = bot.clone();
@@ -94,7 +111,10 @@ async fn main() {
// If the prompt was already deleted, this fails with a
// 400 "message to edit not found" — log and ignore.
if let Err(e) = bot
.edit_message_reply_markup(ChatId(chat_id), MessageId(prompt_message_id as i32))
.edit_message_reply_markup(
ChatId(chat_id),
MessageId(prompt_message_id as i32),
)
.await
{
log::info!("edit-expiry sweep: prompt message gone: {e}");
@@ -116,10 +136,7 @@ async fn main() {
if CONFIG.webhook_enabled {
log::info!("running in webhook mode");
let url = CONFIG
.webhook_url
.clone()
.expect("WEBHOOK_URL is not set");
let url = CONFIG.webhook_url.clone().expect("WEBHOOK_URL is not set");
// `webhooks::axum` calls set_webhook itself (with the full options,
// secret token included) — no explicit registration here.
let listen = CONFIG.webhook_listen.expect("WEBHOOK_LISTEN is not set");
@@ -164,6 +181,7 @@ async fn main() {
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin, drain the queue.
log::info!("Stopping bot");
let _ = stop_tx.send(true);
handlers::stop_url_workers();
if let Some(admin) = CONFIG.admin_ids.first() {
let _ = bot.send_message(ChatId(*admin), "Shutting down...").await;
}
+528
View File
@@ -0,0 +1,528 @@
//! Pure-Rust photo processing: brings a downloaded photo within Telegram's
//! limits (width + height ≤ 10000 px, bytes ≤ 10 MiB) without ffmpeg.
//!
//! Stack: `png` (image-png) for PNG decode/encode, `zune-jpeg` for JPEG
//! decode, `fast_image_resize` (Lanczos3) for downsampling, `jpeg-encoder`
//! for JPEG output.
//!
//! Bit-depth rule: a PNG above 24 bits (32-bit RGBA or 16-bit per channel)
//! is reduced to 24-bit RGB; 24-bit and lower depths are left untouched —
//! gray stays gray, never upconverted. The only upconversion is palette
//! expansion, which resampling requires. Alpha is flattened onto white (JPEG
//! and 24-bit RGB have no alpha channel).
use std::io::Write;
use fast_image_resize as fir;
use tempfile::NamedTempFile;
/// Telegram rejects photos whose width + height exceed this limit
/// (PHOTO_INVALID_DIMENSIONS). Verified empirically: 6300x3730 (sum 10030)
/// fails, 6100x3900 (sum 10000) passes.
pub const PHOTO_MAX_DIMENSION_SUM: u32 = 10000;
/// Resize target with a safety margin so rounding cannot cross the cap.
pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900;
/// Upload cap (bytes): files above this are not uploaded; the bot falls back
/// to a smaller media URL instead.
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024;
/// Decode budget (bytes): a larger intermediate buffer is not worth the peak
/// memory; the photo degrades to the smaller URL instead. Also the cap for
/// downloading photos in the send fallback (they must be downloaded whole).
pub(crate) const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
/// JPEG output quality (1-100).
const JPEG_QUALITY: u8 = 90;
/// What to upload for a downloaded photo.
pub enum PhotoPrep {
/// Upload this file (the original when within limits, else the processed
/// copy).
Upload(NamedTempFile),
/// The photo cannot be brought within Telegram's limits — the caller
/// falls back to the item's smaller URL.
UseFallback,
}
/// A decoded image buffer tagged with its channel layout.
#[derive(Debug)]
enum PixBuf {
Gray(Vec<u8>),
GrayAlpha(Vec<u8>),
Rgb(Vec<u8>),
}
impl PixBuf {
fn pixel_type(&self) -> fir::PixelType {
match self {
PixBuf::Gray(_) => fir::PixelType::U8,
PixBuf::GrayAlpha(_) => fir::PixelType::U8x2,
PixBuf::Rgb(_) => fir::PixelType::U8x3,
}
}
fn into_vec(self) -> Vec<u8> {
match self {
PixBuf::Gray(v) | PixBuf::GrayAlpha(v) | PixBuf::Rgb(v) => v,
}
}
}
/// Entry point: detects the format and processes the photo if needed.
pub fn prepare_photo(file: NamedTempFile) -> Result<PhotoPrep, String> {
let bytes = std::fs::read(file.path()).map_err(|e| format!("prepare read failed: {e}"))?;
if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {
prepare_png(file, bytes)
} else if bytes.starts_with(&[0xFF, 0xD8, 0xFF]) {
prepare_jpeg(file, bytes)
} else {
log::warn!("photo in unsupported format; falling back to smaller media");
Ok(PhotoPrep::UseFallback)
}
}
/// Parses the PNG IHDR (bytes 8..26: signature + length + "IHDR" + width +
/// height + bit depth + color type).
fn parse_png_header(bytes: &[u8]) -> Option<(u32, u32, png::BitDepth, png::ColorType)> {
if !bytes.starts_with(b"\x89PNG\r\n\x1a\n") || bytes.len() < 26 {
return None;
}
let w = u32::from_be_bytes(bytes.get(16..20)?.try_into().ok()?);
let h = u32::from_be_bytes(bytes.get(20..24)?.try_into().ok()?);
let depth = match *bytes.get(24)? {
1 => png::BitDepth::One,
2 => png::BitDepth::Two,
4 => png::BitDepth::Four,
8 => png::BitDepth::Eight,
16 => png::BitDepth::Sixteen,
_ => return None,
};
let color = match *bytes.get(25)? {
0 => png::ColorType::Grayscale,
2 => png::ColorType::Rgb,
3 => png::ColorType::Indexed,
4 => png::ColorType::GrayscaleAlpha,
6 => png::ColorType::Rgba,
_ => return None,
};
Some((w, h, depth, color))
}
/// Output channels of a decoded frame for the given color type (post
/// STRIP_16; palette expands to RGB).
fn output_channels(color: png::ColorType) -> usize {
match color {
png::ColorType::Grayscale => 1,
png::ColorType::GrayscaleAlpha => 2,
png::ColorType::Rgb | png::ColorType::Indexed => 3,
png::ColorType::Rgba => 4,
}
}
/// The 32→24 rule: RGBA (32-bit) becomes RGB with alpha composited onto
/// white; 16-bit per channel was already stripped to 8-bit at decode.
fn flatten_rgba_to_rgb(rgba: &[u8]) -> Vec<u8> {
let mut rgb = Vec::with_capacity(rgba.len() / 4 * 3);
for px in rgba.chunks_exact(4) {
let a = px[3] as u32;
for v in &px[..3] {
// Over white: C = C*a/255 + 255*(1 - a/255).
let v = (*v as u32 * a + 255 * (255 - a)) / 255;
rgb.push(v.min(255) as u8);
}
}
rgb
}
/// Lanczos3 downsampling via fast_image_resize.
fn resize_pix(pix: PixBuf, w: u32, h: u32, nw: u32, nh: u32) -> Result<PixBuf, String> {
let pixel_type = pix.pixel_type();
let src = fir::images::Image::from_vec_u8(w, h, pix.into_vec(), pixel_type)
.map_err(|e| format!("resize input: {e}"))?;
let mut dst = fir::images::Image::new(nw, nh, pixel_type);
let mut resizer = fir::Resizer::new();
let options = fir::ResizeOptions::default()
.resize_alg(fir::ResizeAlg::Convolution(fir::FilterType::Lanczos3));
resizer
.resize(&src, &mut dst, &options)
.map_err(|e| format!("resize: {e}"))?;
let buf = dst.into_vec();
Ok(match pixel_type {
fir::PixelType::U8 => PixBuf::Gray(buf),
fir::PixelType::U8x2 => PixBuf::GrayAlpha(buf),
_ => PixBuf::Rgb(buf),
})
}
fn encode_png(out: &mut Vec<u8>, pix: &PixBuf, w: u32, h: u32) -> Result<(), png::EncodingError> {
let (color, buf) = match pix {
PixBuf::Gray(v) => (png::ColorType::Grayscale, v.as_slice()),
PixBuf::GrayAlpha(v) => (png::ColorType::GrayscaleAlpha, v.as_slice()),
PixBuf::Rgb(v) => (png::ColorType::Rgb, v.as_slice()),
};
let mut encoder = png::Encoder::new(out, w, h);
encoder.set_color(color);
encoder.set_depth(png::BitDepth::Eight);
let mut writer = encoder.write_header()?;
writer.write_image_data(buf)?;
Ok(())
}
fn encode_jpeg(pix: &PixBuf, w: u32, h: u32) -> Result<Vec<u8>, String> {
use jpeg_encoder::{ColorType, Encoder};
let mut out = Vec::new();
let encoder = Encoder::new(&mut out, JPEG_QUALITY);
match pix {
PixBuf::Gray(v) => encoder
.encode(v, w as u16, h as u16, ColorType::Luma)
.map_err(|e| format!("jpeg encode: {e}"))?,
PixBuf::GrayAlpha(v) => {
// JPEG has no alpha: composite onto white, output as gray.
let gray: Vec<u8> = v
.chunks_exact(2)
.map(|px| {
let (g, a) = (px[0] as u32, px[1] as u32);
((g * a + 255 * (255 - a)) / 255).min(255) as u8
})
.collect();
encoder
.encode(&gray, w as u16, h as u16, ColorType::Luma)
.map_err(|e| format!("jpeg encode: {e}"))?;
}
PixBuf::Rgb(v) => encoder
.encode(v, w as u16, h as u16, ColorType::Rgb)
.map_err(|e| format!("jpeg encode: {e}"))?,
}
Ok(out)
}
fn write_temp(bytes: &[u8], ext: &str) -> Result<NamedTempFile, String> {
let mut file = tempfile::Builder::new()
.suffix(&format!(".{ext}"))
.tempfile()
.map_err(|e| format!("temp file failed: {e}"))?;
file.as_file_mut()
.write_all(bytes)
.map_err(|e| format!("temp file write failed: {e}"))?;
Ok(file)
}
fn target_dims(w: u32, h: u32) -> (u32, u32) {
let scale = PHOTO_TARGET_DIMENSION_SUM as f64 / (w + h) as f64;
(
((w as f64 * scale).round() as u32).max(1),
((h as f64 * scale).round() as u32).max(1),
)
}
/// PNG branch: decode (16→8, palette→RGB; gray/GA stay), flatten RGBA to
/// RGB, Lanczos-downscale beyond the dimension cap, encode PNG — a PNG still
/// over the upload cap afterwards becomes JPEG.
fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
let (w, h, _bit_depth, color_type) = parse_png_header(&bytes).ok_or("invalid PNG header")?;
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
return Ok(PhotoPrep::Upload(file));
}
log::info!(
"photo {w}x{h} ({_bit_depth:?} {color_type:?}, {} bytes) needs processing",
bytes.len()
);
let channels = output_channels(color_type);
if (w as u64) * (h as u64) * channels as u64 > MAX_DECODE_BYTES {
log::warn!("photo decode buffer exceeds the memory budget; falling back to smaller media");
return Ok(PhotoPrep::UseFallback);
}
// STRIP_16 drops 16-bit to 8-bit (the depth-reduction step); palette
// expands to RGB (resampling requires it). Gray and gray-alpha are kept.
let transforms = match color_type {
png::ColorType::Indexed => png::Transformations::EXPAND,
_ => png::Transformations::STRIP_16,
};
let mut decoder = png::Decoder::new(std::io::Cursor::new(&bytes));
decoder.set_transformations(transforms);
let mut reader = decoder
.read_info()
.map_err(|e| format!("png decode: {e}"))?;
let out_w = reader.info().width;
let out_h = reader.info().height;
let mut buf = vec![
0u8;
reader
.output_buffer_size()
.ok_or("png output buffer size")?
];
reader
.next_frame(&mut buf)
.map_err(|e| format!("png frame: {e}"))?;
let mut pix = match color_type {
png::ColorType::Rgba => PixBuf::Rgb(flatten_rgba_to_rgb(&buf)),
png::ColorType::Grayscale => PixBuf::Gray(buf),
png::ColorType::GrayscaleAlpha => PixBuf::GrayAlpha(buf),
png::ColorType::Rgb | png::ColorType::Indexed => PixBuf::Rgb(buf),
};
let (mut w, mut h) = (out_w, out_h);
if w + h > PHOTO_MAX_DIMENSION_SUM {
let (nw, nh) = target_dims(w, h);
pix = resize_pix(pix, w, h, nw, nh)?;
(w, h) = (nw, nh);
log::info!("downscaled photo to {w}x{h} (Lanczos3)");
}
let mut png_bytes = Vec::new();
encode_png(&mut png_bytes, &pix, w, h).map_err(|e| format!("png encode: {e}"))?;
if png_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
return Ok(PhotoPrep::Upload(write_temp(&png_bytes, "png")?));
}
log::info!("PNG still over the upload cap after processing; transcoding to JPEG");
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
return Ok(PhotoPrep::Upload(write_temp(&jpeg_bytes, "jpg")?));
}
log::warn!("processed photo still exceeds the upload cap; falling back to smaller media");
Ok(PhotoPrep::UseFallback)
}
/// JPEG branch: zune-jpeg decode → Lanczos downscale → jpeg-encoder output.
fn prepare_jpeg(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(&bytes));
// Decodes to RGB by default. Headers first so dimensions are known before
// the (potentially huge) pixel decode.
decoder
.decode_headers()
.map_err(|e| format!("jpeg headers: {e}"))?;
let info = decoder.info().ok_or("jpeg info unavailable")?;
let (w, h) = (info.width as u32, info.height as u32);
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
return Ok(PhotoPrep::Upload(file));
}
if (w as u64) * (h as u64) * 3 > MAX_DECODE_BYTES {
log::warn!("photo decode buffer exceeds the memory budget; falling back to smaller media");
return Ok(PhotoPrep::UseFallback);
}
let pixels = decoder.decode().map_err(|e| format!("jpeg decode: {e}"))?;
let mut pix = PixBuf::Rgb(pixels);
let (mut w, mut h) = (w, h);
if w + h > PHOTO_MAX_DIMENSION_SUM {
let (nw, nh) = target_dims(w, h);
pix = resize_pix(pix, w, h, nw, nh)?;
(w, h) = (nw, nh);
log::info!("downscaled jpeg to {w}x{h} (Lanczos3)");
}
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
return Ok(PhotoPrep::Upload(write_temp(&jpeg_bytes, "jpg")?));
}
log::warn!("processed photo still exceeds the upload cap; falling back to smaller media");
Ok(PhotoPrep::UseFallback)
}
#[cfg(test)]
mod tests {
use super::*;
fn png_header(w: u32, h: u32, depth: u8, color: u8) -> Vec<u8> {
let mut bytes = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR".to_vec();
bytes.extend(w.to_be_bytes());
bytes.extend(h.to_be_bytes());
bytes.extend([depth, color, 0, 0, 0]);
bytes
}
#[test]
fn parses_png_header() {
let bytes = png_header(8979, 5316, 16, 6); // 16-bit RGBA
let (w, h, depth, color) = parse_png_header(&bytes).unwrap();
assert_eq!((w, h), (8979, 5316));
assert_eq!(depth, png::BitDepth::Sixteen);
assert_eq!(color, png::ColorType::Rgba);
let (_, _, depth, color) = parse_png_header(&png_header(10, 10, 8, 0)).unwrap();
assert_eq!(depth, png::BitDepth::Eight);
assert_eq!(color, png::ColorType::Grayscale);
assert!(parse_png_header(b"not a png").is_none());
}
#[test]
fn flatten_rgba_to_rgb_composites_over_white() {
// opaque red stays red
assert_eq!(flatten_rgba_to_rgb(&[255, 0, 0, 255]), vec![255, 0, 0]);
// fully transparent → white
assert_eq!(flatten_rgba_to_rgb(&[0, 0, 0, 0]), vec![255, 255, 255]);
// half alpha red → (255+255)/2 = 255, (0*128 + 255*127)/255 = 127
let out = flatten_rgba_to_rgb(&[255, 0, 0, 128]);
assert_eq!(out[0], 255);
assert_eq!(out[1], 127);
assert_eq!(out[2], 127);
}
#[test]
fn target_dims_stay_under_the_cap() {
for (w, h) in [(12000u32, 7000u32), (10000, 10000), (8979, 5316)] {
let (nw, nh) = target_dims(w, h);
assert!(nw + nh <= PHOTO_MAX_DIMENSION_SUM, "{w}x{h} -> {nw}x{nh}");
assert!(nw >= 1 && nh >= 1);
}
// already within limits: no change expected from the caller, but the
// helper must not produce zero dimensions.
let (nw, nh) = target_dims(500, 400);
assert!(nw >= 1 && nh >= 1);
}
#[test]
fn resize_pix_changes_dimensions() {
// 300x200 RGB → 100x66
let buf: Vec<u8> = (0..300 * 200 * 3).map(|i| (i % 251) as u8).collect();
let resized = resize_pix(PixBuf::Rgb(buf), 300, 200, 100, 66).unwrap();
match resized {
PixBuf::Rgb(v) => assert_eq!(v.len(), 100 * 66 * 3),
other => panic!("expected rgb, got {other:?}"),
}
}
#[test]
fn png_encode_roundtrip_keeps_gray() {
let gray = vec![128u8; 4 * 4];
let mut out = Vec::new();
encode_png(&mut out, &PixBuf::Gray(gray), 4, 4).unwrap();
assert!(!out.is_empty());
let (_, _, depth, color) = parse_png_header(&out).unwrap();
assert_eq!(depth, png::BitDepth::Eight);
assert_eq!(color, png::ColorType::Grayscale);
}
#[test]
fn jpeg_encode_produces_bytes() {
let rgb = vec![128u8; 8 * 8 * 3];
let out = encode_jpeg(&PixBuf::Rgb(rgb), 8, 8).unwrap();
assert!(out.len() > 100);
assert!(out.starts_with(&[0xFF, 0xD8]));
}
/// Writes a small dimension-oversized PNG (9999x2 → sum 10001) to a temp
/// file and runs the full pipeline.
fn run_pipeline(w: u32, h: u32, color: png::ColorType, fill: u8) -> Result<PhotoPrep, String> {
let (channels, data): (usize, Vec<u8>) = match color {
png::ColorType::Grayscale => (1, vec![fill; (w * h) as usize]),
png::ColorType::Rgb => (3, vec![fill; (w * h * 3) as usize]),
_ => unreachable!(),
};
let mut bytes = Vec::new();
{
let mut encoder = png::Encoder::new(&mut bytes, w, h);
encoder.set_color(color);
encoder.set_depth(png::BitDepth::Eight);
let mut writer = encoder.write_header().unwrap();
writer.write_image_data(&data).unwrap();
}
assert_eq!(data.len(), channels * (w * h) as usize);
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
prepare_photo(file)
}
#[test]
fn pipeline_downscales_oversized_png_keeping_format() {
let prep = run_pipeline(9999, 2, png::ColorType::Rgb, 128).unwrap();
match prep {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
let (w, h, depth, color) = parse_png_header(&out).unwrap();
assert!(w + h <= PHOTO_MAX_DIMENSION_SUM, "{w}x{h}");
assert_eq!(depth, png::BitDepth::Eight);
assert_eq!(color, png::ColorType::Rgb);
}
PhotoPrep::UseFallback => panic!("over-dimension PNG should have been resized"),
}
}
#[test]
fn pipeline_keeps_gray_png_gray() {
let prep = run_pipeline(9999, 2, png::ColorType::Grayscale, 200).unwrap();
match prep {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
let (_, _, _, color) = parse_png_header(&out).unwrap();
assert_eq!(color, png::ColorType::Grayscale, "gray must not upconvert");
}
PhotoPrep::UseFallback => panic!("over-dimension gray PNG should have been resized"),
}
}
#[test]
fn pipeline_resizes_oversized_jpeg() {
// Build a small over-dimension JPEG with jpeg-encoder.
let (w, h) = (9999u16, 2u16);
let rgb = vec![90u8; (w as usize) * (h as usize) * 3];
let mut bytes = Vec::new();
{
let encoder = jpeg_encoder::Encoder::new(&mut bytes, 90);
encoder
.encode(&rgb, w, h, jpeg_encoder::ColorType::Rgb)
.unwrap();
}
let mut file = tempfile::Builder::new().suffix(".jpg").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
match prepare_photo(file).unwrap() {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
assert!(out.starts_with(&[0xFF, 0xD8]), "output must stay jpeg");
// 9999x2 downscaled: the buffer length tells the new dims.
assert!(out.len() > 100);
}
PhotoPrep::UseFallback => panic!("over-dimension JPEG should have been resized"),
}
}
#[test]
#[ignore = "heavy: generates a >10 MiB PNG (run explicitly)"]
fn pipeline_transcodes_oversized_png_to_jpeg() {
// 6000x4000 (sum 10000 — under the dimension cap) smooth gradient with
// small per-pixel noise: PNG-incompressible (delta filters defeated)
// but JPEG-friendly (DCT smooths the small noise). Verified with
// ffmpeg: 8000x6000 amp-5 variant is a 59 MB PNG / 3.3 MB JPEG.
let (w, h) = (6000u32, 4000u32);
let mut rng = 0x1234_5678_9abc_def0u64;
let mut data = Vec::with_capacity((w * h * 3) as usize);
for y in 0..h {
for x in 0..w {
let base = (x + y) * 255 / (w + h);
rng = rng
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
let n = ((rng >> 33) % 11) as i32 - 5; // noise in [-5, 5]
let v = (base as i32 + n).clamp(0, 255) as u8;
data.extend_from_slice(&[v, v, v]);
}
}
let mut bytes = Vec::new();
{
let mut encoder = png::Encoder::new(&mut bytes, w, h);
encoder.set_color(png::ColorType::Rgb);
encoder.set_depth(png::BitDepth::Eight);
let mut writer = encoder.write_header().unwrap();
writer.write_image_data(&data).unwrap();
}
assert!(
bytes.len() as u64 > MAX_UPLOAD_BYTES,
"test needs a >10MiB PNG, got {}",
bytes.len()
);
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
match prepare_photo(file).unwrap() {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
assert!(out.starts_with(&[0xFF, 0xD8]), "must transcode to JPEG");
assert!(out.len() as u64 <= MAX_UPLOAD_BYTES);
}
PhotoPrep::UseFallback => panic!("PNG over the byte cap must transcode to JPEG"),
}
}
}
+167 -84
View File
@@ -5,13 +5,14 @@
//! flow. The Python dict-mutation hack (attempts inside the payload) is
//! replaced by dedicated columns.
use crate::db::now_f64;
use parking_lot::Mutex;
use rusqlite::{params, Connection, TransactionBehavior};
use rusqlite::{Connection, TransactionBehavior, params};
use serde_json::Value;
use std::pin::Pin;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::Duration;
use tokio::sync::Notify;
use tokio::task::JoinHandle;
@@ -29,15 +30,9 @@ const QUEUE_WORKERS: usize = 4;
pub enum QueueError {
/// Reschedule with the given delay; after `MAX_RETRIES` attempts the task
/// is dead-lettered instead.
Retryable {
delay_seconds: f64,
payload: Value,
},
Retryable { delay_seconds: f64, payload: Value },
/// Give up now.
Permanent {
message: String,
payload: Value,
},
Permanent { message: String, payload: Value },
}
type BoxFuture<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
@@ -59,6 +54,7 @@ struct LeasedRow {
}
/// Owned worker state so the spawned loop does not borrow the queue handle.
#[derive(Clone)]
struct QueueWorker {
db_path: String,
notify: Arc<Notify>,
@@ -67,27 +63,31 @@ struct QueueWorker {
dead_letter: Arc<DeadLetter>,
}
fn now_f64() -> f64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
/// Resets rows left `in_progress` with an expired lock TTL back to `pending`
/// so they can be leased again (crash/panic recovery).
fn recover_update(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
conn.execute(
"UPDATE tasks SET status='pending', locked_until=0 WHERE status='in_progress' AND locked_until < ?1",
params![now_f64()],
)?;
Ok(())
}
/// Opens the queue DB with a busy timeout. Handler tasks enqueue while
/// workers lease/update rows concurrently; without the timeout a concurrent
/// write fails immediately with SQLITE_BUSY and the operation is lost.
fn open_db(path: &str) -> rusqlite::Result<Connection> {
let conn = Connection::open(path)?;
conn.busy_timeout(Duration::from_secs(5))?;
Ok(conn)
/// Base delay × 2^attempts (attempts = retries already done), capped at 300s.
/// Applied at the queue layer so the attempt count actually reaches the
/// backoff computation; Telegram `RetryAfter` delays get the same treatment
/// (conservatively larger wait, no API change needed).
fn scaled_retry_delay(base: f64, attempts: i32) -> f64 {
(base * 2f64.powi(attempts)).min(300.0)
}
fn ensure_schema(conn: &Connection) -> rusqlite::Result<()> {
fn ensure_schema(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
"PRAGMA journal_mode=WAL; \
CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
locked_until REAL NOT NULL, created_at REAL NOT NULL);",
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
CREATE INDEX IF NOT EXISTS idx_tasks_pending ON tasks(status, run_after);",
)
}
@@ -129,7 +129,7 @@ impl PersistentTaskQueue {
let dead_letter: Arc<DeadLetter> =
Arc::new(move |payload, message| Box::pin(dead_letter(payload, message)));
self.recover_stale().await;
let mut handles = Vec::with_capacity(QUEUE_WORKERS);
let mut handles = Vec::with_capacity(QUEUE_WORKERS + 1);
for _ in 0..QUEUE_WORKERS {
let worker = QueueWorker {
db_path: self.db_path.clone(),
@@ -138,8 +138,35 @@ impl PersistentTaskQueue {
handler: Arc::clone(&handler),
dead_letter: Arc::clone(&dead_letter),
};
handles.push(tokio::spawn(worker.run_loop()));
handles.push(tokio::spawn(worker.run_loop_supervised()));
}
// Periodic lease-expiry sweep: recovers rows a crashed/panicked
// worker left `in_progress` (the lock TTL bounds the wait). Woken by
// the same notify as the workers, so enqueue and stop interrupt the
// sleep; the first interval tick fires immediately (harmless extra
// recovery at startup).
let sweep_db_path = self.db_path.clone();
let sweep_notify = Arc::clone(&self.notify);
let sweep_stop = Arc::clone(&self.stop);
handles.push(tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(30));
loop {
let notified = sweep_notify.notified();
tokio::pin!(notified);
tokio::select! {
_ = &mut notified => {}
_ = interval.tick() => {}
}
if sweep_stop.load(Ordering::Relaxed) {
break;
}
let result =
crate::db::with_conn(&sweep_db_path, move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue sweep failed: {e}");
}
}
}));
*self.worker.lock() = handles;
}
@@ -162,10 +189,8 @@ impl PersistentTaskQueue {
self.counter.fetch_add(1, Ordering::Relaxed)
);
let payload = payload.to_string();
let db_path = self.db_path.clone();
log::info!("enqueued {id} (run_after {run_after:.1})");
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"INSERT OR REPLACE INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES (?1, ?2, ?3, 0, 'pending', 0, ?4)",
@@ -173,36 +198,46 @@ impl PersistentTaskQueue {
)?;
Ok(())
})
.await
.expect("queue insert worker panicked")?;
// Wake every sleeping worker: with several workers the one that finds
// nothing due must not starve the newly inserted row.
self.notify.notify_waiters();
.await?;
// `notify_one` stores a permit when no worker is registered, so a
// notification fired between a worker's DB reads and its `notified()`
// registration is not lost (notify_waiters would drop it). The
// awakened worker re-leases and finds the new row.
self.notify.notify_one();
Ok(())
}
async fn recover_stale(&self) {
let db_path = self.db_path.clone();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
conn.execute(
"UPDATE tasks SET status='pending', locked_until=0 WHERE status='in_progress' AND locked_until < ?1",
params![now_f64()],
)?;
Ok(())
})
.await
.expect("queue recovery worker panicked")
.unwrap_or_else(|e| log::error!("queue recovery failed: {e}"));
self.recover_sweep().await;
}
async fn recover_sweep(&self) {
let result = crate::db::with_conn(&self.db_path, move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue recovery failed: {e}");
}
}
}
impl QueueWorker {
/// Supervised worker: the inner loop runs in its own task so a panic
/// (e.g. inside a handler or a DB closure) kills only that task; the
/// supervisor respawns it until stop is set. The row a dead worker had
/// leased is recovered by the periodic sweep once its lock TTL expires.
async fn run_loop_supervised(self) {
while !self.stop.load(Ordering::Relaxed) {
let worker = self.clone();
if let Err(e) = tokio::spawn(async move { worker.run_loop().await }).await {
log::error!("queue worker panicked, restarting: {e}");
}
}
}
async fn run_loop(self) {
while !self.stop.load(Ordering::Relaxed) {
match self.lease_next().await {
Some(row) => self.process(row).await,
None => {
Ok(Some(row)) => self.process(row).await,
Ok(None) => {
let wait_until = self.earliest_run_after().await;
let notified = self.notify.notified();
tokio::pin!(notified);
@@ -219,15 +254,20 @@ impl QueueWorker {
}
}
}
// A lease failure while rows are due would otherwise loop
// with sleep(0) and hammer SQLite; back off briefly.
Err(e) => {
log::error!("queue lease failed: {e}");
tokio::time::sleep(Duration::from_secs(1)).await;
}
}
}
}
/// Leases the oldest due row (sets it `in_progress` with a lock TTL).
async fn lease_next(&self) -> Option<LeasedRow> {
let db_path = self.db_path.clone();
tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<LeasedRow>> {
let mut conn = open_db(&db_path)?;
/// Errors are surfaced so the caller can back off instead of spinning.
async fn lease_next(&self) -> Result<Option<LeasedRow>, rusqlite::Error> {
crate::db::with_conn(&self.db_path, |conn| {
// BEGIN IMMEDIATE: with several workers, a deferred transaction
// that read before another worker's lease commit would fail with
// SQLITE_BUSY_SNAPSHOT. Taking the write lock up front serializes
@@ -266,30 +306,26 @@ impl QueueWorker {
}))
})
.await
.expect("queue lease worker panicked")
.unwrap_or_else(|e| {
log::error!("queue lease failed: {e}");
None
})
}
async fn earliest_run_after(&self) -> Option<f64> {
let db_path = self.db_path.clone();
tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<f64>> {
let conn = open_db(&db_path)?;
let mut stmt = conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
let result = crate::db::with_conn(&self.db_path, |conn| {
let mut stmt =
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
let mut rows = stmt.query([])?;
match rows.next()? {
Some(row) => Ok(row.get::<_, Option<f64>>(0)?),
None => Ok(None),
}
})
.await
.expect("queue timing worker panicked")
.unwrap_or_else(|e| {
log::error!("queue timing query failed: {e}");
None
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::error!("queue timing query failed: {e}");
None
}
}
}
async fn process(&self, row: LeasedRow) {
@@ -318,12 +354,13 @@ impl QueueWorker {
self.delete_row(&row.id).await;
(self.dead_letter)(payload, message).await;
} else {
let delay = scaled_retry_delay(delay_seconds, row.attempts);
log::info!(
"task {} rescheduled in {delay_seconds:.1}s (attempt {})",
"task {} rescheduled in {delay:.1}s (attempt {})",
row.id,
row.attempts + 1
);
self.reschedule(&row.id, payload, delay_seconds, row.attempts + 1)
self.reschedule(&row.id, payload, delay, row.attempts + 1)
.await;
}
}
@@ -336,34 +373,33 @@ impl QueueWorker {
}
async fn delete_row(&self, id: &str) {
let db_path = self.db_path.clone();
let id = id.to_string();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
Ok(())
})
.await
.expect("queue delete worker panicked")
.unwrap_or_else(|e| log::error!("queue delete failed: {e}"));
.await;
if let Err(e) = result {
log::error!("queue delete failed: {e}");
}
}
async fn reschedule(&self, id: &str, payload: Value, delay_seconds: f64, attempts: i32) {
let db_path = self.db_path.clone();
let id = id.to_string();
let payload = payload.to_string();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"UPDATE tasks SET payload=?1, run_after=?2, attempts=?3, status='pending', locked_until=0 WHERE id=?4",
params![payload, now_f64() + delay_seconds, attempts, id],
)?;
Ok(())
})
.await
.expect("queue reschedule worker panicked")
.unwrap_or_else(|e| log::error!("queue reschedule failed: {e}"));
self.notify.notify_waiters();
.await;
if let Err(e) = result {
log::error!("queue reschedule failed: {e}");
}
// Same permit semantics as enqueue: never lose the wakeup.
self.notify.notify_one();
}
}
@@ -372,6 +408,16 @@ mod tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering};
#[test]
fn scaled_retry_delay_scales_and_caps() {
assert_eq!(scaled_retry_delay(1.0, 0), 1.0);
assert_eq!(scaled_retry_delay(1.0, 1), 2.0);
assert_eq!(scaled_retry_delay(1.0, 2), 4.0);
assert_eq!(scaled_retry_delay(1.5, 1), 3.0);
assert_eq!(scaled_retry_delay(1.0, 10), 300.0, "capped at 300s");
assert_eq!(scaled_retry_delay(300.0, 0), 300.0);
}
async fn new_queue() -> (PersistentTaskQueue, tempfile::TempDir) {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("queue.db");
@@ -510,4 +556,41 @@ mod tests {
assert_eq!(calls.load(AtomicOrdering::SeqCst), 1);
queue.stop().await;
}
#[tokio::test]
async fn runtime_sweep_recovers_expired_lease() {
let (queue, _dir) = new_queue().await;
let calls = Arc::new(AtomicUsize::new(0));
let c = calls.clone();
queue
.start(
move |payload| {
assert_eq!(payload["s"], 1);
c.fetch_add(1, AtomicOrdering::SeqCst);
async { Ok(()) }
},
|_payload, _message| async {},
)
.await;
// Insert a stale leased row AFTER startup: without a runtime sweep it
// would stay `in_progress` forever (only start() used to recover).
{
let conn = Connection::open(&queue.db_path).unwrap();
ensure_schema(&conn).unwrap();
conn.execute(
"INSERT INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES ('task_stale_runtime', '{\"s\":1}', 0, 0, 'in_progress', ?1, 0)",
params![now_f64() - 1000.0],
)
.unwrap();
}
queue.recover_sweep().await;
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(
calls.load(AtomicOrdering::SeqCst),
1,
"expired lease must be recovered and processed exactly once"
);
queue.stop().await;
}
}
+342 -141
View File
@@ -5,21 +5,27 @@
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE};
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost};
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
use crate::queue::QueueError;
use crate::state::{EditMessage, unix_now};
use rand::Rng;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use tempfile::NamedTempFile;
use std::sync::LazyLock;
use teloxide::prelude::*;
use teloxide::types::{
ChatId, InlineKeyboardButton, InlineKeyboardMarkup, InputFile, InputMedia,
InputMediaAnimation, InputMediaPhoto, InputMediaVideo, Message, MessageId, ParseMode,
ReplyParameters,
ChatId, InlineKeyboardButton, InlineKeyboardMarkup, InputFile, InputMedia, InputMediaAnimation,
InputMediaPhoto, InputMediaVideo, Message, MessageId, ParseMode, ReplyParameters,
};
use teloxide::{ApiError, RequestError};
use tempfile::NamedTempFile;
use x_media::site::FetchError;
/// One process-wide Bot for queue workers. Building a fresh Bot (and its HTTP
/// client) per queue task was pure waste; forced at startup in main so a
/// missing token fails fast instead of on the first task.
pub static BOT: LazyLock<Bot> = LazyLock::new(Bot::from_env);
#[derive(Serialize, Deserialize, Clone, Debug)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum MediaItemPayload {
@@ -61,6 +67,15 @@ impl MediaItemPayload {
MediaItemPayload::Animation { .. } => None,
}
}
/// The cover-frame URL for videos (used by the upload fallback, which
/// otherwise drops the thumbnail the URL-send path applies).
fn thumbnail_url(&self) -> Option<&str> {
match self {
MediaItemPayload::Video { thumbnail, .. } => thumbnail.as_deref(),
MediaItemPayload::Photo { .. } | MediaItemPayload::Animation { .. } => None,
}
}
}
#[derive(Serialize, Deserialize, Clone, Debug)]
@@ -110,16 +125,18 @@ pub enum Task {
impl Task {
fn cache_data(&self) -> Option<&CachedPost> {
match self {
Task::SendMediaSequence { cache_data, .. }
| Task::SendAnimation { cache_data, .. } => cache_data.as_ref(),
Task::SendMediaSequence { cache_data, .. } | Task::SendAnimation { cache_data, .. } => {
cache_data.as_ref()
}
Task::ForwardMessages { .. } => None,
}
}
fn source_url(&self) -> Option<&str> {
match self {
Task::SendMediaSequence { source_url, .. }
| Task::SendAnimation { source_url, .. } => Some(source_url),
Task::SendMediaSequence { source_url, .. } | Task::SendAnimation { source_url, .. } => {
Some(source_url)
}
Task::ForwardMessages { .. } => None,
}
}
@@ -137,9 +154,10 @@ fn file_id_of_message(message: &Message, item: &MediaItemPayload) -> Option<Stri
match item {
// `photo()` returns all sizes, smallest first — the largest carries
// the file id of the sent media.
MediaItemPayload::Photo { .. } => {
message.photo().and_then(|sizes| sizes.last()).map(|p| p.file.id.to_string())
}
MediaItemPayload::Photo { .. } => message
.photo()
.and_then(|sizes| sizes.last())
.map(|p| p.file.id.to_string()),
MediaItemPayload::Video { .. } => message.video().map(|v| v.file.id.to_string()),
MediaItemPayload::Animation { .. } => message.animation().map(|a| a.file.id.to_string()),
}
@@ -210,13 +228,13 @@ pub async fn invalidate_cache(task: &Task) {
}
pub const MAX_MEDIA_GROUP: usize = 9;
/// Upload cap (bytes): files above this are not uploaded; the bot falls back
/// to a smaller media URL instead.
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024; // 10485760
/// Splits media into batches of at most [`MAX_MEDIA_GROUP`] items.
pub fn chunk_media_items<T: Clone>(items: Vec<T>) -> Vec<Vec<T>> {
items.chunks(MAX_MEDIA_GROUP).map(|chunk| chunk.to_vec()).collect()
items
.chunks(MAX_MEDIA_GROUP)
.map(|chunk| chunk.to_vec())
.collect()
}
/// Exponential backoff with jitter, capped at 30s.
@@ -229,12 +247,15 @@ pub fn retry_delay_seconds(attempts: u32) -> f64 {
/// these errors are handled by the download-and-reupload fallback, NOT by a
/// queue retry (resending the URL cannot succeed).
pub fn is_media_fetch_failure(e: &ApiError) -> bool {
const MARKERS: [&str; 5] = [
const MARKERS: [&str; 6] = [
"webpage_media_empty",
"media_empty",
"empty_web_media",
"webpage_curl_failed",
"timeout",
// Oversized photos (width + height > 10000 px) are rejected on URL
// sends too; route them to the download-and-resize fallback.
"PHOTO_INVALID_DIMENSIONS",
];
let description = e.to_string().to_lowercase();
MARKERS.iter().any(|marker| description.contains(marker))
@@ -249,31 +270,41 @@ pub fn is_size_error(e: &ApiError) -> bool {
return true;
}
let description = e.to_string().to_lowercase();
["too large", "too big"].iter().any(|marker| description.contains(marker))
["too large", "too big"]
.iter()
.any(|marker| description.contains(marker))
}
/// Task-free classification of a Telegram request error. The callers attach
/// the (updated) task when building a [`SendError`].
pub enum Classification {
Retryable { delay_seconds: f64 },
Permanent { message: String },
Retryable {
delay_seconds: f64,
},
Permanent {
message: String,
},
/// Handled by the download fallback, not a queue retry.
MediaFetchFailure,
}
pub fn classify_request_error(e: &RequestError) -> Classification {
match e {
RequestError::RetryAfter(seconds) => {
Classification::Retryable { delay_seconds: seconds.seconds() as f64 }
}
RequestError::RetryAfter(seconds) => Classification::Retryable {
delay_seconds: seconds.seconds() as f64,
},
RequestError::Network(_) => Classification::Retryable {
delay_seconds: retry_delay_seconds(0),
},
RequestError::Api(api) if is_media_fetch_failure(api) => Classification::MediaFetchFailure,
RequestError::Api(api) => Classification::Permanent { message: api.to_string() },
RequestError::Api(api) => Classification::Permanent {
message: api.to_string(),
},
RequestError::MigrateToChatId(_)
| RequestError::InvalidJson { .. }
| RequestError::Io(_) => Classification::Permanent { message: e.to_string() },
| RequestError::Io(_) => Classification::Permanent {
message: e.to_string(),
},
}
}
@@ -313,6 +344,11 @@ fn item_url(item: &MediaItemPayload) -> &str {
fn input_file_for(media: &str) -> Result<InputFile, String> {
if media.starts_with("http://") || media.starts_with("https://") {
Ok(InputFile::url(parse_media_url(media)?))
} else if !std::path::Path::new(media).exists() {
// A retried task may reference a temp file the original send's
// TempDir already cleaned up; fail fast and permanent instead of
// burning retries on a file that can never come back.
Err(format!("local media file missing: {media}"))
} else {
Ok(InputFile::file(media))
}
@@ -389,9 +425,9 @@ fn build_media_group(
.map(|(i, item)| {
let item_caption = if i == 0 { caption } else { None };
Ok(match item {
MediaItemPayload::Photo {
has_spoiler, ..
} => photo_media(item.input_file()?, item_caption, *has_spoiler),
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(item.input_file()?, item_caption, *has_spoiler)
}
MediaItemPayload::Video {
has_spoiler,
thumbnail,
@@ -403,9 +439,9 @@ fn build_media_group(
}
video
}
MediaItemPayload::Animation {
has_spoiler, ..
} => animation_media(item.input_file()?, item_caption, *has_spoiler),
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(item.input_file()?, item_caption, *has_spoiler)
}
})
})
.collect()
@@ -430,13 +466,24 @@ fn sniff_ext(bytes: &[u8]) -> &'static str {
}
enum FallbackError {
Retryable { delay_seconds: f64 },
Permanent { message: String },
Retryable {
delay_seconds: f64,
},
Permanent {
message: String,
},
/// The downloaded file exceeds the upload cap; the caller falls back to
/// the item's smaller URL.
MediaTooLarge,
}
/// Brings a downloaded photo within Telegram's limits via the pure-Rust
/// chain in [`crate::photo`] (no ffmpeg): dimension cap / upload cap
/// exceeded photos are decoded, downscaled with Lanczos3, PNG bit depth
/// reduced (>24-bit → 24-bit RGB, ≤24-bit untouched) and transcoded to JPEG
/// only if still too big. Anything that cannot be fixed falls back to the
/// item's smaller URL.
///
/// Downloads one media item to a temp file (deleted on drop). Network errors
/// are retryable; size over the upload cap and other download errors are not.
async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, FallbackError> {
@@ -445,22 +492,30 @@ async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, Fall
| MediaItemPayload::Video { media, .. }
| MediaItemPayload::Animation { media, .. } => media,
};
let bytes = match x_media::site::download_media(media_url).await {
// Photos are downloaded even over the upload cap so `prepare_photo` can
// downscale / transcode them (cap = decode budget); videos/animations
// abort as soon as the upload cap is crossed mid-stream.
let limit = if matches!(item, MediaItemPayload::Photo { .. }) {
photo::MAX_DECODE_BYTES
} else {
MAX_UPLOAD_BYTES + 1
};
let bytes = match x_media::site::download_media_limited(media_url, limit).await {
Ok(bytes) => bytes,
Err(FetchError::Http(_)) => {
return Err(FallbackError::Retryable {
delay_seconds: retry_delay_seconds(0),
});
}
Err(FetchError::TooLarge) => {
return Err(FallbackError::MediaTooLarge);
}
Err(e) => {
return Err(FallbackError::Permanent {
message: format!("download failed: {e}"),
});
}
};
if bytes.len() as u64 > MAX_UPLOAD_BYTES {
return Err(FallbackError::MediaTooLarge);
}
let ext = sniff_ext(&bytes);
let mut file = tempfile::Builder::new()
.suffix(&format!(".{ext}"))
@@ -482,8 +537,9 @@ fn media_from_file(
item: &MediaItemPayload,
path: std::path::PathBuf,
caption: Option<&str>,
) -> InputMedia {
match item {
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(InputFile::file(path), caption, *has_spoiler)
}
@@ -493,7 +549,11 @@ fn media_from_file(
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(InputFile::file(path), caption, *has_spoiler)
}
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
}
/// Builds the media group item from a (smaller) URL.
@@ -501,8 +561,9 @@ fn media_from_url(
item: &MediaItemPayload,
url: &str,
caption: Option<&str>,
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
Ok(match item {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(input_file_for(url)?, caption, *has_spoiler)
}
@@ -512,7 +573,11 @@ fn media_from_url(
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(input_file_for(url)?, caption, *has_spoiler)
}
})
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
}
/// Download-and-reupload fallback for one media batch. Files over the upload
@@ -531,14 +596,16 @@ async fn send_batch_via_upload(
for (i, item) in batch.iter().enumerate() {
let item_caption = if i == 0 { caption } else { None };
// Size check before downloading/uploading: over the cap, use the
// smaller URL instead of the file.
// smaller URL instead of the file. Photos are exempt — they are
// downloaded and processed (downscale / PNG→JPEG) before uploading.
let too_large = match x_media::site::media_size(item_url(item)).await {
Ok(Some(size)) => size > MAX_UPLOAD_BYTES,
_ => false,
};
let too_large = too_large && !matches!(item, MediaItemPayload::Photo { .. });
let media = if too_large {
match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption) {
Some(url) => match media_from_url(item, url, item_caption, item.thumbnail_url()) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
@@ -553,17 +620,63 @@ async fn send_batch_via_upload(
} else {
match download_to_temp(item).await {
Ok(file) => {
let path = file.path().to_path_buf();
files.push(file);
media_from_file(item, path, item_caption)
// Telegram rejects photos wider+taller than 10000 px
// combined (PHOTO_INVALID_DIMENSIONS): downscale the
// downloaded file before uploading; photos that cannot be
// brought within the limits degrade to the smaller URL.
if matches!(item, MediaItemPayload::Photo { .. }) {
// CPU-heavy (decode/resize/encode): run off the async
// executor thread.
let prep = tokio::task::spawn_blocking(move || photo::prepare_photo(file))
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
})?
.map_err(|message| FallbackError::Permanent { message })?;
match prep {
PhotoPrep::Upload(upload) => {
let path = upload.path().to_path_buf();
files.push(upload);
media_from_file(item, path, item_caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?
}
PhotoPrep::UseFallback => match item.fallback_url() {
Some(url) => match media_from_url(
item,
url,
item_caption,
item.thumbnail_url(),
) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
},
None => {
return Err(FallbackError::Permanent {
message:
"photo dimensions exceed Telegram limits and no smaller variant is available"
.into(),
});
}
},
}
} else {
let path = file.path().to_path_buf();
files.push(file);
media_from_file(item, path, item_caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?
}
}
Err(FallbackError::MediaTooLarge) => match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
Some(url) => {
match media_from_url(item, url, item_caption, item.thumbnail_url()) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
}
},
}
None => {
return Err(FallbackError::Permanent {
message: "media too large".into(),
@@ -577,14 +690,16 @@ async fn send_batch_via_upload(
}
let result = bot
.send_media_group(ChatId(chat_id), items)
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply())
.reply_parameters(
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
)
.await;
match result {
Ok(messages) => Ok(messages),
Err(e) => Err(match classify_request_error(&e) {
Classification::Retryable { delay_seconds } => FallbackError::Retryable {
delay_seconds,
},
Classification::Retryable { delay_seconds } => {
FallbackError::Retryable { delay_seconds }
}
Classification::Permanent { message } => FallbackError::Permanent { message },
Classification::MediaFetchFailure => FallbackError::Permanent {
message: "upload failed".into(),
@@ -651,7 +766,11 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
let fresh_send = *batch_index == 0 && sent.is_empty();
for idx in *batch_index..media_batches.len() {
let batch = &media_batches[idx];
let caption = if idx == 0 { Some(caption.as_str()) } else { None };
let caption = if idx == 0 {
Some(caption.as_str())
} else {
None
};
let items = match build_media_group(batch, caption) {
Ok(items) => items,
Err(message) => {
@@ -663,7 +782,9 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
};
match bot
.send_media_group(ChatId(chat_id), items)
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply())
.reply_parameters(
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
)
.await
{
Ok(messages) => {
@@ -675,9 +796,7 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
collect_file_ids(&messages, batch, &mut cached_media);
sent.extend(messages.into_iter().map(|m| m.id.0 as i64));
}
Err(RequestError::Api(api))
if is_media_fetch_failure(&api) || is_size_error(&api) =>
{
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
log::info!(
"Telegram could not fetch media for batch {idx} ({}), downloading and reuploading",
batch.first().map(item_url).unwrap_or("?")
@@ -728,7 +847,9 @@ async fn send_animation_inner(
.send_animation(ChatId(chat_id), file)
.caption(caption)
.parse_mode(ParseMode::Html)
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply());
.reply_parameters(
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
);
if spoiler {
request = request.has_spoiler(true);
}
@@ -751,9 +872,7 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
let reply_to = *reply_to_message_id;
let (media_url, has_spoiler) = match animation {
MediaItemPayload::Animation {
media,
has_spoiler,
..
media, has_spoiler, ..
} => (media, *has_spoiler),
MediaItemPayload::Photo { .. } | MediaItemPayload::Video { .. } => {
unreachable!("SendAnimation carries an Animation payload")
@@ -761,19 +880,20 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
};
let url_file = match input_file_for(media_url) {
Ok(file) => file,
Err(message) => return Err(SendError::Permanent { message, task: task.clone() }),
Err(message) => {
return Err(SendError::Permanent {
message,
task: task.clone(),
});
}
};
match send_animation_inner(bot, chat_id, reply_to, caption, has_spoiler, url_file)
.await
{
match send_animation_inner(bot, chat_id, reply_to, caption, has_spoiler, url_file).await {
Ok(message) => {
let id = message.id.0 as i64;
cache_animation_send(task, &message).await;
Ok(vec![id])
}
Err(RequestError::Api(api))
if is_media_fetch_failure(&api) || is_size_error(&api) =>
{
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
log::info!(
"Telegram could not fetch animation URL, downloading and reuploading: {}",
media_url
@@ -821,21 +941,24 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
Err(e) => Err(classify_to_send_error(&e, task.clone())),
}
}
Err(message) => {
Err(SendError::Permanent { message, task: task.clone() })
}
Err(message) => Err(SendError::Permanent {
message,
task: task.clone(),
}),
},
None => Err(SendError::Permanent {
message: "media too large".into(),
task: task.clone(),
}),
},
Err(FallbackError::Retryable { delay_seconds }) => {
Err(SendError::Retryable { delay_seconds, task: task.clone() })
}
Err(FallbackError::Permanent { message }) => {
Err(SendError::Permanent { message, task: task.clone() })
}
Err(FallbackError::Retryable { delay_seconds }) => Err(SendError::Retryable {
delay_seconds,
task: task.clone(),
}),
Err(FallbackError::Permanent { message }) => Err(SendError::Permanent {
message,
task: task.clone(),
}),
}
}
Err(e) => Err(classify_to_send_error(&e, task.clone())),
@@ -859,7 +982,11 @@ pub async fn forward_messages(bot: &Bot, task: &Task) -> Result<(), SendError> {
.map(|id| MessageId(*id as i32))
.collect::<Vec<_>>();
match bot
.copy_messages(ChatId(*to_chat_id), ChatId(*from_chat_id), message_ids.clone())
.copy_messages(
ChatId(*to_chat_id),
ChatId(*from_chat_id),
message_ids.clone(),
)
.await
{
Ok(_) => {
@@ -893,12 +1020,18 @@ pub fn build_edit_markup(templates: &HashMap<String, String>) -> InlineKeyboardM
/// Notifies a chat about a dead-lettered task (skips when `notify_chat_id` is
/// absent).
pub async fn notify_failure(bot: &Bot, chat_id: Option<i64>, message_id: Option<i64>, message: &str) {
pub async fn notify_failure(
bot: &Bot,
chat_id: Option<i64>,
message_id: Option<i64>,
message: &str,
) {
let Some(chat_id) = chat_id else { return };
let mut request = bot.send_message(ChatId(chat_id), message);
if let Some(message_id) = message_id {
request = request
.reply_parameters(ReplyParameters::new(MessageId(message_id as i32)).allow_sending_without_reply());
request = request.reply_parameters(
ReplyParameters::new(MessageId(message_id as i32)).allow_sending_without_reply(),
);
}
if let Err(e) = request.await {
log::error!("failed to notify about failed task: {e}");
@@ -908,42 +1041,48 @@ pub async fn notify_failure(bot: &Bot, chat_id: Option<i64>, message_id: Option<
/// After a successful send: either open the edit-before-forward prompt or
/// forward to the configured channel (with retry/queue handling).
pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
let (chat_id, reply_to, source_url, edit_before_forward, forward_channel_id, notify_chat_id, notify_message_id) =
match task {
Task::SendMediaSequence {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
}
| Task::SendAnimation {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
} => (
*chat_id,
*reply_to_message_id,
source_url.clone(),
*edit_before_forward,
*forward_channel_id,
*notify_chat_id,
*notify_message_id,
),
Task::ForwardMessages { .. } => return,
};
let (
chat_id,
reply_to,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
) = match task {
Task::SendMediaSequence {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
}
| Task::SendAnimation {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
} => (
*chat_id,
*reply_to_message_id,
source_url.clone(),
*edit_before_forward,
*forward_channel_id,
*notify_chat_id,
*notify_message_id,
),
Task::ForwardMessages { .. } => return,
};
if edit_before_forward {
let mut chat_data = CHAT_STORE.get(chat_id).await;
let keyboard = build_edit_markup(&chat_data.template);
let keyboard = build_edit_markup(&CHAT_STORE.get(chat_id).await.template);
match bot
.send_message(ChatId(chat_id), "Reply to edit message.")
.reply_markup(keyboard)
@@ -958,17 +1097,22 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
prompt.id.0,
message_ids.len()
);
chat_data.edit_message.insert(
prompt.id.0 as i64,
EditMessage {
url: source_url,
chat_id,
forward_message_ids: message_ids,
template: String::new(),
created_at: unix_now(),
},
);
CHAT_STORE.set(chat_id, &chat_data).await;
let prompt_id = prompt.id.0 as i64;
let source_url = source_url.clone();
CHAT_STORE
.update(chat_id, move |data| {
data.edit_message.insert(
prompt_id,
EditMessage {
url: source_url,
chat_id,
forward_message_ids: message_ids,
template: String::new(),
created_at: unix_now(),
},
);
})
.await;
}
Err(e) => log::error!("failed to send edit prompt: {e}"),
}
@@ -976,7 +1120,10 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
}
if let Some(channel_id) = forward_channel_id {
log::info!("forwarding {} message(s) to channel {channel_id}", message_ids.len());
log::info!(
"forwarding {} message(s) to channel {channel_id}",
message_ids.len()
);
let forward_task = Task::ForwardMessages {
from_chat_id: chat_id,
to_chat_id: channel_id,
@@ -986,7 +1133,10 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
};
match forward_messages(bot, &forward_task).await {
Ok(()) => {}
Err(SendError::Retryable { delay_seconds, task }) => {
Err(SendError::Retryable {
delay_seconds,
task,
}) => {
let payload = serde_json::to_value(task).expect("task serializes");
let run_after = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
@@ -1021,12 +1171,27 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
});
}
};
let bot = Bot::from_env();
let bot = BOT.clone();
// A resumed multi-batch send already ran post_send_actions (edit prompt /
// forward) when it first started; running them again on the resume would
// open a duplicate edit prompt and double-forward. SendAnimation is
// atomic (always a fresh run), so only SendMediaSequence can resume.
let resumed = matches!(
&task,
Task::SendMediaSequence {
batch_index,
sent_message_ids,
..
} if *batch_index > 0 || !sent_message_ids.is_empty()
);
match task {
Task::SendMediaSequence { .. } | Task::SendAnimation { .. } => {
let message_ids = match send_media_or_animation(&bot, &task).await {
Ok(ids) => ids,
Err(SendError::Retryable { delay_seconds, task }) => {
Err(SendError::Retryable {
delay_seconds,
task,
}) => {
return Err(QueueError::Retryable {
delay_seconds,
payload: serde_json::to_value(task).expect("task serializes"),
@@ -1040,12 +1205,17 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
});
}
};
post_send_actions(&bot, &task, message_ids).await;
if !resumed {
post_send_actions(&bot, &task, message_ids).await;
}
Ok(())
}
Task::ForwardMessages { .. } => match forward_messages(&bot, &task).await {
Ok(()) => Ok(()),
Err(SendError::Retryable { delay_seconds, task }) => Err(QueueError::Retryable {
Err(SendError::Retryable {
delay_seconds,
task,
}) => Err(QueueError::Retryable {
delay_seconds,
payload: serde_json::to_value(task).expect("task serializes"),
}),
@@ -1070,7 +1240,7 @@ pub async fn dead_letter_notify(payload: serde_json::Value, message: String) {
let notify_chat_id = payload.get("notify_chat_id").and_then(|v| v.as_i64());
let notify_message_id = payload.get("notify_message_id").and_then(|v| v.as_i64());
if notify_chat_id.is_some() {
let bot = Bot::from_env();
let bot = BOT.clone();
notify_failure(
&bot,
notify_chat_id,
@@ -1085,6 +1255,14 @@ pub async fn dead_letter_notify(payload: serde_json::Value, message: String) {
mod tests {
use super::*;
#[test]
fn oversized_photo_boundary() {
// The empirical Telegram limit: sum 10000 passes, 10001 fails.
assert!(crate::photo::PHOTO_MAX_DIMENSION_SUM == 10000);
assert!(6100 + 3900 <= crate::photo::PHOTO_MAX_DIMENSION_SUM);
assert!(6300 + 3730 > crate::photo::PHOTO_MAX_DIMENSION_SUM);
}
#[test]
fn chunk_media_items_sizes() {
assert_eq!(chunk_media_items::<i32>(vec![]), Vec::<Vec<i32>>::new());
@@ -1092,7 +1270,11 @@ mod tests {
assert_eq!(chunk_media_items((0..10).collect()).len(), 2);
assert_eq!(chunk_media_items((0..25).collect()).len(), 3);
assert_eq!(chunk_media_items((0..25).collect())[2].len(), 7);
assert!(chunk_media_items((0..25).collect()).iter().all(|c| c.len() <= 9));
assert!(
chunk_media_items((0..25).collect())
.iter()
.all(|c| c.len() <= 9)
);
}
#[test]
@@ -1116,7 +1298,10 @@ mod tests {
let api = ApiError::Unknown(description.to_string());
assert!(is_media_fetch_failure(&api), "{description}");
}
for description in ["Bad Request: message is not modified", "Forbidden: bot was blocked by the user"] {
for description in [
"Bad Request: message is not modified",
"Forbidden: bot was blocked by the user",
] {
let api = ApiError::Unknown(description.to_string());
assert!(!is_media_fetch_failure(&api), "{description}");
}
@@ -1137,7 +1322,10 @@ mod tests {
assert!(is_size_error(&api), "{description}");
}
// Unrelated errors must not match.
for description in ["Bad Request: WEBPAGE_MEDIA_EMPTY", "Bad Request: message is not modified"] {
for description in [
"Bad Request: WEBPAGE_MEDIA_EMPTY",
"Bad Request: message is not modified",
] {
let api = ApiError::Unknown(description.to_string());
assert!(!is_size_error(&api), "{description}");
}
@@ -1146,9 +1334,16 @@ mod tests {
#[test]
fn media_item_payload_fallback_url_serde_default() {
// Old queued payloads without the field deserialize with None.
let json = serde_json::json!({"kind": "photo", "media": "https://a/b.jpg", "has_spoiler": false});
let json =
serde_json::json!({"kind": "photo", "media": "https://a/b.jpg", "has_spoiler": false});
let photo: MediaItemPayload = serde_json::from_value(json).unwrap();
assert!(matches!(photo, MediaItemPayload::Photo { fallback_url: None, .. }));
assert!(matches!(
photo,
MediaItemPayload::Photo {
fallback_url: None,
..
}
));
assert_eq!(photo.fallback_url(), None);
}
@@ -1227,7 +1422,13 @@ mod tests {
assert_eq!(sent_message_ids, vec![11, 12]);
assert_eq!(forward_channel_id, Some(333));
assert_eq!(media_batches.len(), 2);
assert!(matches!(media_batches[0][0], MediaItemPayload::Photo { has_spoiler: true, .. }));
assert!(matches!(
media_batches[0][0],
MediaItemPayload::Photo {
has_spoiler: true,
..
}
));
}
other => panic!("expected SendMediaSequence, got {other:?}"),
}
+110 -25
View File
@@ -2,10 +2,11 @@
//! `data/task_queue.db`, shared with the task queue).
use parking_lot::Mutex;
use rusqlite::{params, Connection};
use rusqlite::params;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::path::Path;
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
#[derive(Serialize, Deserialize, Default, Clone, Debug)]
@@ -34,6 +35,9 @@ pub struct EditMessage {
pub struct ChatStore {
/// In-memory cache; the DB is the source of truth on first access.
cache: Mutex<HashMap<i64, ChatData>>,
/// Per-chat async locks serializing get→mutate→set so concurrent handler
/// tasks (batch-forwards, callbacks) cannot clobber each other's writes.
locks: Mutex<HashMap<i64, Arc<tokio::sync::Mutex<()>>>>,
db_path: String,
}
@@ -45,7 +49,9 @@ pub fn unix_now() -> i64 {
}
impl ChatStore {
/// Creates the parent directory and both tables (idempotent).
/// Creates the parent directory and the `chat_state` table (idempotent).
/// The shared `tasks` / `link_cache` tables are owned by `queue.rs` and
/// `link_cache.rs` respectively.
pub fn open(path: &str) -> rusqlite::Result<Self> {
if let Some(parent) = Path::new(path).parent()
&& !parent.as_os_str().is_empty()
@@ -53,16 +59,14 @@ impl ChatStore {
std::fs::create_dir_all(parent)
.map_err(|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)))?;
}
let conn = Connection::open(path)?;
let conn = crate::db::open_db(path)?;
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL);",
"CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL);",
)?;
drop(conn);
Ok(ChatStore {
cache: Mutex::new(HashMap::new()),
locks: Mutex::new(HashMap::new()),
db_path: path.to_string(),
})
}
@@ -71,22 +75,19 @@ impl ChatStore {
if let Some(data) = self.cache.lock().get(&chat_id) {
return data.clone();
}
let db_path = self.db_path.clone();
let payload = tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<String>> {
let conn = Connection::open(&db_path)?;
let chat_key = chat_id.to_string();
let payload = crate::db::with_conn(&self.db_path, move |conn| {
// Concurrent handler tasks (batch-forwards) may write chat_state
// while this read runs; without a busy timeout a write lock
// collision fails the query immediately.
conn.busy_timeout(std::time::Duration::from_secs(5))?;
// while this read runs; the shared busy timeout handles the
// write-lock collision instead of failing the query.
let mut stmt = conn.prepare("SELECT payload FROM chat_state WHERE chat_id = ?1")?;
let mut rows = stmt.query(params![chat_id.to_string()])?;
let mut rows = stmt.query(params![chat_key])?;
match rows.next()? {
Some(row) => Ok(Some(row.get(0)?)),
Some(row) => Ok(Some(row.get::<_, String>(0)?)),
None => Ok(None),
}
})
.await
.expect("chat_state worker panicked")
.unwrap_or_else(|e| {
log::error!("chat_state read failed: {e}");
None
@@ -101,19 +102,38 @@ impl ChatStore {
pub async fn set(&self, chat_id: i64, data: &ChatData) {
self.cache.lock().insert(chat_id, data.clone());
let payload = serde_json::to_string(data).expect("chat state serializes");
let db_path = self.db_path.clone();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = Connection::open(&db_path)?;
conn.busy_timeout(std::time::Duration::from_secs(5))?;
let chat_id = chat_id.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
params![chat_id.to_string(), payload],
params![chat_id, payload],
)?;
Ok(())
})
.await
.expect("chat_state worker panicked")
.unwrap_or_else(|e| log::error!("chat_state write failed: {e}"));
.await;
if let Err(e) = result {
log::error!("chat_state write failed: {e}");
}
}
/// Serializes a get→mutate→set cycle per chat: concurrent handler tasks
/// (the batch-forward design spawns several per chat) each snapshot the
/// same `ChatData` and last-writer-wins would silently drop mutations,
/// e.g. a second `edit_message` record. The per-chat lock makes the
/// cycle atomic. Returns the closure's result.
pub async fn update<R>(&self, chat_id: i64, f: impl FnOnce(&mut ChatData) -> R) -> R {
let lock = {
let mut locks = self.locks.lock();
locks
.entry(chat_id)
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
};
let _guard = lock.lock().await;
let mut data = self.get(chat_id).await;
let r = f(&mut data);
self.set(chat_id, &data).await;
r
}
/// Removes edit-before-forward records whose `created_at + ttl` is in the
@@ -123,6 +143,10 @@ impl ChatStore {
let now = unix_now();
let ttl_secs = ttl.as_secs() as i64;
let mut removed = Vec::new();
// Chats with no live edit records: evicted from the cache (and their
// per-chat lock) so the cache stays bounded to active prompts. The DB
// keeps the row; the next get() reloads it.
let mut evicted_chats = Vec::new();
let changed: Vec<(i64, ChatData)> = {
let mut cache = self.cache.lock();
let mut out = Vec::new();
@@ -139,18 +163,79 @@ impl ChatStore {
}
}
if kept.len() != data.edit_message.len() {
// Persist the pruned row (removes expired records from
// the DB too, not just the cache).
data.edit_message = kept;
out.push((*chat_id, data.clone()));
}
if data.edit_message.is_empty() {
evicted_chats.push(*chat_id);
}
}
// Lock order: update() takes the per-chat lock before the cache
// lock, so prune must not hold the cache lock while taking locks.
drop(cache);
out
};
for (chat_id, data) in changed {
self.set(chat_id, &data).await;
}
if !evicted_chats.is_empty() {
let mut cache = self.cache.lock();
let mut locks = self.locks.lock();
for chat_id in &evicted_chats {
cache.remove(chat_id);
locks.remove(chat_id);
}
}
if !removed.is_empty() {
log::info!("pruned {} expired edit-before-forward record(s)", removed.len());
log::info!(
"pruned {} expired edit-before-forward record(s)",
removed.len()
);
}
removed
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn concurrent_updates_do_not_lose_edit_records() {
let dir = tempfile::tempdir().unwrap();
let store = std::sync::Arc::new(
ChatStore::open(dir.path().join("s.db").to_str().unwrap()).unwrap(),
);
let mut handles = Vec::new();
for i in 0..4 {
let store = Arc::clone(&store);
handles.push(tokio::spawn(async move {
store
.update(1001, |data| {
data.edit_message.insert(
i,
EditMessage {
url: format!("https://x.com/u/status/{i}"),
chat_id: 1001,
forward_message_ids: vec![i],
template: String::new(),
created_at: 0,
},
);
})
.await;
}));
}
for h in handles {
h.await.unwrap();
}
let data = store.get(1001).await;
assert_eq!(
data.edit_message.len(),
4,
"concurrent get→mutate→set must not drop records"
);
}
}
+23 -19
View File
@@ -5,30 +5,25 @@ services:
ports:
- '80:80'
- '443:443'
environment:
# Bare-IP access only.
# DEFAULT_HOST: 'bot.example.com'
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./nginx-certs:/etc/nginx/certs:ro
- ./nginx-vhost.d:/etc/nginx/vhost.d:ro
- ./nginx-html:/usr/share/nginx/html:ro
- certs:/etc/nginx/certs:ro
- html:/usr/share/nginx/html:ro
networks: [proxy]
labels:
- 'com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true'
- 'com.github.nginx-proxy.nginx'
container_name: nginx-proxy
acme-companion:
image: nginxproxy/acme-companion
restart: always
environment:
DEFAULT_EMAIL: 'admin@yoursfunny.top'
DEFAULT_EMAIL: ''
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./nginx-certs:/etc/nginx/certs:rw
- ./nginx-vhost.d:/etc/nginx/vhost.d:rw
- ./nginx-html:/usr/share/nginx/html:rw
- ./nginx-acme:/etc/acme.sh
- certs:/etc/nginx/certs:rw
- html:/usr/share/nginx/html:rw
- acme:/etc/acme.sh
networks: [proxy]
container_name: acme-companion
depends_on:
@@ -42,21 +37,17 @@ services:
TELOXIDE_TOKEN: ''
BOT_ADMIN: ''
PIXIV_REFRESH_TOKEN: ''
# Optional: x.com session cookie (auth_token) — fetches NSFW tweets
# that the public syndication endpoint withholds.
TWITTER_AUTH_TOKEN: ''
EDIT_MESSAGE_TTL_SECONDS: '86400'
# Link-result cache TTL (default 604800 = 7 days).
LINK_CACHE_TTL_SECONDS: '604800'
RUST_LOG: 'info'
VIRTUAL_HOST: 'bot.example.com'
VIRTUAL_HOST: '<YOUR_DOMAIN>'
VIRTUAL_PORT: '8443'
# LETSENCRYPT_HOST: 'bot.example.com'
# ACME_HOST: 'your.domain.com'
WEBHOOK: 'true'
WEBHOOK_LISTEN: '0.0.0.0'
WEBHOOK_PORT: '8443'
WEBHOOK_URL: 'https://bot.example.com/'
# WEBHOOK_CERT: './cert/cert.pem'
WEBHOOK_URL: 'https://<YOUR_DOMAIN>/'
WEBHOOK_SECRET_TOKEN: ''
volumes:
- ./data:/app/data
@@ -64,6 +55,19 @@ services:
depends_on:
- nginx-proxy
container_name: tgxmb
# Webhook mode only: the bot listens on WEBHOOK_PORT; nginx-proxy shows
# 502s while this is down, so surface it to the orchestrator.
healthcheck:
test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/8443'"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
volumes:
certs:
html:
acme:
networks:
proxy:
+14 -3
View File
@@ -5,9 +5,20 @@ if [ "$(id -u)" -eq '0' ]
then
USER_ID=${LOCAL_USER_ID:-9001}
useradd --shell /bin/bash -u ${USER_ID} -o -c "" -m user > /dev/null 2>&1
usermod -a -G root user > /dev/null 2>&1
chown -R `id -u user`:`id -u user` /app > /dev/null 2>&1
# `docker compose restart` / `docker restart` reuse the same container, so
# the overlay fs keeps the user created on first boot. A second `useradd`
# then fails with exit code 9, which would trip `set -e` and kill the
# container on every restart. Create only if missing; align the UID
# otherwise so LOCAL_USER_ID changes still apply.
if ! id user > /dev/null 2>&1
then
useradd --shell /bin/bash -u ${USER_ID} -o -c "" -m user > /dev/null 2>&1 || true
else
usermod -u ${USER_ID} -o user > /dev/null 2>&1 || true
fi
# Bind-mounted volumes may not support chown; a failure here must not kill
# the container either.
chown -R `id -u user`:`id -u user` /app > /dev/null 2>&1 || true
export HOME=/home/user
# setpriv (util-linux, present in bookworm-slim) replaces gosu: drop to the