Compare commits

...
40 Commits
Author SHA1 Message Date
YoursFunny ea72516d5c chore: bump version to 1.1.0 2026-08-10 10:32:54 +08:00
YoursFunny 755330e585 chore: clippy and rustfmt cleanup on new code 2026-08-08 20:35:43 +08:00
YoursFunny c40b074b3c docker: fail-closed rebuild, ffmpeg checksum, smaller runtime
- Replace the mtime-touch stub-rebuild hack with cargo clean -p (a
  future-dated host file could silently ship the stub binary)
- Optional FFMPEG_SHA256 build arg verified before extraction
- Drop the redundant libssl3/libcrypto copies and the root
  supplementary group; strip the release binary; add a webhook-mode
  healthcheck to the compose example
2026-08-08 20:34:21 +08:00
YoursFunny 9910da2914 deps: drop unused regex, unify reqwest on 0.12
x-media's reqwest 0.13 dragged in quinn/rustls/aws-lc-rs (cmake C
build) alongside teloxide's 0.12; unifying on 0.12 removes the whole
second TLS/QUIC stack from the build and image. The unused regex dep in
xmedia-bot is gone; x-media keeps url (the bsky HLS remux uses it).
2026-08-08 20:31:03 +08:00
YoursFunny ebc0122264 db: enable WAL and index the pending-task lease query
The lease/earliest_run_after queries full-scanned tasks, and the
rollback journal blocked readers behind worker writes. journal_mode=WAL
(persistent, idempotent) plus idx_tasks_pending(status, run_after)
covers both without a schema migration.
2026-08-08 20:26:07 +08:00
YoursFunny 44cba8abe0 send: reuse one process-wide Bot for queue workers
handle_task and dead_letter_notify built a fresh Bot (env parse + HTTP
client) per queue item. A single LazyLock<Bot> is forced at startup so
a missing TELOXIDE_TOKEN fails fast instead of on the first task.
2026-08-08 20:25:22 +08:00
YoursFunny 99009aae9a db: share one now_f64() instead of four private copies
handlers, queue, send and link_cache each carried the same SystemTime
helper; a single crate::db::now_f64() removes the drift risk.
2026-08-08 20:23:58 +08:00
YoursFunny 72130b9023 caption: escape URLs/handles in HTML captions
Post URLs and author URLs were interpolated raw into <a href> attributes
(and the raw user URL from empty_fetched into caption text), so crafted
links could break the HTML parse and fail the send with a 400. All
attribute interpolations now use encode_double_quoted_attribute; text
stays encode_text.
2026-08-08 20:21:56 +08:00
YoursFunny 734cfc2eb3 handlers: skip inline fetches for non-post queries
Inline queries fire per keystroke and each fetch runs the 3-attempt
retry loop; a user typing any text was pumping requests into
X/Pixiv/BSky and risking rate-limit bans. Queries now pass only if
cache_key recognizes them as a supported post URL.
2026-08-08 20:20:05 +08:00
YoursFunny a8156697fa send: keep video thumbnails in the download-and-reupload fallback
media_from_file/media_from_url never attached the thumbnail, so any
video that tripped the fallback lost its cover frame. Both now take
item.thumbnail_url() and apply it, matching build_media_group.
2026-08-08 20:19:19 +08:00
YoursFunny c093dfe5ac handlers: dedup extracted URLs by normalized post id
Exact-string dedup let https://x.com/u/status/1 and
https://x.com/u/status/1/photo/1 (or the same link in text and caption)
through twice, causing duplicate fetches and sends. Dedup now uses
cache_key, falling back to the raw URL for unsupported links.
2026-08-08 20:17:07 +08:00
YoursFunny ee6f3e4a27 state: evict idle chats from the ChatStore cache
prune_expired only shrank edit_message maps, so the cache kept one
ChatData per chat forever (a leak proportional to chat count). Chats
without live edit records are now dropped from the cache and their
per-chat lock (DB row persists; get() reloads). Lock order kept safe:
prune never holds the cache lock while taking the per-chat locks.
2026-08-08 20:16:26 +08:00
YoursFunny 16ed53fead handlers: replace unbounded per-URL spawn with a bounded job channel
The 8-permit semaphore was acquired inside the spawned task, so a burst
queued unlimited tasks (each cloning Bot+Message) and nothing tracked
them at shutdown — in-flight sends fired after the stop notice. URL work
now flows through a 256-slot mpsc drained by 8 workers started from
main; a full channel backpressures the per-chat handler, and shutdown
sets URL_STOP so workers stop pulling.
2026-08-08 20:15:40 +08:00
YoursFunny 1d9e3629c9 state: serialize per-chat get→mutate→set with ChatStore::update
Concurrent handler tasks (the batch-forward design spawns several per
chat) each snapshotted the same ChatData and last-writer-wins silently
dropped mutations — e.g. a second edit_message record, leaving one
prompt's Forward button dead. All write cycles now run under a per-chat
async lock; read-only callers keep get().
2026-08-08 20:12:51 +08:00
YoursFunny b3d87b4f7d send: fail fast when a retried local media file is gone
A retried ugoira/bsky temp MP4 was already deleted with its TempDir, so
the retry failed at multipart-build time with a confusing Io error and
wasted all three attempts. input_file_for now rejects a missing local
path up front as a clean permanent error.
2026-08-08 20:10:35 +08:00
YoursFunny 98c48b99c0 send: don't repeat post-send actions on queue resumes
A resumed SendMediaSequence (batch_index>0 or already-sent ids) ran
post_send_actions again, opening a second edit prompt and inserting a
second edit_message record for the same messages — both Forward buttons
worked, enabling double forwards. Resumes now skip it.
2026-08-08 20:09:58 +08:00
YoursFunny f40639c799 queue: back off 1s when a lease fails
A lease error (e.g. persistent SQLITE_BUSY) while rows are due made the
worker spin with sleep(0), hammering SQLite and flooding the log.
lease_next now returns the error and the loop sleeps 1s before retrying.
2026-08-08 20:09:04 +08:00
YoursFunny 51cc079a85 queue: use notify_one so wakeups are never lost
notify_waiters drops the notification when every worker is between its
DB reads and registering notified(); a task enqueued in that window sat
until a stale timer fired. notify_one stores a permit, so the next
worker to wait wakes immediately and re-leases. stop() still wakes all
workers with notify_waiters.
2026-08-08 20:08:12 +08:00
YoursFunny aa3083792a queue: wire attempt counts into retry backoff
Every network retry hard-coded retry_delay_seconds(0), so backoff was
flat at 1.2-1.8s regardless of attempt; a multi-minute outage dead-
lettered after three rapid tries. The queue now scales the handler's
delay by 2^attempts (cap 300s) before rescheduling.
2026-08-08 20:07:26 +08:00
YoursFunny 6849006ad7 site: honor TELOXIDE_PROXY for site fetches
The shared HTTP client ignored the proxy the Bot API uses, so on
proxy-required networks (e.g. behind the GFW) every site fetch failed
while the bot itself worked. Explicit proxy overrides reqwest's system
detection; unset keeps direct connections.
2026-08-08 20:05:51 +08:00
YoursFunny 042a04ab6e site: anchor pixiv and bsky URL patterns
Both matched URL substrings anywhere in text, so a link like
https://evil.com/?u=pixiv.net/artworks/1 triggered a real fetch and
cache-key pollution. Prefix ^(?:https?://)? like the twitter pattern.
2026-08-08 20:05:09 +08:00
YoursFunny 9f28af4e6b site: classify HTTP status codes, make transient failures retryable
Twitter (syndication + auth GraphQL) mapped every non-2xx to NotFound,
killing retries on 429/5xx; bsky never checked status; pixiv network
errors arrived wrapped in PixivError and were excluded from the retry
loop. New FetchError::Transient covers 429/5xx from all sites, the
retry loop now also retries Pixiv errors, and 404/410 stay permanent.
2026-08-08 20:04:28 +08:00
YoursFunny d61dba5096 send: stream media downloads with hard size caps
download_to_temp now uses download_media_limited: non-photos abort the
moment the 10 MiB upload cap is crossed mid-stream (no more full-body
buffering before the size check), photos cap at the 512 MiB decode
budget, and the ugoira frame zip gets a 512 MiB cap. MediaTooLarge
routes to the existing smaller-URL fallback.
2026-08-08 20:03:14 +08:00
YoursFunny f6df3e28cb pixiv: drop unneeded mut bindings in ugoira extraction 2026-08-08 20:02:06 +08:00
YoursFunny deb1ef2428 site: add total and connect timeouts to the shared HTTP client 2026-08-08 20:01:39 +08:00
YoursFunny b5e5340edc pixiv: harden ugoira zip extraction, degrade instead of panicking
Sniff the frame extension from magic bytes instead of the entry filename,
cap each frame at 64 MiB (declared size + streamed read), and map a
panicked encode worker to the existing degrade path instead of
expect()-panicking the whole fetch handler.
2026-08-08 20:01:14 +08:00
YoursFunny 425d1505cf handlers: fix /set_forward_channel admin checks
Compare the sender's user id (not the chat id, which only matches in
private chats) and require the bot to actually be an admin with post
rights instead of silently passing when it is missing from the list.
Also stops panicking on get_me network failures.
2026-08-08 19:59:27 +08:00
YoursFunny 6e40f55440 queue: recover expired leases at runtime, supervise workers
Rows left in_progress by a panicked/crashed worker were only recovered at
start(); a runtime sweep (30s interval, woken by the same notify) now
re-queues them once the 120s lock TTL expires. Workers run under a
supervisor that respawns a panicked loop instead of silently shrinking
the pool of 4.
2026-08-08 19:58:28 +08:00
YoursFunny 7998114dc3 bsky: remux HLS video playlists to MP4 via ffmpeg
bsky video embeds expose only an m3u8 playlist, which Telegram cannot
fetch. Download the master/variant playlists and TS segments through the
shared client (proxy-aware, size-capped), then concat-remux locally;
keep the temp dir alive via Fetched._keep_alive like the ugoira path.
Also adds site::download_media_limited (streaming size cap), status
checks on media_size, and moves the ffmpeg probe to site/mod.rs for
pixiv/bsky to share.
2026-08-08 19:56:36 +08:00
YoursFunny 9a96f78177 handlers: fix UTF-8 byte-slice panic in message log preview 2026-08-08 19:52:01 +08:00
YoursFunny b50f794d52 feat: register bot commands with Telegram
Call setMyCommands at startup so clients show the command list in the
/ menu. handlers::register_commands wraps Command::bot_commands()
(teloxide derives it from the #[command(description)] attributes);
a registration failure only warns and does not stop the bot.
2026-08-08 00:08:48 +08:00
YoursFunny d32fa969d6 fix: correct singular 'entry' in clear-cache replies
plural() returned "" for one, rendering '1 entr.'; return "y" so
the suffix composes to '1 entry' / '2 entries'.
2026-08-08 00:07:27 +08:00
YoursFunny 020e2d01a3 chore: bump version to 1.0.8 2026-08-07 16:23:40 +08:00
YoursFunny 3d6f8548c3 style: cargo fmt across the workspace
Apply rustfmt to the 11 files that had drifted (86 hunks): x-media
site modules (bsky/pixiv/twitter) and xmedia-bot (config/main/
photo/send). Formatting only - no semantic changes; full test suite
still green.
2026-08-07 16:12:01 +08:00
YoursFunny 063e910473 feat: add admin-only /clear_cache command
/clear_cache with no argument wipes the whole link_cache table;
with a post URL it removes that single entry (normalized via
site::cache_key so fxtwitter/mobile/photo variants collide with
the write-side key). Non-admins get 'Admin only.'. LinkCache gains
clear(Option<&str>) -> usize reporting removed rows.
2026-08-07 16:10:25 +08:00
YoursFunny b0ced34b4c refactor: share sqlite open/with_conn helpers in db.rs
Converge the duplicated open_db (open + busy_timeout) and the
spawn_blocking + expect ceremony that every table access repeated
into one db.rs module. ChatStore no longer creates the tasks table
(schema ownership: queue.rs owns tasks, state.rs chat_state,
link_cache.rs link_cache). No schema or behavior change - all
CREATE TABLE statements are byte-identical, IF NOT EXISTS stays
idempotent, so existing data/task_queue.db files need no migration.
2026-08-07 16:00:12 +08:00
YoursFunny 4060a88031 fix: expand twitter short links like FxEmbed linkFixer
Replace display_text_range slicing with FxEmbed-style content matching:
expand mapped t.co links to their real URLs (dropping internal
x.com/i/web/status pages), then strip every leftover t.co short link
(appended media link, unmapped links).

The old code cut by display_text_range, whose index unit differs per
endpoint (UTF-16 on the syndication endpoint, code points in the
GraphQL fallback), so slicing by either unit left a partial
"https://t." caption tail on the other path. Content matching is
unit-agnostic and also keeps user-posted/quote links at the end of
the text that the trailing cut previously dropped.
2026-08-07 10:25:54 +08:00
YoursFunny fb43441c56 feat: add command descriptions and document commands in README
All bot commands now carry English descriptions, shown in the Telegram
command menu and by /help (which prints Command::descriptions()). The
README command table explains each command's arguments and behavior:
forward channel (@channel or ID), edit-before-forward flow, template
[] placeholder semantics and per-site caption format placeholders.
2026-08-06 21:21:28 +08:00
YoursFunny bf628dc999 chore: drop label value in compose example 2026-08-06 20:45:35 +08:00
YoursFunny de22aa9b4d chore: leave DEFAULT_EMAIL blank in compose example 2026-08-06 20:03:56 +08:00
25 changed files with 1741 additions and 1098 deletions
Generated
+14 -364
View File
@@ -99,28 +99,6 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
[[package]]
name = "aws-lc-rs"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ec2f1fc3ec205783a5da9a7e6c1509cc69dedf09a1949e412c1e18469326d00"
dependencies = [
"aws-lc-sys",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a2f9779ce85b93ab6170dd940ad0169b5766ff848247aff13bb788b832fe3f4"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
]
[[package]]
name = "axum"
version = "0.8.9"
@@ -264,12 +242,6 @@ version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
[[package]]
name = "chrono"
version = "0.4.44"
@@ -292,15 +264,6 @@ dependencies = [
"inout",
]
[[package]]
name = "cmake"
version = "0.1.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
dependencies = [
"cc",
]
[[package]]
name = "colored"
version = "3.1.1"
@@ -310,16 +273,6 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "combine"
version = "4.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd"
dependencies = [
"bytes",
"memchr",
]
[[package]]
name = "constant_time_eq"
version = "0.3.1"
@@ -530,12 +483,6 @@ dependencies = [
"futures",
]
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
[[package]]
name = "dyn-clone"
version = "1.0.20"
@@ -687,12 +634,6 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "futures"
version = "0.3.32"
@@ -798,10 +739,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [
"cfg-if",
"js-sys",
"libc",
"wasi",
"wasm-bindgen",
]
[[package]]
@@ -1276,55 +1215,6 @@ version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "jni"
version = "0.22.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498"
dependencies = [
"cfg-if",
"combine",
"jni-macros",
"jni-sys",
"log",
"simd_cesu8",
"thiserror",
"walkdir",
"windows-link",
]
[[package]]
name = "jni-macros"
version = "0.22.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3"
dependencies = [
"proc-macro2",
"quote",
"rustc_version",
"simd_cesu8",
"syn",
]
[[package]]
name = "jni-sys"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2"
dependencies = [
"jni-sys-macros",
]
[[package]]
name = "jni-sys-macros"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264"
dependencies = [
"quote",
"syn",
]
[[package]]
name = "jobserver"
version = "0.1.34"
@@ -1409,12 +1299,6 @@ version = "0.4.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897"
[[package]]
name = "lru-slab"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
[[package]]
name = "lzma-rs"
version = "0.3.0"
@@ -1743,62 +1627,6 @@ dependencies = [
"cc",
]
[[package]]
name = "quinn"
version = "0.11.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
dependencies = [
"bytes",
"cfg_aliases",
"pin-project-lite",
"quinn-proto",
"quinn-udp",
"rustc-hash",
"rustls",
"socket2",
"thiserror",
"tokio",
"tracing",
"web-time",
]
[[package]]
name = "quinn-proto"
version = "0.11.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
dependencies = [
"aws-lc-rs",
"bytes",
"getrandom 0.3.4",
"lru-slab",
"rand 0.9.4",
"ring",
"rustc-hash",
"rustls",
"rustls-pki-types",
"slab",
"thiserror",
"tinyvec",
"tracing",
"web-time",
]
[[package]]
name = "quinn-udp"
version = "0.5.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2",
"tracing",
"windows-sys 0.52.0",
]
[[package]]
name = "quote"
version = "1.0.45"
@@ -1827,18 +1655,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a"
dependencies = [
"libc",
"rand_chacha 0.3.1",
"rand_core 0.6.4",
]
[[package]]
name = "rand"
version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea"
dependencies = [
"rand_chacha 0.9.0",
"rand_core 0.9.5",
"rand_chacha",
"rand_core",
]
[[package]]
@@ -1848,17 +1666,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
dependencies = [
"ppv-lite86",
"rand_core 0.6.4",
]
[[package]]
name = "rand_chacha"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
dependencies = [
"ppv-lite86",
"rand_core 0.9.5",
"rand_core",
]
[[package]]
@@ -1870,15 +1678,6 @@ dependencies = [
"getrandom 0.2.17",
]
[[package]]
name = "rand_core"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
dependencies = [
"getrandom 0.3.4",
]
[[package]]
name = "rc-box"
version = "1.3.0"
@@ -1954,16 +1753,20 @@ checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
dependencies = [
"base64",
"bytes",
"encoding_rs",
"futures-core",
"futures-util",
"h2",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-tls",
"hyper-util",
"js-sys",
"log",
"mime",
"mime_guess",
"native-tls",
"percent-encoding",
@@ -1986,47 +1789,6 @@ dependencies = [
"web-sys",
]
[[package]]
name = "reqwest"
version = "0.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62e0021ea2c22aed41653bc7e1419abb2c97e038ff2c33d0e1309e49a97deec0"
dependencies = [
"base64",
"bytes",
"encoding_rs",
"futures-core",
"h2",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-rustls",
"hyper-util",
"js-sys",
"log",
"mime",
"percent-encoding",
"pin-project-lite",
"quinn",
"rustls",
"rustls-pki-types",
"rustls-platform-verifier",
"serde",
"serde_json",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tokio-rustls",
"tower",
"tower-http",
"tower-service",
"url",
"wasm-bindgen",
"wasm-bindgen-futures",
"web-sys",
]
[[package]]
name = "rgb"
version = "0.8.53"
@@ -2064,21 +1826,6 @@ dependencies = [
"smallvec",
]
[[package]]
name = "rustc-hash"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
[[package]]
name = "rustc_version"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
dependencies = [
"semver",
]
[[package]]
name = "rustix"
version = "1.1.4"
@@ -2098,7 +1845,6 @@ version = "0.23.40"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b"
dependencies = [
"aws-lc-rs",
"once_cell",
"rustls-pki-types",
"rustls-webpki",
@@ -2106,62 +1852,21 @@ dependencies = [
"zeroize",
]
[[package]]
name = "rustls-native-certs"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63"
dependencies = [
"openssl-probe",
"rustls-pki-types",
"schannel",
"security-framework",
]
[[package]]
name = "rustls-pki-types"
version = "1.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9"
dependencies = [
"web-time",
"zeroize",
]
[[package]]
name = "rustls-platform-verifier"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
dependencies = [
"core-foundation 0.10.1",
"core-foundation-sys",
"jni",
"log",
"once_cell",
"rustls",
"rustls-native-certs",
"rustls-platform-verifier-android",
"rustls-webpki",
"security-framework",
"security-framework-sys",
"webpki-root-certs",
"windows-sys 0.61.2",
]
[[package]]
name = "rustls-platform-verifier-android"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
[[package]]
name = "rustls-webpki"
version = "0.103.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted",
@@ -2179,15 +1884,6 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "same-file"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
dependencies = [
"winapi-util",
]
[[package]]
name = "schannel"
version = "0.1.29"
@@ -2387,22 +2083,6 @@ version = "0.3.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
[[package]]
name = "simd_cesu8"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94f90157bb87cddf702797c5dadfa0be7d266cdf49e22da2fcaa32eff75b2c33"
dependencies = [
"rustc_version",
"simdutf8",
]
[[package]]
name = "simdutf8"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
[[package]]
name = "slab"
version = "0.4.12"
@@ -2536,7 +2216,7 @@ dependencies = [
"log",
"mime",
"pin-project",
"rand 0.8.6",
"rand",
"serde",
"serde_json",
"teloxide-core",
@@ -2567,7 +2247,7 @@ dependencies = [
"once_cell",
"pin-project",
"rc-box",
"reqwest 0.12.28",
"reqwest",
"rgb",
"serde",
"serde_json",
@@ -2914,16 +2594,6 @@ version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "walkdir"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
dependencies = [
"same-file",
"winapi-util",
]
[[package]]
name = "want"
version = "0.3.1"
@@ -3069,25 +2739,6 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "web-time"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "webpki-root-certs"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c"
dependencies = [
"rustls-pki-types",
]
[[package]]
name = "winapi-util"
version = "0.1.11"
@@ -3351,15 +3002,15 @@ checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
[[package]]
name = "x-media"
version = "1.0.7"
version = "1.1.0"
dependencies = [
"bytes",
"dotenv",
"html-escape",
"log",
"rand 0.8.6",
"rand",
"regex",
"reqwest 0.13.3",
"reqwest",
"serde",
"serde_json",
"tempfile",
@@ -3370,7 +3021,7 @@ dependencies = [
[[package]]
name = "xmedia-bot"
version = "1.0.7"
version = "1.1.0"
dependencies = [
"dotenv",
"fast_image_resize",
@@ -3380,8 +3031,7 @@ dependencies = [
"parking_lot",
"png",
"pretty_env_logger",
"rand 0.8.6",
"regex",
"rand",
"rusqlite",
"serde",
"serde_json",
+4
View File
@@ -1,3 +1,7 @@
[workspace]
members = ["crates/x-media", "crates/xmedia-bot"]
resolver = "3"
# Smaller production binary; debug symbols are not shipped anyway.
[profile.release]
strip = true
+14 -9
View File
@@ -11,6 +11,11 @@ ARG APP_NAME=telegram-twitter-media-bot
# runners. `/redirect/latest/` floats to the newest release build; each build
# also ships a .sha256. Swap `amd64` for `arm64` when building arm64 images.
ARG FFMPEG_URL=https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffmpeg.zip
# Optional sha256 of ffmpeg.zip (pinned releases only): set to verify the
# download. The mirror publishes .sha256 sidecars next to pinned builds, e.g.
# https://ffmpeg.martin-riedl.de/download/linux/amd64/<id>_9.0/ffmpeg.zip.sha256
# (the /redirect/latest/ URL itself has no sidecar — pin the effective URL).
ARG FFMPEG_SHA256=
WORKDIR /build
@@ -30,19 +35,20 @@ RUN mkdir -p crates/x-media/src crates/xmedia-bot/src \
# root. `unzip -t` verifies the archive before extraction so a bad
# download fails loudly here instead of a cryptic later error.
RUN wget -q -O /tmp/ffmpeg.zip "$FFMPEG_URL" \
&& if [ -n "$FFMPEG_SHA256" ]; then echo "$FFMPEG_SHA256 /tmp/ffmpeg.zip" | sha256sum -c -; fi \
&& unzip -tq /tmp/ffmpeg.zip \
&& unzip -q /tmp/ffmpeg.zip -d /usr/local/bin \
&& chmod +x /usr/local/bin/ffmpeg \
&& rm /tmp/ffmpeg.zip \
&& /usr/local/bin/ffmpeg -version >/dev/null
# 3. Real sources last: only our crates recompile on source changes. The
# COPY preserves host mtimes, which predate the stub artifacts from step 1;
# cargo's mtime-based freshness check would otherwise treat the stub build
# as up-to-date and never compile the real sources. `touch` forces cargo to
# see the real files as newer.
# 3. Real sources last: only our crates recompile on source changes.
# `cargo clean -p` drops the two crates' artifacts while keeping the
# compiled dependency layer, forcing a deterministic rebuild of the real
# sources. (The previous `touch`-mtimes hack silently shipped the stub
# binary when host files carried future timestamps.)
COPY crates/ ./crates/
RUN find crates -type f -name '*.rs' -exec touch {} + \
RUN cargo clean -p xmedia-bot -p x-media \
&& cargo build --release -p xmedia-bot
# ---------- runtime stage ----------
@@ -56,10 +62,9 @@ LABEL org.opencontainers.image.title="${APP_NAME}"
# Everything is copied in — no apt in the runtime stage. Privilege dropping is
# done by docker-entrypoint.sh with setpriv (util-linux, already in
# bookworm-slim), so no gosu needed.
# bookworm-slim), so no gosu needed. (libssl3/libcrypto are already in
# bookworm-slim; only ca-certificates and ffmpeg need copying.)
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder /usr/lib/x86_64-linux-gnu/libssl.so.3* /usr/lib/x86_64-linux-gnu/
COPY --from=builder /usr/lib/x86_64-linux-gnu/libcrypto.so.3* /usr/lib/x86_64-linux-gnu/
COPY --from=builder /usr/local/bin/ffmpeg /usr/local/bin/ffmpeg
WORKDIR /app
+7 -5
View File
@@ -101,12 +101,14 @@ Telegram 只接受 443/80/88/8443 端口。
| 命令 | 说明 |
|---|---|
| `/set_forward_channel <频道>` | 设置转发频道 |
| `/start` | 欢迎语 |
| `/help` | 查看全部命令及用法(即本文档的命令表) |
| `/set_forward_channel <频道>` | 设置转发频道,参数为 `@频道名` 或频道 ID;设置后发送的媒体消息会自动转发到该频道 |
| `/remove_forward_channel` | 取消转发频道 |
| `/edit_before_forward` | 开关转发前编辑 |
| `/set_template <名称>` | 将回复的消息(含 `[]`)保存为模板 |
| `/set_format <站点> <格式>` | 自定义 caption 格式(占位符 `{url}` `{title}` `{tags}` 等) |
| `/bot_dict` | 查看聊天状态 |
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/bot_dict` | 查看当前聊天状态(调试用) |
链接处理仅限私聊;命令在任意聊天可用。
+2 -2
View File
@@ -1,10 +1,10 @@
[package]
name = "x-media"
version = "1.0.7"
version = "1.1.0"
edition = "2024"
[dependencies]
reqwest = { version = "0.13", features = ["json", "query", "form"] }
reqwest = { version = "0.12", features = ["json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
regex = "1.12"
+188 -17
View File
@@ -1,12 +1,12 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap()
Regex::new(r"^(?:https?://)?bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap()
});
pub fn enabled() -> bool {
@@ -15,9 +15,169 @@ pub fn enabled() -> bool {
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
let caps = PATTERN.captures(url).ok_or(FetchError::NotFound)?;
let handle = caps.get(1).map(|m| m.as_str()).ok_or(FetchError::NotFound)?;
let rkey = caps.get(2).map(|m| m.as_str()).ok_or(FetchError::NotFound)?;
Ok(fetch(handle, rkey).await?.into())
let handle = caps
.get(1)
.map(|m| m.as_str())
.ok_or(FetchError::NotFound)?;
let rkey = caps
.get(2)
.map(|m| m.as_str())
.ok_or(FetchError::NotFound)?;
let post = fetch(handle, rkey).await?;
let mut fetched: Fetched = post.into();
// bsky video embeds expose only an HLS playlist URL, which Telegram
// cannot fetch; remux it to a single MP4 (mirrors the pixiv ugoira
// encode path — the temp file stays alive via `_keep_alive`). On any
// failure the video item is dropped and the post degrades to its text.
let mut media = Vec::with_capacity(fetched.media.len());
for item in fetched.media {
let is_hls = matches!(&item, Media::Video { url, .. }
if url.contains("playlist") || url.ends_with(".m3u8"));
if !is_hls {
media.push(item);
continue;
}
let url = item.url().to_string();
match resolve_bsky_video(&url).await {
Ok(Some((mp4_path, keep_alive))) => {
let thumbnail_url = match &item {
Media::Video { thumbnail_url, .. } => thumbnail_url.clone(),
_ => String::new(),
};
media.push(Media::Video {
title: None,
url: mp4_path.to_string_lossy().into_owned(),
thumbnail_url,
});
fetched._keep_alive = Some(keep_alive);
}
Ok(None) => log::warn!("bsky video remux unavailable for {url}"),
Err(e) => log::warn!("bsky video remux failed for {url}: {e}"),
}
}
fetched.media = media;
Ok(fetched)
}
/// Downloads an HLS playlist (master or media) and remuxes its segments to a
/// single MP4 via ffmpeg. Returns the MP4 path plus the temp dir that must
/// stay alive until the file is uploaded. `Ok(None)` when ffmpeg is missing.
///
/// Verified live (2026-08): bsky master playlists carry `#EXT-X-STREAM-INF`
/// variant lines (e.g. `720p/video.m3u8?session_id=…`), and the media
/// playlists are VOD MPEG-TS segments (`videoN.ts?…`) without EXT-X-MAP, so
/// a plain `-f concat -c copy` remux is valid.
async fn resolve_bsky_video(
playlist_url: &str,
) -> Result<Option<(std::path::PathBuf, tempfile::TempDir)>, String> {
if !crate::site::ffmpeg_available() {
crate::site::log_once_ffmpeg_missing();
return Ok(None);
}
let master = crate::site::download_media_limited(playlist_url, 1_048_576)
.await
.map_err(|e| format!("bsky video master playlist: {e}"))?;
let master = String::from_utf8_lossy(&master);
// Master playlist: pick the variant with the highest declared bandwidth.
let playlist_url = if master.contains("#EXT-X-STREAM-INF") {
let mut best: Option<(u64, String)> = None;
let mut lines = master.lines();
while let Some(line) = lines.next() {
if !line.starts_with("#EXT-X-STREAM-INF") {
continue;
}
let bandwidth = line
.split_once("BANDWIDTH=")
.and_then(|(_, rest)| rest.split(|c: char| !c.is_ascii_digit()).next())
.and_then(|n| n.parse::<u64>().ok())
.unwrap_or(0);
if let Some(uri) = lines.next().filter(|u| !u.starts_with('#'))
&& bandwidth >= best.as_ref().map(|(b, _)| *b).unwrap_or(0)
{
best = Some((bandwidth, uri.to_string()));
}
}
let Some((_, uri)) = best else {
return Err("bsky video master playlist has no variants".to_string());
};
url::Url::parse(playlist_url)
.and_then(|base| base.join(&uri))
.map_err(|e| format!("bsky video variant URL: {e}"))?
.to_string()
} else {
playlist_url.to_string()
};
let variant = crate::site::download_media_limited(&playlist_url, 1_048_576)
.await
.map_err(|e| format!("bsky video media playlist: {e}"))?;
let variant = String::from_utf8_lossy(&variant);
// Segment URIs: non-#, non-empty lines, resolved relative to the playlist.
let base = url::Url::parse(&playlist_url).map_err(|e| format!("bsky playlist URL: {e}"))?;
let segments: Vec<String> = variant
.lines()
.map(str::trim)
.filter(|l| !l.is_empty() && !l.starts_with('#'))
.map(|l| base.join(l).map(|u| u.to_string()))
.collect::<Result<_, _>>()
.map_err(|e| format!("bsky segment URL: {e}"))?;
if segments.is_empty() {
return Err("bsky video playlist has no segments".to_string());
}
if segments.len() > 500 {
return Err("bsky video has too many segments".to_string());
}
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let mut total: u64 = 0;
let mut list = String::new();
for (i, seg) in segments.iter().enumerate() {
let bytes = crate::site::download_media_limited(seg, 20 * 1024 * 1024)
.await
.map_err(|e| format!("bsky segment {i}: {e}"))?;
total += bytes.len() as u64;
if total > 256 * 1024 * 1024 {
return Err("bsky video exceeds total size cap".to_string());
}
let path = frames_dir.path().join(format!("seg_{i:04}.ts"));
std::fs::write(&path, &bytes).map_err(|e| e.to_string())?;
list.push_str(&format!("file '{}'\n", path.to_string_lossy()));
}
let list_path = frames_dir.path().join("list.txt");
std::fs::write(&list_path, &list).map_err(|e| e.to_string())?;
let output = out_dir.path().join("video.mp4");
let list_str = list_path.to_string_lossy().into_owned();
let output_str = output.to_string_lossy().into_owned();
let status = tokio::task::spawn_blocking(move || {
std::process::Command::new("ffmpeg")
.args([
"-y",
"-f",
"concat",
"-safe",
"0",
"-i",
&list_str,
"-c",
"copy",
"-movflags",
"+faststart",
&output_str,
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
})
.await
.map_err(|e| format!("bsky remux worker panicked: {e}"))?;
match status {
Ok(s) if s.success() => Ok(Some((output, out_dir))),
Ok(s) => Err(format!("ffmpeg exited with {s}")),
Err(e) => Err(format!("ffmpeg spawn failed: {e}")),
}
}
/// Fetches a post thread by handle or DID (`at://` URIs work for both).
@@ -30,8 +190,16 @@ pub async fn fetch(handle: &str, rkey: &str) -> Result<Post, FetchError> {
])
.send()
.await?;
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!("bsky status {status}"))),
};
}
let text = response.text().await?;
Ok(Post::from_json(&text, rkey.to_string())?)
Post::from_json(&text, rkey.to_string())
}
#[derive(Debug)]
@@ -56,8 +224,8 @@ impl Post {
pub fn caption(&self) -> String {
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
url = self.url(),
author_url = self.author_url(),
url = encode_double_quoted_attribute(&self.url()),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
text = encode_text(&self.text),
)
@@ -197,7 +365,10 @@ mod tests {
}));
let post = Post::from_json(&raw.to_string(), "3xxxx".into()).unwrap();
let fetched: Fetched = post.into();
assert_eq!(fetched.source_url, "https://bsky.app/profile/user.bsky.social/post/3xxxx");
assert_eq!(
fetched.source_url,
"https://bsky.app/profile/user.bsky.social/post/3xxxx"
);
assert_eq!(fetched.title, "hello <world>");
assert_eq!(fetched.media.len(), 1);
assert!(!fetched.sensitive);
@@ -246,11 +417,10 @@ mod tests {
#[tokio::test]
async fn live_fetch_with_photos() {
let fetched = fetch_from_url(
"https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m",
)
.await
.unwrap();
let fetched =
fetch_from_url("https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m")
.await
.unwrap();
assert_eq!(
fetched.source_url,
"https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m"
@@ -260,9 +430,10 @@ mod tests {
#[tokio::test]
async fn live_fetch_smoke() {
let fetched = fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
.await
.unwrap();
let fetched =
fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
.await
.unwrap();
assert_eq!(
fetched.source_url,
"https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224"
+81 -13
View File
@@ -6,6 +6,7 @@
use std::fmt;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
pub mod bsky;
@@ -147,6 +148,10 @@ pub enum FetchError {
/// The post exists but its content is withheld (twitter NSFW /
/// age-restricted tweets come back as an empty `{}` from syndication).
Sensitive,
/// A download exceeded the caller's size cap (see [`download_media_limited`]).
TooLarge,
/// A transient server-side failure (429 / 5xx); [`fetch`] retries these.
Transient(String),
}
impl fmt::Display for FetchError {
@@ -158,6 +163,8 @@ impl fmt::Display for FetchError {
FetchError::NotFound => write!(f, "not found"),
FetchError::Blocked => write!(f, "blocked"),
FetchError::Sensitive => write!(f, "content withheld (sensitive)"),
FetchError::TooLarge => write!(f, "media too large"),
FetchError::Transient(message) => write!(f, "transient: {message}"),
}
}
}
@@ -169,6 +176,8 @@ impl std::error::Error for FetchError {
FetchError::Json(e) => Some(e),
FetchError::Pixiv(e) => Some(e),
FetchError::NotFound | FetchError::Blocked | FetchError::Sensitive => None,
FetchError::TooLarge => None,
FetchError::Transient(_) => None,
}
}
}
@@ -179,7 +188,6 @@ impl From<reqwest::Error> for FetchError {
}
}
impl From<serde_json::Error> for FetchError {
fn from(e: serde_json::Error) -> Self {
FetchError::Json(e)
@@ -195,7 +203,22 @@ impl From<PixivError> for FetchError {
/// Shared HTTP client (browser User-Agent) for twitter/bsky fetches and
/// [`download_media`].
pub(crate) static CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
let builder = reqwest::Client::builder().user_agent("Mozilla/5.0");
let mut builder = reqwest::Client::builder()
.user_agent("Mozilla/5.0")
// reqwest has no total timeout by default; a stalled connection
// would otherwise pin a fetch/handler forever.
.timeout(Duration::from_secs(30))
.connect_timeout(Duration::from_secs(10));
// Route site fetches through the same proxy the Bot API uses, so a
// network that needs TELOXIDE_PROXY (e.g. behind the GFW) does not
// leave site fetches dead while the bot itself works.
if let Some(proxy) = std::env::var("TELOXIDE_PROXY")
.ok()
.filter(|s| !s.is_empty())
&& let Ok(p) = reqwest::Proxy::all(&proxy)
{
builder = builder.proxy(p);
}
// Each `#[tokio::test]` runs on its own runtime; the connection pool is
// bound to the runtime that created it, so cross-runtime reuse of idle
// connections fails with DispatchGone. In test builds every request uses
@@ -205,13 +228,38 @@ pub(crate) static CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
builder.build().expect("failed to build HTTP client")
});
/// Whether a usable `ffmpeg` binary is on PATH (probed once). Shared by the
/// pixiv ugoira encoder and the bsky HLS remuxer.
static FFMPEG_AVAILABLE: LazyLock<bool> = LazyLock::new(|| {
std::process::Command::new("ffmpeg")
.arg("-version")
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
});
static FFMPEG_MISSING_LOGGED: AtomicBool = AtomicBool::new(false);
pub(crate) fn ffmpeg_available() -> bool {
*FFMPEG_AVAILABLE
}
pub(crate) fn log_once_ffmpeg_missing() {
if !FFMPEG_MISSING_LOGGED.swap(true, Ordering::Relaxed) {
log::warn!("ffmpeg not found; ugoira and bsky video posts stay unsupported");
}
}
/// Fetches a post from its URL. Returns `Ok(None)` when no site pattern
/// matches (unsupported links are silently ignored by the bot).
///
/// Transient network failures are retried: 3 total attempts with 1s then 2s
/// delays. Non-Http errors (Json/NotFound/Blocked/Pixiv) are not retried.
/// delays. Retried classes: bare HTTP errors, [`FetchError::Transient`]
/// (429/5xx from any site), and pixiv errors (its network failures arrive
/// wrapped as `PixivError`). Non-retried: Json/NotFound/Blocked/Sensitive.
pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
let mut last_http_error = None;
for attempt in 0..3u32 {
match fetch_once(url).await {
Ok(Some(fetched)) => {
@@ -223,18 +271,17 @@ pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
return Ok(Some(fetched));
}
Ok(None) => return Ok(None),
Err(FetchError::Http(e)) => {
last_http_error = Some(e);
Err(e @ (FetchError::Http(_) | FetchError::Transient(_) | FetchError::Pixiv(_))) => {
if attempt < 2 {
tokio::time::sleep(Duration::from_secs(1 << attempt)).await;
} else {
return Err(e);
}
}
Err(other) => return Err(other),
}
}
Err(FetchError::Http(
last_http_error.expect("retry loop always ran 3 attempts"),
))
unreachable!("retry loop always returns")
}
async fn fetch_once(url: &str) -> Result<Option<Fetched>, FetchError> {
@@ -263,18 +310,39 @@ pub async fn media_size(url: &str) -> Result<Option<u64>, FetchError> {
if lower.contains("pximg.net") {
request = request.header("Referer", "https://www.pixiv.net/");
}
let response = request.send().await?;
let response = request.send().await?.error_for_status()?;
Ok(response.content_length())
}
pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
/// Downloads a media file with a hard size cap: the body is streamed and the
/// download aborts with [`FetchError::TooLarge`] the moment the cap is
/// crossed (or when a declared Content-Length already exceeds it). Keeps the
/// bot from buffering arbitrarily large bodies into memory.
pub async fn download_media_limited(url: &str, max_bytes: u64) -> Result<bytes::Bytes, FetchError> {
let mut request = CLIENT.get(url);
let lower = url.to_ascii_lowercase();
if lower.contains("pximg.net") {
request = request.header("Referer", "https://www.pixiv.net/");
}
let response = request.send().await?;
Ok(response.bytes().await?)
let response = request.send().await?.error_for_status()?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let mut response = response;
let mut buf = Vec::new();
while let Some(chunk) = response.chunk().await? {
buf.extend_from_slice(&chunk);
if buf.len() as u64 > max_bytes {
return Err(FetchError::TooLarge);
}
}
Ok(bytes::Bytes::from(buf))
}
pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
download_media_limited(url, u64::MAX).await
}
#[cfg(test)]
+124 -107
View File
@@ -10,8 +10,8 @@ use crate::site::FetchError;
use std::env;
use std::fmt;
use std::io::{Cursor, Read};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::LazyLock;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, SystemTime};
const AUTH_TOKEN_URL: &str = "https://oauth.secure.pixiv.net/auth/token";
@@ -127,13 +127,18 @@ impl PixivAPI {
pub async fn illust_detail(&self, illust_id: u64) -> Result<IllustrationModel, PixivError> {
let access_token = self.get_access_token().await?;
let response = crate::site::CLIENT
.get(format!("{APP_API_URL}/v1/illust/detail?illust_id={illust_id}"))
.get(format!(
"{APP_API_URL}/v1/illust/detail?illust_id={illust_id}"
))
.header("app-os", "ios")
.header("app-os-version", "14.6")
.header("User-Agent", APP_USER_AGENT)
.bearer_auth(access_token)
.send()
.await?;
if !response.status().is_success() {
return Err(PixivError::Api(format!("status {}", response.status())));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
if json.get("error").is_some() {
let message = json
@@ -175,13 +180,18 @@ impl PixivAPI {
pub async fn ugoira_metadata(&self, illust_id: u64) -> Result<UgoiraMetadataModel, PixivError> {
let access_token = self.get_access_token().await?;
let response = crate::site::CLIENT
.get(format!("{APP_API_URL}/v1/ugoira/metadata?illust_id={illust_id}"))
.get(format!(
"{APP_API_URL}/v1/ugoira/metadata?illust_id={illust_id}"
))
.header("app-os", "ios")
.header("app-os-version", "14.6")
.header("User-Agent", APP_USER_AGENT)
.bearer_auth(access_token)
.send()
.await?;
if !response.status().is_success() {
return Err(PixivError::Api(format!("status {}", response.status())));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
if json.get("error").is_some() {
let message = json
@@ -203,8 +213,8 @@ impl PixivAPI {
&self,
illust_id: u64,
) -> Result<Option<(String, tempfile::TempDir)>, PixivError> {
if !ffmpeg_available() {
log_once_ffmpeg_missing();
if !crate::site::ffmpeg_available() {
crate::site::log_once_ffmpeg_missing();
return Ok(None);
}
let metadata = self.ugoira_metadata(illust_id).await?;
@@ -218,87 +228,117 @@ impl PixivAPI {
let Some(zip_url) = zip_url else {
return Ok(None);
};
let zip_bytes = crate::site::download_media(&zip_url).await.map_err(|e| match e {
FetchError::Http(e) => PixivError::Http(e),
other => PixivError::Api(format!("frame zip download failed: {other}")),
})?;
let zip_bytes = crate::site::download_media_limited(&zip_url, 512 * 1024 * 1024)
.await
.map_err(|e| match e {
FetchError::Http(e) => PixivError::Http(e),
other => PixivError::Api(format!("frame zip download failed: {other}")),
})?;
let frame_delays = metadata.frames.iter().map(|f| f.delay).collect::<Vec<_>>();
let result = tokio::task::spawn_blocking(
move || -> Result<(String, tempfile::TempDir), String> {
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let result =
tokio::task::spawn_blocking(move || -> Result<(String, tempfile::TempDir), String> {
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
// Extract frames to canonical zero-padded names; pixiv ugoira
// frames are uniformly jpg or png per artwork.
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
.map_err(|e| format!("unzip: {e}"))?;
// pixiv ugoira frames are uniformly jpg or png per artwork; take
// the extension from the first entry.
let extension = if archive.len() > 0 {
let first_name = archive
.by_index(0)
.map_err(|e| e.to_string())?
.name()
.to_string();
first_name
.rsplit('.')
.next()
.unwrap_or("jpg")
.to_string()
} else {
"jpg".to_string()
};
let mut count = 0usize;
for i in 0..archive.len() {
let mut entry = archive.by_index(i).map_err(|e| e.to_string())?;
let mut bytes = Vec::new();
entry.read_to_end(&mut bytes).map_err(|e| e.to_string())?;
let path = frames_dir.path().join(format!("img_{count:05}.{extension}"));
std::fs::write(&path, bytes).map_err(|e| e.to_string())?;
count += 1;
}
if count == 0 {
return Err("empty frame zip".to_string());
}
// Extract frames to canonical zero-padded names; pixiv ugoira
// frames are uniformly jpg or png per artwork.
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
.map_err(|e| format!("unzip: {e}"))?;
if archive.is_empty() {
return Err("empty frame zip".to_string());
}
// Uniform jpg or png per artwork; sniff the first entry's
// magic bytes instead of trusting its filename.
let first = archive.by_index(0).map_err(|e| e.to_string())?;
let mut first_bytes = Vec::new();
first
.take(64 * 1024 * 1024 + 1)
.read_to_end(&mut first_bytes)
.map_err(|e| e.to_string())?;
if first_bytes.len() > 64 * 1024 * 1024 {
return Err("frame exceeds size cap".to_string());
}
let extension = if first_bytes.starts_with(&[0xFF, 0xD8]) {
"jpg"
} else if first_bytes.starts_with(b"\x89PNG") {
"png"
} else {
"jpg"
};
let mut count = 0usize;
{
let path = frames_dir
.path()
.join(format!("img_{count:05}.{extension}"));
std::fs::write(&path, &first_bytes).map_err(|e| e.to_string())?;
count += 1;
}
for i in 1..archive.len() {
let entry = archive.by_index(i).map_err(|e| e.to_string())?;
if entry.size() > 64 * 1024 * 1024 {
return Err(format!("frame {i} exceeds size cap"));
}
let mut bytes = Vec::new();
entry
.take(64 * 1024 * 1024 + 1)
.read_to_end(&mut bytes)
.map_err(|e| e.to_string())?;
if bytes.len() > 64 * 1024 * 1024 {
return Err(format!("frame {i} exceeds size cap"));
}
let path = frames_dir
.path()
.join(format!("img_{count:05}.{extension}"));
std::fs::write(&path, bytes).map_err(|e| e.to_string())?;
count += 1;
}
if count == 0 {
return Err("empty frame zip".to_string());
}
// Constant rate from the median frame delay (ms).
let mut delays = frame_delays;
delays.sort_unstable();
let median = delays[delays.len() / 2].max(1);
let framerate = 1000.0 / median as f64;
// Constant rate from the median frame delay (ms).
let mut delays = frame_delays;
delays.sort_unstable();
let median = delays[delays.len() / 2].max(1);
let framerate = 1000.0 / median as f64;
let output = out_dir.path().join("ugoira.mp4");
let status = std::process::Command::new("ffmpeg")
.args([
"-y",
"-framerate",
&framerate.to_string(),
"-i",
&frames_dir.path().join(format!("img_%05d.{extension}")).to_string_lossy(),
// libx264 needs even dimensions; pixiv ugoira frames can
// be odd-sized (e.g. 277x405).
"-vf",
"scale=trunc(iw/2)*2:trunc(ih/2)*2",
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
&output.to_string_lossy(),
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map_err(|e| format!("ffmpeg spawn failed: {e}"))?;
if !status.success() {
return Err(format!("ffmpeg exited with {status}"));
}
Ok((output.to_string_lossy().into_owned(), out_dir))
},
)
.await
.expect("ugoira encode worker panicked");
let output = out_dir.path().join("ugoira.mp4");
let status = std::process::Command::new("ffmpeg")
.args([
"-y",
"-framerate",
&framerate.to_string(),
"-i",
&frames_dir
.path()
.join(format!("img_%05d.{extension}"))
.to_string_lossy(),
// libx264 needs even dimensions; pixiv ugoira frames can
// be odd-sized (e.g. 277x405).
"-vf",
"scale=trunc(iw/2)*2:trunc(ih/2)*2",
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
&output.to_string_lossy(),
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map_err(|e| format!("ffmpeg spawn failed: {e}"))?;
if !status.success() {
return Err(format!("ffmpeg exited with {status}"));
}
Ok((output.to_string_lossy().into_owned(), out_dir))
})
.await
.map_err(|e| {
log::error!("ugoira encode worker panicked for {illust_id}: {e}");
PixivError::Api(format!("ugoira worker failed: {e}"))
})?;
match result {
Ok(pair) => Ok(Some(pair)),
Err(message) => {
@@ -309,32 +349,9 @@ impl PixivAPI {
}
}
static FFMPEG_AVAILABLE: LazyLock<bool> = LazyLock::new(|| {
std::process::Command::new("ffmpeg")
.arg("-version")
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
});
static FFMPEG_MISSING_LOGGED: AtomicBool = AtomicBool::new(false);
fn ffmpeg_available() -> bool {
*FFMPEG_AVAILABLE
}
fn log_once_ffmpeg_missing() {
if !FFMPEG_MISSING_LOGGED.swap(true, Ordering::Relaxed) {
log::warn!("ffmpeg not found; pixiv ugoira posts stay unsupported");
}
}
/// pixiv3-rs replacement: `None` when `PIXIV_REFRESH_TOKEN` is unset.
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> = LazyLock::new(|| {
env::var("PIXIV_REFRESH_TOKEN").ok().map(PixivAPI::new)
});
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> =
LazyLock::new(|| env::var("PIXIV_REFRESH_TOKEN").ok().map(PixivAPI::new));
/// Set at startup when the login validation fails; pixiv stays disabled until
/// the next process start.
+74 -24
View File
@@ -1,12 +1,12 @@
use super::model::{IllustrationModel, TypeModel};
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?:www\.)?pixiv\.net/(?:en/)?(?:(?:i|artworks)/|member_illust\.php\?(?:mode=[a-z_]*&)?illust_id=)(\d+)").unwrap()
Regex::new(r"^(?:https?://)?(?:www\.)?pixiv\.net/(?:en/)?(?:(?:i|artworks)/|member_illust\.php\?(?:mode=[a-z_]*&)?illust_id=)(\d+)").unwrap()
});
pub fn enabled() -> bool {
@@ -48,9 +48,9 @@ impl Illustration {
pub fn caption(&self) -> String {
format!(
"<a href=\"{url}\">{title}</a> / <a href=\"{author_url}\">{author}</a>\n{tags}",
url = self.url(),
url = encode_double_quoted_attribute(&self.url()),
title = encode_text(&self.title),
author_url = self.author_url(),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
tags = encode_text(
&self
@@ -82,12 +82,15 @@ impl Illustration {
// keeps media empty when encoding fails or ffmpeg is missing.
} else if model.page_count > 1 {
media.extend(model.meta_pages.iter().filter_map(|page| {
page.image_urls.original.clone().map(|original| Media::Illustration {
title: None,
url: original,
thumbnail_url: Some(page.image_urls.medium.clone()),
fallback_url: Some(page.image_urls.large.clone()),
})
page.image_urls
.original
.clone()
.map(|original| Media::Illustration {
title: None,
url: original,
thumbnail_url: Some(page.image_urls.medium.clone()),
fallback_url: Some(page.image_urls.large.clone()),
})
}));
} else if let Some(original) = model
.meta_single_page
@@ -147,8 +150,8 @@ impl From<Illustration> for Fetched {
#[cfg(test)]
mod tests {
use super::*;
use super::super::model::IllustrationModel;
use super::*;
fn illust_json(
type_: &str,
@@ -203,8 +206,14 @@ mod tests {
("https://pixiv.net/artworks/123456", "123456"),
("https://www.pixiv.net/en/artworks/123456", "123456"),
("https://www.pixiv.net/i/123456", "123456"),
("https://www.pixiv.net/member_illust.php?mode=medium&illust_id=123456", "123456"),
("https://www.pixiv.net/en/member_illust.php?illust_id=123456", "123456"),
(
"https://www.pixiv.net/member_illust.php?mode=medium&illust_id=123456",
"123456",
),
(
"https://www.pixiv.net/en/member_illust.php?illust_id=123456",
"123456",
),
];
for (url, id) in cases {
let caps = PATTERN.captures(url).unwrap_or_else(|| panic!("{url}"));
@@ -225,7 +234,14 @@ mod tests {
#[test]
fn ugoira_yields_empty_media() {
let v = illust_json("ugoira", 1, Some("https://i.pximg.net/orig.jpg"), None, vec![], 0);
let v = illust_json(
"ugoira",
1,
Some("https://i.pximg.net/orig.jpg"),
None,
vec![],
0,
);
let illustration = parse(v);
let fetched: Fetched = illustration.into();
assert!(fetched.media.is_empty());
@@ -296,7 +312,12 @@ mod tests {
let fetched: Fetched = parse(v).into();
assert_eq!(fetched.media.len(), 1);
match &fetched.media[0] {
Media::Illustration { url, thumbnail_url, fallback_url, .. } => {
Media::Illustration {
url,
thumbnail_url,
fallback_url,
..
} => {
assert_eq!(url, "https://i.pximg.net/p2.jpg");
assert_eq!(thumbnail_url.as_deref(), Some("m2.jpg"));
assert_eq!(fallback_url.as_deref(), Some("l2.jpg"));
@@ -307,7 +328,14 @@ mod tests {
#[test]
fn caption_with_escapes_format_and_substitutes() {
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 0);
let v = illust_json(
"illust",
1,
Some("https://i.pximg.net/o.jpg"),
None,
vec![],
0,
);
let fetched: Fetched = parse(v).into();
// Format string is escaped in full, then placeholders substituted.
let out = fetched.caption_with("{title} by {author} <script> {tags}");
@@ -330,7 +358,14 @@ mod tests {
#[test]
fn ai_work_gets_leading_ai_tag() {
// illust_ai_type == 2 is the only AI marker.
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 2);
let v = illust_json(
"illust",
1,
Some("https://i.pximg.net/o.jpg"),
None,
vec![],
2,
);
let fetched: Fetched = parse(v).into();
assert!(
fetched.caption.contains("#AI #tag1 #tag2"),
@@ -338,14 +373,25 @@ mod tests {
fetched.caption
);
// The {tags} placeholder reflects the tag array too.
assert!(fetched.caption_with("{tags}").starts_with("#AI "), "got: {}", fetched.caption_with("{tags}"));
assert!(
fetched.caption_with("{tags}").starts_with("#AI "),
"got: {}",
fetched.caption_with("{tags}")
);
}
#[test]
fn non_ai_work_has_no_ai_tag() {
// 1 = explicitly not AI, 0 = undefined: neither gets the #AI tag.
for ai_type in [0, 1] {
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], ai_type);
let v = illust_json(
"illust",
1,
Some("https://i.pximg.net/o.jpg"),
None,
vec![],
ai_type,
);
let fetched: Fetched = parse(v).into();
assert!(
!fetched.caption.contains("#AI"),
@@ -357,7 +403,14 @@ mod tests {
#[test]
fn caption_escapes_and_links() {
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 0);
let v = illust_json(
"illust",
1,
Some("https://i.pximg.net/o.jpg"),
None,
vec![],
0,
);
let fetched: Fetched = parse(v).into();
assert!(
fetched
@@ -367,9 +420,6 @@ mod tests {
fetched.caption
);
assert!(fetched.caption.contains("#tag1 #tag2"));
assert_eq!(
fetched.source_url,
"https://www.pixiv.net/artworks/123"
);
assert_eq!(fetched.source_url, "https://www.pixiv.net/artworks/123");
}
}
+1 -1
View File
@@ -3,4 +3,4 @@ mod interface;
mod model;
pub use api::{PixivAPI, PixivError, disable, fetch, validate};
pub use interface::{PATTERN, Illustration, enabled, fetch_from_url};
pub use interface::{Illustration, PATTERN, enabled, fetch_from_url};
+30 -20
View File
@@ -23,7 +23,7 @@
use std::sync::LazyLock;
use serde_json::{json, Value};
use serde_json::{Value, json};
use crate::site::FetchError;
@@ -40,8 +40,7 @@ static AUTH_TOKEN: LazyLock<Option<String>> = LazyLock::new(|| {
});
/// Public "logged in" client token used by the x.com web app.
const LOGGED_IN_BEARER: &str =
"Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA";
const LOGGED_IN_BEARER: &str = "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA";
/// `TweetDetail` query id (from nazurin; still valid as of 2026-08,
/// corroborated by the current FxEmbed build — see module caveats).
@@ -103,9 +102,7 @@ pub fn enabled() -> bool {
/// Fetches a tweet as the logged-in user via the private GraphQL API.
/// Returns the syndication-shaped [`Tweet`] (media included for NSFW posts).
pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
let token = AUTH_TOKEN
.as_deref()
.ok_or(FetchError::Sensitive)?;
let token = AUTH_TOKEN.as_deref().ok_or(FetchError::Sensitive)?;
// 16 random bytes as 32 hex chars: X rejects ct0 values of any other
// length with 403 code 353 ("matching csrf cookie and header").
let ct0: String = (0..16)
@@ -129,18 +126,26 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
.header("referer", "https://x.com/")
.send()
.await?;
if !response.status().is_success() {
log::warn!("twitter auth fetch {id}: HTTP {}", response.status());
return Err(FetchError::NotFound);
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
log::warn!("twitter auth fetch {id}: HTTP {status}");
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!(
"twitter auth status {status}"
))),
};
}
let text = response.text().await?;
let json: Value = serde_json::from_str(&text)?;
let result = parse_tweet_result(&json, id)?;
let syndication_shape = to_syndication_shape(&result)
.ok_or_else(|| FetchError::Json(serde_json::Error::io(std::io::Error::new(
let syndication_shape = to_syndication_shape(&result).ok_or_else(|| {
FetchError::Json(serde_json::Error::io(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"missing tweet fields in GraphQL response",
))))?;
)))
})?;
Tweet::from_syndication_json(&syndication_shape.to_string()).map_err(FetchError::Json)
}
@@ -228,7 +233,6 @@ fn to_syndication_shape(tweet: &Value) -> Option<Value> {
"screen_name": user.get("screen_name"),
},
"possibly_sensitive": legacy.get("possibly_sensitive"),
"display_text_range": legacy.get("display_text_range"),
"entities": legacy.get("entities"),
"mediaDetails": legacy.pointer("/extended_entities/media"),
}))
@@ -251,12 +255,12 @@ mod tests {
"legacy": {
"id_str": "2083868672721039569",
"full_text": "nsfw content https://t.co/abc123",
"display_text_range": [0, 12],
"possibly_sensitive": true,
"entities": {
"urls": [
{ "url": "https://t.co/abc123", "expanded_url": "https://example.com/x" }
]
// The appended media link lives in extended_entities.media,
// not entities.urls, so it has no expansion mapping and the
// content-based strip removes it.
"urls": []
},
"extended_entities": {
"media": [
@@ -318,8 +322,11 @@ mod tests {
}
other => panic!("expected video, got {other:?}"),
}
assert_eq!(fetched.source_url, "https://x.com/nsfw_author/status/2083868672721039569");
// display_text_range cuts the trailing t.co link.
assert_eq!(
fetched.source_url,
"https://x.com/nsfw_author/status/2083868672721039569"
);
// The appended media short link (no URL-entity mapping) is stripped.
assert_eq!(fetched.title, "nsfw content");
}
@@ -331,7 +338,10 @@ mod tests {
let json = conversation(rt);
let result = parse_tweet_result(&json, "2083868672721039569").unwrap();
assert!(result.pointer("/legacy/retweeted_status_result").is_none());
assert_eq!(result.pointer("/legacy/id_str").unwrap(), "2083868672721039569");
assert_eq!(
result.pointer("/legacy/id_str").unwrap(),
"2083868672721039569"
);
}
#[test]
+117 -73
View File
@@ -1,7 +1,7 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched};
use html_escape::encode_text;
use html_escape::{encode_double_quoted_attribute, encode_text};
use regex::Regex;
use std::sync::LazyLock;
@@ -35,9 +35,7 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
}
}
} else {
log::info!(
"tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media"
);
log::info!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
Ok(empty_fetched(url))
}
}
@@ -50,7 +48,9 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
fn empty_fetched(url: &str) -> Fetched {
Fetched {
source_url: url.to_string(),
caption: url.to_string(),
// The raw user-supplied URL goes into an HTML caption; escape it so
// crafted links cannot break the parse (Telegram 400).
caption: encode_text(url).into_owned(),
title: String::new(),
media: vec![],
sensitive: true,
@@ -70,8 +70,13 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
))
.send()
.await?;
if !response.status().is_success() {
return Err(FetchError::NotFound);
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
_ => Err(FetchError::Transient(format!("twitter status {status}"))),
};
}
let text = response.text().await?;
// Deleted tweets answer with {"errors": [...]} instead of a tweet.
@@ -148,8 +153,8 @@ impl Tweet {
pub fn caption(&self) -> String {
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
url = self.url(),
author_url = self.author_url(),
url = encode_double_quoted_attribute(&self.url()),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
text = encode_text(&self.text),
)
@@ -158,12 +163,10 @@ impl Tweet {
pub fn from_syndication_json(raw_json: &str) -> Result<Self, serde_json::Error> {
let json: model::SyndicationTweet = serde_json::from_str(raw_json)?;
let id = json.id_str;
// Strip the appended media short link first, then expand the remaining
// t.co short links (the user's own URLs) to their real destinations.
let text = expand_links(
&strip_trailing_short_links(&json.text, json.display_text_range),
&json.entities.urls,
);
// Expand the user's t.co short links to their real destinations and
// strip the appended media short link, mirroring FxEmbed's linkFixer
// (no display_text_range arithmetic — see expand_links).
let text = expand_links(&json.text, &json.entities.urls);
// `name` is the display name, `screen_name` the handle (Python's
// vxtwitter mapping: author = display name, author_id = handle).
let author = json.user.name;
@@ -204,51 +207,48 @@ impl Tweet {
}
}
/// The raw syndication `text` ends with the appended media short link
/// (" https://t.co/wmI8McgXul"). `display_text_range` marks the visible text;
/// a regex strips any remaining trailing t.co link when the range is absent
/// or a tweet ends in a URL short link.
///
/// X reports these indices in Unicode **code points**, not UTF-16 units
/// (verified against GraphQL responses containing emoji: cutting an emoji
/// tweet by UTF-16 units silently drops the character after the emoji).
fn strip_trailing_short_links(text: &str, display_text_range: Option<[usize; 2]>) -> String {
let mut out = match display_text_range {
Some([start, end]) if start < end => {
text.chars().skip(start).take(end - start).collect()
}
_ => text.to_string(),
};
while TRAILING_TCO.is_match(&out) {
out = TRAILING_TCO.replace(&out, "").into_owned();
}
out
}
/// Trailing Twitter short link, optionally preceded by whitespace.
static TRAILING_TCO: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"\s*https?://t\.co/[A-Za-z0-9]+$").unwrap()
});
/// Replaces every t.co short link that has an entity mapping with its
/// expanded URL. Short links without a mapping stay untouched.
/// Mirrors FxEmbed's `linkFixer` (link-fixer.ts): expand every t.co short
/// link that has an entity mapping to its real destination, drop internal
/// `x.com/i/web/status/…` plumbing links, then strip any remaining t.co
/// short link (the appended media link and other unmapped short links).
/// Pure content matching — no `display_text_range` arithmetic, so the
/// endpoint's inconsistent index units (UTF-16 vs code points, see the
/// deleted `strip_trailing_short_links`) never matter.
fn expand_links(text: &str, urls: &[model::SyndicationEntityUrl]) -> String {
let mut out = text.to_string();
for entity in urls {
if let Some(expanded) = &entity.expanded_url {
out = out.replace(&entity.url, expanded);
}
let Some(expanded) = &entity.expanded_url else {
continue;
};
let replacement = if WEB_STATUS_URL.is_match(expanded) {
""
} else {
expanded
};
out = out.replace(&entity.url, replacement);
}
out
TCO_LINK.replace_all(&out, "").into_owned()
}
/// Internal x.com page links (reply / quote plumbing) expand to
/// `x.com/i/web/status/<id>`; FxEmbed drops them — the tweet's own content
/// already carries the information.
static WEB_STATUS_URL: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^https://(?:x\.com|twitter\.com)/i/web/status/\w+").unwrap());
/// A t.co short link, optionally preceded by a space. Any leftover
/// occurrence (unmapped — e.g. the appended media link) is removed,
/// mirroring FxEmbed. Real short-link codes are 10 alphanumerics; the
/// length-agnostic class keeps fixtures and hypothetical odd lengths safe.
static TCO_LINK: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r" ?https?://t\.co/[A-Za-z0-9]+").unwrap());
/// pbs.twimg.com serves a reduced default size without size params; `name=orig`
/// returns the original file (fxtwitter used to hand out the original
/// directly, the syndication API does not). Non-twimg URLs pass through
/// unchanged.
fn original_twimg_url(url: &str) -> String {
if url.starts_with("https://pbs.twimg.com/")
&& (url.ends_with(".jpg") || url.ends_with(".png"))
if url.starts_with("https://pbs.twimg.com/") && (url.ends_with(".jpg") || url.ends_with(".png"))
{
format!("{url}?name=orig")
} else {
@@ -363,24 +363,23 @@ mod tests {
match &fetched.media[0] {
Media::Illustration { url, .. } => {
// Photo URL is rewritten to request the original file.
assert_eq!(
url,
"https://pbs.twimg.com/media/photo.jpg?name=orig"
);
assert_eq!(url, "https://pbs.twimg.com/media/photo.jpg?name=orig");
}
other => panic!("expected illustration, got {other:?}"),
}
match &fetched.media[1] {
Media::Video { url, thumbnail_url, .. } => {
Media::Video {
url, thumbnail_url, ..
} => {
assert_eq!(url, "https://video.twimg.com/v.mp4");
assert_eq!(thumbnail_url, "https://pbs.twimg.com/thumb.jpg");
}
other => panic!("expected video, got {other:?}"),
}
assert!(
fetched
.caption
.contains("<a href=\"https://x.com/author_handle\">Display Name</a>: a &amp; b &lt;c&gt;"),
fetched.caption.contains(
"<a href=\"https://x.com/author_handle\">Display Name</a>: a &amp; b &lt;c&gt;"
),
"caption: {}",
fetched.caption
);
@@ -411,13 +410,12 @@ mod tests {
#[test]
fn syndication_text_strips_trailing_media_short_link() {
// Real syndication shape: the media short link sits after the visible
// text, and display_text_range marks where it begins.
// Real syndication shape: the appended media short link sits after the
// visible text; the unmapped t.co link is stripped by content.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": "hello world https://t.co/abc123",
"display_text_range": [0, 11],
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
@@ -427,8 +425,31 @@ mod tests {
}
#[test]
fn syndication_text_strips_trailing_short_link_without_range() {
// No display_text_range: the regex fallback removes the trailing link.
fn syndication_text_strips_trailing_link_regardless_of_index_units() {
// Real tweet 2084567054481571919: the visible text is 30 code points
// but 41 UTF-16 units, and the two endpoints historically reported
// display_text_range in different units (UTF-16 on syndication, code
// points on GraphQL). The FxEmbed-style content-based strip ignores
// the range entirely, so the appended media link is removed for any
// response shape.
let text = "妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero https://t.co/XnIi83EkEB";
let visible = "妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero";
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "2084567054481571919",
"text": text,
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, visible, "left a partial link");
assert!(!tweet.caption().contains("t.co"));
}
#[test]
fn syndication_text_strips_trailing_short_link_without_entities() {
// No URL entities at all: the leftover t.co link is stripped by the
// content regex.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
@@ -449,7 +470,6 @@ mod tests {
"__typename": "Tweet",
"id_str": "1",
"text": "Test Tweet with @mentionThis $twtr https://t.co/RzmrQ6wAzD #hashtag https://t.co/9r69akA484",
"display_text_range": [0, 67],
"user": { "name": "N", "screen_name": "h" },
"entities": {
"urls": [{
@@ -469,25 +489,47 @@ mod tests {
}
#[test]
fn syndication_text_keeps_unmapped_short_links() {
// No entity mapping for the embedded link: it stays as-is. Only the
// trailing media link is stripped.
fn syndication_text_strips_unmapped_short_links() {
// FxEmbed parity: short links without an entity mapping (appended
// media link, embedded unmapped links) are stripped, not kept.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": "check https://t.co/abc123 #tag https://t.co/def456",
"display_text_range": [0, 30],
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, "check https://t.co/abc123 #tag");
assert_eq!(tweet.text, "check #tag");
}
#[test]
fn syndication_text_utf16_display_range_keeps_multibyte() {
// display_text_range is in UTF-16 units; a Japanese text must not be
// sliced by UTF-8 bytes.
fn syndication_text_drops_internal_web_status_links() {
// FxEmbed parity: a mapped link expanding to an internal
// x.com/i/web/status/... page (reply/quote plumbing) is removed
// instead of being shown.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": "see https://t.co/xyz1234567 for context",
"user": { "name": "N", "screen_name": "h" },
"entities": {
"urls": [{
"url": "https://t.co/xyz1234567",
"expanded_url": "https://x.com/i/web/status/9876543210",
"display_url": "x.com/i/web/status/9876543210"
}]
},
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, "see for context");
assert!(!tweet.caption().contains("t.co"));
}
#[test]
fn syndication_text_keeps_multibyte_text() {
// Text-only tweet: no short links, the multibyte text is untouched.
let text = "コミティア落ちたので、明日は行きません。🙏ごめんなさい";
let units: Vec<u16> = text.encode_utf16().collect();
assert_eq!(units.len(), 28);
@@ -495,7 +537,6 @@ mod tests {
"__typename": "Tweet",
"id_str": "1",
"text": text,
"display_text_range": [0, 28],
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
@@ -547,6 +588,9 @@ mod tests {
async fn live_fetch_deleted_tweet_is_not_found() {
// Deleted tweet: the syndication endpoint answers with errors.
let result = fetch("0").await;
assert!(matches!(result, Err(FetchError::NotFound)), "got {result:?}");
assert!(
matches!(result, Err(FetchError::NotFound)),
"got {result:?}"
);
}
}
-4
View File
@@ -9,10 +9,6 @@ pub struct SyndicationTweet {
pub user: SyndicationUser,
#[serde(default)]
pub possibly_sensitive: Option<bool>,
/// Visible-text span; the raw `text` field has the appended media short
/// link after it. Indices are Unicode code points (not UTF-16 units).
#[serde(default, rename = "display_text_range")]
pub display_text_range: Option<[usize; 2]>,
#[serde(default)]
pub entities: SyndicationEntities,
#[serde(default, rename = "mediaDetails")]
+1 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "xmedia-bot"
version = "1.0.7"
version = "1.1.0"
edition = "2024"
[dependencies]
@@ -12,7 +12,6 @@ log = "0.4"
pretty_env_logger = "0.5"
dotenv = "0.15"
url = "2.5.2"
regex = "1.12"
html-escape = "0.2"
rusqlite = { version = "0.32", features = ["bundled"] }
rand = "0.8"
+28 -30
View File
@@ -24,39 +24,37 @@ pub struct Config {
impl Config {
pub fn load() -> Config {
let admin_ids = env::var("BOT_ADMIN")
.ok()
.map(|s| {
s.split(',')
.filter_map(|part| part.trim().parse::<i64>().ok())
.collect()
})
.unwrap_or_default();
.ok()
.map(|s| {
s.split(',')
.filter_map(|part| part.trim().parse::<i64>().ok())
.collect()
})
.unwrap_or_default();
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(86400));
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(86400));
let link_cache_ttl = env::var("LINK_CACHE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(7 * 24 * 3600));
let link_cache_ttl = env::var("LINK_CACHE_TTL_SECONDS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.map(Duration::from_secs)
.unwrap_or(Duration::from_secs(7 * 24 * 3600));
let webhook_enabled = env::var("WEBHOOK")
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
// value that would otherwise come from `.env`).
let webhook_cert = env::var("WEBHOOK_CERT")
.ok()
.filter(|s| !s.is_empty());
let webhook_secret_token = env::var("WEBHOOK_SECRET_TOKEN")
.ok()
.filter(|s| !s.is_empty());
let webhook_enabled = env::var("WEBHOOK")
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
// value that would otherwise come from `.env`).
let webhook_cert = env::var("WEBHOOK_CERT").ok().filter(|s| !s.is_empty());
let webhook_secret_token = env::var("WEBHOOK_SECRET_TOKEN")
.ok()
.filter(|s| !s.is_empty());
Config {
admin_ids,
+50
View File
@@ -0,0 +1,50 @@
//! Shared SQLite plumbing for the three tables in `data/task_queue.db`
//! (`tasks` in queue.rs, `chat_state` in state.rs, `link_cache` in
//! link_cache.rs).
//!
//! Every operation opens its own short-lived connection with a busy timeout:
//! handler tasks enqueue while workers lease/update rows concurrently, and
//! without the timeout a concurrent write fails immediately with SQLITE_BUSY
//! and the operation is lost. All I/O runs inside `spawn_blocking` via
//! [`with_conn`] — rusqlite connections are not Send-friendly to hold across
//! an await point, and blocking the async executor stalls every handler.
use rusqlite::Connection;
use std::time::Duration;
/// Opens the shared DB with a busy timeout.
pub fn open_db(path: &str) -> rusqlite::Result<Connection> {
let conn = Connection::open(path)?;
conn.busy_timeout(Duration::from_secs(5))?;
// WAL lets readers run alongside writer leases instead of blocking on
// the rollback journal; the mode persists in the DB header, so the
// idempotent pragma here and in ensure_schema only needs to win once.
conn.pragma_update(None, "journal_mode", "WAL")?;
Ok(conn)
}
/// Unix timestamp in fractional seconds. Shared by the queue, chat store and
/// link cache (previously four private copies).
pub fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
}
/// Runs `f` against a fresh connection on a blocking thread, returning the
/// closure's result. Owns the `spawn_blocking` + `expect` ceremony shared by
/// every table access; the caller maps errors to its own log line.
pub async fn with_conn<T, F>(path: &str, f: F) -> rusqlite::Result<T>
where
T: Send + 'static,
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
{
let path = path.to_string();
tokio::task::spawn_blocking(move || {
let mut conn = open_db(&path)?;
f(&mut conn)
})
.await
.expect("db worker panicked")
}
+276 -101
View File
@@ -1,57 +1,105 @@
use crate::config::Config;
use crate::db::now_f64;
use crate::link_cache::{CachedMediaKind, CachedPost, LinkCache};
use crate::queue::PersistentTaskQueue;
use crate::send::{self, MediaItemPayload, Task};
use crate::state::{ChatData, ChatStore, unix_now};
use std::collections::HashSet;
use std::sync::LazyLock;
use teloxide::RequestError;
use teloxide::prelude::*;
use tokio::sync::Semaphore;
use teloxide::types::{
CallbackQuery, ChatAction, ChatId, ChatKind, InlineQuery, InlineQueryResult,
InlineQueryResultMpeg4Gif, InlineQueryResultPhoto, InlineQueryResultVideo, Message,
MessageEntityKind, MessageId, ParseMode, Recipient, ReplyParameters,
};
use teloxide::utils::command::BotCommands;
use teloxide::RequestError;
use x_media::media::Media;
pub static CHAT_STORE: LazyLock<ChatStore> = LazyLock::new(|| {
ChatStore::open("data/task_queue.db").expect("failed to open chat store")
});
/// One URL job: bot handle + the message + the extracted URL.
type UrlJob = (Bot, Message, String);
/// Bounded channel of URL jobs drained by [`start_url_workers`]. The bound
/// caps both queued memory and shutdown backlog; a full channel applies
/// backpressure to the per-chat handler instead of spawning unbounded tasks.
static URL_JOBS: LazyLock<parking_lot::Mutex<Option<tokio::sync::mpsc::Sender<UrlJob>>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
/// Set by main's shutdown sequence; workers stop pulling new jobs.
static URL_STOP: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
/// Worker count draining URL jobs; keeps the old 8-permit concurrency cap
/// while bounding how many jobs can be queued at all.
const URL_WORKERS: usize = 8;
/// Starts the URL job workers (called once from main after the queue starts).
/// teloxide dispatches updates to a per-chat worker that handles them
/// sequentially, so a batch-forward of many messages would otherwise be
/// processed one at a time (fetch + send each, roughly a second per
/// message); the workers add throughput, and FIFO order preserves per-message
/// URL order.
pub async fn start_url_workers() {
let (tx, rx) = tokio::sync::mpsc::channel::<UrlJob>(256);
*URL_JOBS.lock() = Some(tx);
let rx = std::sync::Arc::new(tokio::sync::Mutex::new(rx));
for _ in 0..URL_WORKERS {
let rx = std::sync::Arc::clone(&rx);
tokio::spawn(async move {
while !URL_STOP.load(std::sync::atomic::Ordering::Relaxed) {
let job = rx.lock().await.recv().await;
match job {
Some((bot, message, url)) => url_media(bot, &message, &url).await,
None => break,
}
}
});
}
}
/// Stops URL workers (drains up to the 256 queued jobs, then exits).
pub fn stop_url_workers() {
URL_STOP.store(true, std::sync::atomic::Ordering::Relaxed);
}
pub static CHAT_STORE: LazyLock<ChatStore> =
LazyLock::new(|| ChatStore::open("data/task_queue.db").expect("failed to open chat store"));
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
LazyLock::new(|| PersistentTaskQueue::new("data/task_queue.db"));
pub static LINK_CACHE: LazyLock<LinkCache> =
LazyLock::new(|| LinkCache::open("data/task_queue.db"));
pub static CONFIG: LazyLock<Config> = LazyLock::new(Config::load);
/// Cap on concurrent per-URL processing. teloxide dispatches updates to a
/// per-chat worker that handles them sequentially, so a batch-forward of many
/// messages would otherwise be processed one at a time (fetch + send each,
/// roughly a second per message). Moving the work into spawned tasks trades
/// per-chat reply ordering for throughput; the semaphore bounds how many run
/// at once so a big burst cannot hammer Telegram's rate limits.
static URL_TASKS: LazyLock<Semaphore> = LazyLock::new(|| Semaphore::new(8));
#[derive(BotCommands, Clone)]
#[command(rename_rule = "snake_case", description = "")]
#[command(
rename_rule = "snake_case",
description = "Turn X/Pixiv/Bluesky links into media messages"
)]
enum Command {
#[command(description = "")]
#[command(description = "Get started")]
Start,
#[command(description = "")]
#[command(description = "Show command help")]
Help,
#[command(description = "", parse_with = "split")]
#[command(
description = "Set forward channel (@channel or ID)",
parse_with = "split"
)]
SetForwardChannel(String),
#[command(description = "")]
#[command(description = "Remove forward channel")]
RemoveForwardChannel,
#[command(description = "")]
#[command(description = "Toggle edit-before-forward")]
EditBeforeForward,
#[command(description = "", parse_with = "split")]
#[command(
description = "Reply with [] to save as template",
parse_with = "split"
)]
SetTemplate(String),
#[command(description = "")]
#[command(description = "Show chat state (debug)")]
BotDict,
#[command(description = "", parse_with = "split")]
#[command(description = "Set site caption format", parse_with = "split")]
SetFormat(String),
#[command(
description = "Clear link cache (admin; optional URL, else all)",
parse_with = "split"
)]
ClearCache(String),
}
async fn reply<T>(bot: Bot, message: Message, text: T) -> Result<Message, RequestError>
@@ -63,13 +111,6 @@ where
.await
}
fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
}
/// Extracts URL and text-link entities (text + caption), deduped in order.
pub fn extract_urls(message: &Message) -> Vec<String> {
let mut urls = Vec::new();
@@ -88,7 +129,10 @@ pub fn extract_urls(message: &Message) -> Vec<String> {
}
}
let mut seen = HashSet::new();
urls.retain(|url| seen.insert(url.clone()));
// Dedup by the normalized post id so variant URLs of the same post
// (/status/1 vs /status/1/photo/1) are sent once; unsupported URLs fall
// back to exact-string dedup.
urls.retain(|url| seen.insert(x_media::site::cache_key(url).unwrap_or_else(|| url.clone())));
urls
}
@@ -111,7 +155,7 @@ async fn edit_message_handler(bot: &Bot, message: &Message) -> bool {
};
let link = format!(
"<a href=\"{0}\">{1}</a>",
edit.url,
html_escape::encode_double_quoted_attribute(&edit.url),
html_escape::encode_text(text)
);
let new_text = if edit.template.is_empty() {
@@ -177,19 +221,31 @@ async fn set_forward_channel_handler(
return Err(SetForwardChannelError::NotChannel);
}
let channel_id = chat.id.0;
// The sender must be a channel administrator. Compare against the
// sender's user id, NOT the chat id (they only coincide in private
// chats, so the old check broke group usage).
let Some(sender) = message.from.as_ref() else {
return Err(SetForwardChannelError::NotAdmin);
};
match bot.get_chat_administrators(channel.clone()).await {
Err(e) => {
log::error!("Failed to get channel administrators {}: {}", channel, e);
return Err(SetForwardChannelError::NotBotAdmin(e));
}
Ok(admins) => {
if !admins.iter().any(|admin| admin.user.id == message.chat.id) {
if !admins.iter().any(|admin| admin.user.id == sender.id) {
return Err(SetForwardChannelError::NotAdmin);
}
let bot_id = bot.get_me().await.expect("Failed get bot id").user.id;
if let Some(bot_admin) = admins.iter().find(|admin| admin.user.id == bot_id)
&& !bot_admin.can_post_messages()
{
// The bot itself must be an admin that can post; a missing
// bot entry must not pass silently (copy would fail later).
let bot_id = match bot.get_me().await {
Ok(me) => me.user.id,
Err(e) => return Err(SetForwardChannelError::NotBotAdmin(e)),
};
let bot_ok = admins
.iter()
.any(|admin| admin.user.id == bot_id && admin.can_post_messages());
if !bot_ok {
return Err(SetForwardChannelError::NotBotCanPost);
}
}
@@ -197,7 +253,11 @@ async fn set_forward_channel_handler(
Ok(channel_id)
}
async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Result<(), RequestError> {
async fn execute_command(
bot: &Bot,
message: &Message,
command: Command,
) -> Result<(), RequestError> {
match command {
Command::Start => {
bot.send_message(message.chat.id, "Hello!").await?;
@@ -209,13 +269,16 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
Command::SetForwardChannel(channel) => {
let result = match set_forward_channel_handler(bot, message, channel).await {
Ok(channel_id) => {
let mut chat_data = CHAT_STORE.get(message.chat.id.0).await;
chat_data.forward_channel_id = Some(channel_id);
CHAT_STORE.set(message.chat.id.0, &chat_data).await;
CHAT_STORE
.update(message.chat.id.0, |data| {
data.forward_channel_id = Some(channel_id);
})
.await;
"Add successfully.".to_string()
}
Err(SetForwardChannelError::EmptyParameter) => {
"Receive empty parameter.\nYou should enter a channel id or username".to_string()
"Receive empty parameter.\nYou should enter a channel id or username"
.to_string()
}
Err(SetForwardChannelError::NotChannel) => {
"Given id / username is not a channel".to_string()
@@ -234,31 +297,34 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
}
Command::RemoveForwardChannel => {
let chat_id = message.chat.id.0;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let text = if chat_data.forward_channel_id.is_some() {
chat_data.forward_channel_id = None;
CHAT_STORE.set(chat_id, &chat_data).await;
"Remove successfully.".to_string()
} else {
"No channel to remove.".to_string()
};
let text = CHAT_STORE
.update(chat_id, |data| {
if data.forward_channel_id.is_some() {
data.forward_channel_id = None;
"Remove successfully.".to_string()
} else {
"No channel to remove.".to_string()
}
})
.await;
reply(bot.clone(), message.clone(), text).await?;
}
Command::EditBeforeForward => {
let chat_id = message.chat.id.0;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let text = if chat_data.forward_channel_id.is_none() {
"Please enable forward channel first.".to_string()
} else if chat_data.edit_before_forward {
chat_data.edit_before_forward = false;
chat_data.edit_message.clear();
CHAT_STORE.set(chat_id, &chat_data).await;
"Disable edit before forward.".to_string()
} else {
chat_data.edit_before_forward = true;
CHAT_STORE.set(chat_id, &chat_data).await;
"Enable edit before forward.".to_string()
};
let text = CHAT_STORE
.update(chat_id, |data| {
if data.forward_channel_id.is_none() {
"Please enable forward channel first.".to_string()
} else if data.edit_before_forward {
data.edit_before_forward = false;
data.edit_message.clear();
"Disable edit before forward.".to_string()
} else {
data.edit_before_forward = true;
"Enable edit before forward.".to_string()
}
})
.await;
reply(bot.clone(), message.clone(), text).await?;
}
Command::SetTemplate(name) => {
@@ -272,11 +338,14 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
} else if name.is_empty() {
"Please provide a name for the template.".to_string()
} else {
let mut chat_data = CHAT_STORE.get(chat_id).await;
chat_data
.template
.insert(name, html_escape::encode_text(reply_text).into_owned());
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.template.insert(
name,
html_escape::encode_text(reply_text).into_owned(),
);
})
.await;
"Template set.".to_string()
}
}
@@ -292,7 +361,9 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
Command::SetFormat(arg) => {
let chat_id = message.chat.id.0;
let (site, format) = match arg.split_once(char::is_whitespace) {
Some((site, format)) if !format.trim().is_empty() => (site.trim(), format.trim().to_string()),
Some((site, format)) if !format.trim().is_empty() => {
(site.trim(), format.trim().to_string())
}
_ => {
reply(
bot.clone(),
@@ -312,15 +383,78 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
.await?;
return Ok(());
}
let mut chat_data = CHAT_STORE.get(chat_id).await;
chat_data.message_format.insert(site.to_string(), format);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.message_format.insert(site.to_string(), format);
})
.await;
reply(bot.clone(), message.clone(), "Format set.").await?;
}
Command::ClearCache(arg) => {
let sender_id = message
.from
.as_ref()
.map(|user| user.id.0 as i64)
.unwrap_or(-1);
if !CONFIG.admin_ids.contains(&sender_id) {
reply(bot.clone(), message.clone(), "Admin only.").await?;
return Ok(());
}
let arg = arg.trim();
if arg.is_empty() {
let removed = LINK_CACHE.clear(None).await;
log::info!("cache cleared by {sender_id}: {removed} entries");
reply(
bot.clone(),
message.clone(),
format!("Cleared {removed} cached entr{}.", plural(removed)),
)
.await?;
} else {
let key = match x_media::site::cache_key(arg) {
Some(key) => key,
None => {
reply(
bot.clone(),
message.clone(),
"Unrecognized link. Use a twitter/x, pixiv or bsky post URL.",
)
.await?;
return Ok(());
}
};
let removed = LINK_CACHE.clear(Some(&key)).await;
log::info!("cache entry cleared by {sender_id}: {key} ({removed} rows)");
reply(
bot.clone(),
message.clone(),
format!(
"Cleared cache for {arg} ({} entr{}).",
removed,
plural(removed)
),
)
.await?;
}
}
}
Ok(())
}
/// `"y"` for one, `"ies"` for anything else — "1 entry" / "2 entries".
fn plural(n: usize) -> &'static str {
if n == 1 { "y" } else { "ies" }
}
/// Registers the bot's command list with Telegram so clients show it in the
/// `/` menu (Bot API `setMyCommands`).
pub async fn register_commands(bot: &Bot) -> Result<(), RequestError> {
let commands = Command::bot_commands();
bot.set_my_commands(commands.clone()).await?;
log::info!("registered {} commands", commands.len());
Ok(())
}
/// For locally produced media (encoded ugoira MP4) the thumbnail URL is a
/// hotlink-protected remote URL Telegram may not fetch; let Telegram generate
/// its own thumbnail instead.
@@ -383,7 +517,10 @@ async fn dispatch_send(bot: Bot, message: &Message, task: &Task, url: &str) {
log::info!("sent {} message(s) for {url}", message_ids.len());
send::post_send_actions(&bot, task, message_ids).await;
}
Err(send::SendError::Retryable { delay_seconds, task }) => {
Err(send::SendError::Retryable {
delay_seconds,
task,
}) => {
log::info!("send for {url} failed, queued for retry in {delay_seconds:.1}s");
enqueue_retry(task, delay_seconds).await;
let _ = reply(bot, message.clone(), "Send failed. Task queued for retry.").await;
@@ -444,7 +581,10 @@ fn build_send_task(
async fn url_media(bot: Bot, message: &Message, url: &str) {
let chat_id = message.chat.id.0;
if let Err(e) = bot.send_chat_action(ChatId(chat_id), ChatAction::Typing).await {
if let Err(e) = bot
.send_chat_action(ChatId(chat_id), ChatAction::Typing)
.await
{
log::error!("send_chat_action failed: {e}");
}
@@ -520,7 +660,12 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
// Retries exhausted: notify the user (Rust-only requirement 3).
Err(e) => {
log::error!("fetch {url}: {e}");
let _ = reply(bot, message.clone(), "Failed to fetch media from this link.").await;
let _ = reply(
bot,
message.clone(),
"Failed to fetch media from this link.",
)
.await;
}
Ok(Some(fetched)) => {
if fetched.media.is_empty() {
@@ -542,8 +687,9 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
let caption = fetched.caption_with(&format);
// Raw render data for the link cache; the send fills in the
// Telegram file ids and persists the entry.
let cache_data = fetched.render_fields().map(|(author, author_url, title, tags)| {
CachedPost {
let cache_data = fetched
.render_fields()
.map(|(author, author_url, title, tags)| CachedPost {
url: fetched.source_url.clone(),
caption: fetched.caption.clone(),
title: title.to_string(),
@@ -552,8 +698,7 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
tags: tags.to_string(),
sensitive: fetched.sensitive,
media: vec![],
}
});
});
let items: Vec<MediaItemPayload> = fetched
.media
.iter()
@@ -581,9 +726,15 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
.unwrap_or_else(|| "unknown".to_string());
let text_preview = message
.text()
.map(|t| if t.len() > 120 { &t[..120] } else { t })
.map(|t| {
let end = t.floor_char_boundary(120.min(t.len()));
&t[..end]
})
.unwrap_or("<no text>");
log::info!("message from {sender} in {} (private={is_private}): {text_preview}", message.chat.id);
log::info!(
"message from {sender} in {} (private={is_private}): {text_preview}",
message.chat.id
);
// URL/edit flows only run in private chats; commands run in any chat.
if is_private && edit_message_handler(&bot, &message).await {
return respond(());
@@ -601,13 +752,13 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
log::info!("extracted {} URL(s): {urls:?}", urls.len());
}
for url in urls {
let bot = bot.clone();
let message = message.clone();
tokio::spawn(async move {
// Held for the whole task; the semaphore is never closed.
let _permit = URL_TASKS.acquire().await.expect("URL semaphore closed");
url_media(bot, &message, &url).await;
});
// Clone out of the lock: the parking_lot guard is !Send and must
// not be held across the await below.
let Some(tx) = URL_JOBS.lock().clone() else {
log::warn!("url workers not started; dropping link");
break;
};
let _ = tx.send((bot.clone(), message.clone(), url)).await;
}
}
respond(())
@@ -617,6 +768,12 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
if query.query.is_empty() {
return respond(());
}
// Telegram fires an inline query on every keystroke; only run a fetch
// (3 attempts!) for something that is actually a supported post URL, so
// typing does not hammer the source sites.
if x_media::site::cache_key(&query.query).is_none() {
return respond(());
}
log::info!("inline query: {}", query.query);
match x_media::site::fetch(&query.query).await {
Ok(Some(fetched)) => {
@@ -653,8 +810,8 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
thumbnail,
fetched.title.clone(),
)
.caption(caption)
.parse_mode(ParseMode::Html),
.caption(caption)
.parse_mode(ParseMode::Html),
),
Media::Animated { .. } => InlineQueryResult::Mpeg4Gif(
InlineQueryResultMpeg4Gif::new(id, url, thumbnail)
@@ -683,10 +840,13 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let chat_id = message.chat().id.0;
let prompt_message_id = message.id().0 as i64;
let ttl_secs = CONFIG.edit_message_ttl.as_secs() as i64;
let mut chat_data = CHAT_STORE.get(chat_id).await;
let chat_data = CHAT_STORE.get(chat_id).await;
let edit = chat_data.edit_message.get(&prompt_message_id).cloned();
let Some(edit) = edit else {
log::info!("callback from {}: no edit record for prompt {prompt_message_id}", chat_id);
log::info!(
"callback from {}: no edit record for prompt {prompt_message_id}",
chat_id
);
bot.answer_callback_query(callback_query_id)
.text("Expired")
.await?;
@@ -694,8 +854,11 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
};
// Lazy expiry: a stale record (past the TTL, not yet swept) is dropped.
if edit.created_at + ttl_secs <= unix_now() {
chat_data.edit_message.remove(&prompt_message_id);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
bot.answer_callback_query(callback_query_id)
.text("Expired")
.await?;
@@ -705,7 +868,10 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let Some(data) = data else {
return respond(());
};
log::info!("callback from {} on prompt {prompt_message_id}: {data}", chat_id);
log::info!(
"callback from {} on prompt {prompt_message_id}: {data}",
chat_id
);
if data == "forward" {
match chat_data.forward_channel_id {
Some(channel_id) => {
@@ -728,10 +894,16 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
let _ = bot
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
.await;
chat_data.edit_message.remove(&prompt_message_id);
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
}
Err(send::SendError::Retryable { delay_seconds, task }) => {
Err(send::SendError::Retryable {
delay_seconds,
task,
}) => {
log::info!("forward queued for retry in {delay_seconds:.1}s");
enqueue_retry(task, delay_seconds).await;
bot.answer_callback_query(callback_query_id)
@@ -765,10 +937,13 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
.caption(template_html)
.parse_mode(ParseMode::Html)
.await;
if let Some(entry) = chat_data.edit_message.get_mut(&prompt_message_id) {
entry.template = name.to_string();
}
CHAT_STORE.set(chat_id, &chat_data).await;
CHAT_STORE
.update(chat_id, |data| {
if let Some(entry) = data.edit_message.get_mut(&prompt_message_id) {
entry.template = name.to_string();
}
})
.await;
log::info!("template '{name}' applied to prompt {prompt_message_id}");
}
bot.answer_callback_query(callback_query_id).await?;
+113 -56
View File
@@ -8,7 +8,8 @@
//! [`Config::link_cache_ttl`]; a stale entry is dropped lazily on read and
//! by the periodic prune in `main`.
use rusqlite::{params, Connection};
use crate::db::now_f64;
use rusqlite::{Connection, params};
use serde::{Deserialize, Serialize};
use std::time::Duration;
@@ -49,12 +50,6 @@ pub struct LinkCache {
db_path: String,
}
fn open_db(path: &str) -> rusqlite::Result<Connection> {
let conn = Connection::open(path)?;
conn.busy_timeout(Duration::from_secs(5))?;
Ok(conn)
}
impl LinkCache {
pub fn open(db_path: &str) -> Self {
if let Ok(conn) = Connection::open(db_path)
@@ -73,11 +68,9 @@ impl LinkCache {
/// Returns the cached post if present and not expired; a stale entry is
/// removed on the spot.
pub async fn get(&self, key: &str, ttl: Duration) -> Option<CachedPost> {
let db_path = self.db_path.clone();
let key = key.to_string();
let ttl = ttl.as_secs_f64();
tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<CachedPost>> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
let mut stmt =
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
let mut rows = stmt.query(params![key])?;
@@ -90,74 +83,85 @@ impl LinkCache {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
return Ok(None);
}
serde_json::from_str(&payload).map(Some).map_err(|e| {
rusqlite::Error::ToSqlConversionFailure(Box::new(e))
})
})
.await
.expect("link cache read worker panicked")
.unwrap_or_else(|e| {
log::error!("link cache read failed: {e}");
None
Ok(Some(serde_json::from_str::<CachedPost>(&payload).map_err(
|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)),
)?))
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::error!("link cache read failed: {e}");
None
}
}
}
pub async fn put(&self, key: &str, post: &CachedPost) {
let db_path = self.db_path.clone();
let key = key.to_string();
let payload = serde_json::to_string(post).expect("cached post serializes");
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"INSERT OR REPLACE INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
params![key, payload, now_f64()],
)?;
Ok(())
})
.await
.expect("link cache write worker panicked")
.unwrap_or_else(|e| log::error!("link cache write failed: {e}"));
.await;
if let Err(e) = result {
log::error!("link cache write failed: {e}");
}
}
/// Drops an entry (e.g. a cached file id that turned out invalid).
pub async fn remove(&self, key: &str) {
let db_path = self.db_path.clone();
let key = key.to_string();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Ok(())
})
.await
.expect("link cache delete worker panicked")
.unwrap_or_else(|e| log::error!("link cache delete failed: {e}"));
.await;
if let Err(e) = result {
log::error!("link cache delete failed: {e}");
}
}
/// Removes expired entries; returns how many were deleted.
pub async fn prune(&self, ttl: Duration) -> usize {
let db_path = self.db_path.clone();
let cutoff = now_f64() - ttl.as_secs_f64();
tokio::task::spawn_blocking(move || -> rusqlite::Result<usize> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"DELETE FROM link_cache WHERE created_at < ?1",
params![cutoff],
)
})
.await
.expect("link cache prune worker panicked")
.unwrap_or_else(|e| {
log::error!("link cache prune failed: {e}");
0
})
.await;
match result {
Ok(n) => n,
Err(e) => {
log::error!("link cache prune failed: {e}");
0
}
}
}
}
fn now_f64() -> f64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
/// Deletes one entry (by normalized cache key) or the whole cache when
/// `key` is `None`. Returns how many rows were removed.
pub async fn clear(&self, key: Option<&str>) -> usize {
let key = key.map(str::to_string);
let result = crate::db::with_conn(&self.db_path, move |conn| match &key {
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
None => conn.execute("DELETE FROM link_cache", []),
})
.await;
match result {
Ok(n) => n,
Err(e) => {
log::error!("link cache clear failed: {e}");
0
}
}
}
}
#[cfg(test)]
@@ -200,14 +204,21 @@ mod tests {
// Force the row into the past so a 1s TTL expires it.
{
let conn = Connection::open(dir.path().join("c.db")).unwrap();
conn.execute(
"UPDATE link_cache SET created_at = created_at - 100",
[],
)
.unwrap();
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
.unwrap();
}
assert!(cache.get("twitter:1", Duration::from_secs(1)).await.is_none());
assert!(cache.get("twitter:1", Duration::from_secs(3600)).await.is_none());
assert!(
cache
.get("twitter:1", Duration::from_secs(1))
.await
.is_none()
);
assert!(
cache
.get("twitter:1", Duration::from_secs(3600))
.await
.is_none()
);
}
#[tokio::test]
@@ -217,14 +228,60 @@ mod tests {
cache.put("twitter:1", &entry()).await;
cache.put("pixiv:2", &entry()).await;
cache.remove("twitter:1").await;
assert!(cache.get("twitter:1", Duration::from_secs(3600)).await.is_none());
assert!(cache.get("pixiv:2", Duration::from_secs(3600)).await.is_some());
assert!(
cache
.get("twitter:1", Duration::from_secs(3600))
.await
.is_none()
);
assert!(
cache
.get("pixiv:2", Duration::from_secs(3600))
.await
.is_some()
);
{
let conn = Connection::open(dir.path().join("c.db")).unwrap();
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
.unwrap();
}
assert_eq!(cache.prune(Duration::from_secs(1)).await, 1);
assert!(cache.get("pixiv:2", Duration::from_secs(3600)).await.is_none());
assert!(
cache
.get("pixiv:2", Duration::from_secs(3600))
.await
.is_none()
);
}
#[tokio::test]
async fn clear_one_entry_or_all() {
let dir = tempfile::tempdir().unwrap();
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
cache.put("twitter:1", &entry()).await;
cache.put("pixiv:2", &entry()).await;
// By key: only the matching row is removed.
assert_eq!(cache.clear(Some("twitter:1")).await, 1);
assert!(
cache
.get("twitter:1", Duration::from_secs(3600))
.await
.is_none()
);
assert!(
cache
.get("pixiv:2", Duration::from_secs(3600))
.await
.is_some()
);
// Whole cache: nothing left; removing an absent key deletes 0 rows.
assert_eq!(cache.clear(None).await, 1);
assert!(
cache
.get("pixiv:2", Duration::from_secs(3600))
.await
.is_none()
);
assert_eq!(cache.clear(None).await, 0);
}
}
+25 -8
View File
@@ -1,13 +1,14 @@
use dotenv::dotenv;
use teloxide::dptree::endpoint;
use teloxide::prelude::*;
use teloxide::stop::StopToken;
use teloxide::types::{ChatId, InputFile, MessageId};
use teloxide::update_listeners::{self, webhooks, UpdateListener};
use teloxide::prelude::*;
use teloxide::update_listeners::{self, UpdateListener, webhooks};
use tokio::sync::watch;
use x_media::site;
mod config;
mod db;
mod handlers;
mod link_cache;
mod photo;
@@ -42,6 +43,14 @@ async fn main() {
log::info!("Starting bot");
let bot = Bot::from_env();
// Force the queue workers' shared Bot to initialize now so a missing
// token fails at startup, not on the first queued task.
let _ = &*send::BOT;
// Register the command list with Telegram (client `/` menu).
if let Err(e) = handlers::register_commands(&bot).await {
log::warn!("failed to register commands: {e}");
}
log::info!(
"config: {} admin(s), edit-message TTL {}s",
@@ -56,6 +65,10 @@ async fn main() {
.await;
log::info!("task queue worker started");
// URL job workers: bounded channel + fixed pool for per-URL work.
handlers::start_url_workers().await;
log::info!("url workers started");
// Pixiv login validation (user request): a failed login notifies the
// admin and disables pixiv for this process.
if site::pixiv::enabled() {
@@ -74,7 +87,10 @@ async fn main() {
}
// Edit-expiry sweep: clears the prompt's buttons once the record expires.
log::info!("edit-expiry sweep: every 300s, ttl {}", CONFIG.edit_message_ttl.as_secs());
log::info!(
"edit-expiry sweep: every 300s, ttl {}",
CONFIG.edit_message_ttl.as_secs()
);
let (stop_tx, stop_rx) = watch::channel(false);
{
let bot = bot.clone();
@@ -95,7 +111,10 @@ async fn main() {
// If the prompt was already deleted, this fails with a
// 400 "message to edit not found" — log and ignore.
if let Err(e) = bot
.edit_message_reply_markup(ChatId(chat_id), MessageId(prompt_message_id as i32))
.edit_message_reply_markup(
ChatId(chat_id),
MessageId(prompt_message_id as i32),
)
.await
{
log::info!("edit-expiry sweep: prompt message gone: {e}");
@@ -117,10 +136,7 @@ async fn main() {
if CONFIG.webhook_enabled {
log::info!("running in webhook mode");
let url = CONFIG
.webhook_url
.clone()
.expect("WEBHOOK_URL is not set");
let url = CONFIG.webhook_url.clone().expect("WEBHOOK_URL is not set");
// `webhooks::axum` calls set_webhook itself (with the full options,
// secret token included) — no explicit registration here.
let listen = CONFIG.webhook_listen.expect("WEBHOOK_LISTEN is not set");
@@ -165,6 +181,7 @@ async fn main() {
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin, drain the queue.
log::info!("Stopping bot");
let _ = stop_tx.send(true);
handlers::stop_url_workers();
if let Some(admin) = CONFIG.admin_ids.first() {
let _ = bot.send_message(ChatId(*admin), "Shutting down...").await;
}
+22 -9
View File
@@ -26,8 +26,9 @@ pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900;
/// to a smaller media URL instead.
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024;
/// Decode budget (bytes): a larger intermediate buffer is not worth the peak
/// memory; the photo degrades to the smaller URL instead.
const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
/// memory; the photo degrades to the smaller URL instead. Also the cap for
/// downloading photos in the send fallback (they must be downloaded whole).
pub(crate) const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
/// JPEG output quality (1-100).
const JPEG_QUALITY: u8 = 90;
@@ -216,13 +217,15 @@ fn target_dims(w: u32, h: u32) -> (u32, u32) {
/// RGB, Lanczos-downscale beyond the dimension cap, encode PNG — a PNG still
/// over the upload cap afterwards becomes JPEG.
fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
let (w, h, _bit_depth, color_type) =
parse_png_header(&bytes).ok_or("invalid PNG header")?;
let (w, h, _bit_depth, color_type) = parse_png_header(&bytes).ok_or("invalid PNG header")?;
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
return Ok(PhotoPrep::Upload(file));
}
log::info!("photo {w}x{h} ({_bit_depth:?} {color_type:?}, {} bytes) needs processing", bytes.len());
log::info!(
"photo {w}x{h} ({_bit_depth:?} {color_type:?}, {} bytes) needs processing",
bytes.len()
);
let channels = output_channels(color_type);
if (w as u64) * (h as u64) * channels as u64 > MAX_DECODE_BYTES {
@@ -238,7 +241,9 @@ fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String>
};
let mut decoder = png::Decoder::new(std::io::Cursor::new(&bytes));
decoder.set_transformations(transforms);
let mut reader = decoder.read_info().map_err(|e| format!("png decode: {e}"))?;
let mut reader = decoder
.read_info()
.map_err(|e| format!("png decode: {e}"))?;
let out_w = reader.info().width;
let out_h = reader.info().height;
let mut buf = vec![
@@ -457,7 +462,9 @@ mod tests {
let mut bytes = Vec::new();
{
let encoder = jpeg_encoder::Encoder::new(&mut bytes, 90);
encoder.encode(&rgb, w, h, jpeg_encoder::ColorType::Rgb).unwrap();
encoder
.encode(&rgb, w, h, jpeg_encoder::ColorType::Rgb)
.unwrap();
}
let mut file = tempfile::Builder::new().suffix(".jpg").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
@@ -485,7 +492,9 @@ mod tests {
for y in 0..h {
for x in 0..w {
let base = (x + y) * 255 / (w + h);
rng = rng.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
rng = rng
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
let n = ((rng >> 33) % 11) as i32 - 5; // noise in [-5, 5]
let v = (base as i32 + n).clamp(0, 255) as u8;
data.extend_from_slice(&[v, v, v]);
@@ -499,7 +508,11 @@ mod tests {
let mut writer = encoder.write_header().unwrap();
writer.write_image_data(&data).unwrap();
}
assert!(bytes.len() as u64 > MAX_UPLOAD_BYTES, "test needs a >10MiB PNG, got {}", bytes.len());
assert!(
bytes.len() as u64 > MAX_UPLOAD_BYTES,
"test needs a >10MiB PNG, got {}",
bytes.len()
);
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
+167 -84
View File
@@ -5,13 +5,14 @@
//! flow. The Python dict-mutation hack (attempts inside the payload) is
//! replaced by dedicated columns.
use crate::db::now_f64;
use parking_lot::Mutex;
use rusqlite::{params, Connection, TransactionBehavior};
use rusqlite::{Connection, TransactionBehavior, params};
use serde_json::Value;
use std::pin::Pin;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::Duration;
use tokio::sync::Notify;
use tokio::task::JoinHandle;
@@ -29,15 +30,9 @@ const QUEUE_WORKERS: usize = 4;
pub enum QueueError {
/// Reschedule with the given delay; after `MAX_RETRIES` attempts the task
/// is dead-lettered instead.
Retryable {
delay_seconds: f64,
payload: Value,
},
Retryable { delay_seconds: f64, payload: Value },
/// Give up now.
Permanent {
message: String,
payload: Value,
},
Permanent { message: String, payload: Value },
}
type BoxFuture<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
@@ -59,6 +54,7 @@ struct LeasedRow {
}
/// Owned worker state so the spawned loop does not borrow the queue handle.
#[derive(Clone)]
struct QueueWorker {
db_path: String,
notify: Arc<Notify>,
@@ -67,27 +63,31 @@ struct QueueWorker {
dead_letter: Arc<DeadLetter>,
}
fn now_f64() -> f64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
/// Resets rows left `in_progress` with an expired lock TTL back to `pending`
/// so they can be leased again (crash/panic recovery).
fn recover_update(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
conn.execute(
"UPDATE tasks SET status='pending', locked_until=0 WHERE status='in_progress' AND locked_until < ?1",
params![now_f64()],
)?;
Ok(())
}
/// Opens the queue DB with a busy timeout. Handler tasks enqueue while
/// workers lease/update rows concurrently; without the timeout a concurrent
/// write fails immediately with SQLITE_BUSY and the operation is lost.
fn open_db(path: &str) -> rusqlite::Result<Connection> {
let conn = Connection::open(path)?;
conn.busy_timeout(Duration::from_secs(5))?;
Ok(conn)
/// Base delay × 2^attempts (attempts = retries already done), capped at 300s.
/// Applied at the queue layer so the attempt count actually reaches the
/// backoff computation; Telegram `RetryAfter` delays get the same treatment
/// (conservatively larger wait, no API change needed).
fn scaled_retry_delay(base: f64, attempts: i32) -> f64 {
(base * 2f64.powi(attempts)).min(300.0)
}
fn ensure_schema(conn: &Connection) -> rusqlite::Result<()> {
fn ensure_schema(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
"PRAGMA journal_mode=WAL; \
CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
locked_until REAL NOT NULL, created_at REAL NOT NULL);",
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
CREATE INDEX IF NOT EXISTS idx_tasks_pending ON tasks(status, run_after);",
)
}
@@ -129,7 +129,7 @@ impl PersistentTaskQueue {
let dead_letter: Arc<DeadLetter> =
Arc::new(move |payload, message| Box::pin(dead_letter(payload, message)));
self.recover_stale().await;
let mut handles = Vec::with_capacity(QUEUE_WORKERS);
let mut handles = Vec::with_capacity(QUEUE_WORKERS + 1);
for _ in 0..QUEUE_WORKERS {
let worker = QueueWorker {
db_path: self.db_path.clone(),
@@ -138,8 +138,35 @@ impl PersistentTaskQueue {
handler: Arc::clone(&handler),
dead_letter: Arc::clone(&dead_letter),
};
handles.push(tokio::spawn(worker.run_loop()));
handles.push(tokio::spawn(worker.run_loop_supervised()));
}
// Periodic lease-expiry sweep: recovers rows a crashed/panicked
// worker left `in_progress` (the lock TTL bounds the wait). Woken by
// the same notify as the workers, so enqueue and stop interrupt the
// sleep; the first interval tick fires immediately (harmless extra
// recovery at startup).
let sweep_db_path = self.db_path.clone();
let sweep_notify = Arc::clone(&self.notify);
let sweep_stop = Arc::clone(&self.stop);
handles.push(tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(30));
loop {
let notified = sweep_notify.notified();
tokio::pin!(notified);
tokio::select! {
_ = &mut notified => {}
_ = interval.tick() => {}
}
if sweep_stop.load(Ordering::Relaxed) {
break;
}
let result =
crate::db::with_conn(&sweep_db_path, move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue sweep failed: {e}");
}
}
}));
*self.worker.lock() = handles;
}
@@ -162,10 +189,8 @@ impl PersistentTaskQueue {
self.counter.fetch_add(1, Ordering::Relaxed)
);
let payload = payload.to_string();
let db_path = self.db_path.clone();
log::info!("enqueued {id} (run_after {run_after:.1})");
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"INSERT OR REPLACE INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES (?1, ?2, ?3, 0, 'pending', 0, ?4)",
@@ -173,36 +198,46 @@ impl PersistentTaskQueue {
)?;
Ok(())
})
.await
.expect("queue insert worker panicked")?;
// Wake every sleeping worker: with several workers the one that finds
// nothing due must not starve the newly inserted row.
self.notify.notify_waiters();
.await?;
// `notify_one` stores a permit when no worker is registered, so a
// notification fired between a worker's DB reads and its `notified()`
// registration is not lost (notify_waiters would drop it). The
// awakened worker re-leases and finds the new row.
self.notify.notify_one();
Ok(())
}
async fn recover_stale(&self) {
let db_path = self.db_path.clone();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
conn.execute(
"UPDATE tasks SET status='pending', locked_until=0 WHERE status='in_progress' AND locked_until < ?1",
params![now_f64()],
)?;
Ok(())
})
.await
.expect("queue recovery worker panicked")
.unwrap_or_else(|e| log::error!("queue recovery failed: {e}"));
self.recover_sweep().await;
}
async fn recover_sweep(&self) {
let result = crate::db::with_conn(&self.db_path, move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue recovery failed: {e}");
}
}
}
impl QueueWorker {
/// Supervised worker: the inner loop runs in its own task so a panic
/// (e.g. inside a handler or a DB closure) kills only that task; the
/// supervisor respawns it until stop is set. The row a dead worker had
/// leased is recovered by the periodic sweep once its lock TTL expires.
async fn run_loop_supervised(self) {
while !self.stop.load(Ordering::Relaxed) {
let worker = self.clone();
if let Err(e) = tokio::spawn(async move { worker.run_loop().await }).await {
log::error!("queue worker panicked, restarting: {e}");
}
}
}
async fn run_loop(self) {
while !self.stop.load(Ordering::Relaxed) {
match self.lease_next().await {
Some(row) => self.process(row).await,
None => {
Ok(Some(row)) => self.process(row).await,
Ok(None) => {
let wait_until = self.earliest_run_after().await;
let notified = self.notify.notified();
tokio::pin!(notified);
@@ -219,15 +254,20 @@ impl QueueWorker {
}
}
}
// A lease failure while rows are due would otherwise loop
// with sleep(0) and hammer SQLite; back off briefly.
Err(e) => {
log::error!("queue lease failed: {e}");
tokio::time::sleep(Duration::from_secs(1)).await;
}
}
}
}
/// Leases the oldest due row (sets it `in_progress` with a lock TTL).
async fn lease_next(&self) -> Option<LeasedRow> {
let db_path = self.db_path.clone();
tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<LeasedRow>> {
let mut conn = open_db(&db_path)?;
/// Errors are surfaced so the caller can back off instead of spinning.
async fn lease_next(&self) -> Result<Option<LeasedRow>, rusqlite::Error> {
crate::db::with_conn(&self.db_path, |conn| {
// BEGIN IMMEDIATE: with several workers, a deferred transaction
// that read before another worker's lease commit would fail with
// SQLITE_BUSY_SNAPSHOT. Taking the write lock up front serializes
@@ -266,30 +306,26 @@ impl QueueWorker {
}))
})
.await
.expect("queue lease worker panicked")
.unwrap_or_else(|e| {
log::error!("queue lease failed: {e}");
None
})
}
async fn earliest_run_after(&self) -> Option<f64> {
let db_path = self.db_path.clone();
tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<f64>> {
let conn = open_db(&db_path)?;
let mut stmt = conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
let result = crate::db::with_conn(&self.db_path, |conn| {
let mut stmt =
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
let mut rows = stmt.query([])?;
match rows.next()? {
Some(row) => Ok(row.get::<_, Option<f64>>(0)?),
None => Ok(None),
}
})
.await
.expect("queue timing worker panicked")
.unwrap_or_else(|e| {
log::error!("queue timing query failed: {e}");
None
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::error!("queue timing query failed: {e}");
None
}
}
}
async fn process(&self, row: LeasedRow) {
@@ -318,12 +354,13 @@ impl QueueWorker {
self.delete_row(&row.id).await;
(self.dead_letter)(payload, message).await;
} else {
let delay = scaled_retry_delay(delay_seconds, row.attempts);
log::info!(
"task {} rescheduled in {delay_seconds:.1}s (attempt {})",
"task {} rescheduled in {delay:.1}s (attempt {})",
row.id,
row.attempts + 1
);
self.reschedule(&row.id, payload, delay_seconds, row.attempts + 1)
self.reschedule(&row.id, payload, delay, row.attempts + 1)
.await;
}
}
@@ -336,34 +373,33 @@ impl QueueWorker {
}
async fn delete_row(&self, id: &str) {
let db_path = self.db_path.clone();
let id = id.to_string();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
Ok(())
})
.await
.expect("queue delete worker panicked")
.unwrap_or_else(|e| log::error!("queue delete failed: {e}"));
.await;
if let Err(e) = result {
log::error!("queue delete failed: {e}");
}
}
async fn reschedule(&self, id: &str, payload: Value, delay_seconds: f64, attempts: i32) {
let db_path = self.db_path.clone();
let id = id.to_string();
let payload = payload.to_string();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = open_db(&db_path)?;
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"UPDATE tasks SET payload=?1, run_after=?2, attempts=?3, status='pending', locked_until=0 WHERE id=?4",
params![payload, now_f64() + delay_seconds, attempts, id],
)?;
Ok(())
})
.await
.expect("queue reschedule worker panicked")
.unwrap_or_else(|e| log::error!("queue reschedule failed: {e}"));
self.notify.notify_waiters();
.await;
if let Err(e) = result {
log::error!("queue reschedule failed: {e}");
}
// Same permit semantics as enqueue: never lose the wakeup.
self.notify.notify_one();
}
}
@@ -372,6 +408,16 @@ mod tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering};
#[test]
fn scaled_retry_delay_scales_and_caps() {
assert_eq!(scaled_retry_delay(1.0, 0), 1.0);
assert_eq!(scaled_retry_delay(1.0, 1), 2.0);
assert_eq!(scaled_retry_delay(1.0, 2), 4.0);
assert_eq!(scaled_retry_delay(1.5, 1), 3.0);
assert_eq!(scaled_retry_delay(1.0, 10), 300.0, "capped at 300s");
assert_eq!(scaled_retry_delay(300.0, 0), 300.0);
}
async fn new_queue() -> (PersistentTaskQueue, tempfile::TempDir) {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("queue.db");
@@ -510,4 +556,41 @@ mod tests {
assert_eq!(calls.load(AtomicOrdering::SeqCst), 1);
queue.stop().await;
}
#[tokio::test]
async fn runtime_sweep_recovers_expired_lease() {
let (queue, _dir) = new_queue().await;
let calls = Arc::new(AtomicUsize::new(0));
let c = calls.clone();
queue
.start(
move |payload| {
assert_eq!(payload["s"], 1);
c.fetch_add(1, AtomicOrdering::SeqCst);
async { Ok(()) }
},
|_payload, _message| async {},
)
.await;
// Insert a stale leased row AFTER startup: without a runtime sweep it
// would stay `in_progress` forever (only start() used to recover).
{
let conn = Connection::open(&queue.db_path).unwrap();
ensure_schema(&conn).unwrap();
conn.execute(
"INSERT INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES ('task_stale_runtime', '{\"s\":1}', 0, 0, 'in_progress', ?1, 0)",
params![now_f64() - 1000.0],
)
.unwrap();
}
queue.recover_sweep().await;
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(
calls.load(AtomicOrdering::SeqCst),
1,
"expired lease must be recovered and processed exactly once"
);
queue.stop().await;
}
}
+283 -141
View File
@@ -5,22 +5,27 @@
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE};
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost};
use crate::photo::{self, PhotoPrep, MAX_UPLOAD_BYTES};
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
use crate::queue::QueueError;
use crate::state::{EditMessage, unix_now};
use rand::Rng;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use tempfile::NamedTempFile;
use std::sync::LazyLock;
use teloxide::prelude::*;
use teloxide::types::{
ChatId, InlineKeyboardButton, InlineKeyboardMarkup, InputFile, InputMedia,
InputMediaAnimation, InputMediaPhoto, InputMediaVideo, Message, MessageId, ParseMode,
ReplyParameters,
ChatId, InlineKeyboardButton, InlineKeyboardMarkup, InputFile, InputMedia, InputMediaAnimation,
InputMediaPhoto, InputMediaVideo, Message, MessageId, ParseMode, ReplyParameters,
};
use teloxide::{ApiError, RequestError};
use tempfile::NamedTempFile;
use x_media::site::FetchError;
/// One process-wide Bot for queue workers. Building a fresh Bot (and its HTTP
/// client) per queue task was pure waste; forced at startup in main so a
/// missing token fails fast instead of on the first task.
pub static BOT: LazyLock<Bot> = LazyLock::new(Bot::from_env);
#[derive(Serialize, Deserialize, Clone, Debug)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum MediaItemPayload {
@@ -62,6 +67,15 @@ impl MediaItemPayload {
MediaItemPayload::Animation { .. } => None,
}
}
/// The cover-frame URL for videos (used by the upload fallback, which
/// otherwise drops the thumbnail the URL-send path applies).
fn thumbnail_url(&self) -> Option<&str> {
match self {
MediaItemPayload::Video { thumbnail, .. } => thumbnail.as_deref(),
MediaItemPayload::Photo { .. } | MediaItemPayload::Animation { .. } => None,
}
}
}
#[derive(Serialize, Deserialize, Clone, Debug)]
@@ -111,16 +125,18 @@ pub enum Task {
impl Task {
fn cache_data(&self) -> Option<&CachedPost> {
match self {
Task::SendMediaSequence { cache_data, .. }
| Task::SendAnimation { cache_data, .. } => cache_data.as_ref(),
Task::SendMediaSequence { cache_data, .. } | Task::SendAnimation { cache_data, .. } => {
cache_data.as_ref()
}
Task::ForwardMessages { .. } => None,
}
}
fn source_url(&self) -> Option<&str> {
match self {
Task::SendMediaSequence { source_url, .. }
| Task::SendAnimation { source_url, .. } => Some(source_url),
Task::SendMediaSequence { source_url, .. } | Task::SendAnimation { source_url, .. } => {
Some(source_url)
}
Task::ForwardMessages { .. } => None,
}
}
@@ -138,9 +154,10 @@ fn file_id_of_message(message: &Message, item: &MediaItemPayload) -> Option<Stri
match item {
// `photo()` returns all sizes, smallest first — the largest carries
// the file id of the sent media.
MediaItemPayload::Photo { .. } => {
message.photo().and_then(|sizes| sizes.last()).map(|p| p.file.id.to_string())
}
MediaItemPayload::Photo { .. } => message
.photo()
.and_then(|sizes| sizes.last())
.map(|p| p.file.id.to_string()),
MediaItemPayload::Video { .. } => message.video().map(|v| v.file.id.to_string()),
MediaItemPayload::Animation { .. } => message.animation().map(|a| a.file.id.to_string()),
}
@@ -214,7 +231,10 @@ pub const MAX_MEDIA_GROUP: usize = 9;
/// Splits media into batches of at most [`MAX_MEDIA_GROUP`] items.
pub fn chunk_media_items<T: Clone>(items: Vec<T>) -> Vec<Vec<T>> {
items.chunks(MAX_MEDIA_GROUP).map(|chunk| chunk.to_vec()).collect()
items
.chunks(MAX_MEDIA_GROUP)
.map(|chunk| chunk.to_vec())
.collect()
}
/// Exponential backoff with jitter, capped at 30s.
@@ -250,31 +270,41 @@ pub fn is_size_error(e: &ApiError) -> bool {
return true;
}
let description = e.to_string().to_lowercase();
["too large", "too big"].iter().any(|marker| description.contains(marker))
["too large", "too big"]
.iter()
.any(|marker| description.contains(marker))
}
/// Task-free classification of a Telegram request error. The callers attach
/// the (updated) task when building a [`SendError`].
pub enum Classification {
Retryable { delay_seconds: f64 },
Permanent { message: String },
Retryable {
delay_seconds: f64,
},
Permanent {
message: String,
},
/// Handled by the download fallback, not a queue retry.
MediaFetchFailure,
}
pub fn classify_request_error(e: &RequestError) -> Classification {
match e {
RequestError::RetryAfter(seconds) => {
Classification::Retryable { delay_seconds: seconds.seconds() as f64 }
}
RequestError::RetryAfter(seconds) => Classification::Retryable {
delay_seconds: seconds.seconds() as f64,
},
RequestError::Network(_) => Classification::Retryable {
delay_seconds: retry_delay_seconds(0),
},
RequestError::Api(api) if is_media_fetch_failure(api) => Classification::MediaFetchFailure,
RequestError::Api(api) => Classification::Permanent { message: api.to_string() },
RequestError::Api(api) => Classification::Permanent {
message: api.to_string(),
},
RequestError::MigrateToChatId(_)
| RequestError::InvalidJson { .. }
| RequestError::Io(_) => Classification::Permanent { message: e.to_string() },
| RequestError::Io(_) => Classification::Permanent {
message: e.to_string(),
},
}
}
@@ -314,6 +344,11 @@ fn item_url(item: &MediaItemPayload) -> &str {
fn input_file_for(media: &str) -> Result<InputFile, String> {
if media.starts_with("http://") || media.starts_with("https://") {
Ok(InputFile::url(parse_media_url(media)?))
} else if !std::path::Path::new(media).exists() {
// A retried task may reference a temp file the original send's
// TempDir already cleaned up; fail fast and permanent instead of
// burning retries on a file that can never come back.
Err(format!("local media file missing: {media}"))
} else {
Ok(InputFile::file(media))
}
@@ -390,9 +425,9 @@ fn build_media_group(
.map(|(i, item)| {
let item_caption = if i == 0 { caption } else { None };
Ok(match item {
MediaItemPayload::Photo {
has_spoiler, ..
} => photo_media(item.input_file()?, item_caption, *has_spoiler),
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(item.input_file()?, item_caption, *has_spoiler)
}
MediaItemPayload::Video {
has_spoiler,
thumbnail,
@@ -404,9 +439,9 @@ fn build_media_group(
}
video
}
MediaItemPayload::Animation {
has_spoiler, ..
} => animation_media(item.input_file()?, item_caption, *has_spoiler),
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(item.input_file()?, item_caption, *has_spoiler)
}
})
})
.collect()
@@ -431,8 +466,12 @@ fn sniff_ext(bytes: &[u8]) -> &'static str {
}
enum FallbackError {
Retryable { delay_seconds: f64 },
Permanent { message: String },
Retryable {
delay_seconds: f64,
},
Permanent {
message: String,
},
/// The downloaded file exceeds the upload cap; the caller falls back to
/// the item's smaller URL.
MediaTooLarge,
@@ -453,26 +492,30 @@ async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, Fall
| MediaItemPayload::Video { media, .. }
| MediaItemPayload::Animation { media, .. } => media,
};
let bytes = match x_media::site::download_media(media_url).await {
// Photos are downloaded even over the upload cap so `prepare_photo` can
// downscale / transcode them (cap = decode budget); videos/animations
// abort as soon as the upload cap is crossed mid-stream.
let limit = if matches!(item, MediaItemPayload::Photo { .. }) {
photo::MAX_DECODE_BYTES
} else {
MAX_UPLOAD_BYTES + 1
};
let bytes = match x_media::site::download_media_limited(media_url, limit).await {
Ok(bytes) => bytes,
Err(FetchError::Http(_)) => {
return Err(FallbackError::Retryable {
delay_seconds: retry_delay_seconds(0),
});
}
Err(FetchError::TooLarge) => {
return Err(FallbackError::MediaTooLarge);
}
Err(e) => {
return Err(FallbackError::Permanent {
message: format!("download failed: {e}"),
});
}
};
// Photos are downloaded even over the cap so `prepare_photo` can
// downscale / transcode them; only videos/animations short-circuit.
if !matches!(item, MediaItemPayload::Photo { .. })
&& bytes.len() as u64 > MAX_UPLOAD_BYTES
{
return Err(FallbackError::MediaTooLarge);
}
let ext = sniff_ext(&bytes);
let mut file = tempfile::Builder::new()
.suffix(&format!(".{ext}"))
@@ -494,8 +537,9 @@ fn media_from_file(
item: &MediaItemPayload,
path: std::path::PathBuf,
caption: Option<&str>,
) -> InputMedia {
match item {
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(InputFile::file(path), caption, *has_spoiler)
}
@@ -505,7 +549,11 @@ fn media_from_file(
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(InputFile::file(path), caption, *has_spoiler)
}
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
}
/// Builds the media group item from a (smaller) URL.
@@ -513,8 +561,9 @@ fn media_from_url(
item: &MediaItemPayload,
url: &str,
caption: Option<&str>,
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
Ok(match item {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(input_file_for(url)?, caption, *has_spoiler)
}
@@ -524,7 +573,11 @@ fn media_from_url(
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(input_file_for(url)?, caption, *has_spoiler)
}
})
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
}
/// Download-and-reupload fallback for one media batch. Files over the upload
@@ -552,7 +605,7 @@ async fn send_batch_via_upload(
let too_large = too_large && !matches!(item, MediaItemPayload::Photo { .. });
let media = if too_large {
match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption) {
Some(url) => match media_from_url(item, url, item_caption, item.thumbnail_url()) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
@@ -584,10 +637,16 @@ async fn send_batch_via_upload(
PhotoPrep::Upload(upload) => {
let path = upload.path().to_path_buf();
files.push(upload);
media_from_file(item, path, item_caption)
media_from_file(item, path, item_caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?
}
PhotoPrep::UseFallback => match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption) {
Some(url) => match media_from_url(
item,
url,
item_caption,
item.thumbnail_url(),
) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
@@ -605,16 +664,19 @@ async fn send_batch_via_upload(
} else {
let path = file.path().to_path_buf();
files.push(file);
media_from_file(item, path, item_caption)
media_from_file(item, path, item_caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?
}
}
Err(FallbackError::MediaTooLarge) => match item.fallback_url() {
Some(url) => match media_from_url(item, url, item_caption) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
Some(url) => {
match media_from_url(item, url, item_caption, item.thumbnail_url()) {
Ok(media) => media,
Err(message) => {
return Err(FallbackError::Permanent { message });
}
}
},
}
None => {
return Err(FallbackError::Permanent {
message: "media too large".into(),
@@ -628,14 +690,16 @@ async fn send_batch_via_upload(
}
let result = bot
.send_media_group(ChatId(chat_id), items)
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply())
.reply_parameters(
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
)
.await;
match result {
Ok(messages) => Ok(messages),
Err(e) => Err(match classify_request_error(&e) {
Classification::Retryable { delay_seconds } => FallbackError::Retryable {
delay_seconds,
},
Classification::Retryable { delay_seconds } => {
FallbackError::Retryable { delay_seconds }
}
Classification::Permanent { message } => FallbackError::Permanent { message },
Classification::MediaFetchFailure => FallbackError::Permanent {
message: "upload failed".into(),
@@ -702,7 +766,11 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
let fresh_send = *batch_index == 0 && sent.is_empty();
for idx in *batch_index..media_batches.len() {
let batch = &media_batches[idx];
let caption = if idx == 0 { Some(caption.as_str()) } else { None };
let caption = if idx == 0 {
Some(caption.as_str())
} else {
None
};
let items = match build_media_group(batch, caption) {
Ok(items) => items,
Err(message) => {
@@ -714,7 +782,9 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
};
match bot
.send_media_group(ChatId(chat_id), items)
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply())
.reply_parameters(
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
)
.await
{
Ok(messages) => {
@@ -726,9 +796,7 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
collect_file_ids(&messages, batch, &mut cached_media);
sent.extend(messages.into_iter().map(|m| m.id.0 as i64));
}
Err(RequestError::Api(api))
if is_media_fetch_failure(&api) || is_size_error(&api) =>
{
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
log::info!(
"Telegram could not fetch media for batch {idx} ({}), downloading and reuploading",
batch.first().map(item_url).unwrap_or("?")
@@ -779,7 +847,9 @@ async fn send_animation_inner(
.send_animation(ChatId(chat_id), file)
.caption(caption)
.parse_mode(ParseMode::Html)
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply());
.reply_parameters(
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
);
if spoiler {
request = request.has_spoiler(true);
}
@@ -802,9 +872,7 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
let reply_to = *reply_to_message_id;
let (media_url, has_spoiler) = match animation {
MediaItemPayload::Animation {
media,
has_spoiler,
..
media, has_spoiler, ..
} => (media, *has_spoiler),
MediaItemPayload::Photo { .. } | MediaItemPayload::Video { .. } => {
unreachable!("SendAnimation carries an Animation payload")
@@ -812,19 +880,20 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
};
let url_file = match input_file_for(media_url) {
Ok(file) => file,
Err(message) => return Err(SendError::Permanent { message, task: task.clone() }),
Err(message) => {
return Err(SendError::Permanent {
message,
task: task.clone(),
});
}
};
match send_animation_inner(bot, chat_id, reply_to, caption, has_spoiler, url_file)
.await
{
match send_animation_inner(bot, chat_id, reply_to, caption, has_spoiler, url_file).await {
Ok(message) => {
let id = message.id.0 as i64;
cache_animation_send(task, &message).await;
Ok(vec![id])
}
Err(RequestError::Api(api))
if is_media_fetch_failure(&api) || is_size_error(&api) =>
{
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
log::info!(
"Telegram could not fetch animation URL, downloading and reuploading: {}",
media_url
@@ -872,21 +941,24 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
Err(e) => Err(classify_to_send_error(&e, task.clone())),
}
}
Err(message) => {
Err(SendError::Permanent { message, task: task.clone() })
}
Err(message) => Err(SendError::Permanent {
message,
task: task.clone(),
}),
},
None => Err(SendError::Permanent {
message: "media too large".into(),
task: task.clone(),
}),
},
Err(FallbackError::Retryable { delay_seconds }) => {
Err(SendError::Retryable { delay_seconds, task: task.clone() })
}
Err(FallbackError::Permanent { message }) => {
Err(SendError::Permanent { message, task: task.clone() })
}
Err(FallbackError::Retryable { delay_seconds }) => Err(SendError::Retryable {
delay_seconds,
task: task.clone(),
}),
Err(FallbackError::Permanent { message }) => Err(SendError::Permanent {
message,
task: task.clone(),
}),
}
}
Err(e) => Err(classify_to_send_error(&e, task.clone())),
@@ -910,7 +982,11 @@ pub async fn forward_messages(bot: &Bot, task: &Task) -> Result<(), SendError> {
.map(|id| MessageId(*id as i32))
.collect::<Vec<_>>();
match bot
.copy_messages(ChatId(*to_chat_id), ChatId(*from_chat_id), message_ids.clone())
.copy_messages(
ChatId(*to_chat_id),
ChatId(*from_chat_id),
message_ids.clone(),
)
.await
{
Ok(_) => {
@@ -944,12 +1020,18 @@ pub fn build_edit_markup(templates: &HashMap<String, String>) -> InlineKeyboardM
/// Notifies a chat about a dead-lettered task (skips when `notify_chat_id` is
/// absent).
pub async fn notify_failure(bot: &Bot, chat_id: Option<i64>, message_id: Option<i64>, message: &str) {
pub async fn notify_failure(
bot: &Bot,
chat_id: Option<i64>,
message_id: Option<i64>,
message: &str,
) {
let Some(chat_id) = chat_id else { return };
let mut request = bot.send_message(ChatId(chat_id), message);
if let Some(message_id) = message_id {
request = request
.reply_parameters(ReplyParameters::new(MessageId(message_id as i32)).allow_sending_without_reply());
request = request.reply_parameters(
ReplyParameters::new(MessageId(message_id as i32)).allow_sending_without_reply(),
);
}
if let Err(e) = request.await {
log::error!("failed to notify about failed task: {e}");
@@ -959,42 +1041,48 @@ pub async fn notify_failure(bot: &Bot, chat_id: Option<i64>, message_id: Option<
/// After a successful send: either open the edit-before-forward prompt or
/// forward to the configured channel (with retry/queue handling).
pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
let (chat_id, reply_to, source_url, edit_before_forward, forward_channel_id, notify_chat_id, notify_message_id) =
match task {
Task::SendMediaSequence {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
}
| Task::SendAnimation {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
} => (
*chat_id,
*reply_to_message_id,
source_url.clone(),
*edit_before_forward,
*forward_channel_id,
*notify_chat_id,
*notify_message_id,
),
Task::ForwardMessages { .. } => return,
};
let (
chat_id,
reply_to,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
) = match task {
Task::SendMediaSequence {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
}
| Task::SendAnimation {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
} => (
*chat_id,
*reply_to_message_id,
source_url.clone(),
*edit_before_forward,
*forward_channel_id,
*notify_chat_id,
*notify_message_id,
),
Task::ForwardMessages { .. } => return,
};
if edit_before_forward {
let mut chat_data = CHAT_STORE.get(chat_id).await;
let keyboard = build_edit_markup(&chat_data.template);
let keyboard = build_edit_markup(&CHAT_STORE.get(chat_id).await.template);
match bot
.send_message(ChatId(chat_id), "Reply to edit message.")
.reply_markup(keyboard)
@@ -1009,17 +1097,22 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
prompt.id.0,
message_ids.len()
);
chat_data.edit_message.insert(
prompt.id.0 as i64,
EditMessage {
url: source_url,
chat_id,
forward_message_ids: message_ids,
template: String::new(),
created_at: unix_now(),
},
);
CHAT_STORE.set(chat_id, &chat_data).await;
let prompt_id = prompt.id.0 as i64;
let source_url = source_url.clone();
CHAT_STORE
.update(chat_id, move |data| {
data.edit_message.insert(
prompt_id,
EditMessage {
url: source_url,
chat_id,
forward_message_ids: message_ids,
template: String::new(),
created_at: unix_now(),
},
);
})
.await;
}
Err(e) => log::error!("failed to send edit prompt: {e}"),
}
@@ -1027,7 +1120,10 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
}
if let Some(channel_id) = forward_channel_id {
log::info!("forwarding {} message(s) to channel {channel_id}", message_ids.len());
log::info!(
"forwarding {} message(s) to channel {channel_id}",
message_ids.len()
);
let forward_task = Task::ForwardMessages {
from_chat_id: chat_id,
to_chat_id: channel_id,
@@ -1037,7 +1133,10 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
};
match forward_messages(bot, &forward_task).await {
Ok(()) => {}
Err(SendError::Retryable { delay_seconds, task }) => {
Err(SendError::Retryable {
delay_seconds,
task,
}) => {
let payload = serde_json::to_value(task).expect("task serializes");
let run_after = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
@@ -1072,12 +1171,27 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
});
}
};
let bot = Bot::from_env();
let bot = BOT.clone();
// A resumed multi-batch send already ran post_send_actions (edit prompt /
// forward) when it first started; running them again on the resume would
// open a duplicate edit prompt and double-forward. SendAnimation is
// atomic (always a fresh run), so only SendMediaSequence can resume.
let resumed = matches!(
&task,
Task::SendMediaSequence {
batch_index,
sent_message_ids,
..
} if *batch_index > 0 || !sent_message_ids.is_empty()
);
match task {
Task::SendMediaSequence { .. } | Task::SendAnimation { .. } => {
let message_ids = match send_media_or_animation(&bot, &task).await {
Ok(ids) => ids,
Err(SendError::Retryable { delay_seconds, task }) => {
Err(SendError::Retryable {
delay_seconds,
task,
}) => {
return Err(QueueError::Retryable {
delay_seconds,
payload: serde_json::to_value(task).expect("task serializes"),
@@ -1091,12 +1205,17 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
});
}
};
post_send_actions(&bot, &task, message_ids).await;
if !resumed {
post_send_actions(&bot, &task, message_ids).await;
}
Ok(())
}
Task::ForwardMessages { .. } => match forward_messages(&bot, &task).await {
Ok(()) => Ok(()),
Err(SendError::Retryable { delay_seconds, task }) => Err(QueueError::Retryable {
Err(SendError::Retryable {
delay_seconds,
task,
}) => Err(QueueError::Retryable {
delay_seconds,
payload: serde_json::to_value(task).expect("task serializes"),
}),
@@ -1121,7 +1240,7 @@ pub async fn dead_letter_notify(payload: serde_json::Value, message: String) {
let notify_chat_id = payload.get("notify_chat_id").and_then(|v| v.as_i64());
let notify_message_id = payload.get("notify_message_id").and_then(|v| v.as_i64());
if notify_chat_id.is_some() {
let bot = Bot::from_env();
let bot = BOT.clone();
notify_failure(
&bot,
notify_chat_id,
@@ -1151,7 +1270,11 @@ mod tests {
assert_eq!(chunk_media_items((0..10).collect()).len(), 2);
assert_eq!(chunk_media_items((0..25).collect()).len(), 3);
assert_eq!(chunk_media_items((0..25).collect())[2].len(), 7);
assert!(chunk_media_items((0..25).collect()).iter().all(|c| c.len() <= 9));
assert!(
chunk_media_items((0..25).collect())
.iter()
.all(|c| c.len() <= 9)
);
}
#[test]
@@ -1175,7 +1298,10 @@ mod tests {
let api = ApiError::Unknown(description.to_string());
assert!(is_media_fetch_failure(&api), "{description}");
}
for description in ["Bad Request: message is not modified", "Forbidden: bot was blocked by the user"] {
for description in [
"Bad Request: message is not modified",
"Forbidden: bot was blocked by the user",
] {
let api = ApiError::Unknown(description.to_string());
assert!(!is_media_fetch_failure(&api), "{description}");
}
@@ -1196,7 +1322,10 @@ mod tests {
assert!(is_size_error(&api), "{description}");
}
// Unrelated errors must not match.
for description in ["Bad Request: WEBPAGE_MEDIA_EMPTY", "Bad Request: message is not modified"] {
for description in [
"Bad Request: WEBPAGE_MEDIA_EMPTY",
"Bad Request: message is not modified",
] {
let api = ApiError::Unknown(description.to_string());
assert!(!is_size_error(&api), "{description}");
}
@@ -1205,9 +1334,16 @@ mod tests {
#[test]
fn media_item_payload_fallback_url_serde_default() {
// Old queued payloads without the field deserialize with None.
let json = serde_json::json!({"kind": "photo", "media": "https://a/b.jpg", "has_spoiler": false});
let json =
serde_json::json!({"kind": "photo", "media": "https://a/b.jpg", "has_spoiler": false});
let photo: MediaItemPayload = serde_json::from_value(json).unwrap();
assert!(matches!(photo, MediaItemPayload::Photo { fallback_url: None, .. }));
assert!(matches!(
photo,
MediaItemPayload::Photo {
fallback_url: None,
..
}
));
assert_eq!(photo.fallback_url(), None);
}
@@ -1286,7 +1422,13 @@ mod tests {
assert_eq!(sent_message_ids, vec![11, 12]);
assert_eq!(forward_channel_id, Some(333));
assert_eq!(media_batches.len(), 2);
assert!(matches!(media_batches[0][0], MediaItemPayload::Photo { has_spoiler: true, .. }));
assert!(matches!(
media_batches[0][0],
MediaItemPayload::Photo {
has_spoiler: true,
..
}
));
}
other => panic!("expected SendMediaSequence, got {other:?}"),
}
+110 -25
View File
@@ -2,10 +2,11 @@
//! `data/task_queue.db`, shared with the task queue).
use parking_lot::Mutex;
use rusqlite::{params, Connection};
use rusqlite::params;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::path::Path;
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
#[derive(Serialize, Deserialize, Default, Clone, Debug)]
@@ -34,6 +35,9 @@ pub struct EditMessage {
pub struct ChatStore {
/// In-memory cache; the DB is the source of truth on first access.
cache: Mutex<HashMap<i64, ChatData>>,
/// Per-chat async locks serializing get→mutate→set so concurrent handler
/// tasks (batch-forwards, callbacks) cannot clobber each other's writes.
locks: Mutex<HashMap<i64, Arc<tokio::sync::Mutex<()>>>>,
db_path: String,
}
@@ -45,7 +49,9 @@ pub fn unix_now() -> i64 {
}
impl ChatStore {
/// Creates the parent directory and both tables (idempotent).
/// Creates the parent directory and the `chat_state` table (idempotent).
/// The shared `tasks` / `link_cache` tables are owned by `queue.rs` and
/// `link_cache.rs` respectively.
pub fn open(path: &str) -> rusqlite::Result<Self> {
if let Some(parent) = Path::new(path).parent()
&& !parent.as_os_str().is_empty()
@@ -53,16 +59,14 @@ impl ChatStore {
std::fs::create_dir_all(parent)
.map_err(|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)))?;
}
let conn = Connection::open(path)?;
let conn = crate::db::open_db(path)?;
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL);",
"CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL);",
)?;
drop(conn);
Ok(ChatStore {
cache: Mutex::new(HashMap::new()),
locks: Mutex::new(HashMap::new()),
db_path: path.to_string(),
})
}
@@ -71,22 +75,19 @@ impl ChatStore {
if let Some(data) = self.cache.lock().get(&chat_id) {
return data.clone();
}
let db_path = self.db_path.clone();
let payload = tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<String>> {
let conn = Connection::open(&db_path)?;
let chat_key = chat_id.to_string();
let payload = crate::db::with_conn(&self.db_path, move |conn| {
// Concurrent handler tasks (batch-forwards) may write chat_state
// while this read runs; without a busy timeout a write lock
// collision fails the query immediately.
conn.busy_timeout(std::time::Duration::from_secs(5))?;
// while this read runs; the shared busy timeout handles the
// write-lock collision instead of failing the query.
let mut stmt = conn.prepare("SELECT payload FROM chat_state WHERE chat_id = ?1")?;
let mut rows = stmt.query(params![chat_id.to_string()])?;
let mut rows = stmt.query(params![chat_key])?;
match rows.next()? {
Some(row) => Ok(Some(row.get(0)?)),
Some(row) => Ok(Some(row.get::<_, String>(0)?)),
None => Ok(None),
}
})
.await
.expect("chat_state worker panicked")
.unwrap_or_else(|e| {
log::error!("chat_state read failed: {e}");
None
@@ -101,19 +102,38 @@ impl ChatStore {
pub async fn set(&self, chat_id: i64, data: &ChatData) {
self.cache.lock().insert(chat_id, data.clone());
let payload = serde_json::to_string(data).expect("chat state serializes");
let db_path = self.db_path.clone();
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
let conn = Connection::open(&db_path)?;
conn.busy_timeout(std::time::Duration::from_secs(5))?;
let chat_id = chat_id.to_string();
let result = crate::db::with_conn(&self.db_path, move |conn| {
conn.execute(
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
params![chat_id.to_string(), payload],
params![chat_id, payload],
)?;
Ok(())
})
.await
.expect("chat_state worker panicked")
.unwrap_or_else(|e| log::error!("chat_state write failed: {e}"));
.await;
if let Err(e) = result {
log::error!("chat_state write failed: {e}");
}
}
/// Serializes a get→mutate→set cycle per chat: concurrent handler tasks
/// (the batch-forward design spawns several per chat) each snapshot the
/// same `ChatData` and last-writer-wins would silently drop mutations,
/// e.g. a second `edit_message` record. The per-chat lock makes the
/// cycle atomic. Returns the closure's result.
pub async fn update<R>(&self, chat_id: i64, f: impl FnOnce(&mut ChatData) -> R) -> R {
let lock = {
let mut locks = self.locks.lock();
locks
.entry(chat_id)
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
};
let _guard = lock.lock().await;
let mut data = self.get(chat_id).await;
let r = f(&mut data);
self.set(chat_id, &data).await;
r
}
/// Removes edit-before-forward records whose `created_at + ttl` is in the
@@ -123,6 +143,10 @@ impl ChatStore {
let now = unix_now();
let ttl_secs = ttl.as_secs() as i64;
let mut removed = Vec::new();
// Chats with no live edit records: evicted from the cache (and their
// per-chat lock) so the cache stays bounded to active prompts. The DB
// keeps the row; the next get() reloads it.
let mut evicted_chats = Vec::new();
let changed: Vec<(i64, ChatData)> = {
let mut cache = self.cache.lock();
let mut out = Vec::new();
@@ -139,18 +163,79 @@ impl ChatStore {
}
}
if kept.len() != data.edit_message.len() {
// Persist the pruned row (removes expired records from
// the DB too, not just the cache).
data.edit_message = kept;
out.push((*chat_id, data.clone()));
}
if data.edit_message.is_empty() {
evicted_chats.push(*chat_id);
}
}
// Lock order: update() takes the per-chat lock before the cache
// lock, so prune must not hold the cache lock while taking locks.
drop(cache);
out
};
for (chat_id, data) in changed {
self.set(chat_id, &data).await;
}
if !evicted_chats.is_empty() {
let mut cache = self.cache.lock();
let mut locks = self.locks.lock();
for chat_id in &evicted_chats {
cache.remove(chat_id);
locks.remove(chat_id);
}
}
if !removed.is_empty() {
log::info!("pruned {} expired edit-before-forward record(s)", removed.len());
log::info!(
"pruned {} expired edit-before-forward record(s)",
removed.len()
);
}
removed
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn concurrent_updates_do_not_lose_edit_records() {
let dir = tempfile::tempdir().unwrap();
let store = std::sync::Arc::new(
ChatStore::open(dir.path().join("s.db").to_str().unwrap()).unwrap(),
);
let mut handles = Vec::new();
for i in 0..4 {
let store = Arc::clone(&store);
handles.push(tokio::spawn(async move {
store
.update(1001, |data| {
data.edit_message.insert(
i,
EditMessage {
url: format!("https://x.com/u/status/{i}"),
chat_id: 1001,
forward_message_ids: vec![i],
template: String::new(),
created_at: 0,
},
);
})
.await;
}));
}
for h in handles {
h.await.unwrap();
}
let data = store.get(1001).await;
assert_eq!(
data.edit_message.len(),
4,
"concurrent get→mutate→set must not drop records"
);
}
}
+10 -2
View File
@@ -11,14 +11,14 @@ services:
- html:/usr/share/nginx/html:ro
networks: [proxy]
labels:
- 'com.github.nginx-proxy.nginx=true'
- 'com.github.nginx-proxy.nginx'
container_name: nginx-proxy
acme-companion:
image: nginxproxy/acme-companion
restart: always
environment:
DEFAULT_EMAIL: 'admin@yoursfunny.top'
DEFAULT_EMAIL: ''
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- certs:/etc/nginx/certs:rw
@@ -55,6 +55,14 @@ services:
depends_on:
- nginx-proxy
container_name: tgxmb
# Webhook mode only: the bot listens on WEBHOOK_PORT; nginx-proxy shows
# 502s while this is down, so surface it to the orchestrator.
healthcheck:
test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/8443'"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
volumes:
certs:
-1
View File
@@ -16,7 +16,6 @@ then
else
usermod -u ${USER_ID} -o user > /dev/null 2>&1 || true
fi
usermod -a -G root user > /dev/null 2>&1 || true
# Bind-mounted volumes may not support chown; a failure here must not kill
# the container either.
chown -R `id -u user`:`id -u user` /app > /dev/null 2>&1 || true