Compare commits

..
Author SHA1 Message Date
YoursFunny 8063777962 chore: bump version to 1.9.2 2026-09-25 03:12:07 +08:00
YoursFunny ac10b8715c docs: sync template limits and live-test count 2026-09-25 02:12:17 +08:00
YoursFunny 517e1362a5 docs: document template storage limits 2026-09-25 01:49:31 +08:00
YoursFunny 50299bacbb fix: redact user-controlled log fields 2026-09-25 01:49:31 +08:00
YoursFunny 9d40f3bc50 fix: bound template storage and callback data 2026-09-25 01:49:31 +08:00
YoursFunny 2045c3cea1 fix: keep photo memory permit until processing ends 2026-09-25 01:49:30 +08:00
YoursFunny 09656810fe fix: prune persisted expired prompts 2026-09-25 01:49:30 +08:00
YoursFunny 786bafca42 fix: account for photo processing peak memory 2026-09-25 01:49:30 +08:00
YoursFunny 18da6e97c2 fix: reject oversized local media before upload 2026-09-25 01:49:29 +08:00
YoursFunny 7b816d4153 fix: split long message forwards 2026-09-25 01:49:29 +08:00
YoursFunny d8b9a06453 fix: bound direct command fetches 2026-09-25 01:40:09 +08:00
YoursFunny 4e0f1b834e fix: restrict media download hosts 2026-09-25 01:40:09 +08:00
YoursFunny 21437d1843 fix: bound queue panic retries 2026-09-25 01:11:28 +08:00
YoursFunny a54029c19b fix: propagate startup repair database errors 2026-09-25 01:06:09 +08:00
YoursFunny c6120b7cc0 fix: make sqlite migrations atomic 2026-09-24 21:47:48 +08:00
YoursFunny 7b4e273533 fix: bound site JSON response bodies 2026-09-24 21:25:03 +08:00
YoursFunny 6a87583cc0 fix: honor bsky HLS retry-after 2026-09-24 21:01:36 +08:00
YoursFunny 1ba64e858d fix: enforce inline result limit centrally 2026-09-24 21:00:00 +08:00
YoursFunny 008d31b9bd test: cover inline answer edge cases 2026-09-24 20:57:08 +08:00
YoursFunny 66ec14662c style: simplify inline answer returns 2026-09-24 20:57:08 +08:00
YoursFunny 8ae7d9947d fix: honor media retry-after delay 2026-09-24 20:57:08 +08:00
YoursFunny 28e8fec68d fix: rate limit plain chat operations 2026-09-24 20:57:08 +08:00
YoursFunny b2b01721b2 fix: debounce inline generations and cap results 2026-09-24 20:40:27 +08:00
YoursFunny 85c3be5ef5 fix: always answer inline query 2026-09-24 20:33:23 +08:00
YoursFunny 5e7a87c0dd fix: send text posts as HTML 2026-09-24 20:29:04 +08:00
YoursFunny 6a2ba6a50a fix: handle empty media and prompt persistence 2026-09-24 19:51:50 +08:00
YoursFunny fb354c9434 test: handle chat state Result in callback 2026-09-24 19:46:23 +08:00
YoursFunny bc7f249c20 fix: filter local animation cache sources 2026-09-24 19:44:48 +08:00
YoursFunny 403b84996f test: cover ugoira expansion budgets 2026-09-24 19:42:50 +08:00
YoursFunny f08b187964 fix: drop unsafe upstream media entries 2026-09-24 19:39:46 +08:00
YoursFunny 8428468712 fix: surface chat state read failures 2026-09-24 19:37:29 +08:00
YoursFunny 412359e84c fix: do not cache local media paths 2026-09-24 18:52:18 +08:00
YoursFunny 438ced278f fix: align CI tags and webhook healthcheck 2026-09-24 18:49:20 +08:00
YoursFunny 704b14f05a fix: bound ffmpeg encoding processes 2026-09-24 18:46:09 +08:00
YoursFunny 1f9da14772 fix: bound ugoira archive expansion 2026-09-24 18:44:21 +08:00
YoursFunny 4694e60ee3 fix: reject upstream local media paths 2026-09-24 18:41:23 +08:00
YoursFunny 88dc0ccb9e fix: stop persistent queue before URL workers 2026-09-24 18:37:17 +08:00
YoursFunny b563b949bb fix: avoid overwriting chat state after read failure 2026-09-24 18:35:52 +08:00
YoursFunny e7855b02fe fix: fence queue dead-letter side effects 2026-09-24 18:33:21 +08:00
YoursFunny 5e265f2413 chore(log): align levels with the documented convention
Three places disagreed with AGENTS' own level rules: get_chat/get_chat_administrators failures during /set_channel printed at error even though the usual cause is a mistyped channel name the user fixes on the spot (error is reserved for dead letters and the like — now warn, same as other user-served degradations); the edit-before-forward callback arrival line logged at info while the sibling arrival lines for commands and inline queries log at debug (now debug, still carries the escaped payload); and the two upload-fallback lines logged at info while photo.rs's parallel degradations log at warn (now warn both — Telegram failing to fetch our media is a degradation that leaves the user served). AGENTS' convention sentence is now explicit about all three instead of the ambiguous '(fallback, retry enqueue, dead-letter is error)'.
2026-09-24 15:38:25 +08:00
YoursFunny 5060b760ba fix(fetch): cap one fetch at a 900-second total budget
The per-request CLIENT timeout and the per-chunk DOWNLOAD_IDLE_TIMEOUT both reset on progress, so neither bounded the fetch as a whole: a URL that keeps dripping (a trickle the idle timeout reads as life) could hold one of the eight FETCH_SLOTS for effectively ever, and the queue behind the slots is what then stalls. The attempts loop now runs inside tokio::time::timeout(900 s) — generous for a genuinely large ugoira zip on a honest slow link (minutes), fatal for a drip — and answers Transient with the cache key, so the retry and its jittered backoff take over. The permit is taken outside the timeout: the slot is released on either path. Audit low finding 'url_workers long tail' (process-wide total-deadline option).
2026-09-24 15:38:25 +08:00
YoursFunny 7019f34801 fix(send): refuse a file id at the upload fallback's door
download_to_temp guarded against a file id, but prepare_upload_item answered a non-http media_url first — and item_url maps FileId to the id itself, so the id was presented as a local *path* and died later with a confusing file-open error instead of a classification. The guard now sits at the entry, download_to_temp takes the already-narrowed URL (the inner guard deleted as dead), and the test pins the refusal as Permanent before any I/O. The file also carries item10's download-classification arm (RateLimited is retryable there too) — same file, landed here to keep each tree compiling.
2026-09-24 15:35:55 +08:00
YoursFunny d05450d88a fix(fetch): jitter the retry backoff and honor a429's Retry-After
Two gaps in fetch_with_attempts: the sleep was the bare 1 << attempt, so every worker that failed together (a source coming back, a shared proxy blip) also recovered on the same tick and re-stamped the source; and a429 arrived as an anonymous Transient, its Retry-After header — the one place a source tells you exactly how long it wants silence — dropped on the floor.

retry_wait(attempt, roll, err) is the pure decision: doubling base plus a random slice of itself ([base, 2x base)), floored at a named Retry-After. status_error now takes the response, reads the seconds-form header and returns the new FetchError::RateLimited { site, retry_after_secs } (HTTP-date parses to None and stays transient); the pure table moved to classify_status so tests still build it from bare status codes. The delay is capped at MAX_RETRY_AFTER_SECS = 60 — the header is server-supplied and must not park one of the eight fetch slots.

Everywhere a Transient meant 'retryable' the new variant joins: the Site trait default, pixiv's override (plus its ugoira-zip mapping), bsky's HLS second attempt, the download classifier, and the user-facing message arm. Tests pin the429 rows (with and without the header, cap included) and retry_wait's math; AGENTS' retries bullet and variant list follow.
2026-09-24 15:35:55 +08:00
YoursFunny 053ae0ec25 fix(send): charge the photo download window to the memory budget
A non-photo body charged the process-wide budget for as long as download_to_temp held it; a photo charged nothing until its prepare step, so the download itself — up to MAX_PHOTO_DOWNLOAD_BYTES per item, six items per batch, eight prep slots process-wide — was memory the MEMORY_BUDGET comment promised was bounded but was not (~6x32 MiB on top of the accounted512 MiB). Photos now reserve their own download cap for that window; the prepare step still charges header probe + decode buffer, the only overlap that is actually held in memory together. Net accounting stays within the declared budget instead of exceeding it whenever a batch of large photos downloads at once (audit: upload.rs photo window uncharged).
2026-09-24 15:29:17 +08:00
YoursFunny d1f2ae09b2 fix(rate-limit): charge plain messages and chat actions to the global bucket
send_media_group, send_animation and copy_messages paced themselves against both buckets; send_message and send_chat_action paced against neither. A dead-letter storm — or one error reply per failed item, plus the action refresh loop's repeated typing requests — could therefore burst past Telegram's per-bot30/s ceiling with only429s left to absorb it, exactly what the bot-wide bucket exists to prevent. Both now acquire one global token like their siblings (per-chat pacing for messages already happens at their call sites; chat actions are cheap and frequent, so only the global bucket applies to them).
2026-09-24 15:28:59 +08:00
YoursFunny 715d6a59b0 fix(photo): sum dimensions in u64 so a huge header cannot wrap
plan_photo, target_dims and both pipeline branches added the two u32 dimensions before comparing against PHOTO_MAX_DIMENSION_SUM: u32::MAX + 2 wrapped to 1 and read 'within limits' — the plan said AsIs, and an absurd-sized image would go to Telegram untouched (in debug builds the addition panics instead). The four production sums now widen to u64 first; the test pins the wrap case as TooLarge. PNG caps each dimension at 2^31-1, so a spec-valid file cannot reach this today — the raw header is parsed before any crate validation, which is exactly where a hostile file lands (audit: photo.rs u32 wrap).
2026-09-24 15:25:19 +08:00
YoursFunny ef8a1dae15 fix(handlers): honour the TTL when a caption-edit reply arrives
A button press on an edit-before-forward prompt already dropped records past EDIT_MESSAGE_TTL lazily (before the 300 s sweep clears them); a text reply to the same prompt did not — it rewrote the caption from a record the sweep was about to treat as dead, so the two paths disagreed about what 'expired' means. The reply path now applies the same rule: drop the stale record, return false, and let the message flow on as if no prompt existed. The test seeds a prompt90000 s old (past the TTL under any config a parallel test can hold) and pins all three effects: no edit consumed, no API call, record gone.
2026-09-24 15:23:02 +08:00
YoursFunny ac34759b02 fix(state): report a failed chat-state write to /set_format
set()/update() logged a failed DB write and returned nothing: the cache already held the new value, so /set_format answered 'Format set' for a change that vanishes on the next restart — a promise the retry queue's enqueue path is already forbidden from making. Both now return whether the write landed, and /set_format's two save paths (set and reset) answer 'in memory only … lost on restart' when it did not. The flag travels as the second tuple element; callers that only relay the value destructure it, callers that ignore it are untouched. A false cannot be forced in a test without a pool-failure seam, so the honest path is pinned by the code, not a fixture.
2026-09-24 15:21:05 +08:00
YoursFunny c420c95165 fix(docker): refuse a root or non-numeric LOCAL_USER_ID
LOCAL_USER_ID=0 passed every check and reached `setpriv --reuid=0`: the bot would run root while looking properly configured, and useradd -o accepted the duplicate uid without complaint. A non-numeric value failed later inside useradd behind `|| true`, which hid the real cause. The entrypoint now rejects both up front with a message naming the variable, and the id/uid expansions are quoted so a value with spaces cannot word-split into extra useradd arguments (audit SEC-007).

Verified by sourcing the entrypoint under a faked `id`: uid 0, 'abc' and '12 3' each exit 1 with the refusal line, while a non-root caller passes straight through to exec. AGENTS and .env.example state the constraint.
2026-09-24 15:12:22 +08:00
YoursFunny 4d6ec7924b fix(handlers): escape user-supplied text in log lines
A Telegram display name, callback payload or channel handle is attacker-controlled text, and it went straight into info/error lines: one embedded newline forged a second log entry, and control characters could hide inside a line (log injection, audit SEC-006). handlers::log_escape replaces newlines, carriage returns and other control characters with visible escapes — clean input borrows, so logging allocates nothing extra — and every site that prints such a value uses it: the callback arrival line, the set-forward-channel info line (name + handle) and both get_chat error lines. The test pins that no raw newline can survive; AGENTS' logging convention names the helper.
2026-09-24 15:02:31 +08:00
YoursFunny 9bae46eb9a chore(compose): pin acme-companion to its paired release
The companion was the file's only unpinned image: it runs with a rw docker.sock, so 'latest' let tomorrow's upstream change deploy itself on the next pull, with full visibility into this stack's containers and environment. 2.8.2 is the release paired with the already-pinned nginx-proxy 1.11.6 — the two projects cut releases a day apart (1.11.6 on 2026-07-16, 2.8.2 on 2026-07-17).
2026-09-24 15:00:45 +08:00
YoursFunny cb91913088 docs(agents): correct the CI, coverage and cert-mount claims
Five statements had drifted from the repo:

- the test-suite comment claimed no CI test step exists; ci.yml has run
  cargo test --workspace --locked all along (the audit's H5).
- the untested list still named config.rs as fully untested although three
  parsing tests pin its webhook truth table, TTL fallback and blank secrets.
- the commands.rs executor was said to need a real Bot; its three tests run
  through the scripted MockSender, and it joins the driven-through-mocks list.
- model.rs was called untested although every adapter fixture deserializes
  into those DTOs.
- the volume line claimed ./cert is mounted; the shipped compose mounts only
  ./data, and WEBHOOK_CERT (a Telegram-facing self-signed upload) needs the
  operator to add mount and env line themselves, which .env.example already
  says. The live-test census also gained site/download.rs (2) and the bot
  crate's repair.rs/urls.rs pair.
2026-09-24 14:54:07 +08:00
YoursFunny 2b6cb506c7 refactor(download): fold apply_media_headers into media_request
apply_media_headers had exactly one caller — media_request, the documented 'one choke point' every download already goes through — so the site-header walk sits inside that choke point now instead of one indirection past it, and the two doc comments merge into the one that describes both rules (guard + headers).
2026-09-24 04:37:54 +08:00
YoursFunny c49e2175a1 refactor(site): drop RenderData's url field, source_url already carries it
Every adapter built RenderData { url } from the exact value it then moved into Fetched.source_url — a copy of the same canonical URL held in two places, feeding one placeholder. caption_with reads {url} from self.source_url now and the field is gone from the struct (bsky's and twitter's halves of this change landed with their caption commits; their render builders no longer set it either).
2026-09-24 04:37:54 +08:00
YoursFunny 2ec13a0624 refactor(send): inline hidden_template_count into its two call sites
A one-line saturating_sub wrapper exported for one production line (the prompt text) and two assertions — the expression is shorter than its name at every use.
2026-09-24 04:37:54 +08:00
YoursFunny 0765b7deec refactor(send): media_from reads the thumbnail off the item itself
All three callers passed item.thumbnail_url() as the fourth argument — a parameter that could never vary without defeating its own purpose, and only the video arm ever reads it. The function takes the item already; it asks the item. Three call sites lose an argument.
2026-09-24 04:37:53 +08:00
YoursFunny e72cb1b98f refactor(send): inline the per-kind media builders into media_from
photo_media, video_media and animation_media each wrapped the same five lines (build the kind, attach the caption, attach the spoiler) for exactly one caller — media_from's own match arms. The arms carry those lines now; three pub(super) functions and their call indirection are gone, and the dispatch the doc already described as 'the one place' actually is the one place.
2026-09-24 04:37:53 +08:00
YoursFunny 9fc5533676 refactor(twitter): delegate the tweet caption to the shared builder
Tweet::caption duplicated site::caption byte for byte for non-empty text (and dropped the shared empty-text rule, leaving a dangling ': ' behind the author link when a tweet's text comes out empty after link expansion). It delegates now, like bsky's Post::caption a commit earlier and bilibili/misskey before them; encode_double_quoted_attribute loses its last twitter user.
2026-09-24 04:37:53 +08:00
YoursFunny 46bbe8639d refactor(bsky): delegate the post caption to the shared builder
Post::caption hand-formatted exactly the string site::caption produces — same template, same three escapes (the shared fn even handles the empty-text case these two lines did not: no dangling ': ' after the author link). One line now, matching what bilibili and misskey already do; the encode_double_quoted_attribute import loses its last bsky user.
2026-09-24 04:37:53 +08:00
YoursFunny 2ff213e22f refactor(inline): delete InlineKind, convert straight to CachedMediaKind
InlineKind mirrored CachedMediaKind variant for variant (Photo/Video/Gif) and existed only to feed url_result, with two conversions kept in step: Media → InlineKind at the fetch call and CachedMediaKind → InlineKind at the cache call (plus the enum and the mapper). url_result takes CachedMediaKind now — the type the cache path already carries and the fetch path maps to with the same three-arm match — and the degraded-video skip collapses to a guard before the call. One enum fewer between a media kind and its Telegram result.
2026-09-24 04:37:53 +08:00
YoursFunny 2c45b3491d refactor(main): the sweep takes one AppContext instead of five collaborators
periodic_sweep listed sender, chat_store, link_cache, task_queue and config as separate parameters — exactly the five fields AppContext already carries, and exactly what handle_message and the workers pass around as one value. It takes &AppContext now (production via from_statics(&bot), the test via stores.ctx(&sender), which that test already had); CHAT_STORE and LINK_CACHE lose their last direct use in main and leave the import. The collaborator bundle is unchanged, so the paused-clock test drives the same loop.
2026-09-24 04:37:52 +08:00
YoursFunny a3d03cf30f refactor(site): one ffmpeg gate instead of a probe plus a log call
ffmpeg_available() and log_once_ffmpeg_missing() only ever appeared together (both encoders, three lines each: check, log, return), and calling one without the other was a bug in waiting — a probe that never logged, or a log that never gated. ffmpeg_missing() is the single gate: false when the binary is there, otherwise log once per process and say yes. Both call sites shrink to one condition.
2026-09-24 04:37:52 +08:00
YoursFunny e64b48a453 refactor(site): match the site pattern once, then check enabled
find_site walked the registry with "enabled && matches" and disabled_site walked it again with "!enabled && matches" — two passes re-running the same regexes to answer one question. matching_site finds the first match and the fetch decides what a disabled site means; the patterns are disjoint (one domain each), so "first enabled match" and "first match, then check" never disagree.
2026-09-24 04:37:52 +08:00
YoursFunny 022f63916c refactor(site): drop site_id_from_key, split the key at its caller
site_id_from_key split the prefix out and then walked the registry to confirm the prefix was a registered site id — a round trip over a value cache_key itself produced from that registry: the unknown and no-colon branches were unreachable for any key the bot makes. The single caller (the link-cache hit path) splits the prefix directly; the registry-echo test and the misskey assertion of it go with it.
2026-09-24 04:06:13 +08:00
YoursFunny f984428169 refactor(photo): fold photo_plan into its only caller
photo_plan wrapped the header dispatch for exactly one consumer (decode_budget_bytes); plan_photo itself has three callers and stays. The dispatch is inlined into the budget function with its doc merged, saving the wrapper's signature, doc duplication and call indirection.
2026-09-24 04:03:59 +08:00
YoursFunny 85fb12edc7 refactor(photo): read the PNG header with the png crate
parse_png_header hand-decoded the IHDR — byte offsets, the depth byte's five legal values, the color byte's five — roughly thirty lines the png crate already implements (and validates properly: CRC included). It is now Decoder::new + read_info, which is all the header a plan needs; no pixels are decoded. The synthetic test fixture gained the IHDR CRC and an IDAT header (read_info stops at the first IDAT; the hand parser stopped four bytes earlier and checked no CRC), with a nine-line reflected CRC-32 alongside it; the assertions on width/height/depth/color are unchanged, as are the real-file cases.
2026-09-24 04:03:08 +08:00
YoursFunny 7bdf760c7b ops(compose): align the deploy template defaults and gate the healthcheck
Two template landmines for anyone running docker compose from the tracked files. (1) LOCAL_USER_ID defaulted to 1000 in compose and .env.example while the entrypoint and both READMEs say 9001 — the ./data owner on the host silently depended on which doc you read; all four now say 9001. (2) The healthcheck probed WEBHOOK_PORT unconditionally, so the template's own WEBHOOK=false (polling, no listener) shipped a permanently unhealthy container — the probe is now conditional on the interpolated WEBHOOK value (test '<v>' != true || exec 3<>/dev/tcp/…): polling deployments answer healthy without a port, webhook deployments still surface a dead listener to the orchestrator. Both expressions verified locally; compose YAML parses.
2026-09-24 03:47:34 +08:00
YoursFunny 3dbcb45899 test(config): pin the webhook truth table, TTL fallback and blank secrets
Config::load had exactly one test (BOT_ADMIN parsing) while carrying the branches an operator is most likely to mistype: the WEBHOOK truth table (case-insensitive true|yes|1 — 'on' must not enable it), an unparseable EDIT_MESSAGE_TTL_SECONDS (warn + default, not a silent 0 that expires prompts instantly), and empty WEBHOOK_CERT/WEBHOOK_SECRET_TOKEN (compose injects ${VAR:-} as an empty string, which must read as unset, not as a one-character secret). Three table-style tests reuse the existing env save/restore pattern; none of them asserts a field another test reads, so they are safe under parallel execution.
2026-09-24 03:46:59 +08:00
YoursFunny aee1cf35ae ci(docker): run the image build check on PRs that touch crate sources
The pull_request paths filter listed only Dockerfile/entrypoint/.dockerignore/Cargo.{toml,lock}/workflow/crate manifests, so a PR changing only Rust sources never triggered the build-only check — and the Dockerfile's stub-plus-touch layering is exactly the thing that breaks against source structure it has never seen (a new crate directory, a build script). Such PRs now get the check before master/tag reveals the problem; crates/**/Cargo.toml is subsumed by crates/**.
2026-09-24 03:45:35 +08:00
YoursFunny 7b50ad82b0 ci: make the live job honest about what it ran and skipped
Three holes in the live job. (1) It ran -p x-media only, on the comment that everything network-gated lives there — false: xmedia-bot carries two #[ignore]d live tests (repair refetch, text-only link) that CI never executed; both crates run now, still behind the 'live' name filter, and the bot crate's tests need no token (their sends go through MockSender). (2) BILIBILI_COOKIE was never passed, so the bilibili live tests always hit the risk-control early return. (3) That early return — like the pixiv download test's token gate — printed an indistinguishable 'skipping:' line that --show-output never showed, so with continue-on-error a fully-skipped weekly run read exactly like full coverage; skips now print a 'SKIP ' prefix, the steps tee with --show-output, and a summary step lists every skip in the run summary plus a workflow warning. The redundant 'Run token-gated tests' step (the whole non-ignored suite, a rerun of the test job) goes away with the comment that justified it.
2026-09-24 03:45:17 +08:00
YoursFunny a49bb1d1af test(x-media): gate the pixiv download test as live-network too
The pixiv download test was token-gated but not #[ignore]d, so a shell with PIXIV_REFRESH_TOKEN exported made every local cargo test --workspace hit i.pximg.net — where it demonstrably flaked (twice in one session: TLS EOF mid-body) — and CI's non-ignored step ran it as a side effect of having a secret. It carries the convention's #[ignore = "live network: …"] now and is renamed with the live_ prefix so the live job's --ignored live name filter actually picks it up (the old name did not contain 'live' and would have been filtered out). Its skip prints the SKIP prefix the next commit's live job greps, and AGENTS.md stops claiming a local workspace run is not fully offline.
2026-09-24 03:44:06 +08:00
YoursFunny e99f8b0bc0 build: pin ffmpeg to a versioned URL and verify its sha256 unconditionally
The default build fetched /redirect/latest/ — a floating URL that changes under every build — and skipped the sha256 check whenever FFMPEG_SHA256 was empty, which it always was: neither the Dockerfile nor docker.yml carried a hash, so any binary the CDN served reached the image unverified and then processed untrusted media bytes. Both now pin the 9.0.2 release build with the hash the mirror publishes beside it (verified here by direct digest of the downloaded zip: fa8ecf4a…909d7f matches the sidecar), docker.yml's build-args fall back to the same pair, and the check is unconditional — an FFMPEG_URL override without its matching hash fails the build at the download step. The ffmpeg layer also moves above the dependency layer so a manifest/lock edit no longer re-downloads it (the old comment claimed caching 'unless FFMPEG_URL changes', which was never true), and the stale step reference in the sources comment goes with it. Local note: Git-Bash's sha256sum -c reads files in text mode here and cannot roundtrip anything binary — the -c line itself is standard and is exercised by the PR build in CI.
2026-09-24 03:43:04 +08:00
YoursFunny 3fd04e4420 feat(x-media): gate every fetch entry with one process-wide semaphore
The URL workers bound only their own path (8 workers, bounded channel); inline queries, /debug and /test spawned fetches straight from handler tasks with no limit at all — and the expensive part runs inside the fetch (ugoira: up to 512 MiB plus ffmpeg; bsky: an HLS remux), so N users could mean N concurrent encodes. fetch_with_attempts now takes one permit from an 8-slot gate (the count matching URL_WORKERS, so the bot's own pipeline keeps its full width), which covers every entry at once: message links, queue-side repair, inline, /debug, /test, and retries. Unsupported and disabled links answer before the gate, and the shared-fetch dedup already waits outside x-media, so nothing double-counts.
2026-09-24 03:38:53 +08:00
YoursFunny 0e91753402 perf(x-media): give the slot-holding fallback download its own budget
download_media_limited had one hard-coded total (600s) for every caller, and its heaviest caller — the bot's upload fallback — holds a PREP slot (and its memory reservation) for the whole transfer: six slow-but-alive downloads (a byte every 29s satisfies the idle window) could stall the fallback chain for ten minutes, queue retries included. The budget is a parameter now: the fallback passes 300s of its own (50 MiB in 300s ≈ 1.4 Mbit/s; a slower link is better served by retrying toward the item's smaller URL than by pinning a slot), while bsky's in-fetch HLS segments keep the generous 600s DOWNLOAD_TOTAL_TIMEOUT, now pub(crate) and re-exported for them. download_too_slow reports whichever budget it got.
2026-09-24 03:37:38 +08:00
YoursFunny 8f254c3779 perf(x-media): stream the frame zip through a tokio file handle
download_media_to_file took &mut std::fs::File and wrote every network chunk with a sync write_all on the executor thread — for a ugoira frame zip (up to 512 MiB) that is the whole download stalling a runtime worker, while bsky's remux had already been moved to tokio::fs for exactly this reason (its comment: a multi-megabyte std::fs::write blocks the executor thread). The signature takes &mut tokio::fs::File now; the single caller (pixiv's ugoira path) clones the NamedTempFile's handle — the clone shares the file offset, so the ZipArchive extraction in spawn_blocking reads what the download wrote — and drops it after the download hands the bytes to the OS.
2026-09-24 03:33:23 +08:00
YoursFunny e9dd3aece2 perf(send): write the fallback temp file on a blocking thread
download_to_temp wrote the whole buffered body — up to MAX_MEDIA_UPLOAD_BYTES, 50 MiB — with a sync write_all on whatever executor thread ran the prep task; six PREP slots could stall six runtime threads at once on a slow volume (Docker volume, SD card), pausing unrelated handlers and chat-action refreshes with them. The write now runs in spawn_blocking, the pattern the photo decode in the same file already uses; the failure classes are unchanged (write error = retryable resource exhaustion, panic = permanent, worker bugs must be loud).
2026-09-24 03:30:30 +08:00
YoursFunny 67b6bd3038 fix(commands): stop /debug from nesting blockquotes on long posts
debug_report wrapped every caption in <blockquote>, but preview_caption for a post over CAPTION_QUOTE_TEXT_CHARS already carries quote_long_caption's expandable blockquote — and the API rejects nested entities, so /debug on any long post answered 400. The wrapper now skips a caption that already carries a blockquote, exactly the rule quote_long_caption itself applies; the new test pins that the quoted caption passes through with a single blockquote while the existing test keeps pinning the wrap for ordinary captions.
2026-09-24 02:53:49 +08:00
YoursFunny 3fb6b3b4da fix(fetch): one status table — a persistent 4xx is permanent everywhere
status_error's catch-all called every unlisted status Transient, so a 400/405/418/451 got three retries per link before the same answer (twitter syndication's broken-token 400 being the live example), and download_status_error plus misskey's and bilibili's local fallbacks each carried their own copy of the table — bilibili and misskey classifying a 404 as Transient while the center classified it NotFound. The center now makes any client error except 408/429 a refusal (permanent), the media path delegates to it as status_error("media", ...) and its duplicated fn is deleted, and misskey/bilibili fall through to the center after their own special statuses (misskey's 400 body, bilibili's 412). A table test pins every class.
2026-09-24 02:52:10 +08:00
YoursFunny e6800fd27b fix(pixiv): keep a frame-zip download hiccup retryable
download_media_to_file's non-transport errors all folded into PixivError::Api, and Api is permanent in pixiv_error_is_retryable — so one 429/5xx, stalled transfer or temp-file write failure while fetching the ugoira frame zip permanently failed the whole post, while the bot's own upload downloads classify the very same classes as retryable (classify_download_error). A new PixivError::Transient carries those classes into the existing retry policy (and into startup validation's 'stays enabled' branch); Http keeps its arm and everything else stays permanent. The retryable table and the startup-validation loop pin both halves.
2026-09-24 02:50:22 +08:00
YoursFunny 38e65a3791 fix(queue): close the lost-wakeup window on shutdown
stop() flags the shutdown and fires notify_waiters, but a worker parked between its loop-top stop check and its notified() registration — i.e. inside earliest_run_after's DB await — was not registered when the notification fired, so it slept until the next enqueue that never comes; stop() then blocked until main's 30s shutdown timeout force-killed the drain. The sweep had the same window before recover_expired's await and would sit out a full 30s tick. Both loops now enable() the waiter first and re-check the stop flag: either the stop already happened (recheck returns) or the waiter is registered (notify_waiters reaches it) — no gap. The window itself is a scheduling race with no test seam, so this is pinned by reasoning rather than a regression test; the existing stop tests cover the ordinary path.
2026-09-24 02:49:10 +08:00
YoursFunny eecd4320f8 fix(state): keep failed and stale reads from poisoning the chat cache
Three related races in ChatStore. A read that errored (busy/IO) was indistinguishable from an absent row, so the default got cached — and the next update would write that default back over the chat's real settings (forward channel, templates, formats). A cache-miss backfill inserted unconditionally, letting a stale DB snapshot overwrite the value a concurrent set had just written. And sweep eviction removed per-chat locks unconditionally, so a lock pulled out from under an in-flight update let a second writer create a fresh one and enter the critical section concurrently; eviction now keeps any lock with a holder (the same rule rate_limit's prune applies), which still bounds the map because an uncontended leftover is caught by a later sweep. Two regression tests pin the failed-read and contended-lock cases.
2026-09-24 02:47:49 +08:00
YoursFunny d774f0b37c fix(pixiv): treat an empty refresh token as unset
docker-compose injects PIXIV_REFRESH_TOKEN='' for a blank .env value and enabled() only checked that the variable exists, so a default deployment read pixiv as configured, sent an empty refresh token to OAuth and failed login validation on every boot (error line + admin DM) — and it disagreed with the empty-as-unset gating the tests already use. Filter empty strings in PIXIV_CLIENT and enabled(), like twitter and bilibili; a unit test pins empty == unset.
2026-09-24 00:48:29 +08:00
YoursFunny 5d44946690 perf(db): size the pool to cover every DB consumer
POOL_SIZE = 4 was the process-wide cap on concurrent DB operations while 4 queue workers, 8 URL workers, dispatcher handlers and the sweep all share the pool — WAL readers queued behind writers and every hot-path round trip (3-5 per message) contended for four permits. 16 covers every consumer at once; SQLite's single writer serializes writes regardless.
2026-09-24 00:48:29 +08:00
YoursFunny 33b1f04f0e fix(send): release one keep-alive reference per settled task
release_keep_alive retained every KEEP_ALIVE entry whose path matched the settling task, deleting the shared TempDir out from under a concurrent duplicate of the same post (a shared fetch pushes one Arc per pipeline): the duplicate's queued retry then dead-lettered on local media that no longer existed. Release now removes exactly one matching entry, which requires settle to run once per task — handle_task settled on Permanent right before the queue invoked dead_letter_notify, which settles the same payload again, so the redundant settle is dropped. A regression test pins one settle to one entry removed.
2026-09-24 00:48:28 +08:00
YoursFunny 4c1fa857c4 fix(webhook): refuse to start without a secret token
Webhook mode passed the secret token to the axum listener only when WEBHOOK_SECRET_TOKEN was set, and docker-compose defaults it to empty (an empty string counts as unset) — the default deployment therefore ran its listener on a public port with no check on X-Telegram-Bot-Api-Secret-Token, so anyone could POST forged updates and impersonate admins (/bot_dict, /clear_cache, /test). Startup now fails when webhook mode has no secret; .env.example and AGENTS.md spell out the requirement.
2026-09-24 00:48:21 +08:00
YoursFunny 3ebc1e4a8f feat(inline): answer from the link cache, and put the answer on the trait
Inline results were URL-only: Telegram fetches an inline result's URL itself
and cannot send site headers, so every pixiv item (and every locally encoded
ugoira/bsky MP4) was skipped and such a query answered empty. A post that is
already in the link cache now answers with InlineQueryResultCached* built
from its Telegram file ids — no fetch, no upload, and the hotlink-protected
case simply works. A degraded entry (file ids gone) falls back to URLs, and
there a video with no poster is skipped (Telegram would try to render the
mp4 as its own thumbnail).

The answer call moves onto MediaSender (answer_inline_query, mirroring the
other user-flow methods), which is what makes the path testable at all: the
two new tests drive the cache answer and the degraded/empty answer through
TestStores + MockSender, which recorded nothing about inline before.

The result builders are shared by both paths now (url_result/cached_result +
inline_kind), so the fetch path's behaviour is unchanged.
2026-09-21 20:35:18 +08:00
YoursFunny 20a0cf3ea3 feat(pixiv): fall back to the large variant when a page has no original
A page whose `original` the API left out (the restricted ones) was dropped
whole by the multi-page branch's filter_map — a missing picture in the album
while `large` sat right there — and the single-page branch produced no media
at all when both places it looks for an original were empty. Both now share
one builder that falls back to `large` (the same picture, lower resolution).
The two tests that pinned the old behaviour now pin the fallback.
2026-09-21 20:29:48 +08:00
YoursFunny f6523e021d feat(urls): deliver a media-less post as its text
A post with no media was answered with "No media found or media type is not
supported.", throwing away text the fetch had already parsed, escaped and
built a caption for (the per-site format and the long-post quoting
included). It now goes out as a message through the same caption the media
path would attach — the senders' own quoting is applied here, since there is
no sender to do it. No queue entry: there is no Task shape for text and a
post with nothing to download is cheap to paste again, so a failure is
reported (send::send_text_post) rather than retried.

Proven live: live_a_text_only_link_is_sent_as_text fetches a real text-only
tweet through url_media and asserts one send_message carrying the post link
and no media send.
2026-09-21 20:24:40 +08:00
YoursFunny c549a6d35e feat(upload): give videos and animations Telegram's real 50 MB cap
One MAX_UPLOAD_BYTES (10 MiB) bounded every upload, but that is the *photo*
limit: Telegram's own docs say sendVideo/sendAnimation/sendDocument take up
to 50 MB, and RequestEntityTooLarge is "larger than 50 MB". So a 10-50 MB
video that Telegram refused to fetch by URL was refused a download too, and
a video has no smaller variant — the post was lost. The non-photo cap is now
MAX_MEDIA_UPLOAD_BYTES, and such a body charges the process-wide budget for
the length of the preparation (one 64 MiB unit covers the cap), since
PREP_SLOTS alone no longer bounds their added RAM. A const test pins both
caps against Telegram's numbers.
2026-09-21 20:21:01 +08:00
YoursFunny 9e5b6df21c docs: note the payload's MediaRef in AGENTS.md 2026-09-21 18:44:42 +08:00
YoursFunny aa70b45ae6 refactor(send): model a payload's media as MediaRef, not media + file_id
MediaItemPayload carried one String field with two meanings and a file_id
bool beside it to say which, in all three variants; every reader re-checked
the flag (input_file had a three-arm pattern just to find the file-id case).
MediaRef::Source/FileId says it once, and the readers now match on it —
local_media_paths, item_url, input_file and the download path each shrank to
the one branch they care about.

Fixes a real failure that the flag was hiding: send_animation built its
InputFile with input_file_for(media_url), which read a cached file id as a
local path and answered "local media file missing" — permanent. So the
second request for a single-gif post always failed (the third worked, from
the degraded entry). It now uses the payload's own input_file, and
a_cached_animation_sends_by_file_id fails without that line.

Wire shape: a queued row from before this change no longer parses, and the
queue already handles that shape (handle_task dead-letters it as an invalid
payload, and dead_letter_notify still names the post and drops the stale
cache entry).
2026-09-21 18:44:26 +08:00
YoursFunny 70d4f2ec88 refactor(x-media): move the download stack into site/download.rs
site/mod.rs held the registry, the Site trait, Fetched, the caption helpers,
FetchError and, at the end, the whole media-download stack: two HTTP clients
with different timeouts, the guard that refuses a URL inside the host's own
network (start URL and every redirect hop), and the two streaming entry
points. That stack is its own reason to change and moves whole, with its
tests; CLIENT and the download fns stay re-exported at site::… so no adapter
or bot call site moved. SITES becomes pub(crate) for the header rule.
2026-09-21 18:35:58 +08:00
YoursFunny b7763a6572 refactor(send): move the Bot API error policy into send/error.rs
classify_request_error, the marker tables it matches on, Classification and
SendError (with its fallback conversion) are one policy — which failures are
retried, which are permanent, which the reupload fallback owns — and were
interleaved with the payload types and the senders. They move whole into
send/error.rs and are re-exported, so every existing send::… path is
unchanged.
2026-09-21 18:32:07 +08:00
YoursFunny 459bfe5803 refactor(media_sender): move the test support into its own file
Two thirds of media_sender.rs was cfg(test)-only scaffolding (MockSender and
the fake_api), which made the trait's own surface hard to find. The module
becomes media_sender/mod.rs (trait + the Bot impl, 260 lines) plus
media_sender/test_support.rs (445); the path
crate::media_sender::test_support is unchanged, so no caller moved.
2026-09-21 18:30:01 +08:00
YoursFunny 01e097a8b6 refactor(handlers): split urls.rs into workers, pipeline and startup repair
urls.rs carried five reasons to change: the job channel and its worker pool,
the single-flight fetch, URL parsing, the per-URL pipeline, and the startup
repair of queued retries. The two with their own lifecycle move out:

- url_workers.rs: the bounded channel, its supervised pool and
  start/stop_url_workers (the pipeline stays in urls.rs, which the workers
  call).
- repair.rs: needs_refetch/apply_refresh/refetch/repair_lost_local_media with
  their tests, moved whole (the live one keeps its #[ignore]).

Also folds the two byte-identical render_fields -> CachedPost mappings
(urls.rs and repair.rs) into urls::cached_snapshot, and moves the shared
permanent_error test fixture into ctx::test_support.
2026-09-21 18:28:41 +08:00
YoursFunny 435c8c4cc4 refactor(commands): the executor takes its context instead of the statics
execute_command reached for CHAT_STORE/LINK_CACHE/CONFIG in 16 places, which
is why the one handler body with no test could not have one: those statics
point at the real $DATA_DIR/task_queue.db, so any test would have written to
the developer's state. It now takes &AppContext (the shape handle_message
uses) and answers through the given sender; bot stays for what the
MediaSender surface does not carry (channel admin lookups, the HTML report).

Three tests cover what that unlocks: the settings/set_format round-trip
against the chat's own store (including the refused placeholder), the admin
gate on both admin-only commands, and /debug answering without sending.
2026-09-21 18:25:08 +08:00
YoursFunny df8fd77d29 docs: sync AGENTS.md with the refactors
download_media is gone (the bot uses download_media_limited), the per-site
modules no longer export enabled/is_retryable/media_headers for sites that
inherit the trait defaults, and the shared fetch is shared_fetch.
2026-09-21 17:43:43 +08:00
YoursFunny f7179d65ee refactor(db): open_store no longer creates the directory
handlers::db_path already creates the DATA_DIR before it opens
$DATA_DIR/task_queue.db, and every other caller (tests) passes a path whose
parent exists, so the guard plus its one-caller rusqlite_error mapper were
duplicated work. The doc now says which caller owns the directory.
2026-09-21 17:42:56 +08:00
YoursFunny 65c9aa6c5c refactor(send): inline parse_media_url, drop photos_first's rebind
parse_media_url had a single caller (input_file_for) and read better as the
one expression it wrapped; photos_first rebound its argument only to gain
mut.
2026-09-21 17:41:57 +08:00
YoursFunny c58202e683 refactor(send): call send_animation directly
send_animation_inner only reshuffled its arguments into MediaSender's
send_animation; both call sites (URL send and the reupload fallback) now call
the sender themselves, which also drops the InputFile and MediaSender imports.
2026-09-21 17:40:43 +08:00
YoursFunny ea63dbb9f9 refactor(send): inline Task::media_items into its only caller
local_media_paths was the sole caller; the match that flattens the batches
(or the lone animation) moves into it.
2026-09-21 17:39:24 +08:00
YoursFunny 9db1a6093b refactor(main): drop the no-op dispatcher dependency
dptree::deps![""] inserted a &'static str no handler ever asked for;
Dispatcher::builder already starts from an empty DependencyMap, so the call
was the default written out.
2026-09-21 17:38:11 +08:00
YoursFunny bd98001d42 refactor(urls): inline fetch_shared into its one caller
It bound the process-wide map and the site fetch for shared_fetch and had a
single call site; the caller now passes both directly.
2026-09-21 17:37:31 +08:00
YoursFunny 90663a51cc refactor(handlers): inline reply_html into its only caller
The /debug arm was the sole caller; the HTML parse mode and reply decoration
move there with the Requester-disambiguation comment.
2026-09-21 17:36:44 +08:00
YoursFunny 7ecb61399d refactor(handlers): reply returns unit
Every one of its call sites discarded the message id (a bare ?; or let _ =),
so the Result carried a value nothing could use.
2026-09-21 17:35:50 +08:00
YoursFunny 40ab07882d fix(handlers): restore the group hint's link filter
Replacing is_group(kind) with teloxide's predicates dropped the parentheses:
&& binds tighter than ||, so the branch read as is_group() OR (is_supergroup()
AND has-link) -- a plain group got the hint for any link, supported or not.
The e2e test only covered a group with a supported link (which is true either
way); it now also covers an unsupported link in a group.
2026-09-21 17:35:07 +08:00
YoursFunny 676b4bc2ba refactor(rate_limit): one balance() behind tokens() and is_idle()
Both took the lock, refilled and read the field, differing only in the
comparison.
2026-09-21 17:34:25 +08:00
YoursFunny 2d50440bf1 refactor(commands): one require_admin gate for the admin-only commands
/bot_dict and /clear_cache each resolved the sender id and replied "Admin
only." on their own; the helper returns the id or answers the refusal, so
both arms read as one line.
2026-09-21 17:33:45 +08:00
YoursFunny 6b4da18be5 refactor(commands): one cap_text for the two truncated replies
/bot_dict's dump and /debug's report each cut a string at a byte boundary
and appended an ellipsis, with slightly different bookkeeping (one left a
byte for the ellipsis, one did not). cap_text is the version that keeps the
result within the cap, with a test covering the exact-fit, truncating and
multi-byte cases.
2026-09-21 17:32:46 +08:00
YoursFunny 5222cfa1cb refactor(x-media): hoist the shared caption builder
bilibili and misskey each had a byte-identical caption() (same escaping,
same empty-text early return, differing only in a named format argument).
site::caption holds it once; both adapters call it.
2026-09-21 17:29:18 +08:00
YoursFunny dd98a90a45 refactor(state): add ChatData::format_for for the per-site caption format
Four call sites (cache hit, fresh fetch, startup refetch, /debug preview)
spelled out the same message_format.get(...).cloned().unwrap_or_default();
the accessor names the lookup and keeps the empty-format contract in one
place.
2026-09-21 17:27:42 +08:00
YoursFunny 26581cfca1 refactor(handlers): use teloxide's own chat-kind predicates
is_group hand-rolled what Chat::is_group/is_supergroup/is_private already
answer (verified in teloxide-core's chat.rs), and the private check was a
ChatKind pattern match. The unit test that exercised the helper asserted
teloxide's semantics; the channel case it guarded (a channel must not get the
group hint) is now asserted through handle_message instead, next to the group
case.
2026-09-21 17:26:35 +08:00
YoursFunny d9f3ee99c1 refactor(db): add DbPool::with_conn_or for the default-and-log tails
Ten call sites across link_cache, state and queue repeated the same
match/if-let over a with_conn result with their own log line and default.
with_conn_or takes the level, the operation name and the default; each site
keeps its exact message and the same Ok/Err behaviour.
2026-09-21 17:24:51 +08:00
YoursFunny f21570f579 test: share the photo payload fixture, drop two test-only aliases
The one-photo MediaItemPayload literal was written out at six sites across
the urls and send test modules; ctx::test_support::photo_item holds it once.
permanent_error now delegates to api_error (it stays a fn pointer because
that is what MockSender::scripted takes), misskey's x_media_site_id wrapper
is gone in favour of the function it renamed, and PixivAPI is no longer
re-exported -- nothing outside pixiv/api.rs names it.
2026-09-21 17:22:02 +08:00
YoursFunny 7a61077d80 refactor(pixiv): keep the artwork type as the API's string
TypeModel's Illust and Manga variants existed only so serde would accept
those values -- both call sites ask a single question (is this ugoira?), so
the field is a String and the check is a comparison. Same for a type the API
adds later: it no longer fails the whole parse.
2026-09-21 17:19:26 +08:00
YoursFunny 7dd0e8f2d6 refactor(x-media): drop the download_media wrapper
It existed only to pass u64::MAX to download_media_limited and was called
from this file's tests alone; the bot's own download path already calls
download_media_limited directly. The tests now exercise that function, with
the same uncapped bound.
2026-09-21 17:17:43 +08:00
YoursFunny c8a32d49c9 refactor(x-media): read Fetched::site_id instead of the site_name alias
site_name() only returned the public site_id field; the five callers (the
/debug report, the caption-format lookup on both fetch paths, the per-site
override in urls.rs) read the field now.
2026-09-21 17:16:21 +08:00
YoursFunny 4e42855c59 refactor(x-media): drop Media's write-only title field
Every adapter set title (mostly None) on all three Media variants and
nothing ever read it: the bot's CachedMedia carries kind/file_id/url, and the
one read was misskey's own test. Removing it also drops misskey's
DriveFile.name, which existed only to feed it. Fetched::title (the post's
own headline, which captions do use) is untouched.
2026-09-21 17:15:27 +08:00
YoursFunny 9a7cda05c9 refactor(x-media): drop the per-site fns the Site defaults already cover
twitter, bsky, misskey and bilibili each carried enabled() -> true,
media_headers(url) -> None and is_retryable(err) -> the trait's own default,
with no caller outside their tests (the adapters never override those
methods, so the default was already the production policy). The tests that
only restated the default are gone; the two that pin site-specific classes
(bsky's MediaPrep, bilibili's risk-control codes) now ask the Site impl, and
site/mod.rs keeps one assertion of the shared retry policy. The live
verification notes (no Referer needed for hdslb/twimg) survive as comments.
2026-09-21 17:12:42 +08:00
YoursFunny 5630a86d88 refactor(x-media): share the HTTP-status error mapping
twitter, twitter auth and bsky carried the same 404/410 -> NotFound,
401/403 -> Blocked, else Transient block (comments included).
site::status_error holds it once. bilibili and misskey keep their own
matches: neither maps 404/410 and each has a status the others do not
(412 risk control, 400 + NO_SUCH_NOTE), so routing them through the shared
block would have reclassified those statuses for the user.
2026-09-21 17:09:07 +08:00
YoursFunny b9dd1f4d08 refactor(urls): fold the cache-hit caption branch into one call
caption_from_fields already returns truncate_caption(built_in) for an empty
format, which is exactly what the if-branch did; passing cached.caption as
built_in makes both paths one call. The format path is unmoved -- built_in is
read only when the format is empty.
2026-09-21 17:06:46 +08:00
YoursFunny 9aed6f4a24 refactor(config): one parse_opt helper, BOT_ADMIN parsed once
The three webhook settings repeated the same env -> parse -> warn shape
(warn text unchanged); BOT_ADMIN partitioned and then re-parsed every entry,
building a throwaway vector of the bad ones. The ids an operator gets are
unchanged, pinned by a new test.
2026-09-21 17:05:20 +08:00
YoursFunny 3d377f68fb refactor(photo): one plan decides within-limits and decode budget
decode_budget_bytes mirrored the within-limits early return and the
MAX_DECODE_BYTES guard that prepare_png and prepare_jpeg each spelled out
inline (three copies of the same arithmetic, which the reservation and the
branches had to keep in sync by hand). PhotoPlan/plan_photo now hold that
decision once and all three call it; the log order and the exact bounds are
unchanged. decode_budget_follows_the_processing_decision still pins the
reservation against the branches.
2026-09-21 17:03:22 +08:00
YoursFunny 5166d97545 refactor(db): use rusqlite query_row/optional instead of hand-rolled reads
pending_backlog, earliest_run_after, ChatStore::get and LinkCache::get each
hand-rolled prepare + query + rows.next() for what is a single-row read.
query_row + OptionalExtension::optional is the same statement and the same
error mapping with less scaffolding; the backlog's NULL-on-empty MIN still
goes through the count check, so a pending row with a NULL run_after is not
misread. Also fixes the rustfmt drift from the previous commit.
2026-09-21 17:01:38 +08:00
YoursFunny ebe7b8bdd7 refactor(send): one payload to InputMedia dispatch instead of three
media_from_file, media_from_url and build_media_group's closure each wrote
the same per-kind match plus the same video-thumbnail attach. media_from
takes the already-selected InputFile; the two builders that differ only in
how that file is chosen are now two-line calls to it. The local-file branch
keeps InputFile::file (no existence probe) and every caller still passes the
item's own has_spoiler / thumbnail_url, so what reaches Telegram is the same.
2026-09-21 16:59:54 +08:00
YoursFunny a5187981c7 refactor(send): share one task constructor between fresh sends and startup repair
build_send_task and apply_refresh each wrote the same animation-vs-sequence
branch and the same two 13-field literals; Task::from_items takes the
delivery envelope (chat, reply, forward/edit settings, notify targets) once.
The two callers keep computing that envelope from their own source -- chat
settings vs. the queued row being replaced -- so the repair path's delivery
semantics are unchanged.
2026-09-21 16:58:39 +08:00
53 changed files with 5549 additions and 3508 deletions
+8 -1
View File
@@ -49,11 +49,14 @@ DATA_DIR=data
# --- webhook deployment (docker-compose.yml) ------------------------------
# false = long polling (no public URL needed). true = webhook behind the
# bundled nginx-proxy — and then WEBHOOK_LISTEN/PORT/URL are required.
# The compose healthcheck probes the listener only when this is true.
WEBHOOK=false
# WEBHOOK_LISTEN=0.0.0.0
# WEBHOOK_PORT=8443
# WEBHOOK_URL=https://your.domain/
# Validation token Telegram echoes back as X-Telegram-Bot-Api-Secret-Token.
# Required when WEBHOOK=true: the bot refuses to start without one (use a
# random value of 16+ chars — without it the listener accepts any request).
# WEBHOOK_SECRET_TOKEN=
# Self-signed certificate path, used only for Telegram-side validation (TLS is
# terminated by the reverse proxy); unneeded with acme-companion. Not passed by
@@ -68,7 +71,11 @@ VIRTUAL_PORT=8443
# Certificate notification address for acme-companion.
DEFAULT_EMAIL=
# UID the container runs as; it must be able to write ./data on the host.
LOCAL_USER_ID=1000
# The entrypoint's default (and the README's) is 9001 — keep them equal so
# the file owner on the host matches what you expect. Must be a non-zero
# numeric uid: the entrypoint refuses 0 (the bot would keep root through the
# privilege drop) and anything non-numeric.
LOCAL_USER_ID=9001
# Uncomment (here and the matching line in docker-compose.yml) to have
# acme-companion issue the certificate for VIRTUAL_HOST.
# ACME_HOST=
+23 -8
View File
@@ -28,6 +28,7 @@ name: CI
on:
push:
branches: [master]
tags: ['v*']
pull_request:
schedule:
# Weekly probe of the live endpoints, so external API changes surface.
@@ -140,15 +141,29 @@ jobs:
env:
PIXIV_REFRESH_TOKEN: ${{ secrets.PIXIV_REFRESH_TOKEN }}
TWITTER_AUTH_TOKEN: ${{ secrets.TWITTER_AUTH_TOKEN }}
BILIBILI_COOKIE: ${{ secrets.BILIBILI_COOKIE }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
# Everything network- or secret-gated lives in x-media, and the bot
# crate's suite (MockSender + tempdir stores, no network) already ran in
# the `test` job — rebuilding it here bought nothing.
- name: Run token-gated tests
run: cargo test -p x-media --locked
# The live-network tests, by the "live" name filter (all #[ignore]d).
- name: Run live-network tests
run: cargo test -p x-media --locked -- --ignored live
# All #[ignore]d live tests, by the "live" name filter, in both crates:
# the bot crate has two of its own (repair refetch, text-only link)
# whose fetches are network-bound but whose sends go through MockSender.
# --show-output keeps each test's stdout — the SKIP lines printed by a
# skipped test are what the summary step below greps, so a skip stays
# distinguishable from a pass.
- name: Run live-network tests (x-media)
run: cargo test -p x-media --locked -- --ignored live --show-output 2>&1 | tee live-x-media.log
- name: Run live-network tests (xmedia-bot)
run: cargo test -p xmedia-bot --locked -- --ignored live --show-output 2>&1 | tee live-bot.log
# A green live run must not be able to mean "nothing actually ran"
# (pixiv without its secret, bilibili risk-controlling the runner IP):
# collect every SKIP line into the run summary.
- name: Surface skipped live tests
if: always()
run: |
skips=$(grep -h '^SKIP ' live-*.log 2>/dev/null || true)
if [ -n "$skips" ]; then
echo "::warning::live job skipped tests, see the job summary"
{ echo "### Live tests skipped"; echo "$skips" | sed 's/^/- /'; } >> "$GITHUB_STEP_SUMMARY"
fi
+11 -6
View File
@@ -21,7 +21,10 @@ on:
- Cargo.toml
- Cargo.lock
- .github/workflows/docker.yml
- 'crates/**/Cargo.toml'
# Any crate source, not just manifests: the stub/touch layering only
# breaks against real source structure, which a manifest-only PR never
# exercises.
- 'crates/**'
env:
APP_NAME: telegram-twitter-media-bot
@@ -150,9 +153,11 @@ jobs:
# would give every new tag a cold cache on release builds. PR runs only
# read it (cache-to is empty) so they cannot evict the release cache.
#
# FFMPEG_URL/FFMPEG_SHA256 come from repository variables when set, so a
# release can pin an exact ffmpeg build (the Dockerfile default follows
# the project's `/redirect/latest/` URL, which has no sha256 sidecar).
# FFMPEG_URL/FFMPEG_SHA256 come from repository variables when set —
# both or neither: the Dockerfile checks the sha256 unconditionally, so
# a URL without its matching hash fails the build. The fallbacks pin the
# same 9.0.2 release the Dockerfile defaults to (keep the three in step
# when bumping).
#
# Single-arch (amd64) on purpose: adding arm64 means re-adding
# `docker/setup-qemu-action`, `platforms: linux/amd64,linux/arm64`, and
@@ -164,8 +169,8 @@ jobs:
push: ${{ github.event_name != 'pull_request' }}
build-args: |
APP_NAME=${{ env.APP_NAME }}
FFMPEG_URL=${{ vars.FFMPEG_URL || 'https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffmpeg.zip' }}
FFMPEG_SHA256=${{ vars.FFMPEG_SHA256 }}
FFMPEG_URL=${{ vars.FFMPEG_URL || 'https://ffmpeg.martin-riedl.de/download/linux/amd64/1789931100_9.0.2/ffmpeg.zip' }}
FFMPEG_SHA256=${{ vars.FFMPEG_SHA256 || 'fa8ecf4abbd290d98f7d188b8649cc6b391ae209a98452be955a15aab1909d7f' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=tgxmb-build
+22 -22
View File
@@ -4,7 +4,7 @@
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, Bluesky, Misskey (misskey.io), and Bilibili dynamics into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README is in Chinese; user-facing bot strings are in English. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
Two-crate Cargo workspace (both v1.9.1, edition 2024, resolver 3):
Two-crate Cargo workspace (both v1.9.2, edition 2024, resolver 3):
- **`crates/x-media`** — library that fetches and normalizes media from the four sites. Pure, no Telegram knowledge.
- **`crates/xmedia-bot`** — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.
@@ -18,7 +18,7 @@ Telegram update → Dispatcher (polling or axum webhook) → dptree branches
└─ callback_query → "forward" (copy to channel) / "template|<name>" (apply caption template)
```
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)` → `Fetched` → builds a `Task` → `send::send_media_sequence` (media groups ≤ 10, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue (it reports whether the row was really written, and only then does the user get the "retrying in Ns" notice — an enqueue that fails says so instead) → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s for the bot's own delays, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)` → `Fetched` → builds a `Task` → `send::send_media_sequence` (media groups ≤ 10, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media_limited` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue (it reports whether the row was really written, and only then does the user get the "retrying in Ns" notice — an enqueue that fails says so instead) → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s for the bot's own delays, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
Debug command: `/debug <url>` runs the same `x_media::site::fetch` and replies with `debug_report` (`handlers/commands.rs`) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars and sent with HTML parse mode: raw fields are escaped, and the caption is wrapped in a `<blockquote>` so it renders exactly like the sent media caption (escaped text and links included). The caption it shows is `preview_caption`'s: the chat's per-site format override plus the long-post quoting, i.e. exactly what the send paths produce — showing the raw built-in caption made `/set_format` look like a no-op, and the `/set_format` success reply points users at `/debug` to preview.
@@ -26,7 +26,7 @@ User-facing failure text is a function of the error class, never one generic sen
The `/test <url>` command runs the ordinary link pipeline (`urls::url_media`) with `PostSend::Suppressed`: the media is sent and cached like any other link, but the chat's `forward_channel_id`/`edit_before_forward` are ignored, so a test never forwards to the channel and never opens the edit prompt (retries and dead-letter notifications behave as usual). `/test`, `/debug`, `/set_format` and `/clear_cache` use the custom `parse_arg_remainder` parser (whole remainder, trimmed) because teloxide's built-in `split` parser takes exactly one space-separated token per field: `/set_format <site> <format>` never parsed with it (and `/clear_cache` without an argument did not either), and a command that fails to parse falls through to the URL flow in silence. `commands::tests::every_documented_invocation_parses` pins every documented form against exactly that.
The inline path (`handlers/inline.rs`) hands media URLs straight to Telegram, which fetches them itself and cannot send site-specific headers — so `x_media::site::needs_media_headers(url)` (true exactly where a site's `media_headers` is non-empty, i.e. pixiv's pximg.net) marks the media that must be skipped instead of shipped broken; locally produced media (ugoira MP4, bsky remux) fails `Url::parse` and is skipped the same way. Inline results are therefore URL-only by construction, and a query whose every item was skipped is answered *empty* (with a cache window) rather than left unanswered — an unanswered query keeps the client spinning and, through the debounce's release, re-runs the fetch on every keystroke.
The inline path (`handlers/inline.rs`) answers from the **link cache** first: a post already sent somewhere answers with `InlineQueryResultCached*` built from its Telegram file ids, so no fetch happens and — unlike a URL result — media Telegram could never fetch itself still works (pixiv's pximg.net, a locally encoded ugoira/bsky MP4). Only a cache miss fetches (`fetch_once`), and then the media URLs go straight to Telegram, which fetches them itself and cannot send site-specific headers — so `x_media::site::needs_media_headers(url)` (true exactly where a site's `media_headers` is non-empty, i.e. pixiv's pximg.net) marks the media that must be skipped instead of shipped broken; locally produced media (ugoira MP4, bsky remux) fails `Url::parse` and is skipped the same way. A degraded cache entry keeps only URLs, so it takes the same URL path (a video with no poster is skipped there, Telegram has no thumbnail to show). A query whose every item was skipped is answered *empty* (with a cache window) rather than left unanswered — an unanswered query keeps the client spinning and, through the debounce's release, re-runs the fetch on every keystroke. The answer goes through `MediaSender::answer_inline_query` (the trait carries it so the path is mock-testable; `inline.rs`'s own tests cover the cache and degraded-entry answers offline).
`url_media` is a thin wrapper over `url_media_inner`: `run_with_chat_action` sends the chat action, then re-sends it every `ACTION_REFRESH` (4 s) while the pipeline future is pending, because Telegram drops an action after ~5 s and a fetch (ugoira encode, HLS remux) plus an upload routinely outlasts that. The pipeline flips the shared `ActionHint` from `Typing` to `UploadPhoto`/`UploadVideo` once the media kinds are known. The `select!` is `biased` on the pipeline branch so a finished pipeline never emits a stray action.
@@ -37,17 +37,17 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr
| Path | Purpose |
|---|---|
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media*` (the streaming `download_media_to_file` and the capped `download_media_limited`, which is where a download's size and its total time budget are enforced); `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky\|misskey\|bilibili>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`; without the token a withheld tweet stays `FetchError::Sensitive` and the bot reports it as age-restricted instead of "no media"). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Bilibili fetches dynamics (images/animated images only — an attached video degrades to its cover, and its title stands in for the post text, which AV dynamics do not have) from `/x/polymer/web-dynamic/v1/detail` sent with `features=itemOpusStyle` (without that flag the legacy serialization drops an image/text post's body and headline entirely — `desc` comes back `null`; the adapter still parses the legacy `major.draw`/`desc`/`archive` shapes as a fallback). No WBI signature is involved; device cookies `buvid3`/`buvid4` are fetched automatically from `/x/frontend/finger/spi` because bilibili's `-352` risk control starts rejecting plain requests, `BILIBILI_COOKIE` is the escalation when an IP stays blocked; `b23.tv` short links are deliberately unmatched. Twitter's `from_syndication_value` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`&gt;` `&lt;` `&amp;` `&#39;`) — so the stored text is raw and the caption escap…
| `crates/x-media/src/site/<twitter\|pixiv\|bsky\|misskey\|bilibili>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (`PATTERN`, `fetch_from_url()`, `cache_key`, the unit struct `<Name>Site` implementing `site::Site` — the trait supplies the `enabled`/`is_retryable`/`media_headers` defaults unless the site differs, as pixiv does — and `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`; without the token a withheld tweet stays `FetchError::Sensitive` and the bot reports it as age-restricted instead of "no media"). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Bilibili fetches dynamics (images/animated images only — an attached video degrades to its cover, and its title stands in for the post text, which AV dynamics do not have) from `/x/polymer/web-dynamic/v1/detail` sent with `features=itemOpusStyle` (without that flag the legacy serialization drops an image/text post's body and headline entirely — `desc` comes back `null`; the adapter still parses the legacy `major.draw`/`desc`/`archive` shapes as a fallback). No WBI signature is involved; device cookies `buvid3`/`buvid4` are fetched automatically from `/x/frontend/finger/spi` because bilibili's `-352` risk control starts rejecting plain requests, `BILIBILI_COOKIE` is the escalation when an IP stays blocked; `b23.tv` short links are deliberately unmatched. Twitter's `from_syndication_value` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`&gt;` `&lt;` `&amp;` `&#39;`) — so the stored text is raw and the caption escap…
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands` — `setMyCommands` plus the profile description texts), shared `send::BOT` force-init, startup sweep of this project's leftover temp files (`x_media::TEMP_FILE_PREFIX` + an age gate, since a killed process runs no destructors), startup repair of queued retries whose local media did not survive a restart (`handlers::repair_lost_local_media`, before any worker can lease: those rows are re-fetched from their `source_url`), queue worker start, site login validation (`site::validate_all`), `periodic_sweep` (`SWEEP_INTERVAL` 300 s): expired prompts are rewritten in place to `EDIT_PROMPT_EXPIRED_TEXT` with an empty keyboard — an edit, never a new message, so a background timer cannot wake a chat — plus the link-cache prune, the idle rate-limit buckets and the idle inline-query entries, and the queue backlog line (only when non-empty). Takes its collaborators rather than the statics so its loop is testable with a paused clock, dptree handler tree, webhook vs polling dispatch |
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
| `crates/xmedia-bot/src/db.rs` | `DbPool`: one shared SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) for all three tables over `$DATA_DIR/task_queue.db` (default `data/`) — the three stores share it; `open_store` creates file + schema and then applies the `PRAGMA user_version` migration chain (`MIGRATIONS` + `migrate` — append-only; `schema_init` is the version-0 baseline and must not gain columns an existing database would never receive — `db.rs`'s tests pin a pre-migration database upgrading intact, the shipped migration text frozen (appending is the only allowed change) and a fresh database landing at the latest version), `with_conn` runs all rusqlite I/O in `spawn_blocking` |
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`, the group-only `GROUP_LINK_HINT` for a supported link posted outside a private chat), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. `/test <url>` (send-only) / `/debug <url>` (parse-only) and the admin-only `/bot_dict` state dump; `/set_format` rejects unknown `{…}` placeholders and resets with `-`), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency; one *shared* in-flight fetch per cache key (`fetch_shared`: a second chat, a batch forward or a retry asking for the same post meanwhile waits for the first caller's result, the entry is dropped the moment the fetch settles so nothing is ever answered from an old fetch, and a waiter whose sharer was cancelled fetches for itself); plus the startup repair `repair_lost_local_media`, whose decision (`needs_refetch`) and rewrite (`apply_refresh`) are pure and tested while the fetch itself is a live test), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons, incl. `skip`; a forward that fails retryably is both queued *and* settles the prompt — the queued row carries the message ids itself, and a prompt left live let a second Confirm copy the same messages twice and let Skip answer "nothing was forwarded" while the row still delivered), `statics.rs` (global statics) |
| `crates/xmedia-bot/src/db.rs` | `DbPool`: one shared SQLite connection pool (`POOL_SIZE = 16`, WAL, busy_timeout) for all three tables over `$DATA_DIR/task_queue.db` (default `data/`) — the three stores share it; `open_store` creates file + schema and then applies the `PRAGMA user_version` migration chain (`MIGRATIONS` + `migrate` — append-only; `schema_init` is the version-0 baseline and must not gain columns an existing database would never receive — `db.rs`'s tests pin a pre-migration database upgrading intact, the shipped migration text frozen (appending is the only allowed change) and a fresh database landing at the latest version), `with_conn` runs all rusqlite I/O in `spawn_blocking` |
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`, the group-only `GROUP_LINK_HINT` for a supported link posted outside a private chat), `url_workers.rs`/`repair.rs` (worker pool; startup repair), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. `/test <url>` (send-only) / `/debug <url>` (parse-only) and the admin-only `/bot_dict` state dump; `/set_format` rejects unknown `{…}` placeholders and resets with `-`), `urls.rs` (URL extraction + the per-URL pipeline; one *shared* in-flight fetch per cache key (`shared_fetch`+`IN_FLIGHT_FETCHES`: a second chat, a batch forward or a retry asking for the same post meanwhile waits for the first caller's result, the entry is dropped the moment the fetch settles so nothing is ever answered from an old fetch, and a waiter whose sharer was cancelled fetches for itself); plus the startup repair `repair_lost_local_media`, whose decision (`needs_refetch`) and rewrite (`apply_refresh`) are pure and tested while the fetch itself is a live test), `url_workers.rs` (the bounded job channel (256) and its `URL_WORKERS = 8` supervised workers, `start_url_workers`/`stop_url_workers` — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential, so batch-forwards need this concurrency), `repair.rs` (startup `repair_lost_local_media` with its `needs_refetch`/`apply_refresh`/`refetch`), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons, incl. `skip`; a forward that fails retryably is both queued *and* settles the prompt — the queued row carries the message ids itself, and a prompt left live let a second Confirm copy the same messages twice and let Skip answer "nothing was forwarded" while the row still delivered), `statics.rs` (global statics) |
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table); the 300 s sweep's `prune_expired` evicts any chat with no live edit-before-forward prompt, so the cache (and the per-chat lock map) stays bounded to active prompts — durable settings reload from the DB on next use |
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + the source media URLs + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune; a permanent send failure *degrades* the entry instead of dropping it (the file ids go, the URLs stay, so the next request re-sends from those without a fetch), and a degraded entry that fails again is removed |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, a `lease_token` fence: `lease_next` stamps a random token and every write-back (heartbeat, `delete`, `reschedule`, `mark_done`) is guarded by it, so a lease that expired and was re-leased cannot be written by its former holder — a lost lease stops the attempt instead; a finished row's `DELETE`/reschedule retried and a failed delete falling back to a `done` tombstone (the lease query and the sweep only look at `pending`/`in_progress`, so a task that already ran cannot be resurrected and re-run), `runnable_rows`/`replace_payload` (the startup repair's read/rewrite path: it runs before the workers exist, which is why it needs no lease token), `notify_one` worker wakeup plus a separate `Notify` for the 30 s lease-expiry sweep (a shared one let the sweep steal the workers' wakeup permit; the sweep does notify the workers after it actually recovered a row, since a recovered task is due immediately while every worker may be parked on `notify` with no pending row to sleep on), `busy_timeout` on all connections |
| `crates/xmedia-bot/src/ctx.rs` | `AppContext`: the injected collaborators (`sender` + `ChatStore`/`PersistentTaskQueue`/`LinkCache`/`Config`), `from_statics` for production and the `CONTEXT` static the worker closures hold. `test_support::TestStores` backs handler tests with a tempdir store set, and the module also carries the fixtures those tests share — the canonical cached post (`cached_photo`), the edit-before-forward prompt (`seed_prompt` with its `PROMPT_ID`/`FORWARDED_ID`) and a scripted API error (`api_error`) — so no two test modules keep their own copies |
| `crates/xmedia-bot/src/send/` | `send/mod.rs`: `Task`/`MediaItemPayload` payloads, `SendError`/`Classification`, `send_media_sequence`/`send_animation`/`forward_messages`; `send/input_media.rs`: payload → `InputFile`/`InputMedia` + `build_media_group` (caption on the first item only); `send/upload.rs`: the download-and-reupload fallback (`prepare_upload_item`/`send_batch_via_upload`, photo downscale handoff); `send/post_send.rs`: link-cache write, `KEEP_ALIVE` registry, `settle_task`, `post_send_actions`, `handle_task`/`dead_letter_notify` |
| `crates/xmedia-bot/src/media_sender.rs` | `MediaSender` trait: the user-flow surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`answer_callback_query`/`edit_message_text`/`edit_message_caption`/`delete_message`/`send_chat_action`) implemented by teloxide `Bot` (per-chat rate-limited) and by a recording `MockSender` in tests. Admin/setup APIs (`get_chat`, `set_my_commands`, …) stay on the concrete `Bot`. `test_support` holds the scripted `MockSender` and `fake_api` (the stand-in API the real-`Bot` tests drive) |
| `crates/xmedia-bot/src/send/` | `send/mod.rs`: `Task`/`MediaItemPayload` payloads (`media: MediaRef`, i.e. `Source` URL-or-path vs `FileId` — one field used to carry both with a flag), `send_media_sequence`/`send_animation`/`forward_messages`; `send/error.rs`: the Bot API error policy (`SendError`/`Classification`, `classify_request_error`, the media-fetch/size markers); `send/input_media.rs`: payload → `InputFile`/`InputMedia` + `build_media_group` (caption on the first item only); `send/upload.rs`: the download-and-reupload fallback (`prepare_upload_item`/`send_batch_via_upload`, photo downscale handoff); `send/post_send.rs`: link-cache write, `KEEP_ALIVE` registry, `settle_task`, `post_send_actions`, `handle_task`/`dead_letter_notify` |
| `crates/xmedia-bot/src/media_sender/{mod.rs,test_support.rs}` | `MediaSender` trait: the user-flow surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`answer_callback_query`/`edit_message_text`/`edit_message_caption`/`delete_message`/`send_chat_action`/`answer_inline_query`) implemented by teloxide `Bot` (per-chat rate-limited) and by a recording `MockSender` in tests. Admin/setup APIs (`get_chat`, `set_my_commands`, …) stay on the concrete `Bot`. `test_support.rs` (cfg(test)-only) holds the scripted `MockSender` and `fake_api` (the stand-in API the real-`Bot` tests drive) |
| `crates/xmedia-bot/src/rate_limit.rs` | Two token buckets paced before sends reach the API so batch forwards don't trip flood control: one per chat (`CAPACITY = 20`, ~20 msg/min refill) and one bot-wide (`acquire_global`, 30/s — Telegram's per-bot ceiling, invisible to any per-chat bucket and only binding when a batch fans out over many chats). `prune_idle` drops the per-chat buckets that refilled while unheld |
## Development Commands
@@ -56,7 +56,7 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr
export TELOXIDE_TOKEN=<token> # required; PIXIV_REFRESH_TOKEN optional (Pixiv disabled without it)
cargo run -p xmedia-bot # run the bot (polling by default)
cargo run -p x-media --example fetch -- <url> # test a link through the fetch library
cargo test --workspace # full test suite (no CI test step exists — run locally)
cargo test --workspace # full test suite (CI runs the same, with --locked)
cargo build --release -p xmedia-bot # release build (Dockerfile does this)
cargo clippy --workspace --all-targets # lint (Clippy is the configured IDE linter)
cargo fmt --check # formatting
@@ -66,28 +66,28 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
## Code Conventions & Common Patterns
- **Errors via `thiserror` derive** (no anyhow): the public, stringified errors — `FetchError` (`Http`/`Json`/`Pixiv`/`Site`/`NotFound`/`Blocked`/`Disabled`/`Sensitive`/`TooLarge`/`MediaPrep`/`Transient`/`Io`) and `PixivError` — derive `thiserror::Error` with `#[from]` conversions; `Display`/`source()` come from the derive. The internal control-flow enums — `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `Classification`, `FallbackError` — carry no `Display` and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive `thiserror`, internal flow enums stay plain.
- **Errors via `thiserror` derive** (no anyhow): the public, stringified errors — `FetchError` (`Http`/`Json`/`Pixiv`/`Site`/`NotFound`/`Blocked`/`Disabled`/`Sensitive`/`TooLarge`/`MediaPrep`/`Transient`/`RateLimited`/`Io`) and `PixivError` — derive `thiserror::Error` with `#[from]` conversions; `Display`/`source()` come from the derive. The internal control-flow enums — `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `Classification`, `FallbackError` — carry no `Display` and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive `thiserror`, internal flow enums stay plain.
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers/statics.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers share the process-wide `send::BOT` (`LazyLock<Bot>`, force-initialized in `main` so a missing token fails at startup).
- **Async**: tokio multi-thread runtime (`#[tokio::main]` default). All rusqlite I/O inside `tokio::task::spawn_blocking`. Long loops use `tokio::select!` with `tokio::sync::{watch, Notify}` stop/wake channels. No streams.
- **Blocking sync primitives**: `parking_lot::Mutex` for hot caches, `tokio::sync::Mutex` for async-shared state (pixiv token cache), `AtomicBool` for feature gates.
- **Site adapter convention**: each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`, plus `cache_key`/`is_retryable`/`media_headers`, and a unit struct `<Name>Site` implementing `site::Site`; the central dispatcher (`site/mod.rs`) only iterates the `SITES` registry. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one `Box::new(...)` entry in `SITES` — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return `SiteFuture` (a boxed `Pin<Box<dyn Future + Send>>`) because `async fn` in traits is not dyn-compatible.
- **Site adapter convention**: each site module exports `PATTERN: LazyLock<Regex>`, `fetch_from_url(url) -> Result<Fetched, FetchError>` and `cache_key`, plus a unit struct `<Name>Site` implementing `site::Site`; `enabled`/`is_retryable`/`media_headers` come from the trait's defaults unless the site overrides them (only pixiv does); the central dispatcher (`site/mod.rs`) only iterates the `SITES` registry. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one `Box::new(...)` entry in `SITES` — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return `SiteFuture` (a boxed `Pin<Box<dyn Future + Send>>`) because `async fn` in traits is not dyn-compatible.
- **Serde**: per-site `model.rs` are pure `Deserialize` DTOs mirroring API JSON; site structs in `interface.rs` have private fields, a `caption()` builder, and `impl From<SiteStruct> for Fetched`. Persisted payloads use internally-tagged enums (`#[serde(tag = "kind")]` / `type`).
- **Naming**: module-per-concern, snake_case files, `CamelCase` types, `snake_case` fns. `//!` module docs and `///` docs on non-obvious logic (syndication token, ugoira encoding, `display_text_range`).
- **Retries**: only `x-media::site::fetch` retries (3 attempts, `1 << attempt` backoff, HTTP errors only); `site::fetch_once` is the same code path with a single attempt, used by inline queries whose answer window is shorter than the backoff. A status a site answers with is classified by what a *retry* can change: 404/410 are `NotFound` and 401/403 are `Blocked` (permanent, reported at once), 429/5xx are `Transient` and retried. Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`), scaled per attempt by `scaled_retry_delay` — which only ever scales **up**, so a delay the server asked for (Telegram `retry_after`) is never shortened. `send::classify_request_error` is the send-side counterpart: `RetryAfter` and `Network` are retryable, and so is a 5xx — teloxide sleeps 10 s on a server error and then parses the body, so by then the HTTP status is gone and the condition is recognised by shape instead (a JSON server-error description, or an `InvalidJson` whose raw body is not JSON, i.e. a proxy/error page).
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`). `main.rs` initializes the **timed** builder with a default filter of `info,hyper_util=warn,reqwest=warn` when `RUST_LOG` is unset: the plain `init` had no timestamps and fell back to `error`, so a deployment that forgot the variable logged nothing at all, and at `debug` the HTTP client's own lines outnumbered the bot's two to one. An explicit `RUST_LOG` overrides the default wholesale. Level convention: `info` = lifecycle + per-post business results (`sent`/`forwarded`/`copied`, with `chat=` and the total `ms`), admin/operator actions and anomalies (fallback, retry enqueue, dead-letter is `error`); `debug` = per-request detail (URL extraction, `fetching`/`fetched` with the fetch duration, batch sends, queue processing with the row's `chat=`/`key=` and per-attempt `ms`, photo processing, inline queries); `trace` = user data (the full URL, the message text, the inline query). At `debug` and above links are printed via the normalized cache key (`handlers::log_key`, e.g. `[key=twitter:123...]`), so a `debug` log can be shared without echoing what users pasted, and degradations that leave the user served (a failed cache read/write, a failed chat action) are `warn`, not `error`. The only queue/sweep aggregate is the 300 s sweep's queue line, and it speaks only when the queue is non-empty.
- **Retries**: only `x-media::site::fetch` retries (3 attempts, a doubling backoff widened by a random slice of itself so workers that failed together do not recover together, with a 429's `Retry-After` honored up to `MAX_RETRY_AFTER_SECS` = 60 s, over HTTP errors only); `site::fetch_once` is the same code path with a single attempt, used by inline queries whose answer window is shorter than the backoff. A status a site answers with is classified by what a *retry* can change: 404/410 are `NotFound` and 401/403 are `Blocked` (permanent, reported at once), 429/5xx are `Transient` and retried. Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`), scaled per attempt by `scaled_retry_delay` — which only ever scales **up**, so a delay the server asked for (Telegram `retry_after`) is never shortened. `send::classify_request_error` is the send-side counterpart: `RetryAfter` and `Network` are retryable, and so is a 5xx — teloxide sleeps 10 s on a server error and then parses the body, so by then the HTTP status is gone and the condition is recognised by shape instead (a JSON server-error description, or an `InvalidJson` whose raw body is not JSON, i.e. a proxy/error page).
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`). `main.rs` initializes the **timed** builder with a default filter of `info,hyper_util=warn,reqwest=warn` when `RUST_LOG` is unset: the plain `init` had no timestamps and fell back to `error`, so a deployment that forgot the variable logged nothing at all, and at `debug` the HTTP client's own lines outnumbered the bot's two to one. An explicit `RUST_LOG` overrides the default wholesale. Level convention: `info` = lifecycle + per-post business results (`sent`/`forwarded`/`copied`, with `chat=` and the total `ms`), admin/operator actions and anomalies (the upload fallback and other user-served degradations are `warn`; retry enqueue and dead-letter are `error`); `debug` = per-request detail (URL extraction, `fetching`/`fetched` with the fetch duration, batch sends, queue processing with the row's `chat=`/`key=` and per-attempt `ms`, photo processing, inline queries); `trace` = user data (the full URL, the message text, the inline query). At `debug` and above links are printed via the normalized cache key (`handlers::log_key`, e.g. `[key=twitter:123...]`), so a `debug` log can be shared without echoing what users pasted; user-supplied text that does reach a line (display names, callback data, channel handles) goes through `handlers::log_escape`, whose escapes keep a crafted value from splitting or forging a log entry, and degradations that leave the user served (a failed cache read/write, a failed chat action) are `warn`, not `error`. The only queue/sweep aggregate is the 300 s sweep's queue line, and it speaks only when the queue is non-empty.
## Important Files
| File | Why it matters |
|---|---|
| `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) |
| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `mod.rs` also holds `apply_caption_edit`, the one place a caption edit is applied and its failure classified: a short retryable delay is retried once, anything else is reported to the user instead of being swallowed (`callback.rs`'s template button answers its toast with the failure and leaves the record alone); `commands.rs` = command dispatch (incl. `/test <url>` send-only, `/debug <url>` parse-only, the read-only `/settings` every chat member can read — unlike the admin-only `/bot_dict` raw dump — and template removal; `/start`/`/help` carry the guidance teloxide's `descriptions()` cannot render, and `/set_format` rejects unknown `{…}` placeholders, resetting with `-`); `urls.rs` = URL extraction + the per-URL pipeline (`url_media` takes a `PostSend` mode: chat settings vs `/test`'s suppressed actions); `inline.rs` = debounced inline queries (hotlink-protected and local media skipped); `callback.rs` = edit-before-forward buttons (dptree entry + testable `handle_callback` core, incl. `skip`) |
| `crates/xmedia-bot/src/send/` | `mod.rs`: constants `MAX_MEDIA_GROUP = 10`; `classify_request_error` (5xx/non-JSON bodies retry, see the Retries bullet) and the media-fetch markers that route a URL send into the reupload fallback — including `failed to get HTTP url content`, the description single-media URL sends answer with; the senders. `upload.rs`: download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`), with a download's class from `classify_download_error` (transport/429/5xx retry; 4xx is permanent — the media itself is gone or refused — and a temp-file *write* failure retries, being resource exhaustion far more often than a broken temp dir). Item preparation is bounded **process-wide** (`PREP_SLOTS` in `upload.rs`: URL workers and queue workers can each be inside a batch, so a per-batch bound is not a memory bound), and the check that routes an oversized item to `fallback_url` is the download's own declared-Content-Length abort (`FetchError::TooLarge` → `MediaTooLarge`) — there is no separate size probe, which used to cost a second request per item. `post_send.rs`: settlement (`settle_task`), cache write, post-send actions (dead-letter text via `failure_text`: post key + cause, since the raw error alone does not say which link died), queue handlers. `input_media.rs`: payload → `InputMedia` |
| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `mod.rs` also holds `apply_caption_edit`, the one place a caption edit is applied and its failure classified: a short retryable delay is retried once, anything else is reported to the user instead of being swallowed (`callback.rs`'s template button answers its toast with the failure and leaves the record alone); `commands.rs` = command dispatch (incl. `/test <url>` send-only, `/debug <url>` parse-only, the read-only `/settings` every chat member can read — unlike the admin-only `/bot_dict` raw dump — and template removal; `/start`/`/help` carry the guidance teloxide's `descriptions()` cannot render, and `/set_format` rejects unknown `{…}` placeholders, resetting with `-`; `/set_template` enforces 50 templates per chat, 55 UTF-8-byte names, and 1024-character escaped bodies; `/settings` output is capped at 4000 characters); `urls.rs` = URL extraction + the per-URL pipeline (`url_media` takes a `PostSend` mode: chat settings vs `/test`'s suppressed actions); `inline.rs` = debounced inline queries (hotlink-protected and local media skipped); `callback.rs` = edit-before-forward buttons (dptree entry + testable `handle_callback` core, incl. `skip`) |
| `crates/xmedia-bot/src/send/` | `mod.rs`: constants `MAX_MEDIA_GROUP = 10` and the senders; `error.rs`: `classify_request_error` (5xx/non-JSON bodies retry, see the Retries bullet) and the media-fetch markers that route a URL send into the reupload fallback — including `failed to get HTTP url content`, the description single-media URL sends answer with. `upload.rs`: download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`), the per-kind upload cap (`MAX_MEDIA_UPLOAD_BYTES` = 50 MB for video/animation/other, Telegram's multipart limit; `photo::MAX_UPLOAD_BYTES` stays the 10 MiB photo one) with a download's class from `classify_download_error` (transport/429/5xx retry; 4xx is permanent — the media itself is gone or refused — and a temp-file *write* failure retries, being resource exhaustion far more often than a broken temp dir). Item preparation is bounded **process-wide** (`PREP_SLOTS` in `upload.rs`: URL workers and queue workers can each be inside a batch, so a per-batch bound is not a memory bound), and the check that routes an oversized item to `fallback_url` is the download's own declared-Content-Length abort (`FetchError::TooLarge` → `MediaTooLarge`) — there is no separate size probe, which used to cost a second request per item. `post_send.rs`: settlement (`settle_task`), cache write, post-send actions (dead-letter text via `failure_text`: post key + cause, since the raw error alone does not say which link died), queue handlers. `input_media.rs`: payload → `InputMedia` |
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL. Two budgets, not one: `MAX_PHOTO_DOWNLOAD_BYTES` (32 MiB) caps the *download* in the send fallback — the whole body is buffered, once per prep slot — while `MAX_DECODE_BYTES` (512 MiB) stays the pre-allocation guard that decides whether a decoded photo can be processed at all; over either one the item degrades to its smaller URL |
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection), `needs_media_headers` (the same per-site rule, asked by the inline path to skip what Telegram cannot fetch) |
| `crates/x-media/src/site/{mod,download}.rs` | `mod.rs`: dispatcher, `Fetched`/`FetchError`, `needs_media_headers` (the per-site rule, asked by the inline path to skip what Telegram cannot fetch). `download.rs`: the media-download stack — the metadata vs. media HTTP clients, the host-network guard (applied to the start URL and every redirect hop) and `download_media_limited`/`download_media_to_file` (which add the site's headers, e.g. `Referer: https://www.pixiv.net/` for `pximg.net`) |
| `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` |
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime bump (cargo's freshness is mtime-based and `cargo clean -p` removes 0 files — the touch is what forces the real sources to rebuild while deps stay cached), static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, optional `FFMPEG_SHA256` checksum, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint. Runtime ships **no libssl/libcrypto/CA bundle** — rustls webpki-roots handles all TLS, and the static ffmpeg only processes local files (downloads go through reqwest) |
| `docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim) |
|`docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim); rejects a non-numeric or `0` `LOCAL_USER_ID`, which would otherwise survive the drop and run the bot root |
| `docker-compose.yml` | The deployment composition, committed as-is: every instance value (token, admins, site credentials, domain) is a `${VAR}` substitution read from the gitignored `.env` beside it, so the file needs no per-deployment edit — and a variable not listed in a service's `environment:` never reaches that container. Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) |
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master, plus a build-only check on PRs touching the build inputs; **no test step**; verifies a release tag matches both crate versions; buildx gha cache (`cache-from` always, `cache-to` except on PRs, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs; `FFMPEG_URL`/`FFMPEG_SHA256` come from repo variables when set |
| `README.md` | Feature docs + command table (Chinese) |
@@ -98,8 +98,8 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
- Package manager: **Cargo** (workspace with path dep `x-media` ← `xmedia-bot`). No `[workspace.package]`/shared deps — each crate lists deps independently.
- **TLS is rustls end-to-end** (no native-tls/openssl in the tree, no libssl in the Docker runtime image): `teloxide` is declared `default-features = false` with `["webhooks-axum", "macros", "rustls", "ctrlc_handler"]` (the removed `default` also carried `native-tls` and `ctrlc_handler` — the latter must stay); x-media's reqwest is `default-features = false` with `["json", "rustls-tls", "gzip", "http2"]` (webpki-roots baked in, so the image ships no CA bundle; `gzip` because the site APIs answer their JSON compressed — twitter's syndication body is 4469 bytes identity vs 1066 gzipped — and `http2` because every site CDN here negotiates h2). One reqwest 0.12.28 in the lock.
- **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md`, `README.en.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag **in one push** (`git push origin master vX.Y.Z`; the tag push triggers the Docker Hub build). Pushing them separately with the branch first makes the master run of `docker.yml` build the same commit as the tag run — its duplicate check can only see the tags that already exist on the remote. The tag must equal both crate versions: `.github/workflows/docker.yml` verifies that before building, and `--locked` verifies the lock file.
- Config is **environment-variable driven** (dotenv loads `.env`, which is gitignored; `.env.example` is the tracked template — `cp .env.example .env` — and is also the file `docker compose` substitutes `${VAR}` from, so every variable the compose passes must be documented there). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BILIBILI_COOKIE` (optional; whole bilibili cookie string — bilibili dynamics fetch anonymously and add their own device cookies, this only rescues an egress IP that bilibili has hard-flagged with `-352`/412), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `CAPTION_QUOTE_TEXT_CHARS` (default 200; a post whose text — the `title` plus `content` joined, see `site::compose_text` — reaches this length gets that text wrapped in an expandable blockquote inside its caption, the URL and author line staying outside; `0` disables it. Applied at the send boundary in `send::quote_long_caption`, which locates the text as what follows the author link, so a `/set_format` that moves `{title}`/`{content}` elsewhere and pixiv's title-inside-a-link layout opt out; `copy_messages` forwards and queued retries inherit the wrap, while the edit-before-forward rewrite stays unquoted by design), `DATA_DIR` (default `data`, CWD-relative; the SQLite dir, auto-created), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `$DATA_DIR/task_queue.db` (default `data/task_queue.db`, CWD-relative — run from the workspace root, or `/app` in Docker; set `DATA_DIR` to pin state anywhere). Mount `./data` and `./cert` volumes.
- Config is **environment-variable driven** (dotenv loads `.env`, which is gitignored; `.env.example` is the tracked template — `cp .env.example .env` — and is also the file `docker compose` substitutes `${VAR}` from, so every variable the compose passes must be documented there). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BILIBILI_COOKIE` (optional; whole bilibili cookie string — bilibili dynamics fetch anonymously and add their own device cookies, this only rescues an egress IP that bilibili has hard-flagged with `-352`/412), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `CAPTION_QUOTE_TEXT_CHARS` (default 200; a post whose text — the `title` plus `content` joined, see `site::compose_text` — reaches this length gets that text wrapped in an expandable blockquote inside its caption, the URL and author line staying outside; `0` disables it. Applied at the send boundary in `send::quote_long_caption`, which locates the text as what follows the author link, so a `/set_format` that moves `{title}`/`{content}` elsewhere and pixiv's title-inside-a-link layout opt out; `copy_messages` forwards and queued retries inherit the wrap, while the edit-before-forward rewrite stays unquoted by design), `DATA_DIR` (default `data`, CWD-relative; the SQLite dir, auto-created), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port **and the secret token**, all `.expect`ed — a listener without a secret accepts unauthenticated updates; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `$DATA_DIR/task_queue.db` (default `data/task_queue.db`, CWD-relative — run from the workspace root, or `/app` in Docker; set `DATA_DIR` to pin state anywhere). Mount the `./data` volume (the shipped compose mounts only that); `./cert` matters solely for a self-signed `WEBHOOK_CERT` you wire in yourself — add both its mount and the env line to compose then, as the template's own comment says.
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `nginx-*` (proxy state), `/target`, `.idea/` (the compose file is tracked; only `.env` carries the deployment's own values).
- Docs are in Chinese (README, AGENTS.md); user-facing bot strings are in English. Keep that split when editing user-facing strings and docs.
@@ -107,8 +107,8 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
- **~180 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches. Tests that must go through a **real `Bot`** (its URL/multipart building, the per-chat limiter and the bot-wide budget) talk to a stand-in API instead (`media_sender::test_support::fake_api::FakeApi`, a `tokio` TCP listener that records every call and answers the smallest result each method needs — teloxide keys methods by payload type, so the recorded name is `SendMediaGroup`, not `sendMediaGroup`): a media group, the edit-before-forward prompt through the real callback path, and `handlers::handle_message` (the context-taking body of `message_handler`, split out for exactly this).
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (1), `site/misskey/interface.rs` (1), `site/bilibili/interface.rs` (5), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/mod.rs` also has a **token-gated but not `#[ignore]`d** pixiv download test (`download_media_pixiv_original_with_referer`): it hits `i.pximg.net` whenever `PIXIV_REFRESH_TOKEN` is set, so a local `cargo test --workspace` is not fully offline and can flake on a pixiv CDN body timeout. `disabled_site_is_reported_not_ignored` (same file) is gated the other way round: it asserts `fetch` answers `FetchError::Disabled { site: "pixiv" }` for a pixiv link and early-returns when `PIXIV_REFRESH_TOKEN` **is** set (the site is then enabled). Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail), and the bilibili live tests early-return when the API answers risk control (`-352`, which bilibili applies per IP by request volume). Run the full offline suite with `cargo test --workspace`.
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (1), `site/misskey/interface.rs` (1), `site/bilibili/interface.rs` (5), `site/pixiv/api.rs` (1) and `site/download.rs` (1: the pixiv download below); the bot crate adds one live test each in `handlers/repair.rs` and `handlers/urls.rs`; `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/download.rs`'s pixiv download test (`live_download_media_pixiv_original_with_referer`) is gated **both ways** — `#[ignore = "live network: …"]` *and* an early return without `PIXIV_REFRESH_TOKEN` — so a local `cargo test --workspace` stays fully offline and the pixiv CDN flake surfaces only in the live job. `disabled_site_is_reported_not_ignored` (same file) is gated the other way round: it asserts `fetch` answers `FetchError::Disabled { site: "pixiv" }` for a pixiv link and early-returns when `PIXIV_REFRESH_TOKEN` **is** set (the site is then enabled). Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail), and the bilibili live tests early-return when the API answers risk control (`-352`, which bilibili applies per IP by request volume). Run the full offline suite with `cargo test --workspace`.
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
- **CI** — `.github/workflows/ci.yml` (actions pinned to commit SHAs, `--locked` on every cargo invocation, `concurrency` cancels superseded runs, `RUST_BACKTRACE=1`) runs (behind a `changes` gate job, so a push/PR whose entire diff is markdown skips it instead of burning four minutes on nothing) `cargo fmt --check` + `cargo clippy --workspace --all-targets --locked -- -D warnings` + `cargo test --workspace --locked` + a release-profile `cargo build --release --locked` + an `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `-p x-media` since every network/secret-gated test lives there, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds and pushes the image on master/tag and runs a **build-only check on pull requests touching the build inputs**; its `should-build` gate skips a branch push that is already tagged (`git tag --points-at` — the tag run builds it, so push both refs together) or that touched no build input at all, while a tag push always builds (`Dockerfile`, entrypoint, manifests, `.dockerignore`); a release tag must match both crate versions or the build stops, and `FFMPEG_URL`/`FFMPEG_SHA256` are taken from repository variables when set (a release can pin an exact ffmpeg build). `.github/dependabot.yml` keeps crates, the pinned actions and the Docker base images current.
- Untested and hard to test without a mock seam: `config.rs`, `handlers/statics.rs`; `db.rs` is covered for the migration chain but not for pool behaviour under contention; `main.rs` is covered where it was split out (`periodic_sweep`, `sweep_temp_dir`) but not for startup/shutdown or its `dptree` branch tree (the handlers themselves are, through the stand-in API); in `x-media`: `media.rs`, `lib.rs`, all `model.rs`. The `commands.rs` *executor* needs a real `Bot` (only its pure report builder is tested). Everything else — `handlers/{mod,callback,inline,urls}.rs`, `send/*`, `ctx.rs`, `state.rs`, `queue.rs`, `link_cache.rs`, `rate_limit.rs` — is driven through `TestStores`/`ctx::test_support` and the scripted `MockSender`.
- **CI** — `.github/workflows/ci.yml` (actions pinned to commit SHAs, `--locked` on every cargo invocation, `concurrency` cancels superseded runs, `RUST_BACKTRACE=1`) runs (behind a `changes` gate job, so a push/PR whose entire diff is markdown skips it instead of burning four minutes on nothing) `cargo fmt --check` + `cargo clippy --workspace --all-targets --locked -- -D warnings` + `cargo test --workspace --locked` + a release-profile `cargo build --release --locked` + an `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `continue-on-error`) that runs the `#[ignore]`d `live` tests in **both** crates — `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN`/`BILIBILI_COOKIE` pass through as secrets, and a summary step lists every `SKIP` a test printed so a green live run cannot mean zero coverage. `.github/workflows/docker.yml` builds and pushes the image on master/tag and runs a **build-only check on pull requests touching the build inputs**; its `should-build` gate skips a branch push that is already tagged (`git tag --points-at` — the tag run builds it, so push both refs together) or that touched no build input at all, while a tag push always builds (`Dockerfile`, entrypoint, manifests, `.dockerignore`); a release tag must match both crate versions or the build stops, and `FFMPEG_URL`/`FFMPEG_SHA256` are taken from repository variables when set (a release can pin an exact ffmpeg build). `.github/dependabot.yml` keeps crates, the pinned actions and the Docker base images current.
- Untested and hard to test without a mock seam: `handlers/statics.rs`; `config.rs` only partly (the webhook truth table, TTL fallback and blank-secret parsing are pinned, the rest of the env parsing is not); `db.rs` is covered for the migration chain but not for pool behaviour under contention; `main.rs` is covered where it was split out (`periodic_sweep`, `sweep_temp_dir`) but not for startup/shutdown or its `dptree` branch tree (the handlers themselves are, through the stand-in API); in `x-media`: `media.rs`, `lib.rs`, all `model.rs` (their serde shapes are exercised indirectly by the adapter fixtures that deserialize into them). Everything else — `handlers/{mod,callback,inline,urls}.rs`, `send/*`, `ctx.rs`, `state.rs`, `queue.rs`, `link_cache.rs`, `rate_limit.rs`, `commands.rs` (its executor, through scripted outcomes) — is driven through `TestStores`/`ctx::test_support` and the scripted `MockSender`.
- No coverage tracking.
Generated
+14 -14
View File
@@ -275,7 +275,7 @@ version = "3.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
dependencies = [
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -553,7 +553,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -1153,7 +1153,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
dependencies = [
"hermit-abi",
"libc",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -1620,7 +1620,7 @@ dependencies = [
"bytes",
"getrandom 0.4.3",
"lru-slab",
"rand 0.10.3",
"rand 0.10.2",
"rand_pcg",
"ring",
"rustc-hash",
@@ -1644,7 +1644,7 @@ dependencies = [
"once_cell",
"socket2",
"tracing",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -1675,9 +1675,9 @@ dependencies = [
[[package]]
name = "rand"
version = "0.10.3"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af"
checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
dependencies = [
"chacha20",
"getrandom 0.4.3",
@@ -1892,7 +1892,7 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -2318,7 +2318,7 @@ dependencies = [
"getrandom 0.4.3",
"once_cell",
"rustix",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -2727,7 +2727,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -2879,13 +2879,13 @@ checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc"
[[package]]
name = "x-media"
version = "1.9.1"
version = "1.9.2"
dependencies = [
"bytes",
"dotenv",
"html-escape",
"log",
"rand 0.10.3",
"rand 0.10.2",
"regex",
"reqwest",
"serde",
@@ -2899,7 +2899,7 @@ dependencies = [
[[package]]
name = "xmedia-bot"
version = "1.9.1"
version = "1.9.2"
dependencies = [
"bytes",
"dotenv",
@@ -2910,7 +2910,7 @@ dependencies = [
"parking_lot",
"png",
"pretty_env_logger",
"rand 0.10.3",
"rand 0.10.2",
"rusqlite",
"serde",
"serde_json",
+28 -24
View File
@@ -8,20 +8,36 @@ ARG APP_NAME=telegram-twitter-media-bot
# encoding. Served from https://ffmpeg.martin-riedl.de (Cloudflare CDN,
# built on Debian 12 — glibc-compatible with the bookworm-slim runtime).
# johnvansickle.com throttles datacenter IPs and served garbage from GitHub
# runners. `/redirect/latest/` floats to the newest release build; each build
# also ships a .sha256. Swap `amd64` for `arm64` when building arm64 images.
ARG FFMPEG_URL=https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffmpeg.zip
# Arm64 images need this URL swapped for the `linux/arm64` build (currently
# hardcoded amd64; the workflow builds amd64 only — see docker.yml).
# Optional sha256 of ffmpeg.zip (pinned releases only): set to verify the
# download. The mirror publishes .sha256 sidecars next to pinned builds, e.g.
# https://ffmpeg.martin-riedl.de/download/linux/amd64/<id>_9.0/ffmpeg.zip.sha256
# (the /redirect/latest/ URL itself has no sidecar — pin the effective URL).
ARG FFMPEG_SHA256=
# runners.
#
# Pinned to one release build instead of `/redirect/latest/`: the floating
# URL changes under every build and ships no sha256 sidecar, while this pair
# (zip + the sha256 the mirror publishes beside it, `<url>.sha256`) is
# verified on every run. Bump both together — the site lists the current
# ids, e.g. https://ffmpeg.martin-riedl.de. Swap `amd64` for `arm64` when
# building arm64 images (the workflow builds amd64 only — see docker.yml).
ARG FFMPEG_URL=https://ffmpeg.martin-riedl.de/download/linux/amd64/1789931100_9.0.2/ffmpeg.zip
# sha256 of that zip, checked unconditionally: an FFMPEG_URL override must
# pair with the new zip's sha256 or the build fails here, so an unverifiable
# binary never reaches the image.
ARG FFMPEG_SHA256=fa8ecf4abbd290d98f7d188b8649cc6b391ae209a98452be955a15aab1909d7f
WORKDIR /build
# 1. Rust dependencies first: only the manifests plus stub sources, so the
# 1. Static ffmpeg first: only the two ARGs above invalidate this layer, so a
# manifest or source edit never re-downloads it. The zip contains a single
# `ffmpeg` binary at the root. `unzip -t` verifies the archive before
# extraction so a bad download fails loudly here instead of a cryptic
# later error.
RUN wget -q -O /tmp/ffmpeg.zip "$FFMPEG_URL" \
&& echo "$FFMPEG_SHA256 /tmp/ffmpeg.zip" | sha256sum -c - \
&& unzip -tq /tmp/ffmpeg.zip \
&& unzip -q /tmp/ffmpeg.zip -d /usr/local/bin \
&& chmod +x /usr/local/bin/ffmpeg \
&& rm /tmp/ffmpeg.zip \
&& /usr/local/bin/ffmpeg -version >/dev/null
# 2. Rust dependencies next: only the manifests plus stub sources, so the
# expensive dependency fetch + compile lives in a layer invalidated only by
# manifest/lock changes.
COPY Cargo.toml Cargo.lock ./
@@ -32,21 +48,9 @@ RUN mkdir -p crates/x-media/src crates/xmedia-bot/src \
&& : > crates/x-media/src/lib.rs \
&& cargo build --release --locked -p xmedia-bot
# 2. Static ffmpeg next (cached unless FFMPEG_URL changes), so source edits
# never re-download it. The zip contains a single `ffmpeg` binary at the
# root. `unzip -t` verifies the archive before extraction so a bad
# download fails loudly here instead of a cryptic later error.
RUN wget -q -O /tmp/ffmpeg.zip "$FFMPEG_URL" \
&& if [ -n "$FFMPEG_SHA256" ]; then echo "$FFMPEG_SHA256 /tmp/ffmpeg.zip" | sha256sum -c -; fi \
&& unzip -tq /tmp/ffmpeg.zip \
&& unzip -q /tmp/ffmpeg.zip -d /usr/local/bin \
&& chmod +x /usr/local/bin/ffmpeg \
&& rm /tmp/ffmpeg.zip \
&& /usr/local/bin/ffmpeg -version >/dev/null
# 3. Real sources last: only our crates recompile on source changes. Cargo's
# freshness check is mtime-based; the COPY'd host files usually predate the
# step-1 stub build, so cargo would consider the stub up to date and never
# stub build, so cargo would consider the stub up to date and never
# compile the real sources. `touch` makes every .rs newer than the stub
# artifacts, forcing a rebuild of just the two crates while the compiled
# dependency layer stays cached. (`cargo clean -p` does NOT work here — it
+1 -1
View File
@@ -122,7 +122,7 @@ Telegram only accepts ports 443/80/88/8443.
| `/set_forward_channel <channel>` | Set the forward channel: `@channel` or channel ID; media messages are forwarded to it automatically afterwards |
| `/remove_forward_channel` | Remove the forward channel |
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or tapping a template button applies one), then `↩️ Confirm` forwards and `🛑 Skip` drops this forward; the prompt states its expiry and is marked expired in place when it lapses (nothing is forwarded) |
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward") |
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward"). Names are limited to 55 UTF-8 bytes, bodies to 1024 escaped characters, and 50 templates per chat |
| `/remove_template <name>` | Remove a template (names are listed by `/settings`; the prompt's keyboard shows at most 60) |
| `/settings` | Show this chat's configuration: forward channel, edit-before-forward, per-site caption formats, saved templates |
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv` / `misskey` / `bilibili`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{content}` `{tags}`; unknown placeholders are rejected with the list of valid ones, and `-` restores the site's built-in format (preview with `/debug <link>`) |
+1 -1
View File
@@ -122,7 +122,7 @@ Telegram 只接受 443/80/88/8443 端口。
| `/set_forward_channel <频道>` | 设置转发频道,参数为 `@频道名` 或频道 ID;设置后发送的媒体消息会自动转发到该频道 |
| `/remove_forward_channel` | 取消转发频道 |
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板),再点 `↩️ Confirm` 才会真正转发,`🛑 Skip` 放弃本次转发;提示消息写明过期时间,过期后原地标记为已过期且不会转发 |
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用)。模板名称最长 55 个 UTF-8 字节、正文最长 1024 个转义后字符,每聊天最多 50 个模板 |
| `/remove_template <名称>` | 删除某个模板(名称见 `/settings`;提示消息的模板按钮最多显示 60 个) |
| `/settings` | 查看本聊天配置:转发频道、转发前编辑开关、各站点 caption 格式、模板列表 |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv` / `misskey` / `bilibili`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{content}` `{tags}`;未识别的占位符会被拒绝并列出可用项,格式填 `-` 恢复站点默认格式(可用 `/debug <链接>` 预览效果) |
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "x-media"
version = "1.9.1"
version = "1.9.2"
edition = "2024"
[dependencies]
-3
View File
@@ -37,18 +37,15 @@ impl Media {
#[derive(Debug)]
pub enum Media {
Illustration {
title: Option<String>,
url: String,
thumbnail_url: Option<String>,
fallback_url: Option<String>,
},
Video {
title: Option<String>,
url: String,
thumbnail_url: String,
},
Animated {
title: Option<String>,
url: String,
thumbnail_url: String,
},
+14 -51
View File
@@ -30,7 +30,7 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched, RenderData, Site, SiteFuture, compose_text};
use html_escape::{encode_double_quoted_attribute, encode_text};
use html_escape::encode_text;
use regex::Regex;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicBool, Ordering};
@@ -111,10 +111,6 @@ pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
.unwrap()
});
pub fn enabled() -> bool {
true
}
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
let dynamic_id = PATTERN
.captures(url)
@@ -133,18 +129,9 @@ pub fn cache_key(url: &str) -> Option<String> {
.map(|caps| format!("bilibili:{}", &caps[1]))
}
/// Bilibili's fetch-retry policy: transient classes only. Not-found, blocked
/// and parse failures are permanent.
pub fn is_retryable(err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
}
/// hdslb media serves without a `Referer` (verified live 2026-09-17 on
/// `i0.hdslb.com` image URLs, requested both with and without one), so no
/// extra headers.
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
// hdslb media serves without a `Referer` (verified live 2026-09-17 on
// `i0.hdslb.com` image URLs, requested both with and without one), so this
// adapter does not override `Site::media_headers`.
/// `Cookie` header for bilibili requests: the operator's `BILIBILI_COOKIE`
/// when set, otherwise the anonymous device cookies.
@@ -187,10 +174,7 @@ async fn cookie() -> Option<String> {
/// requests go out without a cookie.
async fn fetch_buvid() -> Result<Option<String>, FetchError> {
let response = crate::site::CLIENT.get(SPI_URL).send().await?;
let fingerprint: model::Fingerprint = response.json().await.map_err(|e| FetchError::Site {
site: "bilibili",
error: Box::new(e),
})?;
let fingerprint: model::Fingerprint = crate::site::response_json(response, "bilibili").await?;
Ok(buvid_cookie(&fingerprint))
}
@@ -226,16 +210,10 @@ pub async fn fetch(dynamic_id: &str) -> Result<model::Item, FetchError> {
if !status.is_success() {
return Err(match status.as_u16() {
412 => risk_control("412"),
// A refusal or an auth demand is not a bad moment (412 above is
// bilibili's risk control, which does clear on its own).
401 | 403 => FetchError::Blocked,
_ => FetchError::Transient(format!("bilibili status {status}")),
_ => crate::site::status_error("bilibili", &response),
});
}
let detail: model::Detail = response.json().await.map_err(|e| FetchError::Site {
site: "bilibili",
error: Box::new(e),
})?;
let detail: model::Detail = crate::site::response_json(response, "bilibili").await?;
if let Some(err) = code_error(detail.code, detail.message.as_deref().unwrap_or_default()) {
return Err(err);
}
@@ -289,7 +267,7 @@ impl From<model::Item> for Fetched {
let text = compose_text(&title, &content);
let tags = topic_name(&item).to_string();
let caption = caption(&url, &author_url, &author, &text);
let caption = crate::site::caption(&url, &author_url, &author, &text);
let media = media_of(&item);
Fetched {
@@ -301,7 +279,6 @@ impl From<model::Item> for Fetched {
sensitive: false,
site_id: "bilibili",
render_data: Some(RenderData {
url,
author: encode_text(&author).into_owned(),
author_url,
title: encode_text(&title).into_owned(),
@@ -471,7 +448,6 @@ fn image(url: &str) -> Option<Media> {
}
Some(if url.ends_with(".gif") {
Media::Animated {
title: None,
url,
// Left empty on purpose: the `@518w.jpg` variant is unverified for
// animated sources, and Telegram generates a frame preview itself.
@@ -479,7 +455,6 @@ fn image(url: &str) -> Option<Media> {
}
} else {
Media::Illustration {
title: None,
// Written before `url` moves so the formatting borrows it.
thumbnail_url: Some(format!("{url}{THUMB_SUFFIX}")),
url,
@@ -500,19 +475,6 @@ fn to_https(url: &str) -> String {
}
}
fn caption(url: &str, author_url: &str, author: &str, text: &str) -> String {
let url = encode_double_quoted_attribute(url);
let author_url = encode_double_quoted_attribute(author_url);
let author = encode_text(author);
if text.is_empty() {
return format!("{url}\n<a href=\"{author_url}\">{author}</a>");
}
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {}",
encode_text(text)
)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -908,7 +870,7 @@ mod tests {
// dropping the post.
for code in [-352, -412] {
let err = code_error(code, "-352").unwrap();
assert!(is_retryable(&err), "{err}");
assert!(BilibiliSite.is_retryable(&err), "{err}");
}
// A removed dynamic is permanent.
assert!(matches!(code_error(500, ""), Some(FetchError::NotFound)));
@@ -917,7 +879,7 @@ mod tests {
Some(FetchError::NotFound)
));
let err = code_error(-400, "param parsing failed").unwrap();
assert!(!is_retryable(&err), "{err}");
assert!(!BilibiliSite.is_retryable(&err), "{err}");
assert!(err.to_string().contains("-400"), "{err}");
}
@@ -1034,13 +996,14 @@ mod tests {
/// Fetches a live dynamic, skipping the assertion when bilibili
/// risk-controls this IP (the site blocks datacenter/over-used addresses
/// with `-352` regardless of cookies — a real failure would surface as a
/// parse error or a not-found instead). Mirrors the token-gated pixiv
/// tests' "skipping: …" convention.
/// parse error or a not-found instead). Mirrors the pixiv download
/// test's `SKIP …` convention — CI's live job greps that prefix to list
/// the skips in the run summary instead of showing a silently green run.
async fn live_fetch(url: &str) -> Option<Fetched> {
match fetch_from_url(url).await {
Ok(fetched) => Some(fetched),
Err(e) if e.to_string().contains("risk control") => {
eprintln!("skipping: {e}");
eprintln!("SKIP (bilibili risk control): {e}");
None
}
Err(e) => panic!("{e}"),
+1 -3
View File
@@ -1,6 +1,4 @@
mod interface;
mod model;
pub use interface::{
BilibiliSite, PATTERN, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
};
pub use interface::{BilibiliSite, PATTERN, cache_key, fetch_from_url};
+40 -48
View File
@@ -1,7 +1,7 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched, Site, SiteFuture};
use html_escape::{encode_double_quoted_attribute, encode_text};
use html_escape::encode_text;
use regex::Regex;
use std::sync::LazyLock;
@@ -30,10 +30,6 @@ pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"^(?:https?://)?bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap()
});
pub fn enabled() -> bool {
true
}
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
let caps = PATTERN.captures(url).ok_or(FetchError::NotFound)?;
let handle = caps
@@ -77,7 +73,6 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
_ => String::new(),
};
media.push(Media::Video {
title: None,
url: mp4_path.to_string_lossy().into_owned(),
thumbnail_url,
});
@@ -112,17 +107,6 @@ pub fn cache_key(url: &str) -> Option<String> {
.map(|caps| format!("bsky:{}/{}", &caps[1], &caps[2]))
}
/// Bluesky's fetch-retry policy: transient classes only. Not-found, blocked
/// and parse failures are permanent.
pub fn is_retryable(err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
}
/// bsky media (cdn.bsky.app) needs no extra headers.
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// Segments fetched (and written) at once while remuxing an HLS video. Small
/// on purpose: a segment can be up to 20 MiB and the whole playlist is capped
/// at 256 MiB, so this is also what bounds the remux's peak memory.
@@ -147,10 +131,18 @@ fn concat_list(files: &mut [(usize, std::path::PathBuf)]) -> String {
/// the end of a 500-segment video meant downloading the entire thing twice
/// more, so the second attempt belongs on the request that actually failed.
async fn fetch_hls(url: &str, cap: u64) -> Result<bytes::Bytes, String> {
match crate::site::download_media_limited(url, cap).await {
match crate::site::download_media_limited(url, cap, crate::site::DOWNLOAD_TOTAL_TIMEOUT).await {
Err(FetchError::RateLimited {
retry_after_secs, ..
}) => {
tokio::time::sleep(std::time::Duration::from_secs(retry_after_secs)).await;
crate::site::download_media_limited(url, cap, crate::site::DOWNLOAD_TOTAL_TIMEOUT)
.await
.map_err(|e| e.to_string())
}
Err(FetchError::Http(_) | FetchError::Transient(_)) => {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
crate::site::download_media_limited(url, cap)
crate::site::download_media_limited(url, cap, crate::site::DOWNLOAD_TOTAL_TIMEOUT)
.await
.map_err(|e| e.to_string())
}
@@ -169,8 +161,7 @@ async fn fetch_hls(url: &str, cap: u64) -> Result<bytes::Bytes, String> {
async fn resolve_bsky_video(
playlist_url: &str,
) -> Result<Option<(std::path::PathBuf, tempfile::TempDir)>, String> {
if !crate::site::ffmpeg_available() {
crate::site::log_once_ffmpeg_missing();
if crate::site::ffmpeg_missing() {
return Ok(None);
}
let master = fetch_hls(playlist_url, 1_048_576)
@@ -280,7 +271,7 @@ async fn resolve_bsky_video(
let list_str = list_path.to_string_lossy().into_owned();
let output_str = output.to_string_lossy().into_owned();
let status = tokio::task::spawn_blocking(move || {
std::process::Command::new("ffmpeg")
let mut child = std::process::Command::new("ffmpeg")
.args([
"-y",
"-f",
@@ -297,15 +288,30 @@ async fn resolve_bsky_video(
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.spawn()
.map_err(|e| format!("ffmpeg spawn failed: {e}"))?;
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(300);
loop {
match child
.try_wait()
.map_err(|e| format!("ffmpeg wait failed: {e}"))?
{
Some(status) => break Ok(status),
None if std::time::Instant::now() >= deadline => {
let _ = child.kill();
let _ = child.wait();
break Err("ffmpeg exceeded 300s".to_string());
}
None => std::thread::sleep(std::time::Duration::from_millis(50)),
}
}
})
.await
.map_err(|e| format!("bsky remux worker panicked: {e}"))?;
match status {
Ok(s) if s.success() => Ok(Some((output, out_dir))),
Ok(s) => Err(format!("ffmpeg exited with {s}")),
Err(e) => Err(format!("ffmpeg spawn failed: {e}")),
.map_err(|e| format!("bsky remux worker panicked: {e}"))??;
if !status.success() {
return Err(format!("ffmpeg exited with {status}"));
}
Ok(Some((output, out_dir)))
}
/// Fetches a post thread by handle or DID (`at://` URIs work for both).
@@ -321,15 +327,9 @@ pub async fn fetch(handle: &str, rkey: &str) -> Result<Post, FetchError> {
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
// A refusal or an auth demand is not a bad moment: retrying it
// three times only delays an error the user has to see.
401 | 403 => Err(FetchError::Blocked),
_ => Err(FetchError::Transient(format!("bsky status {status}"))),
};
return Err(crate::site::status_error("bsky", &response));
}
let text = response.text().await?;
let text = crate::site::response_text(response, "bsky").await?;
Post::from_json(&text, rkey.to_string())
}
@@ -353,13 +353,7 @@ impl Post {
}
pub fn caption(&self) -> String {
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
url = encode_double_quoted_attribute(&self.url()),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
text = encode_text(&self.text),
)
crate::site::caption(&self.url(), &self.author_url(), &self.author, &self.text)
}
pub fn from_json(raw_json: &str, id: String) -> Result<Self, FetchError> {
@@ -375,7 +369,6 @@ impl Post {
match embed {
model::Media::Images { images } => {
media.extend(images.into_iter().map(|image| Media::Illustration {
title: None,
url: image.fullsize,
thumbnail_url: Some(image.thumb),
fallback_url: None,
@@ -386,7 +379,6 @@ impl Post {
thumbnail,
} => {
media.push(Media::Video {
title: None,
url: playlist,
thumbnail_url: thumbnail,
});
@@ -418,7 +410,6 @@ impl From<Post> for Fetched {
let url = post.url();
let author_url = post.author_url();
let render_data = Some(crate::site::RenderData {
url: url.clone(),
author: encode_text(&post.author).into_owned(),
author_url: author_url.clone(),
// A post has no title: its text is all content.
@@ -505,10 +496,11 @@ mod tests {
/// ([`fetch_hls`]). The classes below are the ones still retried there.
#[test]
fn media_prep_failure_is_not_retried() {
assert!(!is_retryable(&FetchError::MediaPrep(
use crate::site::Site as _;
assert!(!BskySite.is_retryable(&FetchError::MediaPrep(
"bsky video remux failed: segment 400: 503".into()
)));
assert!(is_retryable(&FetchError::Transient("429".into())));
assert!(BskySite.is_retryable(&FetchError::Transient("429".into())));
}
#[test]
+1 -3
View File
@@ -1,6 +1,4 @@
mod interface;
mod model;
pub use interface::{
BskySite, PATTERN, Post, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
};
pub use interface::{BskySite, PATTERN, Post, cache_key, fetch_from_url};
+388
View File
@@ -0,0 +1,388 @@
//! The media-download stack: the two HTTP clients (site metadata vs. media,
//! which need different timeouts), the CDN allowlist that keeps a download
//! out of the host's own network, and the two streaming entry points — a capped
//! body in memory ([`download_media_limited`]) and a large one written as it
//! arrives ([`download_media_to_file`]).
//!
//! Site-specific headers come from each adapter's `Site::media_headers`; no
//! code here knows about a particular site.
use super::{FetchError, SITES};
use std::sync::LazyLock;
use std::time::Duration;
/// How long a download may make no progress: the response head, and then each
/// individual chunk, must arrive within this window. Not a total timeout — see
/// [`DOWNLOAD_TOTAL_TIMEOUT`].
const DOWNLOAD_IDLE_TIMEOUT: Duration = Duration::from_secs(30);
/// Absolute ceiling for one media download, on top of the idle window: a
/// server that drips a byte every 29 s keeps [`next_chunk`] satisfied
/// indefinitely, and a transfer that trickles forever holds whatever the
/// caller pinned to it — a fetch permit for an in-flight post, a prep slot
/// for the bot's upload fallback. Generous on purpose: the legitimate cases
/// are big — an ugoira frame zip runs to hundreds of MB and an HLS remux
/// pulls a whole video — so this is the budget for downloads *inside a
/// fetch*, while the slot-holding fallback passes its own shorter one (see
/// [`download_media_limited`]'s `total`). Checked between chunks, so a
/// transfer that completes just over the budget is kept rather than thrown
/// away.
pub(crate) const DOWNLOAD_TOTAL_TIMEOUT: Duration = Duration::from_secs(600);
/// The error a download reports when it spends its whole budget without
/// finishing. Retryable: the transfer may simply have been unlucky, and a retry
/// of the post restarts the download.
fn download_too_slow(total: Duration) -> FetchError {
FetchError::Transient(format!("download exceeded {}s", total.as_secs()))
}
/// Builds a client with the shared configuration (browser User-Agent, the
/// Bot API's proxy, per-runtime pools under test). `total_timeout` is what
/// differs between the two clients below.
fn build_client(total_timeout: Option<Duration>) -> reqwest::Client {
let mut builder = reqwest::Client::builder()
.user_agent("Mozilla/5.0")
.connect_timeout(Duration::from_secs(10));
// Redirects stay allowed for allowlisted CDN hops, but every hop goes
// through the same policy as the initial URL; a third-party response must
// not be able to introduce a new host.
builder = builder.redirect(reqwest::redirect::Policy::custom(|attempt| {
if !media_url_allowed(attempt.url()) {
log::warn!("refusing a media redirect outside the CDN allowlist");
return attempt.error(FetchError::Blocked);
}
if attempt.previous().len() >= 10 {
return attempt.stop();
}
attempt.follow()
}));
if let Some(total) = total_timeout {
// reqwest has no total timeout by default; a stalled connection
// would otherwise pin a fetch/handler forever.
builder = builder.timeout(total);
}
// Route site fetches through the same proxy the Bot API uses, so a
// network that needs TELOXIDE_PROXY (e.g. behind the GFW) does not
// leave site fetches dead while the bot itself works.
if let Some(proxy) = std::env::var("TELOXIDE_PROXY")
.ok()
.filter(|s| !s.is_empty())
&& let Ok(p) = reqwest::Proxy::all(&proxy)
{
builder = builder.proxy(p);
}
// Each `#[tokio::test]` runs on its own runtime; the connection pool is
// bound to the runtime that created it, so cross-runtime reuse of idle
// connections fails with DispatchGone. In test builds every request uses
// a fresh connection. Production runs on one runtime and keeps pooling.
#[cfg(test)]
let builder = builder.pool_max_idle_per_host(0);
builder.build().expect("failed to build HTTP client")
}
/// Shared HTTP client (browser User-Agent) for the site fetches — metadata
/// requests, where 30s is generous.
pub(crate) static CLIENT: LazyLock<reqwest::Client> =
LazyLock::new(|| build_client(Some(Duration::from_secs(30))));
/// Client for media *downloads*, with no reqwest-level total timeout: a 10 MiB
/// fallback download, or an ugoira frame zip that may be hundreds of MB,
/// legitimately takes minutes on a slow link — a 30s total cap made those posts
/// impossible to deliver at all (the size cap said 512 MiB, the clock said 30s).
/// What a stalled connection cannot do is hang a worker: the head and every
/// chunk are bounded by [`DOWNLOAD_IDLE_TIMEOUT`] (see [`next_chunk`]), and a
/// transfer that keeps trickling but never finishes is bounded by the
/// caller's total budget (see [`download_media_limited`]).
static MEDIA_CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| build_client(None));
/// The error a download reports when it stops making progress.
fn download_stalled() -> FetchError {
FetchError::Transient(format!(
"download stalled for {}s",
DOWNLOAD_IDLE_TIMEOUT.as_secs()
))
}
/// Sends a media-download request: the response head must arrive within the
/// idle window, and a non-2xx status is classified by
/// [`super::status_error`] with `"media"` as the name — the same table the
/// site adapters use, so a dead URL and a bad moment read the same everywhere.
/// A transport error never reaches that table — it fails in `send()` and
/// stays [`FetchError::Http`].
async fn send_download(request: reqwest::RequestBuilder) -> Result<reqwest::Response, FetchError> {
let response = match tokio::time::timeout(DOWNLOAD_IDLE_TIMEOUT, request.send()).await {
Ok(Ok(response)) => response,
Ok(Err(e)) => return Err(e.into()),
Err(_) => return Err(download_stalled()),
};
if response.status().is_success() {
Ok(response)
} else {
Err(super::status_error("media", &response))
}
}
/// One body chunk, or `None` at the end. A body that stops delivering is a
/// transient download error rather than a hang.
async fn next_chunk(response: &mut reqwest::Response) -> Result<Option<bytes::Bytes>, FetchError> {
match tokio::time::timeout(DOWNLOAD_IDLE_TIMEOUT, response.chunk()).await {
Ok(Ok(chunk)) => Ok(chunk),
Ok(Err(e)) => Err(e.into()),
Err(_) => Err(download_stalled()),
}
}
/// Reads a successful API response body with a hard byte cap.
pub(crate) async fn send_json_response(
mut response: reqwest::Response,
site: &'static str,
) -> Result<bytes::Bytes, FetchError> {
if let Some(len) = response.content_length()
&& len > crate::site::MAX_SITE_JSON_BYTES as u64
{
return Err(FetchError::Site {
site,
error: "site response exceeds JSON size cap".into(),
});
}
let mut body = Vec::new();
while let Some(chunk) = next_chunk(&mut response).await? {
if body.len().saturating_add(chunk.len()) > crate::site::MAX_SITE_JSON_BYTES {
return Err(FetchError::Site {
site,
error: "site response exceeds JSON size cap".into(),
});
}
body.extend_from_slice(&chunk);
}
Ok(bytes::Bytes::from(body))
}
/// `localhost` (and anything under it) plus the mDNS `.local` suffix.
fn is_local_name(name: &str) -> bool {
let name = name.trim_end_matches('.').to_ascii_lowercase();
name == "localhost" || name.ends_with(".localhost") || name.ends_with(".local")
}
/// Media is fetched only from the CDN families used by the site adapters.
/// IP literals are rejected as well: a public IP is not a member of that
/// allowlist, and accepting one would turn the bot into a generic proxy.
fn media_host_allowed(name: &str) -> bool {
let name = name.trim_end_matches('.').to_ascii_lowercase();
name == "misskey.io"
|| name.ends_with(".misskey.io")
|| name == "misskeyusercontent.jp"
|| name.ends_with(".misskeyusercontent.jp")
|| name == "bsky.app"
|| name.ends_with(".bsky.app")
|| name == "twimg.com"
|| name.ends_with(".twimg.com")
|| name == "pximg.net"
|| name.ends_with(".pximg.net")
|| name == "hdslb.com"
|| name.ends_with(".hdslb.com")
}
fn media_url_allowed(url: &url::Url) -> bool {
if !matches!(url.scheme(), "http" | "https") {
return false;
}
matches!(url.host(), Some(url::Host::Domain(name)) if !is_local_name(name) && media_host_allowed(name))
}
/// Prepares a media download: refuses a URL outside the CDN allowlist
/// ([`FetchError::Blocked`], permanent — the same URL would be refused again),
/// then applies every site's media-header rule (pixiv's `Referer` for pximg.net
/// hotlink protection; sites contribute via `media_headers(url)`, so the
/// central download code carries no other per-site logic). One choke point so
/// every download path gets both.
fn media_request(url: &str) -> Result<reqwest::RequestBuilder, FetchError> {
let parsed = url::Url::parse(url).map_err(|e| {
log::warn!("media url is not a url: {e}");
FetchError::Blocked
})?;
if !media_url_allowed(&parsed) {
log::warn!("refusing media URL outside the CDN allowlist");
return Err(FetchError::Blocked);
}
let mut request = MEDIA_CLIENT.get(parsed);
for site in SITES.iter() {
if let Some(headers) = site.media_headers(url) {
for (name, value) in headers {
request = request.header(name, value);
}
}
}
Ok(request)
}
/// Downloads a media file with a hard size cap: the body is streamed and the
/// download aborts with [`FetchError::TooLarge`] the moment the cap is
/// crossed (or when a declared Content-Length already exceeds it). Keeps the
/// bot from buffering arbitrarily large bodies into memory — the size check
/// the bot's upload fallback needs is the one here, not a probe of its own.
///
/// This is the bot's download path for the upload fallback: when Telegram
/// cannot fetch a media URL itself (hotlink protection), the bot downloads
/// the file and uploads it via multipart. Site-appropriate headers come from
/// each site's `media_headers` (pixiv image hosts need `Referer`).
///
/// `total` is this caller's whole-transfer budget. The bot's upload fallback
/// holds a prep slot (and its memory reservation) while this runs, so it
/// passes a shorter one of its own; bsky's in-fetch segments take the
/// generous [`super::DOWNLOAD_TOTAL_TIMEOUT`].
pub async fn download_media_limited(
url: &str,
max_bytes: u64,
total: Duration,
) -> Result<bytes::Bytes, FetchError> {
let response = send_download(media_request(url)?).await?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let mut response = response;
let mut buf = Vec::new();
let started = std::time::Instant::now();
while let Some(chunk) = next_chunk(&mut response).await? {
if started.elapsed() > total {
return Err(download_too_slow(total));
}
buf.extend_from_slice(&chunk);
if buf.len() as u64 > max_bytes {
return Err(FetchError::TooLarge);
}
}
Ok(bytes::Bytes::from(buf))
}
/// Streams a download to `out`, aborting with [`FetchError::TooLarge`] the
/// moment the body crosses `max_bytes` (or when a declared Content-Length
/// already exceeds it). Unlike [`download_media_limited`] the body is never
/// buffered in memory — used for large files (e.g. the pixiv ugoira frame
/// zip, which can be hundreds of MB) that would otherwise spike RAM. Writes
/// go through the tokio handle so a sync write never stalls an executor
/// thread for the length of the download. Returns the number of bytes written.
pub async fn download_media_to_file(
url: &str,
max_bytes: u64,
out: &mut tokio::fs::File,
) -> Result<u64, FetchError> {
use tokio::io::AsyncWriteExt;
let response = send_download(media_request(url)?).await?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let mut response = response;
let mut total: u64 = 0;
let started = std::time::Instant::now();
while let Some(chunk) = next_chunk(&mut response).await? {
if started.elapsed() > DOWNLOAD_TOTAL_TIMEOUT {
return Err(download_too_slow(DOWNLOAD_TOTAL_TIMEOUT));
}
total += chunk.len() as u64;
if total > max_bytes {
return Err(FetchError::TooLarge);
}
out.write_all(&chunk).await.map_err(FetchError::Io)?;
}
Ok(total)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::site::{Fetched, pixiv};
#[test]
fn media_urls_outside_the_allowlist_are_refused() {
for url in [
"http://127.0.0.1:9/x",
"http://169.254.169.254/latest/meta-data/",
"http://[::1]:9/x",
"https://1.1.1.1/x",
"https://[2606:4700::1111]/x",
"https://localhost/",
"https://prompt.localhost/x",
"https://printer.local/x",
"https://example.com/a",
"https://evil.pximg.net.attacker.example/a",
"file:///etc/passwd",
"gopher://example.com/1",
] {
let parsed = url::Url::parse(url).unwrap();
assert!(!media_url_allowed(&parsed), "{url}");
}
for url in [
"https://i.pximg.net/img-original/img/1.jpg",
"https://cdn.bsky.app/img/feed_thumbnail/plain/x",
"https://pbs.twimg.com/media/1.jpg",
"https://media.misskeyusercontent.jp/io/1.jpg",
"https://i0.hdslb.com/bfs/1.jpg",
] {
let parsed = url::Url::parse(url).unwrap();
assert!(media_url_allowed(&parsed), "{url}");
}
}
#[tokio::test]
async fn a_download_from_a_refused_host_is_blocked() {
// Refused on the URL alone: nothing has to be listening (or leaking) at
// the metadata endpoint for this to hold, and the class is permanent so
// the send path does not retry it.
for url in [
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:9/secret",
"http://8.8.8.8/x",
] {
let err = download_media_limited(url, u64::MAX, DOWNLOAD_TOTAL_TIMEOUT)
.await
.unwrap_err();
assert!(matches!(err, FetchError::Blocked), "{url}: got {err:?}");
}
// A malformed URL is refused the same way instead of becoming a
// retryable transport error.
assert!(matches!(
download_media_limited("not a url", u64::MAX, DOWNLOAD_TOTAL_TIMEOUT)
.await
.unwrap_err(),
FetchError::Blocked
));
}
#[tokio::test]
#[ignore = "live network: requires PIXIV_REFRESH_TOKEN and i.pximg.net"]
async fn live_download_media_pixiv_original_with_referer() {
// Proves the Referer header is attached for i.pximg.net: a header-less
// GET to a pixiv original URL is rejected with 403. `#[ignore]` as
// well as the token gate: this hit the CDN on every `cargo test
// --workspace` in a token-exported shell (and flaked on a CDN body
// timeout there), and the `live_` name puts it inside the CI live
// job's `--ignored live` filter. Empty-string check too: an unset CI
// secret arrives as "" (GitHub Actions), which would otherwise run
// the test tokenless and fail — the `SKIP` prefix is what the live
// job greps to tell a skip from a pass.
if std::env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|s| !s.is_empty())
.is_none()
{
eprintln!("SKIP (no PIXIV_REFRESH_TOKEN): not running the pixiv download test");
return;
}
let illustration = pixiv::fetch(126839080).await.unwrap();
let fetched: Fetched = illustration.into();
let url = match fetched.media.first() {
Some(crate::media::Media::Illustration { url, .. }) => url.clone(),
other => panic!("expected illustration media, got {other:?}"),
};
assert!(url.contains("i.pximg.net"));
let bytes = download_media_limited(&url, u64::MAX, DOWNLOAD_TOTAL_TIMEOUT)
.await
.unwrap();
assert!(!bytes.is_empty());
}
}
+10 -51
View File
@@ -4,7 +4,7 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched, RenderData, Site, SiteFuture};
use html_escape::{encode_double_quoted_attribute, encode_text};
use html_escape::encode_text;
use regex::Regex;
use std::sync::LazyLock;
@@ -34,10 +34,6 @@ impl Site for MisskeySite {
pub static PATTERN: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^(?:https?://)?misskey\.io/notes/([\w.\-~]+)").unwrap());
pub fn enabled() -> bool {
true
}
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
let caps = PATTERN.captures(url).ok_or(FetchError::NotFound)?;
let note_id = caps.get(1).ok_or(FetchError::NotFound)?.as_str();
@@ -53,20 +49,10 @@ pub fn cache_key(url: &str) -> Option<String> {
.map(|caps| format!("misskey:{}", &caps[1]))
}
/// Misskey's fetch-retry policy: transient classes only. Not-found, blocked
/// and parse failures are permanent.
pub fn is_retryable(err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
}
/// misskey.io media hosts need no extra headers (verified: direct GET works).
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// Fetches a note from misskey.io by id. The API answers client failures
/// with HTTP 400 + `{"error":{"code":...}}` (NO_SUCH_NOTE → NotFound);
/// everything else non-success is transient and retried by [`crate::site::fetch`].
/// every other non-success status falls through to the shared classes in
/// [`crate::site::status_error`] — persistent 4xx permanent, 429/5xx retried.
pub async fn fetch(note_id: &str) -> Result<model::Note, FetchError> {
let response = crate::site::CLIENT
.post(API_URL)
@@ -77,21 +63,19 @@ pub async fn fetch(note_id: &str) -> Result<model::Note, FetchError> {
if !status.is_success() {
return Err(match status.as_u16() {
400 => not_found_or_invalid(response).await,
// A refusal or an auth demand is not a bad moment.
401 | 403 => FetchError::Blocked,
_ => FetchError::Transient(format!("misskey status {status}")),
// The local fallback used to disagree with the center: a misskey
// 404 came back Transient here and was fetched three more times
// for a note that is simply gone.
_ => crate::site::status_error("misskey", &response),
});
}
response.json().await.map_err(|e| FetchError::Site {
site: "misskey",
error: Box::new(e),
})
crate::site::response_json(response, "misskey").await
}
/// Maps a 400 response: NO_SUCH_NOTE is permanent NotFound, any other 400 is
/// a site error (permanent — retrying a rejected request cannot succeed).
async fn not_found_or_invalid(response: reqwest::Response) -> FetchError {
match response.json::<serde_json::Value>().await {
match crate::site::response_json::<serde_json::Value>(response, "misskey").await {
Ok(v) if v["error"]["code"] == "NO_SUCH_NOTE" => FetchError::NotFound,
_ => FetchError::Site {
site: "misskey",
@@ -132,7 +116,7 @@ impl From<model::Note> for Fetched {
text.push_str(content.text.as_deref().unwrap_or_default().trim());
let text = text.trim().to_string();
let caption = caption(&url, &author_url, &author, &text);
let caption = crate::site::caption(&url, &author_url, &author, &text);
let sensitive = content.cw.is_some() || content.files.iter().any(|f| f.is_sensitive);
let media: Vec<Media> = content.files.iter().filter_map(media_from_file).collect();
@@ -146,7 +130,6 @@ impl From<model::Note> for Fetched {
sensitive,
site_id: "misskey",
render_data: Some(RenderData {
url,
author: encode_text(&author).into_owned(),
author_url: author_url.clone(),
title: String::new(),
@@ -158,38 +141,21 @@ impl From<model::Note> for Fetched {
}
}
fn caption(url: &str, author_url: &str, author: &str, text: &str) -> String {
let url = encode_double_quoted_attribute(url);
let author_url = encode_double_quoted_attribute(author_url);
let author = encode_text(author);
if text.is_empty() {
return format!("{url}\n<a href=\"{author_url}\">{author}</a>");
}
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
text = encode_text(text),
)
}
/// Maps a Misskey DriveFile to a [`Media`] item; unknown/audio/other types
/// are skipped (twitter's `_ => {}` precedent). GIF must be matched before
/// the generic image arm.
fn media_from_file(file: &model::DriveFile) -> Option<Media> {
let title = file.name.clone();
match file.mime_type.as_str() {
"image/gif" => Some(Media::Animated {
title,
url: file.url.clone(),
thumbnail_url: file.thumbnail_url.clone().unwrap_or_default(),
}),
mime if mime.starts_with("image/") => Some(Media::Illustration {
title,
url: file.url.clone(),
thumbnail_url: file.thumbnail_url.clone(),
fallback_url: None,
}),
mime if mime.starts_with("video/") => Some(Media::Video {
title,
url: file.url.clone(),
thumbnail_url: file.thumbnail_url.clone().unwrap_or_default(),
}),
@@ -239,11 +205,6 @@ mod tests {
cache_key("https://misskey.io/notes/aotihl10lqrs015s"),
Some("misskey:aotihl10lqrs015s".to_string())
);
assert_eq!(x_media_site_id("misskey:abc"), "misskey");
}
fn x_media_site_id(key: &str) -> &'static str {
crate::site::site_id_from_key(key)
}
#[test]
@@ -268,12 +229,10 @@ mod tests {
assert_eq!(fetched.media.len(), 1);
match &fetched.media[0] {
Media::Illustration {
title,
url,
thumbnail_url,
fallback_url,
} => {
assert_eq!(title.as_deref(), Some("pic.webp"));
assert_eq!(url, "https://media.misskeyusercontent.jp/io/a.webp");
assert_eq!(
thumbnail_url.as_deref(),
+1 -3
View File
@@ -1,6 +1,4 @@
mod interface;
mod model;
pub use interface::{
MisskeySite, PATTERN, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
};
pub use interface::{MisskeySite, PATTERN, cache_key, fetch_from_url};
-2
View File
@@ -30,6 +30,4 @@ pub(crate) struct DriveFile {
pub(crate) thumbnail_url: Option<String>,
#[serde(default, rename = "isSensitive")]
pub(crate) is_sensitive: bool,
#[serde(default)]
pub(crate) name: Option<String>,
}
+303 -496
View File
@@ -15,12 +15,16 @@ use thiserror::Error;
pub mod bilibili;
pub mod bsky;
mod download;
pub mod misskey;
pub mod pixiv;
pub mod twitter;
pub use pixiv::PixivError;
pub(crate) use download::{CLIENT, DOWNLOAD_TOTAL_TIMEOUT};
pub use download::{download_media_limited, download_media_to_file};
/// The result of fetching a post: canonical URL, HTML caption, the post's
/// title and body, media list and spoiler flag. Produced by [`fetch`].
#[derive(Debug)]
@@ -65,13 +69,14 @@ pub struct Fetched {
///
/// `author`, `title`, `content` and `tags` come from the site API (post
/// text, display names, descriptions) and are HTML-escaped at construction.
/// `url` and `author_url` stay raw: they are canonical URLs the adapter
/// builds from numeric ids and API-constrained handles/DIDs, so they carry
/// no escapable character — the bot's `/test` report relies on that when it
/// embeds them.
/// `author_url` stays raw: it is a canonical URL the adapter builds from
/// numeric ids and API-constrained handles/DIDs, so it carries no escapable
/// character — the bot's `/test` report relies on that when it embeds it.
/// `{url}` needs no copy here: [`Fetched::source_url`] is the same canonical
/// URL every adapter would have handed this struct, and `caption_with` reads
/// it from there.
#[derive(Debug)]
pub(crate) struct RenderData {
pub url: String,
pub author: String,
pub author_url: String,
pub title: String,
@@ -79,6 +84,24 @@ pub(crate) struct RenderData {
pub tags: String,
}
/// The built-in caption for a post that has no user-supplied format: the
/// canonical URL, the author as a link, then the post's text after a colon.
/// The two URLs are escaped for an HTML attribute and the text as HTML text,
/// so site-supplied content cannot inject markup. Shared by the adapters whose
/// captions have exactly this shape (bilibili, misskey).
pub fn caption(url: &str, author_url: &str, author: &str, text: &str) -> String {
let url = html_escape::encode_double_quoted_attribute(url);
let author_url = html_escape::encode_double_quoted_attribute(author_url);
let author = html_escape::encode_text(author);
if text.is_empty() {
return format!("{url}\n<a href=\"{author_url}\">{author}</a>");
}
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {}",
html_escape::encode_text(text)
)
}
/// The post's text as one string: title and content joined by a line break,
/// each only when it is non-empty. This is what the sites' built-in captions
/// show after the author line, and what the bot quotes when it is long.
@@ -92,13 +115,6 @@ pub fn compose_text(title: &str, content: &str) -> String {
}
impl Fetched {
/// The site this post came from (used for per-site format overrides).
/// A thin alias over [`Fetched::site_id`] kept for callers that read the
/// site off a fetched post.
pub fn site_name(&self) -> &'static str {
self.site_id
}
/// Renders a user-supplied caption format. The format string is
/// HTML-escaped in full, then the (already-escaped) placeholder values
/// are substituted — users can structure text but never inject raw HTML
@@ -110,7 +126,7 @@ impl Fetched {
(Some(data), false) => caption_from_fields(
format,
"",
&data.url,
&self.source_url,
&data.author,
&data.author_url,
&data.title,
@@ -208,6 +224,36 @@ pub fn caption_from_fields(
)
}
/// Maximum decoded JSON response accepted from a site API. Metadata is
/// expected to be much smaller; this keeps a compromised or malformed API
/// from growing an unbounded `String` before serde gets a chance to reject it.
pub(crate) const MAX_SITE_JSON_BYTES: usize = 8 * 1024 * 1024;
/// Reads a successful site response as a bounded UTF-8 JSON value.
pub(crate) async fn response_json<T: serde::de::DeserializeOwned>(
response: reqwest::Response,
site: &'static str,
) -> Result<T, FetchError> {
let body = crate::site::download::send_json_response(response, site).await?;
serde_json::from_slice(&body).map_err(|e| FetchError::Site {
site,
error: Box::new(e),
})
}
/// Same bounded response reader for endpoints that need a text body before
/// classification or parsing.
pub(crate) async fn response_text(
response: reqwest::Response,
site: &'static str,
) -> Result<String, FetchError> {
let body = crate::site::download::send_json_response(response, site).await?;
String::from_utf8(body.to_vec()).map_err(|e| FetchError::Site {
site,
error: Box::new(e),
})
}
/// Stable per-post cache key derived from any supported URL, so variant
/// domains (x.com / twitter.com / fxtwitter.com, mobile, `/photo/N`
/// suffixes) map to the same post. Delegates to each registered site's
@@ -216,19 +262,6 @@ pub fn cache_key(url: &str) -> Option<String> {
SITES.iter().find_map(|site| site.cache_key(url))
}
/// The site id carried by a cache key (`"twitter:123"` → `"twitter"`).
/// Unknown prefixes fall back to `"unknown"`. The bot uses this on the
/// link-cache hit path, where no [`Fetched`] is available — the same value
/// a fresh fetch would read from [`Fetched::site_id`].
pub fn site_id_from_key(key: &str) -> &'static str {
let prefix = key.split(':').next().unwrap_or("");
SITES
.iter()
.map(|site| site.id())
.find(|id| *id == prefix)
.unwrap_or("unknown")
}
#[derive(Debug, Error)]
pub enum FetchError {
#[error("http error: {0}")]
@@ -275,127 +308,64 @@ pub enum FetchError {
/// A transient server-side failure (429 / 5xx); [`fetch`] retries these.
#[error("transient: {0}")]
Transient(String),
/// The source answered 429 *with* a `Retry-After` and named its own
/// delay: [`fetch`] sleeps at least that long instead of guessing one
/// (capped by [`MAX_RETRY_AFTER_SECS`] — the header is server-supplied
/// and must not park one of the fetch slots).
#[error("{site} rate limited, retry after {retry_after_secs}s")]
RateLimited {
site: &'static str,
retry_after_secs: u64,
},
/// A local I/O failure while streaming a download to disk
/// (see [`download_media_to_file`]).
#[error("io error: {0}")]
Io(std::io::Error),
}
/// How long a download may make no progress: the response head, and then each
/// individual chunk, must arrive within this window. Not a total timeout — see
/// [`DOWNLOAD_TOTAL_TIMEOUT`].
const DOWNLOAD_IDLE_TIMEOUT: Duration = Duration::from_secs(30);
/// Cap on a server-supplied `Retry-After`: honored so a retry stops hammering
/// a source that asked for air, bounded so the same untrusted header cannot
/// park a fetch slot for an hour.
pub const MAX_RETRY_AFTER_SECS: u64 = 60;
/// Absolute ceiling for one media download, on top of the idle window. A server
/// that drips a byte every 29 s keeps [`next_chunk`] satisfied indefinitely, and
/// on the bot's side each such download holds one of the process-wide upload-prep
/// slots (`send::upload`'s `PREP_SLOTS`) for as long as it lasts. Generous on
/// purpose: the legitimate cases are big — an ugoira frame zip runs to hundreds
/// of MB and an HLS remux pulls a whole video — and a slow link is not an error.
/// Checked between chunks, so a transfer that completes just over the budget is
/// kept rather than thrown away.
const DOWNLOAD_TOTAL_TIMEOUT: Duration = Duration::from_secs(600);
/// The error a download reports when it spends its whole budget without
/// finishing. Retryable: the transfer may simply have been unlucky, and a retry
/// of the post restarts the download.
fn download_too_slow() -> FetchError {
FetchError::Transient(format!(
"download exceeded {}s",
DOWNLOAD_TOTAL_TIMEOUT.as_secs()
))
/// The error class for a non-success HTTP status, shared by the site
/// adapters, the media downloads and twitter's auth fallback: 404/410 mean
/// the post is gone (permanent), any other client error the source answers
/// on sight is a refusal (permanent too — three retries only delay the same
/// answer), 408/429/5xx are a bad moment retried by [`fetch`], and a 429
/// that carries `Retry-After` keeps the delay the source asked for (the
/// seconds form only — a HTTP-date value parses to `None` and falls back to
/// the plain transient path).
/// `site` only names the adapter in the message (`"media"` for downloads);
/// a site whose statuses mean something else (bilibili's 412 risk control,
/// misskey's 400 with `NO_SUCH_NOTE`) maps those before falling back here.
pub fn status_error(site: &'static str, response: &reqwest::Response) -> FetchError {
let retry_after = response
.headers()
.get(reqwest::header::RETRY_AFTER)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.trim().parse::<u64>().ok());
classify_status(site, response.status(), retry_after)
}
/// Builds a client with the shared configuration (browser User-Agent, the
/// Bot API's proxy, per-runtime pools under test). `total_timeout` is what
/// differs between the two clients below.
fn build_client(total_timeout: Option<Duration>) -> reqwest::Client {
let mut builder = reqwest::Client::builder()
.user_agent("Mozilla/5.0")
.connect_timeout(Duration::from_secs(10));
// Redirects stay allowed (site CDNs use them), but every hop goes through
// the same guard as the initial URL, and the cap stays reqwest's default:
// a third-party response must not be able to walk the bot into the host's
// own network.
builder = builder.redirect(reqwest::redirect::Policy::custom(|attempt| {
if !media_url_allowed(attempt.url()) {
log::warn!("refusing a media redirect into the host's own network");
return attempt.error(FetchError::Blocked);
}
if attempt.previous().len() >= 10 {
return attempt.stop();
}
attempt.follow()
}));
if let Some(total) = total_timeout {
// reqwest has no total timeout by default; a stalled connection
// would otherwise pin a fetch/handler forever.
builder = builder.timeout(total);
}
// Route site fetches through the same proxy the Bot API uses, so a
// network that needs TELOXIDE_PROXY (e.g. behind the GFW) does not
// leave site fetches dead while the bot itself works.
if let Some(proxy) = std::env::var("TELOXIDE_PROXY")
.ok()
.filter(|s| !s.is_empty())
&& let Ok(p) = reqwest::Proxy::all(&proxy)
{
builder = builder.proxy(p);
}
// Each `#[tokio::test]` runs on its own runtime; the connection pool is
// bound to the runtime that created it, so cross-runtime reuse of idle
// connections fails with DispatchGone. In test builds every request uses
// a fresh connection. Production runs on one runtime and keeps pooling.
#[cfg(test)]
let builder = builder.pool_max_idle_per_host(0);
builder.build().expect("failed to build HTTP client")
}
/// Shared HTTP client (browser User-Agent) for the site fetches — metadata
/// requests, where 30s is generous.
pub(crate) static CLIENT: LazyLock<reqwest::Client> =
LazyLock::new(|| build_client(Some(Duration::from_secs(30))));
/// Client for media *downloads*, with no reqwest-level total timeout: a 10 MiB
/// fallback download, or an ugoira frame zip that may be hundreds of MB,
/// legitimately takes minutes on a slow link — a 30s total cap made those posts
/// impossible to deliver at all (the size cap said 512 MiB, the clock said 30s).
/// What a stalled connection cannot do is hang a worker: the head and every
/// chunk are bounded by [`DOWNLOAD_IDLE_TIMEOUT`] (see [`next_chunk`]), and a
/// transfer that keeps trickling but never finishes is bounded by
/// [`DOWNLOAD_TOTAL_TIMEOUT`].
static MEDIA_CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| build_client(None));
/// The error a download reports when it stops making progress.
fn download_stalled() -> FetchError {
FetchError::Transient(format!(
"download stalled for {}s",
DOWNLOAD_IDLE_TIMEOUT.as_secs()
))
}
/// Sends a media-download request: the response head must arrive within the
/// idle window, and a non-2xx status is classified by [`download_status_error`].
async fn send_download(request: reqwest::RequestBuilder) -> Result<reqwest::Response, FetchError> {
let response = match tokio::time::timeout(DOWNLOAD_IDLE_TIMEOUT, request.send()).await {
Ok(Ok(response)) => response,
Ok(Err(e)) => return Err(e.into()),
Err(_) => return Err(download_stalled()),
};
if response.status().is_success() {
Ok(response)
} else {
Err(download_status_error(response.status()))
}
}
/// One body chunk, or `None` at the end. A body that stops delivering is a
/// transient download error rather than a hang.
async fn next_chunk(response: &mut reqwest::Response) -> Result<Option<bytes::Bytes>, FetchError> {
match tokio::time::timeout(DOWNLOAD_IDLE_TIMEOUT, response.chunk()).await {
Ok(Ok(chunk)) => Ok(chunk),
Ok(Err(e)) => Err(e.into()),
Err(_) => Err(download_stalled()),
/// [`status_error`]'s table, split out so tests reach it without building an
/// HTTP response — the retry delay only ever shapes the 429 arm.
pub(crate) fn classify_status(
site: &'static str,
status: reqwest::StatusCode,
retry_after: Option<u64>,
) -> FetchError {
match status.as_u16() {
404 | 410 => FetchError::NotFound,
code if status.is_client_error() && !matches!(code, 408 | 429) => FetchError::Blocked,
429 => match retry_after {
Some(retry_after_secs) => FetchError::RateLimited {
site,
retry_after_secs: retry_after_secs.min(MAX_RETRY_AFTER_SECS),
},
None => FetchError::Transient(format!("{site} status {status}")),
},
_ => FetchError::Transient(format!("{site} status {status}")),
}
}
@@ -413,14 +383,17 @@ static FFMPEG_AVAILABLE: LazyLock<bool> = LazyLock::new(|| {
static FFMPEG_MISSING_LOGGED: AtomicBool = AtomicBool::new(false);
pub(crate) fn ffmpeg_available() -> bool {
*FFMPEG_AVAILABLE
}
pub(crate) fn log_once_ffmpeg_missing() {
/// Whether the encode step must be skipped: no ffmpeg on PATH, logged once
/// per process. The one gate both encoders check — the probe and the
/// log-once used to be two functions that only ever appeared together.
pub(crate) fn ffmpeg_missing() -> bool {
if *FFMPEG_AVAILABLE {
return false;
}
if !FFMPEG_MISSING_LOGGED.swap(true, Ordering::Relaxed) {
log::warn!("ffmpeg not found; ugoira and bsky video posts stay unsupported");
}
true
}
/// Site adapter: one impl per supported site (twitter / bsky / misskey /
@@ -450,9 +423,13 @@ pub trait Site: Send + Sync {
fn cache_key(&self, url: &str) -> Option<String>;
/// Fetches and normalizes a post.
fn fetch_from_url<'a>(&'a self, url: &'a str) -> SiteFuture<'a, Fetched>;
/// Retry policy for fetch errors: transient classes only.
/// Retry policy for fetch errors: transient classes only (a 429's
/// named `Retry-After` included — it is a bad moment, just a louder one).
fn is_retryable(&self, err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
matches!(
err,
FetchError::Http(_) | FetchError::Transient(_) | FetchError::RateLimited { .. }
)
}
/// Extra headers for downloading this site's media (hotlink protection,
/// e.g. pixiv's Referer for pximg.net). Matched on the media URL, not
@@ -475,7 +452,7 @@ type SiteFuture<'a, T, E = FetchError> = Pin<Box<dyn Future<Output = Result<T, E
/// The one registry of supported sites, in dispatch order (twitter → bsky →
/// misskey → pixiv → bilibili). Adding a site = new module + one
/// `Box::new(...)` entry here; the bot crate never lists sites itself.
static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| {
pub(crate) static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| {
vec![
Box::new(twitter::TwitterSite),
Box::new(bsky::BskySite),
@@ -485,25 +462,16 @@ static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| {
]
});
/// The first enabled site whose pattern matches `url`, in dispatch order.
fn find_site(url: &str) -> Option<&'static dyn Site> {
/// The first site whose pattern matches `url`, in dispatch order — enabled
/// or not. The caller decides what a disabled match means; no match at all is
/// the "unsupported link" case.
fn matching_site(url: &str) -> Option<&'static dyn Site> {
SITES
.iter()
.find(|site| site.enabled() && site.pattern().is_match(url))
.find(|site| site.pattern().is_match(url))
.map(|site| site.as_ref())
}
/// The site whose pattern matches `url` but which is disabled right now.
/// `None` when no site matches the URL at all, or when the matching site is
/// enabled. Lets the dispatcher tell "unsupported link" (silently ignored)
/// apart from "this bot has that site switched off" (reported to the user).
fn disabled_site(url: &str) -> Option<&'static str> {
SITES
.iter()
.find(|site| !site.enabled() && site.pattern().is_match(url))
.map(|site| site.id())
}
/// Every supported site id, in dispatch order. The bot's SetFormat whitelist
/// derives from this list.
pub fn site_ids() -> Vec<&'static str> {
@@ -548,45 +516,104 @@ pub async fn fetch_once(url: &str) -> Result<Option<Fetched>, FetchError> {
fetch_with_attempts(url, 1).await
}
/// A small admission gate for `/test` and `/debug`, which run directly in
/// dispatcher handlers instead of the URL worker pool.
pub async fn acquire_command_fetch_slot() -> tokio::sync::OwnedSemaphorePermit {
std::sync::Arc::clone(&COMMAND_FETCH_SLOTS)
.acquire_owned()
.await
.expect("command fetch gate closed")
}
/// Total attempts of the retried [`fetch`] (3: the initial try plus two).
const MAX_FETCH_ATTEMPTS: u32 = 3;
/// How many fetches may run at once, process-wide. The URL workers already
/// bound their own path (8 workers on a bounded channel), but inline queries,
/// `/debug` and `/test` reach [`fetch`]/[`fetch_once`] straight from handler
/// and debounce tasks with no limit at all — and the heavy part runs *inside*
/// the fetch: a ugoira encode is a 512 MiB download plus ffmpeg, a bsky video
/// an HLS remux, so N users meant N encodes. Every entry waits on this one
/// gate instead; the count matches `URL_WORKERS` so the bot's own pipeline
/// keeps its full width. A retry's backoff (1s, then 2s) holds its permit —
/// deliberately simple: the wait is bounded by the same retries.
static FETCH_SLOTS: LazyLock<tokio::sync::Semaphore> =
LazyLock::new(|| tokio::sync::Semaphore::new(8));
static COMMAND_FETCH_SLOTS: LazyLock<std::sync::Arc<tokio::sync::Semaphore>> =
LazyLock::new(|| std::sync::Arc::new(tokio::sync::Semaphore::new(2)));
async fn fetch_with_attempts(url: &str, attempts: u32) -> Result<Option<Fetched>, FetchError> {
// Wall time of the whole fetch, retry backoff included: the ugoira encode
// and the HLS remux live inside it, so this is where a slow fetch shows.
let started = std::time::Instant::now();
let Some(site) = find_site(url) else {
// A registered-but-disabled site (pixiv without a token) is not an
// unsupported link: report it, so the bot answers the user instead of
// ignoring the message.
return match disabled_site(url) {
Some(site) => Err(FetchError::Disabled { site }),
None => Ok(None),
};
let Some(site) = matching_site(url) else {
return Ok(None); // unsupported link: silently ignored
};
for attempt in 0..attempts.max(1) {
match site.fetch_from_url(url).await {
Ok(fetched) => {
// Per-request detail: debug only, keyed by the post id.
log::debug!(
"fetched [key={}]: site {} returned {} media in {}ms",
cache_key(url).unwrap_or_else(|| "?".into()),
fetched.site_name(),
fetched.media.len(),
started.elapsed().as_millis()
);
return Ok(Some(fetched));
}
Err(err) => {
if site.is_retryable(&err) && attempt + 1 < attempts {
tokio::time::sleep(Duration::from_secs(1 << attempt)).await;
} else {
return Err(err);
// A registered-but-disabled site (pixiv without a token) is not an
// unsupported link: report it, so the bot answers the user instead of
// ignoring the message. One match is the whole lookup — the site patterns
// are disjoint (one domain each), so "first enabled match" and "first
// match, then check" never disagree.
if !site.enabled() {
return Err(FetchError::Disabled { site: site.id() });
}
// Gate every network attempt process-wide (see FETCH_SLOTS).
let _permit = FETCH_SLOTS.acquire().await.expect("fetch gate closed");
let fetch = async {
// One hard ceiling for the whole fetch, backoff naps included (the
// timeout below): the idle timeouts restart on every chunk, so a
// drip-feeding URL could otherwise pin one fetch slot effectively
// forever. Generous for a genuinely large ugoira zip on a slow link
// — minutes, not hours — and the deadline the audit's low finding
// asked for.
for attempt in 0..attempts.max(1) {
match site.fetch_from_url(url).await {
Ok(fetched) => {
// Per-request detail: debug only, keyed by the post id.
log::debug!(
"fetched [key={}]: site {} returned {} media in {}ms",
cache_key(url).unwrap_or_else(|| "?".into()),
fetched.site_id,
fetched.media.len(),
started.elapsed().as_millis()
);
return Ok(Some(fetched));
}
Err(err) => {
if site.is_retryable(&err) && attempt + 1 < attempts {
tokio::time::sleep(retry_wait(attempt, rand::random::<u64>(), &err)).await;
} else {
return Err(err);
}
}
}
}
unreachable!("retry loop always returns")
};
match tokio::time::timeout(Duration::from_secs(900), fetch).await {
Ok(result) => result,
Err(_) => Err(FetchError::Transient(format!(
"fetch exceeded its 900s total budget [key={}]",
cache_key(url).unwrap_or_else(|| "?".into())
))),
}
unreachable!("retry loop always returns")
}
/// How long to sleep before retrying `attempt` (0-based) after `err`: the
/// doubling base plus a random slice of it (roll in [0, base) → [base, 2×base))
/// so workers that failed together do not recover together, floored at the
/// delay a 429's `Retry-After` named — already capped by the classifier at
/// [`MAX_RETRY_AFTER_SECS`], so an untrusted server cannot park a slot.
pub(crate) fn retry_wait(attempt: u32, roll: u64, err: &FetchError) -> Duration {
let base = 1u64 << attempt.min(16);
let mut secs = base + roll % base;
if let FetchError::RateLimited {
retry_after_secs, ..
} = err
{
secs = secs.max(*retry_after_secs);
}
Duration::from_secs(secs)
}
/// Whether fetching `url` requires site-specific headers (pixiv's `Referer`
@@ -598,181 +625,6 @@ pub fn needs_media_headers(url: &str) -> bool {
SITES.iter().any(|site| site.media_headers(url).is_some())
}
/// Applies every site's media-header rule to a download request (pixiv's
/// `Referer` for pximg.net hotlink protection). Sites contribute via their
/// `media_headers(url)` — the central download code carries no per-site logic.
/// Whether an address must never be fetched. Media URLs come from a site's own
/// API response and the bytes are uploaded to Telegram, so following one into
/// the host's own network would turn the bot into a proxy for it: a cloud
/// metadata endpoint read back into a chat.
fn blocked_ip(addr: std::net::IpAddr) -> bool {
use std::net::IpAddr;
match addr {
IpAddr::V4(v4) => {
let [a, b, ..] = v4.octets();
v4.is_private() // 10/8, 172.16/12, 192.168/16
|| v4.is_loopback() // 127/8
|| v4.is_link_local() // 169.254/16 — the cloud metadata range
|| v4.is_unspecified()
|| v4.is_broadcast()
|| v4.is_documentation()
|| v4.is_multicast()
// Ranges the std helpers do not cover: carrier-grade NAT and
// benchmarking.
|| (a == 100 && (64..=127).contains(&b))
|| (a == 198 && (18..=19).contains(&b))
}
IpAddr::V6(v6) => {
let [first, ..] = v6.segments();
v6.is_loopback()
|| v6.is_unspecified()
|| v6.is_multicast()
|| (first & 0xfe00) == 0xfc00 // unique local fc00::/7
|| (first & 0xffc0) == 0xfe80 // link local fe80::/10
|| v6.to_ipv4_mapped().is_some_and(|v4| blocked_ip(IpAddr::V4(v4)))
}
}
}
/// `localhost` (and anything under it) plus the mDNS `.local` suffix: names that
/// only ever mean this machine.
fn is_local_name(name: &str) -> bool {
let name = name.trim_end_matches('.').to_ascii_lowercase();
name == "localhost" || name.ends_with(".localhost") || name.ends_with(".local")
}
/// Whether a media URL may be requested at all: http(s), and a host that is no
/// address or name of the host's own network. Applied to the URL a download
/// starts from *and* to every redirect hop.
///
/// The residual gap is DNS rebinding — a name the site controls that resolves to
/// a private address. Closing it needs a `reqwest::dns::Resolve` wrapper
/// filtering resolved addresses; it is deliberately not installed, because the
/// same resolver also resolves the operator's proxy host and `TELOXIDE_PROXY`
/// is routinely a LAN address, so the guard would take down a working
/// deployment to block a far less likely attack.
fn media_url_allowed(url: &url::Url) -> bool {
if !matches!(url.scheme(), "http" | "https") {
return false;
}
match url.host() {
Some(url::Host::Ipv4(v4)) => !blocked_ip(v4.into()),
Some(url::Host::Ipv6(v6)) => !blocked_ip(v6.into()),
Some(url::Host::Domain(name)) => !is_local_name(name),
None => false,
}
}
/// Prepares a media download: refuses a URL pointing inside the host's own
/// network ([`FetchError::Blocked`], permanent — the same URL would be refused
/// again), then applies the site's media headers. One choke point so every
/// download path gets the guard.
fn media_request(url: &str) -> Result<reqwest::RequestBuilder, FetchError> {
let parsed = url::Url::parse(url).map_err(|e| {
log::warn!("media url is not a url: {e}");
FetchError::Blocked
})?;
if !media_url_allowed(&parsed) {
log::warn!("refusing to fetch media from the host's own network");
return Err(FetchError::Blocked);
}
Ok(apply_media_headers(MEDIA_CLIENT.get(parsed), url))
}
fn apply_media_headers(mut request: reqwest::RequestBuilder, url: &str) -> reqwest::RequestBuilder {
for site in SITES.iter() {
if let Some(headers) = site.media_headers(url) {
for (name, value) in headers {
request = request.header(name, value);
}
}
}
request
}
/// Maps a media download's HTTP status onto the same classes the site
/// adapters use, so callers can tell "try again" from "this URL is dead":
/// 4xx is a property of the media (gone, refused by the host), while 429/5xx
/// is a property of the moment. A transport error never reaches this — it
/// fails in `send()` and stays [`FetchError::Http`].
fn download_status_error(status: reqwest::StatusCode) -> FetchError {
match status.as_u16() {
401 | 403 => FetchError::Blocked,
404 | 410 => FetchError::NotFound,
_ => FetchError::Transient(format!("media status {status}")),
}
}
/// Downloads a media file with a hard size cap: the body is streamed and the
/// download aborts with [`FetchError::TooLarge`] the moment the cap is
/// crossed (or when a declared Content-Length already exceeds it). Keeps the
/// bot from buffering arbitrarily large bodies into memory — the size check
/// the bot's upload fallback needs is the one here, not a probe of its own.
///
/// This is the bot's download path for the upload fallback: when Telegram
/// cannot fetch a media URL itself (hotlink protection), the bot downloads
/// the file and uploads it via multipart. Site-appropriate headers come from
/// each site's `media_headers` (pixiv image hosts need `Referer`).
pub async fn download_media_limited(url: &str, max_bytes: u64) -> Result<bytes::Bytes, FetchError> {
let response = send_download(media_request(url)?).await?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let mut response = response;
let mut buf = Vec::new();
let started = std::time::Instant::now();
while let Some(chunk) = next_chunk(&mut response).await? {
if started.elapsed() > DOWNLOAD_TOTAL_TIMEOUT {
return Err(download_too_slow());
}
buf.extend_from_slice(&chunk);
if buf.len() as u64 > max_bytes {
return Err(FetchError::TooLarge);
}
}
Ok(bytes::Bytes::from(buf))
}
pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
download_media_limited(url, u64::MAX).await
}
/// Streams a download to `out`, aborting with [`FetchError::TooLarge`] the
/// moment the body crosses `max_bytes` (or when a declared Content-Length
/// already exceeds it). Unlike [`download_media_limited`] the body is never
/// buffered in memory — used for large files (e.g. the pixiv ugoira frame
/// zip, which can be hundreds of MB) that would otherwise spike RAM.
/// Returns the number of bytes written.
pub async fn download_media_to_file(
url: &str,
max_bytes: u64,
out: &mut std::fs::File,
) -> Result<u64, FetchError> {
use std::io::Write;
let response = send_download(media_request(url)?).await?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
return Err(FetchError::TooLarge);
}
let mut response = response;
let mut total: u64 = 0;
let started = std::time::Instant::now();
while let Some(chunk) = next_chunk(&mut response).await? {
if started.elapsed() > DOWNLOAD_TOTAL_TIMEOUT {
return Err(download_too_slow());
}
total += chunk.len() as u64;
if total > max_bytes {
return Err(FetchError::TooLarge);
}
out.write_all(&chunk).map_err(FetchError::Io)?;
}
Ok(total)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -838,27 +690,17 @@ mod tests {
assert_eq!(cache_key("https://example.com/not-a-post"), None);
}
#[test]
fn site_id_from_key_parses_prefix() {
assert_eq!(site_id_from_key("twitter:123"), "twitter");
assert_eq!(site_id_from_key("pixiv:123"), "pixiv");
assert_eq!(site_id_from_key("bsky:handle.example/3lorem"), "bsky");
assert_eq!(site_id_from_key("bilibili:123"), "bilibili");
assert_eq!(site_id_from_key("unknown:1"), "unknown");
assert_eq!(site_id_from_key("no-colon"), "unknown");
}
#[test]
fn registry_lists_all_sites_in_dispatch_order() {
assert_eq!(
site_ids(),
vec!["twitter", "bsky", "misskey", "pixiv", "bilibili"]
);
// Enabled sites dispatch; unsupported URLs never match.
assert!(find_site("https://x.com/u/status/1").is_some());
assert!(find_site("https://misskey.io/notes/abc").is_some());
assert!(find_site("https://t.bilibili.com/1245284537985925159").is_some());
assert!(find_site("https://example.com/x").is_none());
// Patterns dispatch; unsupported URLs never match.
assert!(matching_site("https://x.com/u/status/1").is_some());
assert!(matching_site("https://misskey.io/notes/abc").is_some());
assert!(matching_site("https://t.bilibili.com/1245284537985925159").is_some());
assert!(matching_site("https://example.com/x").is_none());
// Cache keys are pattern-driven, independent of the enabled() gate
// (pixiv is disabled in tests without PIXIV_REFRESH_TOKEN).
assert_eq!(
@@ -876,8 +718,10 @@ mod tests {
};
assert_eq!(err.to_string(), "example error: boom");
assert!(err.source().is_some());
// Permanent by default: no site's is_retryable matches it.
assert!(!twitter::is_retryable(&err));
// Permanent by default: no site's is_retryable matches it (the trait
// default is the policy for every site that does not override it).
assert!(!twitter::TwitterSite.is_retryable(&err));
assert!(twitter::TwitterSite.is_retryable(&FetchError::Transient("429".into())));
}
#[test]
@@ -964,98 +808,6 @@ mod tests {
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
}
#[test]
fn blocked_addresses_are_the_hosts_own_network() {
for addr in [
"127.0.0.1",
"10.0.0.1",
"172.16.0.1",
"192.168.1.1",
"169.254.169.254", // cloud metadata
"0.0.0.0",
"255.255.255.255",
"100.64.0.1", // carrier-grade NAT
"198.18.0.1", // benchmarking
"::1",
"::",
"fc00::1",
"fe80::1",
"::ffff:127.0.0.1",
] {
assert!(blocked_ip(addr.parse().unwrap()), "{addr}");
}
for addr in [
"1.1.1.1",
"93.184.216.34",
"2606:4700::1111",
"::ffff:1.1.1.1",
] {
assert!(!blocked_ip(addr.parse().unwrap()), "{addr}");
}
}
#[test]
fn media_urls_inside_the_host_are_refused() {
for url in [
"http://127.0.0.1:9/x",
"http://169.254.169.254/latest/meta-data/",
"http://[::1]:9/x",
"https://localhost/",
"https://prompt.localhost/x",
"https://printer.local/x",
"file:///etc/passwd",
"gopher://example.com/1",
] {
let parsed = url::Url::parse(url).unwrap();
assert!(!media_url_allowed(&parsed), "{url}");
}
// Real media hosts and any public address stay fetchable.
for url in [
"https://i.pximg.net/img-original/img/1.jpg",
"https://cdn.bsky.app/img/feed_thumbnail/plain/x",
"http://example.com/a",
"https://93.184.216.34/a",
] {
let parsed = url::Url::parse(url).unwrap();
assert!(media_url_allowed(&parsed), "{url}");
}
}
/// The redirect-hop guard, against a public redirector: the initial URL is
/// checked by [`media_request`], but a redirect is the part of the path a
/// third-party response actually controls.
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to httpbin.org"]
async fn live_redirect_into_the_hosts_network_is_refused() {
let url = "https://httpbin.org/redirect-to?url=http://169.254.169.254/latest/meta-data/";
match download_media(url).await.unwrap_err() {
// A policy refusal reaches the caller wrapped by reqwest.
FetchError::Http(e) => assert!(e.is_redirect(), "got {e}"),
FetchError::Blocked => {}
other => panic!("expected a refusal, got {other:?}"),
}
}
#[tokio::test]
async fn a_download_into_the_hosts_network_is_refused() {
// Refused on the URL alone: nothing has to be listening (or leaking) at
// the metadata endpoint for this to hold, and the class is permanent so
// the send path does not retry it.
for url in [
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:9/secret",
] {
let err = download_media(url).await.unwrap_err();
assert!(matches!(err, FetchError::Blocked), "{url}: got {err:?}");
}
// A malformed URL is refused the same way instead of becoming a
// retryable transport error.
assert!(matches!(
download_media("not a url").await.unwrap_err(),
FetchError::Blocked
));
}
#[tokio::test]
async fn unsupported_urls_return_none() {
// Neither a URL no site pattern matches nor a string that is no URL at
@@ -1086,6 +838,86 @@ mod tests {
}
}
#[test]
fn persistent_client_statuses_are_permanent() {
use reqwest::StatusCode;
// The one table every caller shares now: only 408, 429 and 5xx can
// answer differently on a retry. A 400 used to be Transient here and
// in two local fallbacks — twitter syndication's broken-token 400, for
// one, burned three retries per link before saying the same thing.
assert!(matches!(
classify_status("x", StatusCode::NOT_FOUND, None),
FetchError::NotFound
));
assert!(matches!(
classify_status("x", StatusCode::BAD_REQUEST, None),
FetchError::Blocked
));
assert!(matches!(
classify_status("x", StatusCode::PAYLOAD_TOO_LARGE, None),
FetchError::Blocked
));
assert!(matches!(
classify_status("x", StatusCode::REQUEST_TIMEOUT, None),
FetchError::Transient(_)
));
assert!(matches!(
classify_status("x", StatusCode::TOO_MANY_REQUESTS, None),
FetchError::Transient(_)
));
assert!(matches!(
classify_status("x", StatusCode::INTERNAL_SERVER_ERROR, None),
FetchError::Transient(_)
));
// The download path delegates under its own name, same classes.
assert!(matches!(
classify_status("media", StatusCode::BAD_REQUEST, None),
FetchError::Blocked
));
// A 429 that named its delay keeps it — and the cap means the
// (server-supplied) header cannot park a fetch slot for an hour.
assert!(matches!(
classify_status("x", StatusCode::TOO_MANY_REQUESTS, Some(12)),
FetchError::RateLimited {
retry_after_secs: 12,
..
}
));
assert!(matches!(
classify_status("x", StatusCode::TOO_MANY_REQUESTS, Some(9999)),
FetchError::RateLimited {
retry_after_secs: crate::site::MAX_RETRY_AFTER_SECS,
..
}
));
}
#[test]
fn retry_wait_jitters_and_respects_a_named_delay() {
// Doubling base plus a random slice: attempt 0 → exactly 1 s (any
// slice of 1 is 0), attempt 2 with roll 3 → 4 + 3 s.
assert_eq!(
retry_wait(0, 0, &FetchError::Transient("x".into())),
Duration::from_secs(1)
);
assert_eq!(
retry_wait(2, 3, &FetchError::Transient("x".into())),
Duration::from_secs(7)
);
// A named delay floors the wait: roll 0 would sleep 1 s, the source said 60.
assert_eq!(
retry_wait(
0,
0,
&FetchError::RateLimited {
site: "x",
retry_after_secs: 60
}
),
Duration::from_secs(60)
);
}
#[tokio::test]
async fn disabled_site_is_reported_not_ignored() {
// pixiv is the only token-gated site; with PIXIV_REFRESH_TOKEN set it
@@ -1110,29 +942,4 @@ mod tests {
Some("pixiv:1".into())
);
}
#[tokio::test]
async fn download_media_pixiv_original_with_referer() {
// Proves the Referer header is attached for i.pximg.net: a header-less
// GET to a pixiv original URL is rejected with 403.
// Empty-string check too: an unset CI secret arrives as "" (GitHub
// Actions), which would otherwise run the test tokenless and fail.
if std::env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|s| !s.is_empty())
.is_none()
{
eprintln!("skipping: no PIXIV_REFRESH_TOKEN");
return;
}
let illustration = pixiv::fetch(126839080).await.unwrap();
let fetched: Fetched = illustration.into();
let url = match fetched.media.first() {
Some(crate::media::Media::Illustration { url, .. }) => url.clone(),
other => panic!("expected illustration media, got {other:?}"),
};
assert!(url.contains("i.pximg.net"));
let bytes = download_media(&url).await.unwrap();
assert!(!bytes.is_empty());
}
}
+156 -58
View File
@@ -4,7 +4,7 @@
//! `app-api.pixiv.net`, deserialized with the kept `model.rs` types.
use super::interface::Illustration;
use super::model::{IllustrationModel, TypeModel, UgoiraMetadataModel};
use super::model::{IllustrationModel, UgoiraMetadataModel};
use crate::media::Media;
use crate::site::FetchError;
use std::env;
@@ -23,6 +23,19 @@ const APP_USER_AGENT: &str = "PixivIOSApp/7.13.3 (iOS 14.6; iPhone13,2)";
/// Token refresh safe margin (seconds).
const TOKEN_REFRESH_SAFE_MARGIN: u64 = 300;
const MAX_UGOIRA_FRAMES: usize = 5_000;
const MAX_UGOIRA_UNPACKED_BYTES: u64 = 512 * 1024 * 1024;
fn check_ugoira_archive_size(entries: usize, unpacked: u64) -> Result<(), &'static str> {
if entries > MAX_UGOIRA_FRAMES {
return Err("ugoira has too many frames");
}
if unpacked > MAX_UGOIRA_UNPACKED_BYTES {
return Err("ugoira exceeds total unpacked size cap");
}
Ok(())
}
#[derive(Debug, Error)]
pub enum PixivError {
/// No refresh token available (PIXIV_REFRESH_TOKEN unset).
@@ -39,6 +52,20 @@ pub enum PixivError {
Status(u16),
#[error("pixiv api error: {0}")]
Api(String),
/// A bad moment while preparing media: a transient download status
/// (429 / 5xx), a stalled transfer or a temp-file write failure. A retry
/// can change the answer, so the pixiv retry policy re-fetches these.
#[error("transient pixiv error: {0}")]
Transient(String),
}
fn map_response_error(error: FetchError) -> PixivError {
match error {
FetchError::Http(e) => PixivError::Http(e),
FetchError::Transient(message) => PixivError::Transient(message),
FetchError::RateLimited { .. } => PixivError::Transient("rate limited".into()),
other => PixivError::Api(other.to_string()),
}
}
/// Native pixiv app-API client.
@@ -76,14 +103,12 @@ impl PixivAPI {
.header("User-Agent", AUTH_USER_AGENT)
.send()
.await?;
// Check the status *before* reading the body: a 429/5xx from the
// token endpoint is worth retrying (the class comes from
// `is_retryable`), while parsing a maintenance page as JSON turned it
// into a permanent `Api`/`Json` error with no retry at all.
if !response.status().is_success() {
return Err(PixivError::Status(response.status().as_u16()));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
let json: serde_json::Value = crate::site::response_json(response, "pixiv")
.await
.map_err(map_response_error)?;
let access_token = json
.get("access_token")
.and_then(|v| v.as_str())
@@ -122,7 +147,9 @@ impl PixivAPI {
if !response.status().is_success() {
return Err(PixivError::Status(response.status().as_u16()));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
let json: serde_json::Value = crate::site::response_json(response, "pixiv")
.await
.map_err(map_response_error)?;
if json.get("error").is_some() {
let message = json
.get("message")
@@ -139,7 +166,7 @@ impl PixivAPI {
pub async fn fetch(&self, illust_id: u64) -> Result<Illustration, FetchError> {
let model = self.illust_detail(illust_id).await?;
let mut illustration = Illustration::from_model(&model);
if matches!(&model.r#type, TypeModel::Ugoira) {
if model.r#type == "ugoira" {
// Real ugoira support: download the frame zip and encode an MP4.
// Without ffmpeg the post stays unsupported (empty media, like
// Python) — but a *failed* download/encode is reported instead:
@@ -149,7 +176,6 @@ impl PixivAPI {
match self.ugoira_video(illust_id).await {
Ok(Some((mp4_path, _keep_alive))) => {
illustration.media.push(Media::Video {
title: None,
url: mp4_path,
thumbnail_url: model.image_urls.medium.clone(),
});
@@ -181,7 +207,9 @@ impl PixivAPI {
if !response.status().is_success() {
return Err(PixivError::Status(response.status().as_u16()));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
let json: serde_json::Value = crate::site::response_json(response, "pixiv")
.await
.map_err(map_response_error)?;
if json.get("error").is_some() {
let message = json
.get("message")
@@ -202,14 +230,15 @@ impl PixivAPI {
&self,
illust_id: u64,
) -> Result<Option<(String, tempfile::TempDir)>, PixivError> {
if !crate::site::ffmpeg_available() {
crate::site::log_once_ffmpeg_missing();
if crate::site::ffmpeg_missing() {
return Ok(None);
}
let metadata = self.ugoira_metadata(illust_id).await?;
if metadata.frames.is_empty() {
return Ok(None);
}
check_ugoira_archive_size(metadata.frames.len(), 0)
.map_err(|e| PixivError::Api(e.to_string()))?;
let zip_url = metadata
.zip_url
.clone()
@@ -220,17 +249,39 @@ impl PixivAPI {
// Stream the frame zip to a temp file instead of buffering it in
// memory: ugoira zips can be hundreds of MB, and the old
// download_media_limited path spiked RAM up to the size cap.
let mut zip_file = tempfile::Builder::new()
let zip_file = tempfile::Builder::new()
.prefix(crate::TEMP_FILE_PREFIX)
.suffix(".zip")
.tempfile()
.map_err(|e| PixivError::Api(format!("temp zip failed: {e}")))?;
crate::site::download_media_to_file(&zip_url, 512 * 1024 * 1024, zip_file.as_file_mut())
// Stream through a tokio handle: a sync write per chunk would stall
// an executor thread for the whole (up to 512 MiB) download. The
// clone shares the file offset with `zip_file`, so the extraction
// below reads what was written, and dropping it after the download
// hands every byte to the OS.
let mut zip_out = tokio::fs::File::from_std(
zip_file
.as_file()
.try_clone()
.map_err(|e| PixivError::Api(format!("temp zip clone failed: {e}")))?,
);
crate::site::download_media_to_file(&zip_url, 512 * 1024 * 1024, &mut zip_out)
.await
.map_err(|e| match e {
FetchError::Http(e) => PixivError::Http(e),
// A bad moment (429/5xx, a stalled transfer, a temp-file
// write failure) must stay retryable: folding it into Api
// made one hiccup permanently fail the whole ugoira post,
// while the bot's own upload downloads retry the same
// classes.
transient @ (FetchError::Transient(_)
| FetchError::RateLimited { .. }
| FetchError::Io(_)) => {
PixivError::Transient(format!("frame zip download failed: {transient}"))
}
other => PixivError::Api(format!("frame zip download failed: {other}")),
})?;
drop(zip_out);
let frame_delays = metadata.frames.iter().map(|f| f.delay).collect::<Vec<_>>();
let result =
tokio::task::spawn_blocking(move || -> Result<(String, tempfile::TempDir), String> {
@@ -242,10 +293,6 @@ impl PixivAPI {
.prefix(crate::TEMP_FILE_PREFIX)
.tempdir()
.map_err(|e| e.to_string())?;
// Extract frames to canonical zero-padded names; pixiv ugoira
// frames are uniformly jpg or png per artwork. The zip is read
// from disk; `zip_file` stays alive for the whole extraction.
let mut archive = zip::ZipArchive::new(
std::fs::File::open(zip_file.path()).map_err(|e| e.to_string())?,
)
@@ -253,33 +300,12 @@ impl PixivAPI {
if archive.is_empty() {
return Err("empty frame zip".to_string());
}
// Uniform jpg or png per artwork; sniff the first entry's
// magic bytes instead of trusting its filename.
let first = archive.by_index(0).map_err(|e| e.to_string())?;
let mut first_bytes = Vec::new();
first
.take(64 * 1024 * 1024 + 1)
.read_to_end(&mut first_bytes)
.map_err(|e| e.to_string())?;
if first_bytes.len() > 64 * 1024 * 1024 {
return Err("frame exceeds size cap".to_string());
}
let extension = if first_bytes.starts_with(&[0xFF, 0xD8]) {
"jpg"
} else if first_bytes.starts_with(b"\x89PNG") {
"png"
} else {
"jpg"
};
check_ugoira_archive_size(archive.len(), 0)?;
let mut count = 0usize;
{
let path = frames_dir
.path()
.join(format!("img_{count:05}.{extension}"));
std::fs::write(&path, &first_bytes).map_err(|e| e.to_string())?;
count += 1;
}
for i in 1..archive.len() {
let mut unpacked = 0u64;
let mut extension = "jpg";
for i in 0..archive.len() {
let entry = archive.by_index(i).map_err(|e| e.to_string())?;
if entry.size() > 64 * 1024 * 1024 {
return Err(format!("frame {i} exceeds size cap"));
@@ -292,6 +318,19 @@ impl PixivAPI {
if bytes.len() > 64 * 1024 * 1024 {
return Err(format!("frame {i} exceeds size cap"));
}
unpacked = unpacked
.checked_add(bytes.len() as u64)
.ok_or_else(|| "ugoira unpacked size overflow".to_string())?;
check_ugoira_archive_size(archive.len(), unpacked)?;
if i == 0 {
extension = if bytes.starts_with(&[0xFF, 0xD8]) {
"jpg"
} else if bytes.starts_with(b"\x89PNG") {
"png"
} else {
"jpg"
};
}
let path = frames_dir
.path()
.join(format!("img_{count:05}.{extension}"));
@@ -301,7 +340,6 @@ impl PixivAPI {
if count == 0 {
return Err("empty frame zip".to_string());
}
// Constant rate from the median frame delay (ms).
let mut delays = frame_delays;
delays.sort_unstable();
@@ -309,7 +347,7 @@ impl PixivAPI {
let framerate = 1000.0 / median as f64;
let output = out_dir.path().join("ugoira.mp4");
let status = std::process::Command::new("ffmpeg")
let mut child = std::process::Command::new("ffmpeg")
.args([
"-y",
"-framerate",
@@ -333,10 +371,27 @@ impl PixivAPI {
])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.spawn()
.map_err(|e| format!("ffmpeg spawn failed: {e}"))?;
if !status.success() {
return Err(format!("ffmpeg exited with {status}"));
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(300);
loop {
match child
.try_wait()
.map_err(|e| format!("ffmpeg wait failed: {e}"))?
{
Some(status) => {
if !status.success() {
return Err(format!("ffmpeg exited with {status}"));
}
break;
}
None if std::time::Instant::now() >= deadline => {
let _ = child.kill();
let _ = child.wait();
return Err("ffmpeg exceeded 300s".to_string());
}
None => std::thread::sleep(std::time::Duration::from_millis(50)),
}
}
Ok((output.to_string_lossy().into_owned(), out_dir))
})
@@ -355,16 +410,23 @@ impl PixivAPI {
}
}
/// pixiv3-rs replacement: `None` when `PIXIV_REFRESH_TOKEN` is unset.
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> =
LazyLock::new(|| env::var("PIXIV_REFRESH_TOKEN").ok().map(PixivAPI::new));
/// pixiv3-rs replacement: `None` when `PIXIV_REFRESH_TOKEN` is unset or empty
/// (compose injects an empty string for a blank `.env` value; an empty token
/// must mean "not configured" instead of being sent to OAuth).
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> = LazyLock::new(|| {
env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|token| !token.is_empty())
.map(PixivAPI::new)
});
/// Set at startup when the login validation fails; pixiv stays disabled until
/// the next process start.
static DISABLED: AtomicBool = AtomicBool::new(false);
pub fn enabled() -> bool {
!DISABLED.load(Ordering::Relaxed) && env::var("PIXIV_REFRESH_TOKEN").is_ok()
!DISABLED.load(Ordering::Relaxed)
&& env::var("PIXIV_REFRESH_TOKEN").is_ok_and(|token| !token.is_empty())
}
/// Permanently disables pixiv until the next process start.
@@ -398,15 +460,33 @@ mod tests {
use super::*;
use dotenv::dotenv;
/// An empty `PIXIV_REFRESH_TOKEN` (what compose injects for a blank
/// `.env` value, and what an unset GitHub secret looks like) must read as
/// "not configured", exactly like unset — otherwise a default deployment
/// sends an empty refresh token to OAuth and fails login validation on
/// every boot.
#[test]
fn empty_refresh_token_reads_as_unset() {
// SAFETY: the value is restored before returning; `enabled()` keys on
// this variable alone and no other test mutates it. Concurrent readers
// see unset or empty, which this very fix makes the same answer.
let previous = env::var("PIXIV_REFRESH_TOKEN").ok();
unsafe { env::set_var("PIXIV_REFRESH_TOKEN", "") };
let empty = enabled();
unsafe { env::remove_var("PIXIV_REFRESH_TOKEN") };
let unset = enabled();
match previous {
Some(value) => unsafe { env::set_var("PIXIV_REFRESH_TOKEN", value) },
None => unsafe { env::remove_var("PIXIV_REFRESH_TOKEN") },
}
assert!(!empty, "an empty token must not enable pixiv");
assert_eq!(empty, unset, "empty must read exactly like unset");
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to oauth.secure.pixiv.net"]
async fn live_validate_with_bogus_token_fails() {
dotenv().ok();
// A rejected credential must surface as a permanent status, not a panic
// and not a retryable class: the exchange answers 4xx and the status is
// checked before the body is read (api.rs, `get_access_token`). This
// used to assert `Api`, which that check made unreachable — `Api` is
// only reached from a 2xx body without an `access_token`.
let client = PixivAPI::new("bogus_token_for_testing".to_string());
let result = client.get_access_token().await;
assert!(
@@ -414,4 +494,22 @@ mod tests {
"got {result:?}"
);
}
#[test]
fn response_read_transport_errors_stay_retryable() {
let mapped = map_response_error(FetchError::Transient("reset".into()));
assert!(matches!(mapped, PixivError::Transient(_)));
}
#[test]
fn ugoira_budget_rejects_too_many_frames() {
assert!(check_ugoira_archive_size(MAX_UGOIRA_FRAMES, 0).is_ok());
assert!(check_ugoira_archive_size(MAX_UGOIRA_FRAMES + 1, 0).is_err());
}
#[test]
fn ugoira_budget_rejects_too_many_unpacked_bytes() {
assert!(check_ugoira_archive_size(1, MAX_UGOIRA_UNPACKED_BYTES).is_ok());
assert!(check_ugoira_archive_size(1, MAX_UGOIRA_UNPACKED_BYTES + 1).is_err());
}
}
+72 -37
View File
@@ -1,4 +1,4 @@
use super::model::{IllustrationModel, TypeModel};
use super::model::{IllustrationModel, ImageUrlsModel};
use crate::media::Media;
use crate::site::{FetchError, Fetched, PixivError, Site, SiteFuture};
use html_escape::{encode_double_quoted_attribute, encode_text};
@@ -91,7 +91,7 @@ pub fn cache_key(url: &str) -> Option<String> {
/// API/auth errors, unparseable bodies and missing auth are not retried.
pub fn is_retryable(err: &FetchError) -> bool {
match err {
FetchError::Http(_) | FetchError::Transient(_) => true,
FetchError::Http(_) | FetchError::Transient(_) | FetchError::RateLimited { .. } => true,
FetchError::Pixiv(e) => pixiv_error_is_retryable(e),
_ => false,
}
@@ -102,7 +102,7 @@ pub fn is_retryable(err: &FetchError) -> bool {
/// rejected credential is not.
fn pixiv_error_is_retryable(err: &PixivError) -> bool {
match err {
PixivError::Http(_) => true,
PixivError::Http(_) | PixivError::Transient(_) => true,
PixivError::Status(code) if *code == 429 || *code >= 500 => true,
PixivError::Status(_) | PixivError::Api(_) | PixivError::Json(_) | PixivError::NoAuth => {
false
@@ -225,34 +225,28 @@ impl Illustration {
tags.insert(0, "AI".to_string());
}
let mut media = vec![];
if matches!(&model.r#type, TypeModel::Ugoira) {
if model.r#type == "ugoira" {
// No static images for ugoira; the fetch path encodes an MP4 via
// ffmpeg and appends it as a Video item (api.rs). This fallback
// keeps media empty when encoding fails or ffmpeg is missing.
} else if model.page_count > 1 {
// Every page is kept: `original` is the only URL the API may leave
// out (typically the restricted ones), and a page without it used
// to be dropped whole — losing a page of the work while `large`
// sat right there.
media.extend(model.meta_pages.iter().filter_map(|page| {
page.image_urls
.original
.clone()
.map(|original| Media::Illustration {
title: None,
url: original,
thumbnail_url: Some(page.image_urls.medium.clone()),
fallback_url: Some(page.image_urls.large.clone()),
})
page_illustration(page.image_urls.original.clone(), &page.image_urls)
}));
} else if let Some(original) = model
.meta_single_page
.original_image_url
.clone()
.or(model.image_urls.original.clone())
{
media.push(Media::Illustration {
title: None,
url: original,
thumbnail_url: Some(model.image_urls.medium.clone()),
fallback_url: Some(model.image_urls.large.clone()),
});
} else {
// The single page names its original in one of two places, and
// `large` is the last resort.
let urls = &model.image_urls;
let original = model
.meta_single_page
.original_image_url
.clone()
.or_else(|| urls.original.clone());
media.extend(page_illustration(original, urls));
}
let nsfw = model.sanity_level > 5;
Self {
@@ -269,6 +263,21 @@ impl Illustration {
}
}
/// One artwork page as a media item: `original` when the API sent one, else the
/// `large` variant (the same picture at a lower resolution), with `medium` as
/// the thumbnail. `None` when the API gave no usable URL at all.
fn page_illustration(original: Option<String>, urls: &ImageUrlsModel) -> Option<Media> {
let url = original.unwrap_or_else(|| urls.large.clone());
if url.is_empty() {
return None;
}
Some(Media::Illustration {
url,
thumbnail_url: Some(urls.medium.clone()),
fallback_url: Some(urls.large.clone()),
})
}
impl From<Illustration> for Fetched {
fn from(illustration: Illustration) -> Self {
let url = illustration.url();
@@ -280,7 +289,6 @@ impl From<Illustration> for Fetched {
.collect::<Vec<_>>()
.join(" ");
let render_data = Some(crate::site::RenderData {
url: url.clone(),
author: encode_text(&illustration.author).into_owned(),
author_url: author_url.clone(),
title: encode_text(&illustration.title).into_owned(),
@@ -434,7 +442,11 @@ mod tests {
// it calls `disable()`, a process-wide flag with no reset, so a test
// touching it would order-couple every other pixiv test (the predicate
// it keys on is covered by the table below).
for err in [PixivError::Status(429), PixivError::Status(503)] {
for err in [
PixivError::Status(429),
PixivError::Status(503),
PixivError::Transient("frame zip download failed: transient".into()),
] {
let enabled_before = api::enabled();
let message = startup_validation(Err(err)).unwrap_err();
assert!(message.contains("stays enabled"), "{message}");
@@ -449,11 +461,19 @@ mod tests {
#[test]
fn is_retryable_classifies_transient_and_permanent() {
// Transient: network errors, explicit transient, pixiv 429/5xx.
// Transient: network errors, explicit transient, pixiv 429/5xx, and a
// failed media download (the frame zip's own bad moment).
assert!(is_retryable(&FetchError::Transient("429".into())));
assert!(is_retryable(&FetchError::RateLimited {
site: "pixiv",
retry_after_secs: 30
}));
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(429))));
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(500))));
assert!(is_retryable(&FetchError::Pixiv(PixivError::Status(503))));
assert!(is_retryable(&FetchError::Pixiv(PixivError::Transient(
"frame zip download failed: transient: media status 429".into()
))));
// Permanent: pixiv 4xx (bad/expired token, forbidden, not found),
// api/auth errors, unparseable bodies, not-found/blocked/sensitive.
assert!(!is_retryable(&FetchError::Pixiv(PixivError::Status(400))));
@@ -542,14 +562,19 @@ mod tests {
}
#[test]
fn single_page_without_any_original_is_empty() {
fn single_page_without_any_original_falls_back_to_large() {
let v = illust_json("illust", 1, None, None, vec![], 0);
let fetched: Fetched = parse(v).into();
assert!(fetched.media.is_empty());
// Neither `meta_single_page.original_image_url` nor `image_urls.
// original` is set: the work is still deliverable as `large`.
match fetched.media.as_slice() {
[Media::Illustration { url, .. }] => assert_eq!(url, "large.jpg"),
other => panic!("expected the large variant, got {other:?}"),
}
}
#[test]
fn multi_page_skips_pages_without_original() {
fn multi_page_keeps_pages_without_original() {
let v = illust_json(
"illust",
2,
@@ -562,17 +587,27 @@ mod tests {
0,
);
let fetched: Fetched = parse(v).into();
assert_eq!(fetched.media.len(), 1);
// Both pages arrive: the restricted one (no `original`) sends its
// `large` instead of vanishing — a dropped page is a missing picture.
let urls: Vec<&str> = fetched
.media
.iter()
.map(|media| match media {
Media::Illustration { url, .. } => url.as_str(),
other => panic!("expected Illustration, got {other:?}"),
})
.collect();
assert_eq!(urls, vec!["l1.jpg", "https://i.pximg.net/p2.jpg"]);
match &fetched.media[0] {
Media::Illustration {
url,
thumbnail_url,
fallback_url,
..
} => {
assert_eq!(url, "https://i.pximg.net/p2.jpg");
assert_eq!(thumbnail_url.as_deref(), Some("m2.jpg"));
assert_eq!(fallback_url.as_deref(), Some("l2.jpg"));
assert_eq!(thumbnail_url.as_deref(), Some("m1.jpg"));
// `large` is the item itself here, so it is not also a
// smaller variant of itself.
assert_eq!(fallback_url.as_deref(), Some("l1.jpg"));
}
other => panic!("expected Illustration, got {other:?}"),
}
@@ -604,7 +639,7 @@ mod tests {
);
// Empty format falls back to the built-in caption.
assert_eq!(fetched.caption_with(""), fetched.caption);
assert_eq!(fetched.site_name(), "pixiv");
assert_eq!(fetched.site_id, "pixiv");
}
#[test]
+1 -1
View File
@@ -2,7 +2,7 @@ mod api;
mod interface;
mod model;
pub use api::{PixivAPI, PixivError, disable, fetch, validate};
pub use api::{PixivError, disable, fetch, validate};
pub use interface::{
Illustration, PATTERN, PixivSite, cache_key, enabled, fetch_from_url, is_retryable,
media_headers,
+4 -11
View File
@@ -10,7 +10,10 @@ pub struct IllustrationModel {
/// works (`<br />`, `<a href>`, sometimes `<p>`), empty for many.
#[serde(default)]
pub caption: String,
pub r#type: TypeModel,
/// `"illust"` / `"manga"` / `"ugoira"`; only ugoira changes how the
/// artwork is fetched (a zip of frames to encode), so the rest is kept as
/// the string the API sent rather than as variants nothing matches.
pub r#type: String,
pub image_urls: ImageUrlsModel,
pub user: UserInfoModel,
pub tags: Vec<IllustrationTagModel>,
@@ -22,16 +25,6 @@ pub struct IllustrationModel {
pub meta_pages: Vec<MetaPageModel>,
}
#[derive(Deserialize, Debug)]
pub enum TypeModel {
#[serde(rename = "illust")]
Illust,
#[serde(rename = "manga")]
Manga,
#[serde(rename = "ugoira")]
Ugoira,
}
#[derive(Deserialize, Debug)]
pub struct UserInfoModel {
pub id: u64,
+2 -10
View File
@@ -130,17 +130,9 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
let status = response.status();
if !status.is_success() {
log::warn!("twitter auth fetch {id}: HTTP {status}");
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
// A stale/refused `auth_token` is not a bad moment: retrying it
// three times only delays the report.
401 | 403 => Err(FetchError::Blocked),
_ => Err(FetchError::Transient(format!(
"twitter auth status {status}"
))),
};
return Err(crate::site::status_error("twitter auth", &response));
}
let text = response.text().await?;
let text = crate::site::response_text(response, "twitter auth").await?;
let json: Value = serde_json::from_str(&text)?;
let result = parse_tweet_result(&json, id)?;
let syndication_shape = to_syndication_shape(&result).ok_or_else(|| {
+10 -56
View File
@@ -1,7 +1,7 @@
use super::model;
use crate::media::Media;
use crate::site::{FetchError, Fetched, Site, SiteFuture};
use html_escape::{decode_html_entities, encode_double_quoted_attribute, encode_text};
use html_escape::{decode_html_entities, encode_text};
use regex::Regex;
use std::sync::LazyLock;
@@ -30,8 +30,12 @@ pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"^(?:https?://)?(?:www\.|mobile\.)?(?:x|twitter|fixvx|vxtwitter|fixupx|fxtwitter)\.com/[^.]+/status/(\d+)").unwrap()
});
pub fn enabled() -> bool {
true
/// Cache key for a twitter URL: `"twitter:<id>"`. The prefix is the site id
/// used for caption-format lookup and link-cache keys.
pub fn cache_key(url: &str) -> Option<String> {
PATTERN
.captures(url)
.map(|caps| format!("twitter:{}", &caps[1]))
}
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
@@ -68,26 +72,6 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
}
}
/// Cache key for a twitter URL: `"twitter:<id>"`. The prefix is the site id
/// used for caption-format lookup and link-cache keys.
pub fn cache_key(url: &str) -> Option<String> {
PATTERN
.captures(url)
.map(|caps| format!("twitter:{}", &caps[1]))
}
/// Twitter's fetch-retry policy: transient classes only. Not-found, blocked,
/// sensitive (NSFW withholding) and parse failures are permanent — retrying
/// them only wastes attempts against the syndication endpoint.
pub fn is_retryable(err: &FetchError) -> bool {
matches!(err, FetchError::Http(_) | FetchError::Transient(_))
}
/// twimg URLs need no extra headers (no hotlink protection).
pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// Fetches a tweet from the syndication endpoint. Deleted/blocked tweets
/// surface as `FetchError::NotFound`; withheld content (empty tombstone,
/// age-restricted) as `FetchError::Sensitive`.
@@ -103,15 +87,9 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
// 404/410 = gone (permanent); 429/5xx = transient and retried by fetch.
let status = response.status();
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
// A refusal or an auth demand is not a bad moment: retrying it
// three times only delays an error the user has to see.
401 | 403 => Err(FetchError::Blocked),
_ => Err(FetchError::Transient(format!("twitter status {status}"))),
};
return Err(crate::site::status_error("twitter", &response));
}
let text = response.text().await?;
let text = crate::site::response_text(response, "twitter").await?;
// Classify before building the tweet (see [`parse_syndication_body`]), and
// build it from the value that classification already parsed: this used to
// scan and allocate the whole body twice.
@@ -221,13 +199,7 @@ impl Tweet {
}
pub fn caption(&self) -> String {
format!(
"{url}\n<a href=\"{author_url}\">{author}</a>: {text}",
url = encode_double_quoted_attribute(&self.url()),
author_url = encode_double_quoted_attribute(&self.author_url()),
author = encode_text(&self.author),
text = encode_text(&self.text),
)
crate::site::caption(&self.url(), &self.author_url(), &self.author, &self.text)
}
/// Builds a tweet from an already-parsed syndication body. Takes the value
@@ -258,7 +230,6 @@ impl Tweet {
for item in json.media_details {
match item.media_type.as_str() {
"photo" => media.push(Media::Illustration {
title: None,
url: original_twimg_url(&item.media_url_https),
thumbnail_url: None,
// The param-less base URL is a reduced-size variant;
@@ -266,12 +237,10 @@ impl Tweet {
fallback_url: Some(item.media_url_https.clone()),
}),
"video" => media.push(Media::Video {
title: None,
url: mp4_variant(&item),
thumbnail_url: item.media_url_https,
}),
"animated_gif" => media.push(Media::Animated {
title: None,
url: mp4_variant(&item),
thumbnail_url: item.media_url_https,
}),
@@ -357,7 +326,6 @@ impl From<Tweet> for Fetched {
let author_url = tweet.author_url();
// A tweet has no title: its text is all content.
let render_data = Some(crate::site::RenderData {
url: url.clone(),
author: encode_text(&tweet.author).into_owned(),
author_url: author_url.clone(),
title: String::new(),
@@ -468,20 +436,6 @@ mod tests {
assert_eq!(cache_key("https://example.com/1"), None);
}
#[test]
fn is_retryable_classifies_transient_and_permanent() {
// Transient: network errors and explicit transient statuses (the
// `Http` arm shares this match arm with `Transient`).
assert!(is_retryable(&FetchError::Transient("429".into())));
// Permanent: gone, blocked, withheld, oversized, unparseable.
assert!(!is_retryable(&FetchError::NotFound));
assert!(!is_retryable(&FetchError::Blocked));
assert!(!is_retryable(&FetchError::Sensitive));
assert!(!is_retryable(&FetchError::TooLarge));
let json_err = serde_json::from_str::<serde_json::Value>("x").unwrap_err();
assert!(!is_retryable(&FetchError::Json(json_err)));
}
#[test]
fn syndication_json_converts_to_fetched() {
let raw = fixture(serde_json::json!([
+1 -3
View File
@@ -2,6 +2,4 @@ mod auth;
mod interface;
mod model;
pub use interface::{
PATTERN, Tweet, TwitterSite, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
};
pub use interface::{PATTERN, Tweet, TwitterSite, cache_key, fetch_from_url};
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "xmedia-bot"
version = "1.9.1"
version = "1.9.2"
edition = "2024"
[dependencies]
+144 -30
View File
@@ -34,29 +34,45 @@ impl Config {
fn parse_u64(name: &str, default: u64) -> u64 {
match env::var(name) {
Ok(v) => v.parse::<u64>().unwrap_or_else(|_| {
log::warn!("invalid {name}={v:?}; using default {default}");
log::warn!("invalid {name}; using default {default}");
default
}),
Err(_) => default,
}
}
let admin_ids = match env::var("BOT_ADMIN") {
Ok(s) => {
let (ids, bad): (Vec<_>, Vec<_>) = s
/// A setting that must parse when it is set: an unparseable value warns
/// (naming the variable) and counts as unset.
fn parse_opt<T: std::str::FromStr>(name: &str) -> Option<T> {
env::var(name).ok().and_then(|s| {
s.parse::<T>().ok().or_else(|| {
log::warn!("invalid {name}");
None
})
})
}
let admin_ids = env::var("BOT_ADMIN")
.map(|s| {
let mut bad = Vec::new();
let ids: Vec<i64> = s
.split(',')
.map(str::trim)
.filter(|part| !part.is_empty())
.partition(|part| part.parse::<i64>().is_ok());
.filter_map(|part| match part.parse::<i64>() {
Ok(id) => Some(id),
Err(_) => {
bad.push(part);
None
}
})
.collect();
if !bad.is_empty() {
log::warn!("BOT_ADMIN: ignoring non-numeric ids: {bad:?}");
log::warn!("BOT_ADMIN: ignoring {} non-numeric id(s)", bad.len());
}
ids.into_iter()
.filter_map(|p| p.parse::<i64>().ok())
.collect()
}
Err(_) => Vec::new(),
};
ids
})
.unwrap_or_default();
let edit_message_ttl =
Duration::from_secs(parse_u64("EDIT_MESSAGE_TTL_SECONDS", 24 * 3600));
@@ -69,24 +85,9 @@ impl Config {
// The webhook settings are consumed by `.expect()` in main when
// WEBHOOK=true, so an unparseable value fails fast at startup with a
// clear message; still log here for the WEBHOOK=false case.
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| {
s.parse::<url::Url>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_URL={s:?}");
None
})
});
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| {
s.parse::<IpAddr>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_LISTEN={s:?}");
None
})
});
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| {
s.parse::<u16>().ok().or_else(|| {
log::warn!("invalid WEBHOOK_PORT={s:?}");
None
})
});
let webhook_url = parse_opt::<url::Url>("WEBHOOK_URL");
let webhook_listen = parse_opt::<IpAddr>("WEBHOOK_LISTEN");
let webhook_port = parse_opt::<u16>("WEBHOOK_PORT");
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
// value that would otherwise come from `.env`).
let webhook_cert = env::var("WEBHOOK_CERT").ok().filter(|s| !s.is_empty());
@@ -108,3 +109,116 @@ impl Config {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// The ids an operator's `BOT_ADMIN` yields: blanks dropped, non-numeric
/// entries warned about and skipped, the rest kept in order. Parsed once —
/// the split used to parse every entry twice.
#[test]
fn bot_admin_keeps_the_numeric_ids_in_order() {
// SAFETY: no other test reads BOT_ADMIN, and the value is restored
// before this test returns.
let previous = env::var("BOT_ADMIN").ok();
unsafe { env::set_var("BOT_ADMIN", " 7 ,abc,42, ,") };
let ids = Config::load().admin_ids;
match previous {
Some(value) => unsafe { env::set_var("BOT_ADMIN", value) },
None => unsafe { env::remove_var("BOT_ADMIN") },
}
assert_eq!(ids, vec![7, 42]);
}
/// The webhook truth table: `Config::load` enables webhook mode only for
/// a case-insensitive `true|yes|1`, and everything else — including the
/// classic misspelling "on", which an operator would expect to work — is
/// polling. Without this pin a typo silently ran a different transport
/// (with P0's fail-fast secret check, or with no listener at all).
#[test]
fn webhook_flag_is_a_case_insensitive_truth_table() {
// SAFETY: no other test *mutates* WEBHOOK, the value is restored
// before this test returns, and concurrent Config::load callers in
// other tests assert fields other than webhook_enabled.
let previous = env::var("WEBHOOK").ok();
for (value, expected) in [
("true", true),
("TRUE", true),
("Yes", true),
("1", true),
("false", false),
("on", false),
("", false),
] {
unsafe { env::set_var("WEBHOOK", value) };
assert_eq!(
Config::load().webhook_enabled,
expected,
"WEBHOOK={value:?}"
);
}
match previous {
Some(value) => unsafe { env::set_var("WEBHOOK", value) },
None => unsafe { env::remove_var("WEBHOOK") },
}
}
/// A malformed TTL warns and falls back to the default instead of being
/// parsed as 0 — the difference between a 24h edit-prompt expiry and a
/// prompt that expires instantly, which an operator would only notice
/// when the buttons stop working.
#[test]
fn invalid_ttl_falls_back_to_the_default() {
// SAFETY: no other test *mutates* EDIT_MESSAGE_TTL_SECONDS; restored
// below, and no other test asserts the TTL field.
let previous = env::var("EDIT_MESSAGE_TTL_SECONDS").ok();
unsafe { env::set_var("EDIT_MESSAGE_TTL_SECONDS", "not-a-number") };
let invalid = Config::load().edit_message_ttl;
unsafe { env::set_var("EDIT_MESSAGE_TTL_SECONDS", "120") };
let valid = Config::load().edit_message_ttl;
match previous {
Some(value) => unsafe { env::set_var("EDIT_MESSAGE_TTL_SECONDS", value) },
None => unsafe { env::remove_var("EDIT_MESSAGE_TTL_SECONDS") },
}
assert_eq!(
invalid,
Duration::from_secs(24 * 3600),
"an unparseable value falls back to the default"
);
assert_eq!(
valid,
Duration::from_secs(120),
"a valid value is taken as-is"
);
}
/// A blank WEBHOOK_CERT / WEBHOOK_SECRET_TOKEN counts as unset — compose
/// injects `${VAR:-}` as an empty string for a commented-out template
/// line — while a present value is kept (the `-e VAR=` disable idiom).
#[test]
fn blank_webhook_cert_and_secret_count_as_unset() {
// SAFETY: no other test *mutates* these two, both are restored
// below, and no other test asserts them.
let prev_cert = env::var("WEBHOOK_CERT").ok();
let prev_secret = env::var("WEBHOOK_SECRET_TOKEN").ok();
unsafe { env::set_var("WEBHOOK_CERT", "") };
unsafe { env::set_var("WEBHOOK_SECRET_TOKEN", "") };
let blank = Config::load();
unsafe { env::set_var("WEBHOOK_CERT", "/x/cert.pem") };
unsafe { env::set_var("WEBHOOK_SECRET_TOKEN", "s3cret") };
let present = Config::load();
match prev_cert {
Some(value) => unsafe { env::set_var("WEBHOOK_CERT", value) },
None => unsafe { env::remove_var("WEBHOOK_CERT") },
}
match prev_secret {
Some(value) => unsafe { env::set_var("WEBHOOK_SECRET_TOKEN", value) },
None => unsafe { env::remove_var("WEBHOOK_SECRET_TOKEN") },
}
assert_eq!(blank.webhook_cert, None, "blank must read as unset");
assert_eq!(blank.webhook_secret_token, None, "blank must read as unset");
assert_eq!(present.webhook_cert.as_deref(), Some("/x/cert.pem"));
assert_eq!(present.webhook_secret_token.as_deref(), Some("s3cret"));
}
}
+31 -1
View File
@@ -50,6 +50,7 @@ pub static CONTEXT: LazyLock<AppContext<'static>> =
pub(crate) mod test_support {
use super::*;
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost};
use crate::send::{MediaItemPayload, MediaRef};
use crate::state::EditMessage;
use std::sync::Arc;
use teloxide::{ApiError, RequestError};
@@ -64,6 +65,28 @@ pub(crate) mod test_support {
RequestError::Api(ApiError::Unknown(message.to_string()))
}
/// The API error a caption edit that changes nothing answers with — what
/// the mocks script for a permanent send failure. A `fn` pointer, so it can
/// be handed to `MockSender::scripted` as-is.
pub(crate) fn permanent_error() -> RequestError {
api_error("Bad Request: message is not modified")
}
/// One photo payload item: `media` in the two flags the tests vary (no
/// smaller variant, since that is the field most tests leave alone —
/// `send`'s own tests build that case directly).
pub(crate) fn photo_item(media: &str, has_spoiler: bool, file_id: bool) -> MediaItemPayload {
MediaItemPayload::Photo {
media: if file_id {
MediaRef::FileId(media.to_string())
} else {
MediaRef::Source(media.to_string())
},
has_spoiler,
fallback_url: None,
}
}
/// The cached post every test that touches the link cache starts from: one
/// photo with a Telegram file id at the canonical URL (key `twitter:1`).
/// Tests that need another field mutate the returned value.
@@ -108,7 +131,8 @@ pub(crate) mod test_support {
},
);
})
.await;
.await
.expect("seed prompt state");
}
pub(crate) struct TestStores {
@@ -162,6 +186,12 @@ pub(crate) mod test_support {
&self.task_queue
}
/// Path to the shared test database, for tests that need to corrupt or
/// inspect schema through a separate connection.
pub(crate) fn db_path(&self) -> &str {
self.pool.path()
}
/// Rows persisted in the task queue: what "queued for retry" looks like
/// from the outside.
pub(crate) async fn queued_tasks(&self) -> i64 {
+32 -16
View File
@@ -17,9 +17,11 @@ use std::sync::Arc;
use std::time::Duration;
/// Upper bound on pooled (reused) connections and on concurrent DB
/// operations per store. Small on purpose: the queue's `BEGIN IMMEDIATE`
/// leases serialize writes anyway, and WAL readers rarely need more.
const POOL_SIZE: usize = 4;
/// operations. Sized to cover every consumer at once — 4 queue workers +
/// 8 URL workers, plus dispatcher handlers and the sweep — so the semaphore
/// stops queueing operations behind each other; SQLite's single writer
/// serializes writes regardless, and WAL readers rarely block.
const POOL_SIZE: usize = 16;
/// A tiny connection pool for one SQLite file. Connections are checked out
/// on a blocking thread and returned afterwards; `acquire` opens a new
@@ -82,6 +84,26 @@ impl DbPool {
pub fn path(&self) -> &str {
&self.inner.path
}
/// [`with_conn`] for the many callers that answer a failed statement with
/// a default plus one log line: `what` names the operation and `level`
/// says how bad it is (`Error` when the failure loses work the caller
/// expected, `Warn` when the user is still served).
///
/// [`with_conn`]: DbPool::with_conn
pub async fn with_conn_or<T, F>(&self, level: log::Level, what: &str, default: T, f: F) -> T
where
T: Send + 'static,
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
{
match self.with_conn(f).await {
Ok(value) => value,
Err(e) => {
log::log!(level, "{what}: {e}");
default
}
}
}
}
impl PoolInner {
@@ -115,13 +137,10 @@ pub fn open_db(path: &str) -> rusqlite::Result<Connection> {
/// Opens the shared DB file, runs the merged schema for all three tables and
/// returns a pool for it. One call per process in production (the stores
/// share the returned pool); tests call it per tempdir.
/// share the returned pool); tests call it per tempdir. The file's directory
/// must exist already — [`crate::handlers::db_path`] is what creates it, and
/// it is the only caller that takes a path it did not get from a tempdir.
pub fn open_store(path: &str) -> rusqlite::Result<Arc<DbPool>> {
if let Some(parent) = std::path::Path::new(path).parent()
&& !parent.as_os_str().is_empty()
{
std::fs::create_dir_all(parent).map_err(rusqlite_error)?;
}
let conn = open_db(path)?;
schema_init(&conn)?;
migrate(&conn)?;
@@ -155,17 +174,14 @@ fn migrate(conn: &Connection) -> rusqlite::Result<()> {
if version >= target {
continue;
}
conn.execute_batch(statement)?;
// `PRAGMA` does not take bind parameters; the value is our own index.
conn.execute_batch(&format!("PRAGMA user_version = {target}"))?;
let tx = conn.unchecked_transaction()?;
tx.execute_batch(statement)?;
tx.execute_batch(&format!("PRAGMA user_version = {target}"))?;
tx.commit()?;
}
Ok(())
}
fn rusqlite_error(e: std::io::Error) -> rusqlite::Error {
rusqlite::Error::ToSqlConversionFailure(Box::new(e))
}
/// Creates the `tasks`, `chat_state` and `link_cache` tables (idempotent).
/// The three stores used to own their own schema; keeping it in one place
/// means one initialization for the whole database file.
+24 -8
View File
@@ -59,7 +59,8 @@ async fn handle_callback(
};
// Lazy expiry: a stale record (past the TTL, not yet swept) is dropped.
if edit.created_at + ttl_secs <= unix_now() {
ctx.chat_store
let _ = ctx
.chat_store
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
@@ -71,13 +72,17 @@ async fn handle_callback(
return;
}
log::info!("callback from {chat_id} on prompt {prompt_message_id}: {data}");
log::debug!(
"callback from {chat_id} on prompt {prompt_message_id}: {}",
super::log_escape(data)
);
if data == SKIP {
// Skip works with or without a forward channel: it is the explicit
// "do not forward this" answer, and it drops the record so the forward
// can never happen later.
log::info!("edit-before-forward prompt {prompt_message_id} skipped");
ctx.chat_store
let _ = ctx
.chat_store
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
@@ -102,6 +107,7 @@ async fn handle_callback(
from_chat_id: edit.chat_id,
to_chat_id: channel_id,
message_ids: edit.forward_message_ids.clone(),
forward_offset: 0,
notify_chat_id: Some(chat_id),
notify_message_id: Some(prompt_message_id),
};
@@ -147,7 +153,8 @@ async fn handle_callback(
.sender
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
.await;
ctx.chat_store
let _ = ctx
.chat_store
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
@@ -187,21 +194,29 @@ async fn handle_callback(
.await
{
super::EditOutcome::Applied => {
ctx.chat_store
let _ = ctx
.chat_store
.update(chat_id, |data| {
if let Some(entry) = data.edit_message.get_mut(&prompt_message_id) {
entry.template = name.to_string();
}
})
.await;
log::info!("template '{name}' applied to prompt {prompt_message_id}");
log::info!(
"template '{}' applied to prompt {prompt_message_id}",
super::log_escape(name)
);
}
// Nothing was applied, so nothing is recorded either: the
// prompt keeps rendering through whatever it used before, and
// the toast says why (a silently "successful" press left the
// caption unchanged).
super::EditOutcome::Failed(reason) => {
log::error!("template '{name}' could not be applied: {reason}");
log::error!(
"template '{}' could not be applied: {}",
super::log_escape(name),
super::log_escape(&reason)
);
answer = Some(format!("Could not apply the template: {reason}"));
}
}
@@ -350,7 +365,8 @@ mod tests {
seed_prompt(&ctx, "", crate::db::unix_now()).await;
ctx.chat_store
.update(1, |data| data.forward_channel_id = None)
.await;
.await
.unwrap();
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
+503 -141
View File
@@ -2,12 +2,12 @@
//! registration. URL/inline/callback flows live in their own modules.
use super::urls::{PostSend, url_media};
use super::{CHAT_STORE, CONFIG, LINK_CACHE, log_key, reply, reply_html};
use super::{log_key, reply};
use crate::ctx::AppContext;
use crate::state::ChatData;
use teloxide::RequestError;
use teloxide::prelude::*;
use teloxide::types::{ChatId, Message, Recipient};
use teloxide::types::{ChatId, Message, ParseMode, Recipient, ReplyParameters};
use teloxide::utils::command::{BotCommands, ParseError};
#[derive(BotCommands, Clone)]
@@ -74,6 +74,8 @@ fn parse_arg_remainder(s: String) -> Result<(String,), ParseError> {
/// `x_media::site::caption_from_fields` substitutes.
const FORMAT_PLACEHOLDERS: [&str; 6] = ["url", "author", "author_url", "title", "content", "tags"];
const CHAT_STATE_READ_ERROR: &str = "Couldn't read chat settings; try again.";
/// `/start`'s welcome: what the bot is for, where links work, where to look
/// next. The old "Hello!" left a first-time user with nothing.
const START_TEXT: &str = "\
@@ -104,6 +106,17 @@ Links are handled in private chats only; in a group use inline mode.";
/// Cap on template names echoed by `/settings`: a chat with hundreds of
/// templates must not produce a message Telegram rejects for length.
const MAX_SETTINGS_TEMPLATE_NAMES: usize = 30;
/// Telegram's callback data limit is 64 bytes. Reserve the `template|` prefix
/// so a name can always be carried by a prompt button.
const MAX_TEMPLATE_NAME_BYTES: usize = 64 - "template|".len();
/// Keep the persisted map bounded well below the prompt keyboard's 60-button
/// cap so every stored template remains usable in a prompt.
const MAX_TEMPLATES: usize = 50;
/// Keep the persisted template body within a caption-sized value. Validation
/// applies to the escaped body after `html_escape::encode_text` (before the
/// `[]` placeholder is substituted at apply time).
const MAX_TEMPLATE_BODY_CHARS: usize = x_media::site::MAX_CAPTION_CHARS;
const MAX_SETTINGS_CHARS: usize = 4000;
/// Sorted template names: the order `/settings`, `/remove_template` and the
/// prompt's buttons all show.
@@ -160,7 +173,7 @@ fn settings_text(data: &ChatData) -> String {
}
),
});
lines.join("\n")
cap_text(lines.join("\n"), MAX_SETTINGS_CHARS)
}
/// The first `{…}` token in a caption format that is not a known placeholder
@@ -205,14 +218,18 @@ async fn set_forward_channel_handler(
if let Some(from) = &message.from {
log::info!(
"Set forward channel for {} ({}) to {}",
from.full_name(),
super::log_escape(&from.full_name()),
message.chat.id,
channel
super::log_escape(&channel.to_string())
);
}
let chat = match bot.get_chat(channel.clone()).await {
Err(e) => {
log::error!("Failed to get channel {}: {}", channel, e);
log::warn!(
"Failed to get channel {}: {}",
super::log_escape(&channel.to_string()),
e
);
return Err(SetForwardChannelError::NotBotAdmin(e));
}
Ok(chat) => chat,
@@ -223,13 +240,17 @@ async fn set_forward_channel_handler(
let channel_id = chat.id.0;
// The sender must be a channel administrator. Compare against the
// sender's user id, NOT the chat id (they only coincide in private
// chats, so the old check broke group usage).
// chats), so the old check broke group usage).
let Some(sender) = message.from.as_ref() else {
return Err(SetForwardChannelError::NotAdmin);
};
match bot.get_chat_administrators(channel.clone()).await {
Err(e) => {
log::error!("Failed to get channel administrators {}: {}", channel, e);
log::warn!(
"Failed to get channel administrators {}: {}",
super::log_escape(&channel.to_string()),
e
);
return Err(SetForwardChannelError::NotBotAdmin(e));
}
Ok(admins) => {
@@ -253,35 +274,52 @@ async fn set_forward_channel_handler(
Ok(channel_id)
}
/// Runs one parsed command. Takes its context (stores + the sender) and the
/// `Bot`, the same shape [`crate::handlers::handle_message`] uses: `bot` is
/// only for the calls the [`MediaSender`] surface does not carry (channel
/// admin lookups, the HTML-parse-mode report).
///
/// [`MediaSender`]: crate::media_sender::MediaSender
pub(crate) async fn execute_command(
ctx: &AppContext<'_>,
bot: &Bot,
message: &Message,
command: Command,
) -> Result<(), RequestError> {
match command {
Command::Start => {
bot.send_message(message.chat.id, START_TEXT).await?;
ctx.sender
.send_message(message.chat.id, START_TEXT.to_string(), None, None)
.await?;
}
Command::Help => {
// The command list plus the parts teloxide's `descriptions()`
// cannot show: argument syntax, caption placeholders, and where a
// link actually works.
bot.send_message(
message.chat.id,
format!("{}\n\n{}", Command::descriptions(), HELP_FOOTER),
)
.await?;
ctx.sender
.send_message(
message.chat.id,
format!("{}\n\n{}", Command::descriptions(), HELP_FOOTER),
None,
None,
)
.await?;
}
Command::SetForwardChannel(channel) => {
let result = match set_forward_channel_handler(bot, message, channel).await {
Ok(channel_id) => {
CHAT_STORE
.update(message.chat.id.0, |data| {
data.forward_channel_id = Some(channel_id);
})
.await;
"Add successfully.".to_string()
}
Ok(channel_id) => match ctx
.chat_store
.update(message.chat.id.0, |data| {
data.forward_channel_id = Some(channel_id);
})
.await
{
Ok((_, true)) => "Add successfully.".to_string(),
Ok((_, false)) => {
"Forward channel set only in memory; retry later.".to_string()
}
Err(()) => CHAT_STATE_READ_ERROR.to_string(),
},
Err(SetForwardChannelError::EmptyParameter) => {
"Receive empty parameter.\nYou should enter a channel id or username"
.to_string()
@@ -299,11 +337,12 @@ pub(crate) async fn execute_command(
"Bot can't post messages to the channel".to_string()
}
};
reply(bot, message.chat.id.0, message.id, result).await?;
reply(ctx.sender, message.chat.id.0, message.id, result).await?;
}
Command::RemoveForwardChannel => {
let chat_id = message.chat.id.0;
let text = CHAT_STORE
let text = match ctx
.chat_store
.update(chat_id, |data| {
if data.forward_channel_id.is_some() {
data.forward_channel_id = None;
@@ -312,12 +351,17 @@ pub(crate) async fn execute_command(
"No channel to remove.".to_string()
}
})
.await;
reply(bot, message.chat.id.0, message.id, text).await?;
.await
{
Ok((text, _)) => text,
Err(()) => CHAT_STATE_READ_ERROR.to_string(),
};
reply(ctx.sender, message.chat.id.0, message.id, text).await?;
}
Command::EditBeforeForward => {
let chat_id = message.chat.id.0;
let text = CHAT_STORE
let text = match ctx
.chat_store
.update(chat_id, |data| {
if data.forward_channel_id.is_none() {
"Please enable forward channel first.".to_string()
@@ -330,11 +374,16 @@ pub(crate) async fn execute_command(
"Enable edit before forward.".to_string()
}
})
.await;
reply(bot, message.chat.id.0, message.id, text).await?;
.await
{
Ok((text, _)) => text,
Err(()) => CHAT_STATE_READ_ERROR.to_string(),
};
reply(ctx.sender, message.chat.id.0, message.id, text).await?;
}
Command::SetTemplate(name) => {
let chat_id = message.chat.id.0;
let name = name.trim().to_string();
let text = match message.reply_to_message() {
None => "Please reply to a message to set as template.".to_string(),
Some(reply) => {
@@ -343,27 +392,49 @@ pub(crate) async fn execute_command(
"Please reply to a message with [] to set as template.".to_string()
} else if name.is_empty() {
"Please provide a name for the template.".to_string()
} else if name.len() > MAX_TEMPLATE_NAME_BYTES {
format!("Template name is too long (max {MAX_TEMPLATE_NAME_BYTES} bytes).")
} else {
CHAT_STORE
.update(chat_id, |data| {
data.template.insert(
name,
html_escape::encode_text(reply_text).into_owned(),
);
})
.await;
"Template set.".to_string()
let template = html_escape::encode_text(reply_text);
if template.chars().count() > MAX_TEMPLATE_BODY_CHARS {
format!(
"Template is too long (max {MAX_TEMPLATE_BODY_CHARS} characters)."
)
} else {
match ctx
.chat_store
.update(chat_id, |data| {
if data.template.len() >= MAX_TEMPLATES
&& !data.template.contains_key(&name)
{
return Err(());
}
data.template.insert(name.clone(), template.into_owned());
Ok(())
})
.await
{
Ok((Ok(()), true)) => "Template set.".to_string(),
Ok((Ok(()), false)) => {
"Template set only in memory; retry later.".to_string()
}
Ok((Err(()), _)) => format!(
"This chat already has the maximum of {MAX_TEMPLATES} templates."
),
Err(()) => CHAT_STATE_READ_ERROR.to_string(),
}
}
}
}
};
reply(bot, message.chat.id.0, message.id, text).await?;
reply(ctx.sender, message.chat.id.0, message.id, text).await?;
}
Command::RemoveTemplate(name) => {
let chat_id = message.chat.id.0;
let name = name.trim().to_string();
if name.is_empty() {
reply(
bot,
ctx.sender,
chat_id,
message.id,
"Usage: /remove_template <name> (see /settings for the saved names)",
@@ -371,52 +442,42 @@ pub(crate) async fn execute_command(
.await?;
return Ok(());
}
let removed = CHAT_STORE
let text = match ctx
.chat_store
.update(chat_id, |data| data.template.remove(&name).is_some())
.await;
let text = if removed {
format!("Template '{name}' removed.")
} else {
// Name the live templates: a typo would otherwise look like a
// successful delete.
let names = sorted_template_names(&CHAT_STORE.get(chat_id).await);
if names.is_empty() {
format!("No template named '{name}'. None are saved yet.")
} else {
format!("No template named '{name}'. Saved: {}", names.join(", "))
.await
{
Ok((true, _)) => format!("Template '{name}' removed."),
Ok((false, _)) => {
let names = sorted_template_names(&ctx.chat_store.get(chat_id).await);
if names.is_empty() {
format!("No template named '{name}'. None are saved yet.")
} else {
format!("No template named '{name}'. Saved: {}", names.join(", "))
}
}
Err(()) => CHAT_STATE_READ_ERROR.to_string(),
};
reply(bot, chat_id, message.id, text).await?;
reply(ctx.sender, chat_id, message.id, text).await?;
}
Command::Settings => {
let chat_id = message.chat.id.0;
let data = CHAT_STORE.get(chat_id).await;
reply(bot, chat_id, message.id, settings_text(&data)).await?;
let data = ctx.chat_store.get(chat_id).await;
reply(ctx.sender, chat_id, message.id, settings_text(&data)).await?;
}
Command::BotDict => {
// Debug dump of the chat's persisted state: admin only (it echoes
// forward-channel ids and templates to whoever asks).
let sender_id = message
.from
.as_ref()
.map(|user| user.id.0 as i64)
.unwrap_or(-1);
if !CONFIG.admin_ids.contains(&sender_id) {
reply(bot, message.chat.id.0, message.id, "Admin only.").await?;
if require_admin(ctx, message).await?.is_none() {
return Ok(());
}
let chat_data = CHAT_STORE.get(message.chat.id.0).await;
let chat_data = ctx.chat_store.get(message.chat.id.0).await;
let debug = html_escape::encode_text(&format!("{chat_data:?}")).into_owned();
// A chat with many templates/edit records exceeds Telegram's 4096
// char message limit; the dump is plain text (no parse mode), so a
// plain byte-boundary cut is safe.
let end = debug.floor_char_boundary(MAX_DEBUG_DUMP_CHARS.min(debug.len()));
let text = if end < debug.len() {
format!("{}…", &debug[..end])
} else {
debug
};
reply(bot, message.chat.id.0, message.id, text).await?;
let text = cap_text(debug, MAX_DEBUG_DUMP_CHARS);
reply(ctx.sender, message.chat.id.0, message.id, text).await?;
}
Command::SetFormat(arg) => {
let chat_id = message.chat.id.0;
@@ -426,7 +487,7 @@ pub(crate) async fn execute_command(
}
_ => {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
"Usage: /set_format <site> <format>",
@@ -437,7 +498,7 @@ pub(crate) async fn execute_command(
};
if !x_media::site::site_ids().contains(&site) {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
"Unknown site. Use twitter, bsky, pixiv, misskey or bilibili.",
@@ -449,18 +510,18 @@ pub(crate) async fn execute_command(
// set a format once could never get back to the default (the
// built-in format string is not something a user can retype).
if format == "-" {
CHAT_STORE
let text = match ctx
.chat_store
.update(chat_id, |data| {
data.message_format.remove(site);
})
.await;
reply(
bot,
message.chat.id.0,
message.id,
"Format reset to the built-in one.",
)
.await?;
.await
{
Ok((_, true)) => "Format reset to the built-in one.".to_string(),
Ok((_, false)) => "Reset in memory only; retry later.".to_string(),
Err(()) => CHAT_STATE_READ_ERROR.to_string(),
};
reply(ctx.sender, message.chat.id.0, message.id, text).await?;
return Ok(());
}
// A typo like {titel} would otherwise be rendered literally into
@@ -468,7 +529,7 @@ pub(crate) async fn execute_command(
// exact keys), which is invisible until a post arrives.
if let Some(token) = unknown_placeholder(&format) {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
format!(
@@ -483,35 +544,31 @@ pub(crate) async fn execute_command(
.await?;
return Ok(());
}
CHAT_STORE
let text = match ctx
.chat_store
.update(chat_id, |data| {
data.message_format.insert(site.to_string(), format);
})
.await;
reply(
bot,
message.chat.id.0,
message.id,
"Format set. Use /debug <link> to preview the caption.",
)
.await?;
.await
{
Ok((_, true)) => {
"Format set. Use /debug <link> to preview the caption.".to_string()
}
Ok((_, false)) => "Format set only in memory; retry later.".to_string(),
Err(()) => CHAT_STATE_READ_ERROR.to_string(),
};
reply(ctx.sender, message.chat.id.0, message.id, text).await?;
}
Command::ClearCache(arg) => {
let sender_id = message
.from
.as_ref()
.map(|user| user.id.0 as i64)
.unwrap_or(-1);
if !CONFIG.admin_ids.contains(&sender_id) {
reply(bot, message.chat.id.0, message.id, "Admin only.").await?;
let Some(sender_id) = require_admin(ctx, message).await? else {
return Ok(());
}
};
let arg = arg.trim();
if arg.is_empty() {
let removed = LINK_CACHE.clear(None).await;
let removed = ctx.link_cache.clear(None).await;
log::info!("cache cleared by {sender_id}: {removed} entries");
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
format!("Cleared {removed} cached entr{}.", plural(removed)),
@@ -522,7 +579,7 @@ pub(crate) async fn execute_command(
Some(key) => key,
None => {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
"Unrecognized link. Use a twitter/x, pixiv, bsky, misskey or bilibili post URL.",
@@ -531,10 +588,10 @@ pub(crate) async fn execute_command(
return Ok(());
}
};
let removed = LINK_CACHE.clear(Some(&key)).await;
let removed = ctx.link_cache.clear(Some(&key)).await;
log::info!("cache entry cleared by {sender_id}: {key} ({removed} rows)");
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
format!(
@@ -550,7 +607,7 @@ pub(crate) async fn execute_command(
let url = arg.trim();
if url.is_empty() {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
"Usage: /test <post url>",
@@ -560,7 +617,7 @@ pub(crate) async fn execute_command(
}
if x_media::site::cache_key(url).is_none() {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
"No enabled site matches this link (twitter/x, pixiv, bsky, misskey or bilibili).",
@@ -568,27 +625,23 @@ pub(crate) async fn execute_command(
.await?;
return Ok(());
}
// The ordinary link pipeline with the chat's post-send actions
// suppressed: the media is sent (and cached) like a normal link,
// but nothing is forwarded to the channel and no
// edit-before-forward prompt opens. Info level echoes the
// normalized key (never the raw URL) per the logging convention.
let _command_fetch = x_media::site::acquire_command_fetch_slot().await;
log::info!("test: sending [key={}]", log_key(url));
let ctx = AppContext::from_statics(bot);
url_media(
&ctx,
ctx,
message.chat.id.0,
message.id.0 as i64,
url,
PostSend::Suppressed,
)
.await;
drop(_command_fetch);
}
Command::Debug(arg) => {
let url = arg.trim();
if url.is_empty() {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
"Usage: /debug <post url>",
@@ -596,13 +649,12 @@ pub(crate) async fn execute_command(
.await?;
return Ok(());
}
// Debug tool: report the parse result only — nothing is sent,
// cached or forwarded.
let _command_fetch = x_media::site::acquire_command_fetch_slot().await;
log::info!("debug: parsing [key={}]", log_key(url));
match x_media::site::fetch(url).await {
Ok(None) => {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
"No enabled site matches this link (twitter/x, pixiv, bsky, misskey or bilibili).",
@@ -611,7 +663,7 @@ pub(crate) async fn execute_command(
}
Err(e) => {
reply(
bot,
ctx.sender,
message.chat.id.0,
message.id,
format!("Fetch failed: {e}"),
@@ -623,23 +675,21 @@ pub(crate) async fn execute_command(
// the chat's per-site format override plus the long-post
// quoting. Rendering the raw built-in caption here made
// `/set_format` look like it did nothing.
let format = CHAT_STORE
let format = ctx
.chat_store
.get(message.chat.id.0)
.await
.message_format
.get(fetched.site_name())
.cloned()
.unwrap_or_default();
.format_for(fetched.site_id);
let caption = preview_caption(
&format,
&fetched.caption,
&fetched.source_url,
fetched.render_fields(),
CONFIG.caption_quote_text_chars,
ctx.config.caption_quote_text_chars,
);
let report = debug_report(
url,
fetched.site_name(),
fetched.site_id,
&fetched.source_url,
&fetched.title,
&fetched.content,
@@ -650,7 +700,15 @@ pub(crate) async fn execute_command(
);
// HTML report: the caption renders inside a <blockquote>
// exactly as it will appear in the sent media message.
reply_html(bot, message.chat.id.0, message.id, report).await?;
// `<Bot as Requester>::` disambiguates from the
// MediaSender trait's same-named method (see
// media_sender.rs).
<Bot as Requester>::send_message(bot, ChatId(message.chat.id.0), report)
.parse_mode(ParseMode::Html)
.reply_parameters(
ReplyParameters::new(message.id).allow_sending_without_reply(),
)
.await?;
}
}
}
@@ -658,6 +716,24 @@ pub(crate) async fn execute_command(
Ok(())
}
/// The gate the admin-only commands share: `Some(sender_id)` for an admin,
/// `None` after the refusal has been sent (the command then returns).
async fn require_admin(
ctx: &AppContext<'_>,
message: &Message,
) -> Result<Option<i64>, RequestError> {
let sender_id = message
.from
.as_ref()
.map(|user| user.id.0 as i64)
.unwrap_or(-1);
if ctx.config.admin_ids.contains(&sender_id) {
return Ok(Some(sender_id));
}
reply(ctx.sender, message.chat.id.0, message.id, "Admin only.").await?;
Ok(None)
}
/// `"y"` for one, `"ies"` for anything else — "1 entry" / "2 entries".
fn plural(n: usize) -> &'static str {
if n == 1 { "y" } else { "ies" }
@@ -700,6 +776,17 @@ const MAX_DEBUG_REPORT_CHARS: usize = 4000;
/// message, so it must stay under Telegram's 4096-char limit.
const MAX_DEBUG_DUMP_CHARS: usize = 3500;
/// Truncates `text` to at most `max` characters. The byte boundary keeps the
/// result valid UTF-8; Telegram's message limit is character-based, so this
/// remains conservative for non-ASCII text.
fn cap_text(text: String, max: usize) -> String {
if text.len() <= max {
return text;
}
let end = text.floor_char_boundary(max.saturating_sub(1));
format!("{}…", &text[..end])
}
/// The caption a link would actually send for this chat: the per-site format
/// override (empty = the site's built-in caption) and, on a long post, the
/// same text quoting the send paths apply. `/debug` shows this so the preview
@@ -776,11 +863,17 @@ fn debug_report(
lines.push(format!("sensitive: {sensitive}"));
// The caption is wrapped in a <blockquote> so the report (an HTML
// message) shows it exactly as it will render in the sent media caption
// — escaped text and links included.
lines.push(format!(
"caption: <blockquote>{}</blockquote>",
x_media::site::truncate_caption(caption)
));
// — escaped text and links included. A long post's caption already
// carries quote_long_caption's expandable blockquote and the API rejects
// nested ones (the same rule quote_long_caption applies), so that caption
// is shown unwrapped instead of failing to send.
let caption = x_media::site::truncate_caption(caption);
let caption = if caption.contains("<blockquote") {
caption
} else {
format!("<blockquote>{caption}</blockquote>")
};
lines.push(format!("caption: {caption}"));
lines.push(format!("media ({}):", media.len()));
for (i, item) in media.iter().enumerate() {
let kind = match item {
@@ -794,35 +887,183 @@ fn debug_report(
html_escape::encode_text(item.url())
));
}
let mut out = lines.join(
let out = lines.join(
"
",
);
if out.chars().count() > MAX_DEBUG_REPORT_CHARS {
let end = out.floor_char_boundary(MAX_DEBUG_REPORT_CHARS - 1);
out = format!("{}…", &out[..end]);
}
out
cap_text(out, MAX_DEBUG_REPORT_CHARS)
}
#[cfg(test)]
mod tests {
use super::{
MAX_DEBUG_REPORT_CHARS, debug_report, preview_caption, settings_text, unknown_placeholder,
Command, MAX_DEBUG_REPORT_CHARS, cap_text, debug_report, execute_command, preview_caption,
settings_text, unknown_placeholder,
};
use crate::ctx::test_support::{TestStores, api_error, cached_photo};
use crate::media_sender::test_support::{MockSender, Outcome};
use std::time::Duration;
use teloxide::Bot;
use teloxide::types::Message;
use x_media::media::Media;
/// A private message from `user_id`, as the dispatcher would hand it over.
fn message_from(user_id: i64, text: &str) -> Message {
serde_json::from_value(serde_json::json!({
"message_id": 2,
"date": 0,
"chat": { "id": 1, "type": "private" },
"from": { "id": user_id, "is_bot": false, "first_name": "u" },
"text": text,
}))
.expect("a minimal message deserializes")
}
/// The executor's wiring, which had no test while it reached for the
/// process-wide statics: each command reads and writes the chat's own
/// store and answers through the sender it was given.
#[tokio::test]
async fn the_executor_uses_the_context_it_is_given() {
let sender = MockSender::scripted(vec![Outcome::MessageOk], || api_error("boom"));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
let bot = Bot::new("42:TEST");
let message = message_from(5, "/settings");
execute_command(&ctx, &bot, &message, Command::Settings)
.await
.unwrap();
assert!(
sender.messages()[0].contains("Forward channel: not set"),
"{:?}",
sender.messages()
);
// `/set_format` writes the chat's override, the `-` form removes it
// again; neither touches another chat.
execute_command(
&ctx,
&bot,
&message,
Command::SetFormat("twitter {author}: {content}".into()),
)
.await
.unwrap();
assert_eq!(
stores.chat_store().get(1).await.format_for("twitter"),
"{author}: {content}"
);
execute_command(&ctx, &bot, &message, Command::SetFormat("twitter -".into()))
.await
.unwrap();
assert_eq!(stores.chat_store().get(1).await.format_for("twitter"), "");
// A typo'd placeholder is refused (and not stored): it would otherwise
// render literally into every caption of that site.
execute_command(
&ctx,
&bot,
&message,
Command::SetFormat("twitter {titel}".into()),
)
.await
.unwrap();
let last = sender.messages().last().unwrap().clone();
assert!(last.contains("Unknown placeholder {titel}"), "{last}");
assert_eq!(stores.chat_store().get(1).await.format_for("twitter"), "");
}
/// The admin gate: the two admin-only commands answer a refusal instead of
/// acting, and act for an admin.
#[tokio::test]
async fn the_admin_only_commands_refuse_a_non_admin() {
let sender = MockSender::scripted(vec![Outcome::MessageOk], || api_error("boom"));
let mut stores = TestStores::new();
stores.config_mut().admin_ids = vec![5];
stores.link_cache().put("twitter:1", &cached_photo()).await;
let ctx = stores.ctx(&sender);
let bot = Bot::new("42:TEST");
let outsider = message_from(9, "/clear_cache");
for command in [
Command::BotDict,
Command::ClearCache(String::new()),
Command::ClearCache("https://x.com/u/status/1".into()),
] {
execute_command(&ctx, &bot, &outsider, command)
.await
.unwrap();
}
assert_eq!(
sender.messages(),
vec!["Admin only."; 3],
"every admin-only command answers the refusal"
);
assert!(
stores
.link_cache()
.get("twitter:1", Duration::from_secs(60))
.await
.is_some(),
"a refusal must not clear the cache"
);
// The admin's `/clear_cache` does clear it, by link and wholesale.
let admin = message_from(5, "/clear_cache");
execute_command(
&ctx,
&bot,
&admin,
Command::ClearCache("https://x.com/u/status/1".into()),
)
.await
.unwrap();
assert!(
stores
.link_cache()
.get("twitter:1", Duration::from_secs(60))
.await
.is_none(),
"the admin's /clear_cache must clear the entry"
);
}
/// `/debug` answers the parse result and sends nothing: an unsupported link
/// gets the explanation the group/private paths also use.
#[tokio::test]
async fn debug_replies_without_sending_media() {
let sender = MockSender::scripted(vec![Outcome::MessageOk], || api_error("boom"));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
let bot = Bot::new("42:TEST");
let message = message_from(5, "/debug https://example.com/x");
execute_command(
&ctx,
&bot,
&message,
Command::Debug("https://example.com/x".into()),
)
.await
.unwrap();
assert_eq!(sender.calls(), vec!["send_message"]);
assert!(
sender.messages()[0].contains("No enabled site matches this link"),
"{:?}",
sender.messages()
);
}
#[test]
fn debug_report_renders_fields_and_media() {
let media = vec![
Media::Illustration {
title: None,
url: "https://cdn.example/1.jpg".into(),
thumbnail_url: None,
fallback_url: None,
},
Media::Video {
title: None,
url: "https://cdn.example/2.mp4".into(),
thumbnail_url: "https://cdn.example/2.jpg".into(),
},
@@ -910,12 +1151,48 @@ mod tests {
);
}
#[test]
fn debug_report_does_not_nest_a_quoted_caption() {
// A long post's preview_caption already carries quote_long_caption's
// <blockquote expandable>; wrapping it again produced nested
// blockquotes, which the API rejects — /debug on any long post 400'd.
let quoted = "intro <blockquote expandable>long text</blockquote>";
let report = debug_report(
"https://x.com/u/status/1",
"twitter",
"https://x.com/u/status/1",
"t",
"c",
None,
false,
quoted,
&[],
);
assert!(
report.contains(&format!("caption: {quoted}")),
"the quoted caption must be shown as-is: {report}"
);
assert_eq!(
report.matches("<blockquote").count(),
1,
"no outer wrapper may be added: {report}"
);
}
#[test]
fn cap_text_cuts_on_a_char_boundary() {
assert_eq!(cap_text("short".into(), 10), "short");
assert_eq!(cap_text("exactly".into(), 7), "exactly");
assert_eq!(cap_text("truncated".into(), 5), "trun…");
// A multi-byte character at the cut is dropped whole, not split.
assert_eq!(cap_text("aaaa漢bb".into(), 6), "aaaa…");
}
#[test]
fn debug_report_is_capped() {
// 200 media lines ≈ 8 KB, comfortably over the cap.
let media: Vec<Media> = (0..200)
.map(|i| Media::Illustration {
title: None,
url: format!("https://cdn.example/{i}.jpg"),
thumbnail_url: None,
fallback_url: None,
@@ -960,6 +1237,91 @@ mod tests {
assert!(text.contains("Templates (2): a, b"), "{text}");
}
#[test]
fn settings_text_is_capped_before_telegram_limit() {
use crate::state::ChatData;
let data = ChatData {
message_format: [("twitter", "x".repeat(4000))]
.into_iter()
.map(|(site, format)| (site.to_string(), format))
.collect(),
template: (0..super::MAX_TEMPLATES)
.map(|i| (format!("t{i}"), "[]".to_string()))
.collect(),
..ChatData::default()
};
let text = settings_text(&data);
assert!(
text.chars().count() <= super::MAX_SETTINGS_CHARS,
"{}",
text.chars().count()
);
assert!(text.ends_with('…'), "{text}");
}
#[tokio::test]
async fn template_limits_reject_unusable_names_bodies_and_overflow() {
let sender = MockSender::scripted(vec![Outcome::MessageOk; 4], || api_error("boom"));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
let bot = Bot::new("42:TEST");
let message = |reply: &str| {
serde_json::from_value::<Message>(serde_json::json!({
"message_id": 2,
"date": 0,
"chat": { "id": 1, "type": "private" },
"from": { "id": 5, "is_bot": false, "first_name": "u" },
"reply_to_message": {
"message_id": 1,
"date": 0,
"chat": { "id": 1, "type": "private" },
"text": reply,
},
"text": "/set_template x",
}))
.unwrap()
};
let short = message("before [] after");
execute_command(&ctx, &bot, &short, Command::SetTemplate("漢".repeat(22)))
.await
.unwrap();
assert!(sender.messages()[0].contains("name is too long"));
assert!(stores.chat_store().get(1).await.template.is_empty());
let long_body = message(&format!(
"{} []",
"<".repeat(super::MAX_TEMPLATE_BODY_CHARS)
));
execute_command(&ctx, &bot, &long_body, Command::SetTemplate("long".into()))
.await
.unwrap();
assert!(sender.messages()[1].contains("Template is too long"));
assert!(stores.chat_store().get(1).await.template.is_empty());
execute_command(&ctx, &bot, &short, Command::SetTemplate("ok".into()))
.await
.unwrap();
stores
.chat_store()
.update(1, |data| {
for i in 0..super::MAX_TEMPLATES - 1 {
data.template.insert(format!("t{i}"), "[]".into());
}
})
.await
.unwrap();
execute_command(&ctx, &bot, &short, Command::SetTemplate("overflow".into()))
.await
.unwrap();
assert!(sender.messages().last().unwrap().contains("maximum"));
assert_eq!(
stores.chat_store().get(1).await.template.len(),
super::MAX_TEMPLATES
);
}
#[test]
fn help_and_start_cover_what_the_command_list_cannot() {
// The placeholders the renderer substitutes must be the ones the help
+413 -147
View File
@@ -3,13 +3,16 @@
//! inline cache instead of re-fetching.
use super::log_key;
use crate::ctx::AppContext;
use crate::link_cache::{CachedMediaKind, CachedPost};
use std::collections::HashMap;
use std::sync::LazyLock;
use teloxide::RequestError;
use teloxide::prelude::*;
use teloxide::types::{
InlineQuery, InlineQueryResult, InlineQueryResultMpeg4Gif, InlineQueryResultPhoto,
InlineQueryResultVideo, ParseMode,
FileId, InlineQuery, InlineQueryResult, InlineQueryResultCachedMpeg4Gif,
InlineQueryResultCachedPhoto, InlineQueryResultCachedVideo, InlineQueryResultMpeg4Gif,
InlineQueryResultPhoto, InlineQueryResultVideo, ParseMode,
};
use x_media::media::Media;
@@ -33,54 +36,55 @@ const INLINE_STATE_TTL: std::time::Duration = std::time::Duration::from_secs(300
/// different user's query) cancel another user's pending answer.
struct InlineDebounceState {
query: String,
generation: u64,
answered: bool,
/// When a query last touched this entry, so the periodic sweep can drop
/// one per user who ever used inline mode (the map had no eviction at all,
/// unlike the rate limiter's buckets and the chat store).
last_seen: std::time::Instant,
}
#[derive(Default)]
struct DebounceStates(HashMap<u64, InlineDebounceState>);
struct DebounceStates {
entries: HashMap<u64, InlineDebounceState>,
generation: u64,
}
impl DebounceStates {
/// Records `query` as the user's newest query. Returns false when it is a
/// repeat whose answer already went out (Telegram's inline cache serves
/// it; re-fetching would only hit the source site again).
fn note(&mut self, user_id: u64, query: &str) -> bool {
if let Some(prev) = self.0.get(&user_id)
fn note(&mut self, user_id: u64, query: &str) -> (bool, u64) {
if let Some(prev) = self.entries.get(&user_id)
&& prev.query == query
&& prev.answered
{
return false;
return (false, prev.generation);
}
self.0.insert(
self.generation = self.generation.wrapping_add(1);
let generation = self.generation;
self.entries.insert(
user_id,
InlineDebounceState {
query: query.to_string(),
generation,
answered: false,
last_seen: std::time::Instant::now(),
},
);
true
(true, generation)
}
/// Drops entries no query has touched for `idle_for`. Split from the clock
/// so the boundary is testable without ageing a monotonic instant.
fn prune_idle_at(&mut self, now: std::time::Instant, idle_for: std::time::Duration) -> usize {
let before = self.0.len();
self.0
let before = self.entries.len();
self.entries
.retain(|_, state| now.saturating_duration_since(state.last_seen) < idle_for);
before - self.0.len()
before - self.entries.len()
}
/// Claims the answer for the user's newest query; false when a newer query
/// superseded it or the answer was already claimed.
fn claim(&mut self, user_id: u64, query: &str) -> bool {
let Some(state) = self.0.get_mut(&user_id) else {
fn claim(&mut self, user_id: u64, query: &str, generation: u64) -> bool {
let Some(state) = self.entries.get_mut(&user_id) else {
return false;
};
if state.query != query || state.answered {
if state.query != query || state.generation != generation || state.answered {
return false;
}
state.answered = true;
@@ -88,10 +92,10 @@ impl DebounceStates {
true
}
/// Releases a claimed-but-unsent answer so a repeat can retry the fetch.
fn release(&mut self, user_id: u64, query: &str) {
if let Some(state) = self.0.get_mut(&user_id)
fn release(&mut self, user_id: u64, query: &str, generation: u64) {
if let Some(state) = self.entries.get_mut(&user_id)
&& state.query == query
&& state.generation == generation
{
state.answered = false;
state.last_seen = std::time::Instant::now();
@@ -111,60 +115,72 @@ static INLINE_DEBOUNCE_STATE: LazyLock<parking_lot::Mutex<DebounceStates>> =
LazyLock::new(|| parking_lot::Mutex::new(DebounceStates::default()));
pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), RequestError> {
if query.query.is_empty() {
return respond(());
let ctx = AppContext::from_statics(&bot);
if query.query.is_empty() || x_media::site::cache_key(&query.query).is_none() {
return answer_inline_query(&ctx, query).await.map(|_| ());
}
// Only run a fetch for something that is actually a supported post URL.
if x_media::site::cache_key(&query.query).is_none() {
return respond(());
}
// Debounce: record the query and answer only after it has been stable for
// INLINE_DEBOUNCE (the timer below). An already-answered repeat of the
// same query is left to Telegram's inline cache instead of re-fetching.
let user_id = query.from.id.0;
if !INLINE_DEBOUNCE_STATE.lock().note(user_id, &query.query) {
let (should_answer, generation) = INLINE_DEBOUNCE_STATE.lock().note(user_id, &query.query);
if !should_answer {
return respond(());
}
let query_text = query.query.clone();
tokio::spawn(async move {
tokio::time::sleep(INLINE_DEBOUNCE).await;
// Only the user's last query of a typing burst survives: earlier
// timers see the query changed and give up without answering.
if !INLINE_DEBOUNCE_STATE.lock().claim(user_id, &query_text) {
if !INLINE_DEBOUNCE_STATE
.lock()
.claim(user_id, &query_text, generation)
{
return;
}
match answer_inline_query(bot, query).await {
let ctx = AppContext::from_statics(&bot);
match answer_inline_query(&ctx, query).await {
Ok(true) => {}
// The fetch or the answer call failed: release so a repeat of the
// same query may retry it. An *empty* answer is a real answer
// (`Ok(true)`), so a link whose media Telegram cannot fetch is not
// re-fetched on every keystroke.
Ok(false) | Err(_) => INLINE_DEBOUNCE_STATE.lock().release(user_id, &query_text),
Ok(false) | Err(_) => {
INLINE_DEBOUNCE_STATE
.lock()
.release(user_id, &query_text, generation);
}
}
});
respond(())
}
/// Fetches the post behind an inline query and answers it. The caller has
/// already applied the debounce. Returns `true` when an answer was sent.
async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, RequestError> {
/// Answers the inline query behind a post URL. The caller has already applied
/// the debounce. Returns `true` when an answer was sent.
async fn answer_inline_query(
ctx: &AppContext<'_>,
query: InlineQuery,
) -> Result<bool, RequestError> {
// The query is user input: `debug` keeps only its normalized key, the
// text itself is `trace` (same split as the message handler).
log::debug!("inline query [key={}]", log_key(&query.query));
log::trace!("inline query: {}", query.query);
log::trace!("inline query: {}", super::log_escape(&query.query));
let Some(key) = x_media::site::cache_key(&query.query) else {
answer(ctx.sender, query.id, Vec::new()).await?;
return Ok(true);
};
// A post that was already sent to some chat is answered from the link
// cache: its Telegram file ids make the answer instant, and — unlike a URL
// result, which Telegram must fetch itself — they carry media that a
// hotlink-protected host (pixiv's pximg.net) or a locally encoded file
// (ugoira MP4, bsky remux) can never serve inline. That media used to be
// skipped outright, so a pixiv link answered empty.
if let Some(cached) = ctx.link_cache.get(&key, ctx.config.link_cache_ttl).await {
let caption = inline_caption(&cached, ctx.config.caption_quote_text_chars);
let results = cached_inline_results(&cached, &caption);
answer(ctx.sender, query.id, results).await?;
return Ok(true);
}
// No retries: the debounce plus a 1s/2s backoff would outlast the inline
// query the answer belongs to.
match x_media::site::fetch_once(&query.query).await {
Ok(Some(fetched)) => {
let mut results: Vec<InlineQueryResult> = Vec::new();
// Inline results have the same 1024-char caption limit as regular
// messages; truncate once here for all items, then apply the same
// long-post quoting as the send paths. `answer_inline_query` has no
// `AppContext` (the debounce spawns it), so the parsed config comes
// from the process-wide static, and the text is the *escaped*
// title/content the built-in caption embeds (the raw
// `Fetched.title`/`content` differ whenever the post contains
// `<`/`&`).
// long-post quoting as the send paths. The built-in caption is what
// an inline answer can use: there is no chat whose per-site format
// could apply, so the render fields come from the fetch itself.
let caption = x_media::site::truncate_caption(&fetched.caption);
let text = fetched
.render_fields()
@@ -173,17 +189,17 @@ async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, Reque
let caption = crate::send::quote_long_caption(
&caption,
&text,
super::CONFIG.caption_quote_text_chars,
ctx.config.caption_quote_text_chars,
);
let mut results: Vec<InlineQueryResult> = Vec::new();
for (i, media) in fetched.media.iter().enumerate() {
let id = format!("{i}");
// Telegram fetches an inline result's URL itself and cannot
// send site-specific headers, so hotlink-protected media
// (pixiv's pximg.net) would render as a broken file there.
// Locally produced media (ugoira MP4, bsky remux) is a local
// path and does not parse as a URL at all — same skip.
if x_media::site::needs_media_headers(media.url()) {
log::debug!("inline: skipping hotlink-protected media {id}");
log::debug!("inline: skipping hotlink-protected media {i}");
continue;
}
let Some(url) = url::Url::parse(media.url()).ok() else {
@@ -193,132 +209,382 @@ async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, Reque
.thumbnail_url()
.and_then(|t| url::Url::parse(t).ok())
.unwrap_or_else(|| url.clone());
let caption = caption.clone().into_owned();
let result = match media {
Media::Illustration { .. } => {
// Inline photo results have their own (smaller) size
// cap; use the reduced variant when one exists.
let photo_url = media
.smaller_url()
.and_then(|u| url::Url::parse(u).ok())
.unwrap_or_else(|| url.clone());
InlineQueryResult::Photo(
InlineQueryResultPhoto::new(id, photo_url, thumbnail)
.caption(caption)
.parse_mode(ParseMode::Html),
)
}
Media::Video { .. } => InlineQueryResult::Video(
InlineQueryResultVideo::new(
id,
url,
"video/mp4".parse().expect("valid mime"),
thumbnail,
fetched.title.clone(),
)
.caption(caption)
.parse_mode(ParseMode::Html),
),
Media::Animated { .. } => InlineQueryResult::Mpeg4Gif(
InlineQueryResultMpeg4Gif::new(id, url, thumbnail)
.caption(caption)
.parse_mode(ParseMode::Html),
),
};
results.push(result);
// Inline photo results have their own (smaller) size cap; use
// the reduced variant when one exists.
let url = media
.smaller_url()
.and_then(|u| url::Url::parse(u).ok())
.unwrap_or(url);
results.push(url_result(
i.to_string(),
match media {
Media::Illustration { .. } => CachedMediaKind::Photo,
Media::Video { .. } => CachedMediaKind::Video,
Media::Animated { .. } => CachedMediaKind::Animation,
},
url,
thumbnail,
fetched.title.clone(),
caption.clone().into_owned(),
));
}
if !results.is_empty() {
// Explicit cache window: repeats of the same query within 5
// minutes are served by Telegram without hitting the bot.
bot.answer_inline_query(query.id, results)
.cache_time(300)
.await?;
return Ok(true);
answer(ctx.sender, query.id, results).await?;
Ok(true)
}
Ok(None) | Err(_) => {
if let Err(e) = answer(ctx.sender, query.id, Vec::new()).await {
log::error!(
"inline empty answer failed for [key={}]: {e}",
log_key(&query.query)
);
return Err(e);
}
// Every item was skipped: Telegram fetches an inline result's URL
// itself, so pixiv's hotlink-protected media (and a local ugoira /
// bsky MP4) can never be one. Answer *empty* — the client stops
// spinning, and the same query is not re-fetched on every
// keystroke: an unanswered query releases the debounce below
// (`Ok(false)`), which is what made this re-run the fetch each
// time, and the window lets Telegram serve the repeats itself.
log::debug!("inline: nothing Telegram can fetch for the query; answering empty");
bot.answer_inline_query(query.id, Vec::new())
.cache_time(300)
.await?;
return Ok(true);
Ok(true)
}
Ok(None) => {}
Err(e) => log::error!("inline fetch [key={}]: {e}", log_key(&query.query)),
}
Ok(false)
}
/// The caption of an inline answer, from a cached post: the caption that was
/// sent (the site's built-in one, truncated) plus the long-post quoting the
/// send paths apply.
fn inline_caption(cached: &CachedPost, quote_chars: usize) -> String {
let text = x_media::site::compose_text(&cached.title, &cached.content);
crate::send::quote_long_caption(
&x_media::site::truncate_caption(&cached.caption),
&text,
quote_chars,
)
.into_owned()
}
/// One inline result pointing Telegram at a URL it fetches itself.
fn url_result(
id: String,
kind: CachedMediaKind,
url: url::Url,
thumbnail: url::Url,
title: String,
caption: String,
) -> InlineQueryResult {
let parse_mode = ParseMode::Html;
match kind {
CachedMediaKind::Photo => InlineQueryResult::Photo(
InlineQueryResultPhoto::new(id, url, thumbnail)
.caption(caption)
.parse_mode(parse_mode),
),
CachedMediaKind::Video => InlineQueryResult::Video(
InlineQueryResultVideo::new(
id,
url,
"video/mp4".parse().expect("valid mime"),
thumbnail,
title,
)
.caption(caption)
.parse_mode(parse_mode),
),
CachedMediaKind::Animation => InlineQueryResult::Mpeg4Gif(
InlineQueryResultMpeg4Gif::new(id, url, thumbnail)
.caption(caption)
.parse_mode(parse_mode),
),
}
}
/// One inline result served from a Telegram file id.
fn cached_result(
id: String,
kind: CachedMediaKind,
file_id: String,
title: String,
caption: String,
) -> InlineQueryResult {
let parse_mode = ParseMode::Html;
let file_id = FileId(file_id);
match kind {
CachedMediaKind::Photo => InlineQueryResult::CachedPhoto(
InlineQueryResultCachedPhoto::new(id, file_id)
.caption(caption)
.parse_mode(parse_mode),
),
CachedMediaKind::Video => InlineQueryResult::CachedVideo(
InlineQueryResultCachedVideo::new(id, file_id, title)
.caption(caption)
.parse_mode(parse_mode),
),
CachedMediaKind::Animation => InlineQueryResult::CachedMpeg4Gif(
InlineQueryResultCachedMpeg4Gif::new(id, file_id)
.caption(caption)
.parse_mode(parse_mode),
),
}
}
/// The inline results a cached post answers with, one per media item: from the
/// file id when the entry has one, else from the source URL (a degraded entry
/// keeps only URLs). A URL item that needs site headers is skipped as in the
/// fetch path; a *file id* needs no headers, which is what makes a pixiv post
/// answerable inline.
fn cached_inline_results(cached: &CachedPost, caption: &str) -> Vec<InlineQueryResult> {
cached
.media
.iter()
.enumerate()
.filter_map(|(i, media)| {
let id = i.to_string();
let caption = || caption.to_string();
if !media.file_id.is_empty() {
return Some(cached_result(
id,
media.kind,
media.file_id.clone(),
cached.title.clone(),
caption(),
));
}
if x_media::site::needs_media_headers(&media.url) {
log::debug!("inline: skipping hotlink-protected cached media {i}");
return None;
}
let url = url::Url::parse(&media.url).ok()?;
if matches!(media.kind, CachedMediaKind::Video) {
log::debug!("inline: skipping a cached video with no thumbnail {i}");
return None;
}
Some(url_result(
id,
media.kind,
url.clone(),
url,
cached.title.clone(),
caption(),
))
})
.collect()
}
/// Answers with `results` (an empty vec is a real answer: it stops the client
/// spinning and lets Telegram serve repeats itself) under the cache window
/// [`INLINE_STATE_TTL`] mirrors.
async fn answer(
sender: &dyn crate::media_sender::MediaSender,
id: teloxide::types::InlineQueryId,
mut results: Vec<InlineQueryResult>,
) -> Result<(), RequestError> {
results.truncate(50);
if results.is_empty() {
log::debug!("inline: nothing Telegram can serve for the query; answering empty");
}
sender.answer_inline_query(id, results, 300).await
}
#[cfg(test)]
mod tests {
use super::{DebounceStates, INLINE_STATE_TTL};
use super::{DebounceStates, INLINE_STATE_TTL, answer_inline_query, cached_inline_results};
use crate::ctx::test_support::{TestStores, api_error, cached_photo};
use crate::link_cache::{CachedMedia, CachedMediaKind};
use crate::media_sender::test_support::MockSender;
use teloxide::types::InlineQuery;
const URL_A: &str = "https://x.com/a/status/1";
const URL_B: &str = "https://x.com/b/status/2";
fn inline_query(url: &str) -> InlineQuery {
serde_json::from_value(serde_json::json!({
"id": "42",
"from": { "id": 5, "is_bot": false, "first_name": "u" },
"query": url,
"offset": "",
}))
.expect("a minimal inline query deserializes")
}
/// A post already in the link cache is answered from its file ids: no
/// fetch, and — unlike a URL result — media Telegram could never fetch
/// itself (a pixiv pximg URL) can be served.
#[tokio::test]
async fn a_cached_post_answers_from_its_file_ids() {
let sender = MockSender::scripted(vec![], || api_error("boom"));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
let mut entry = cached_photo();
entry.media = vec![
CachedMedia {
kind: CachedMediaKind::Photo,
file_id: "AgAC-photo".into(),
url: "https://i.pximg.net/img-original/img/1.jpg".into(),
},
CachedMedia {
kind: CachedMediaKind::Animation,
file_id: "AgAC-gif".into(),
url: "https://i.pximg.net/img-original/img/1.gif".into(),
},
];
stores.link_cache().put("twitter:1", &entry).await;
let answered = answer_inline_query(&ctx, inline_query("https://x.com/u/status/1"))
.await
.unwrap();
assert!(answered);
assert_eq!(
sender.inline_answers(),
vec![vec!["cached_photo:AgAC-photo", "cached_gif:AgAC-gif"]],
"every item goes out as its cached file id, hotlink protection and all"
);
}
/// A degraded entry has no file ids left, so its URLs are used — and an
/// item Telegram must not fetch (needs site headers) or cannot render (a
/// video with no poster) is skipped. Nothing left means an *empty* answer:
/// leaving the query unanswered makes the client spin and re-fetch on every
/// keystroke.
#[tokio::test]
async fn a_degraded_cached_post_answers_with_urls_or_empty() {
let sender = MockSender::scripted(vec![], || api_error("boom"));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
let mut entry = cached_photo();
entry.media = vec![
CachedMedia {
kind: CachedMediaKind::Photo,
file_id: String::new(),
url: "https://p/1.jpg".into(),
},
CachedMedia {
kind: CachedMediaKind::Video,
file_id: String::new(),
url: "https://v/1.mp4".into(),
},
];
stores.link_cache().put("twitter:1", &entry).await;
answer_inline_query(&ctx, inline_query("https://x.com/u/status/1"))
.await
.unwrap();
assert_eq!(
sender.inline_answers(),
vec![vec!["photo:https://p/1.jpg"]],
"the degradable photo goes out by URL, the poster-less video is skipped"
);
// Nothing servable: a pixiv original needs a Referer Telegram does not
// send.
stores.link_cache().remove("twitter:1").await;
let mut entry = cached_photo();
entry.media = vec![CachedMedia {
kind: CachedMediaKind::Photo,
file_id: String::new(),
url: "https://i.pximg.net/img-original/img/1.jpg".into(),
}];
stores.link_cache().put("twitter:1", &entry).await;
answer_inline_query(&ctx, inline_query("https://x.com/u/status/1"))
.await
.unwrap();
assert_eq!(
sender.inline_answers(),
vec![vec!["photo:https://p/1.jpg".to_string()], Vec::new()],
"a query with nothing servable is still answered, with no results"
);
}
#[tokio::test]
async fn unsupported_inline_query_answers_empty() {
let sender = MockSender::scripted(vec![], || api_error("boom"));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
let answered = answer_inline_query(&ctx, inline_query("not a supported post"))
.await
.unwrap();
assert!(answered);
assert_eq!(sender.inline_answers(), vec![Vec::<String>::new()]);
}
#[tokio::test]
async fn answer_caps_cached_results_at_telegram_limit() {
let sender = MockSender::scripted(vec![], || api_error("boom"));
let mut entry = cached_photo();
entry.media = (0..51)
.map(|i| CachedMedia {
kind: CachedMediaKind::Photo,
file_id: format!("id-{i}"),
url: format!("https://p/{i}.jpg"),
})
.collect();
let results = cached_inline_results(&entry, "caption");
assert!(results.len() > 50, "the builder itself may keep all items");
super::answer(
&sender,
inline_query("https://x.com/u/status/1").id,
results,
)
.await
.unwrap();
assert_eq!(sender.inline_answers()[0].len(), 50);
}
#[test]
fn debounce_state_is_per_user() {
let mut states = DebounceStates::default();
// Two users query different links: both proceed, and neither timer
// cancels the other (a single shared slot dropped one of them).
assert!(states.note(1, URL_A));
assert!(states.note(2, URL_B));
assert!(states.claim(1, URL_A), "user 1's answer was cancelled");
assert!(states.claim(2, URL_B), "user 2's answer was cancelled");
assert!(states.note(1, URL_A).0);
assert!(states.note(2, URL_B).0);
let (_, generation_a) = states.note(1, URL_A);
let (_, generation_b) = states.note(2, URL_B);
assert!(states.claim(1, URL_A, generation_a));
assert!(states.claim(2, URL_B, generation_b));
}
#[test]
fn answered_query_is_suppressed_per_user_only() {
let mut states = DebounceStates::default();
assert!(states.note(1, URL_A));
assert!(states.claim(1, URL_A));
// A repeat of the answered query by the same user is left to
// Telegram's inline cache.
assert!(!states.note(1, URL_A));
// Another user pasting the same link still gets an answer.
assert!(states.note(2, URL_A));
assert!(states.claim(2, URL_A));
assert!(states.note(1, URL_A).0);
let (_, generation) = states.note(1, URL_A);
assert!(states.claim(1, URL_A, generation));
assert!(!states.note(1, URL_A).0);
assert!(states.note(2, URL_A).0);
let (_, generation) = states.note(2, URL_A);
assert!(states.claim(2, URL_A, generation));
}
#[test]
fn idle_states_are_pruned_and_live_ones_kept() {
let mut states = DebounceStates::default();
assert!(states.note(1, URL_A));
let first = states.0[&1].last_seen;
// Entry 2 is strictly newer, so one timestamp can sit exactly on the
// window's edge for one and comfortably inside it for the other.
assert!(states.note(1, URL_A).0);
let first = states.entries[&1].last_seen;
std::thread::sleep(std::time::Duration::from_millis(2));
assert!(states.note(2, URL_B));
assert!(states.note(2, URL_B).0);
assert_eq!(
states.prune_idle_at(first + INLINE_STATE_TTL, INLINE_STATE_TTL),
1
);
assert!(
!states.0.contains_key(&1),
"the entry past the window must go"
);
assert!(states.0.contains_key(&2), "the live entry must stay");
// A pruned user's repeat is answered fresh instead of suppressed.
assert!(states.note(1, URL_A));
assert!(!states.entries.contains_key(&1));
assert!(states.entries.contains_key(&2));
assert!(states.note(1, URL_A).0);
}
#[test]
fn stale_same_query_generation_cannot_claim_after_a_b_a() {
let mut states = DebounceStates::default();
assert!(states.note(1, URL_A).0);
assert!(states.note(1, URL_B).0);
let (_, stale_generation) = states.note(1, URL_A);
let (_, current_generation) = states.note(1, URL_A);
assert_ne!(stale_generation, current_generation);
assert!(!states.claim(1, URL_A, stale_generation));
assert!(states.claim(1, URL_A, current_generation));
}
#[test]
fn newer_query_supersedes_and_failed_answer_is_released() {
let mut states = DebounceStates::default();
assert!(states.note(1, URL_A));
assert!(states.note(1, URL_B));
// The stale timer for the half-typed query gives up…
assert!(!states.claim(1, URL_A));
// …and the newest one answers.
assert!(states.claim(1, URL_B));
// No results → release so a repeat may retry the fetch.
states.release(1, URL_B);
assert!(states.claim(1, URL_B));
assert!(states.note(1, URL_A).0);
let (_, generation_a) = states.note(1, URL_B);
assert!(!states.claim(1, URL_A, generation_a));
let (_, generation_b) = states.note(1, URL_B);
assert!(states.claim(1, URL_B, generation_b));
states.release(1, URL_B, generation_b);
assert!(states.claim(1, URL_B, generation_b));
}
}
+121 -75
View File
@@ -9,59 +9,43 @@
mod callback;
mod commands;
mod inline;
mod repair;
mod statics;
mod url_workers;
mod urls;
pub use callback::callback_query_handler;
pub use commands::register_commands;
pub use inline::inline_query_handler;
pub(crate) use inline::prune_idle_states;
pub(crate) use repair::repair_lost_local_media;
/// The resolved `$DATA_DIR/task_queue.db` path, for the startup config line.
pub(crate) use statics::db_path;
pub use statics::{CHAT_STORE, CONFIG, LINK_CACHE, TASK_QUEUE};
pub(crate) use urls::repair_lost_local_media;
pub use urls::{start_url_workers, stop_url_workers};
pub use url_workers::{start_url_workers, stop_url_workers};
use crate::ctx::AppContext;
use crate::media_sender::MediaSender;
use commands::{Command, execute_command};
use teloxide::RequestError;
use teloxide::prelude::*;
use teloxide::types::{
ChatId, ChatKind, Message, MessageId, ParseMode, PublicChatKind, ReplyParameters,
};
use teloxide::types::{ChatId, Message, MessageId};
use teloxide::utils::command::BotCommands;
use urls::{URL_JOBS, extract_urls};
use url_workers::URL_JOBS;
use urls::extract_urls;
/// Reply to a message by id, keeping the reply decoration even if the
/// original was already deleted. Returns the reply's message id.
/// original was already deleted.
pub(crate) async fn reply(
sender: &dyn MediaSender,
chat_id: i64,
reply_to: MessageId,
text: impl Into<String>,
) -> Result<i64, RequestError> {
) -> Result<(), RequestError> {
sender
.send_message(ChatId(chat_id), text.into(), Some(reply_to), None)
.await
}
/// Reply to a message by id with HTML parse mode (same reply decoration as
/// [`reply`]). Used by `/test`, whose report is an HTML message (the caption
/// is wrapped in a `<blockquote>` to show it exactly as it will render).
pub(crate) async fn reply_html(
bot: &Bot,
chat_id: i64,
reply_to: MessageId,
text: String,
) -> Result<i64, RequestError> {
// `<Bot as Requester>::` disambiguates from the MediaSender trait's
// same-named method (see media_sender.rs).
<Bot as Requester>::send_message(bot, ChatId(chat_id), text)
.parse_mode(ParseMode::Html)
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply())
.await
.map(|message| message.id.0 as i64)
.map(|_| ())
}
/// Log prefix tying the whole lifecycle of one link (fetch → send → cache →
@@ -72,6 +56,26 @@ pub fn log_key(url: &str) -> String {
x_media::site::cache_key(url).unwrap_or_else(|| "<unsupported>".to_string())
}
/// Makes user-supplied text (a display name, callback data, a channel handle)
/// fit one log line: newlines and other control characters are escaped, so a
/// crafted value cannot forge a second log entry or hide inside one. Tab is
/// kept — it cannot break the line.
pub(crate) fn log_escape(s: &str) -> std::borrow::Cow<'_, str> {
if !s.chars().any(|c| c.is_control() && c != '\t') {
return std::borrow::Cow::Borrowed(s);
}
let mut out = String::with_capacity(s.len());
for c in s.chars() {
match c {
'\n' => out.push_str("\\n"),
'\r' => out.push_str("\\r"),
c if c != '\t' && c.is_control() => out.push_str(&format!("\\u{:04x}", c as u32)),
c => out.push(c),
}
}
std::borrow::Cow::Owned(out)
}
/// How long a caption edit may sleep before it gives up on retrying: the reply
/// (or button press) that carried the text is already consumed, so the update
/// must not stall the chat's queue behind a long flood-control wait — the user
@@ -137,6 +141,18 @@ async fn edit_message_handler(
let Some(edit) = chat_data.edit_message.get(&reply_to_message_id) else {
return false;
};
// Lazy expiry, the same rule a button press gets: a record past the TTL
// (not yet swept) is dropped and the reply falls through to the normal
// message flow instead of rewriting a caption from a dead prompt.
if edit.created_at + ctx.config.edit_message_ttl.as_secs() as i64 <= crate::db::unix_now() {
let _ = ctx
.chat_store
.update(chat_id, |data| {
data.edit_message.remove(&reply_to_message_id);
})
.await;
return false;
}
let Some(first_forward_id) = edit.forward_message_ids.first() else {
return false;
};
@@ -196,7 +212,7 @@ pub(crate) async fn handle_message(
bot: &Bot,
message: Message,
) -> Result<(), RequestError> {
let is_private = matches!(message.chat.kind, ChatKind::Private(_));
let is_private = message.chat.is_private();
let sender = message
.from
.as_ref()
@@ -206,14 +222,15 @@ pub(crate) async fn handle_message(
.text()
.map(|t| {
let end = t.floor_char_boundary(120.min(t.len()));
&t[..end]
log_escape(&t[..end])
})
.unwrap_or("<no text>");
.unwrap_or_else(|| std::borrow::Cow::Borrowed("<no text>"));
// Per-request detail: who and where at `debug`; the message text itself is
// user data and only ever appears at `trace`, so a `debug` log can be
// shared without leaking what people pasted.
log::debug!(
"message from {sender} in {} (private={is_private})",
"message from {} in {} (private={is_private})",
log_escape(&sender),
message.chat.id
);
log::trace!("message text: {text_preview}");
@@ -233,10 +250,10 @@ pub(crate) async fn handle_message(
log::debug!(
"command from {}: {}",
message.chat.id,
text.split_whitespace().next().unwrap_or("<empty>")
log_escape(text.split_whitespace().next().unwrap_or("<empty>"))
);
log::trace!("command text: {text_preview}");
execute_command(bot, &message, command).await?;
execute_command(ctx, bot, &message, command).await?;
return respond(());
}
if is_private {
@@ -267,7 +284,7 @@ pub(crate) async fn handle_message(
break;
}
}
} else if is_group(&message.chat.kind)
} else if (message.chat.is_group() || message.chat.is_supergroup())
&& extract_urls(&message)
.iter()
.any(|url| x_media::site::cache_key(url).is_some())
@@ -287,18 +304,6 @@ pub(crate) async fn handle_message(
const GROUP_LINK_HINT: &str =
"Links are handled in private chat only — send me this link there, or use inline mode here.";
/// Groups and supergroups, as opposed to private chats and channels.
fn is_group(kind: &ChatKind) -> bool {
matches!(
kind,
ChatKind::Public(chat)
if matches!(
chat.kind,
PublicChatKind::Group | PublicChatKind::Supergroup(_)
)
)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -310,6 +315,42 @@ mod tests {
/// edit (the prompt was deleted).
const API_ERROR: &str = "Bad Request: message not found";
#[test]
fn log_escape_cannot_forge_a_second_log_line() {
let forged = log_escape("alice\nINFO injected entry");
assert!(!forged.contains('\n'), "no raw newline may survive");
assert!(
forged.contains("\\n"),
"the break stays visible as an escape"
);
// The common case (clean input) borrows — logging must not allocate.
assert!(matches!(
log_escape("plain text"),
std::borrow::Cow::Borrowed(_)
));
}
#[tokio::test]
async fn a_reply_to_an_expired_prompt_is_not_edited() {
// 90 000 s ago: past the TTL under any config a test can hold.
let sender = MockSender::scripted(vec![], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "tpl", crate::db::unix_now() - 90_000).await;
let consumed = edit_message_handler(&ctx, 1, PROMPT_ID, "new caption").await;
assert!(!consumed, "an expired prompt must not consume the reply");
assert!(
sender.captions().is_empty(),
"no caption edit may reach a dead prompt"
);
assert!(
stores.chat_store().get(1).await.edit_message.is_empty(),
"the stale record must be dropped for good"
);
}
#[tokio::test]
async fn reply_to_a_prompt_swaps_the_caption_through_its_template() {
let sender = MockSender::scripted(vec![Outcome::EditOk], || api_error(API_ERROR));
@@ -464,38 +505,43 @@ mod tests {
assert_eq!(api.methods(), vec!["EditMessageCaption", "SendMessage"]);
assert_eq!(api.body("SendMessage")["text"], GROUP_LINK_HINT);
}
#[test]
fn the_link_hint_is_for_groups_only() {
use teloxide::types::{ChatPrivate, ChatPublic, PublicChatChannel, PublicChatSupergroup};
// A channel stays silent: the hint reply would be posted into the
// channel itself, so the same link must produce no further call.
let channel: Message = serde_json::from_value(serde_json::json!({
"message_id": 3,
"date": 0,
"chat": { "id": -1001234567890i64, "type": "channel", "title": "c" },
"text": "https://x.com/u/status/1",
"entities": [{ "type": "url", "offset": 0, "length": 24 }],
}))
.expect("a minimal channel message deserializes");
let group = ChatKind::Public(ChatPublic {
title: None,
kind: PublicChatKind::Group,
});
let supergroup = ChatKind::Public(ChatPublic {
title: None,
kind: PublicChatKind::Supergroup(PublicChatSupergroup {
username: None,
is_forum: false,
}),
});
// A channel must stay silent: the hint reply would be posted into the
// channel itself.
let channel = ChatKind::Public(ChatPublic {
title: None,
kind: PublicChatKind::Channel(PublicChatChannel { username: None }),
});
let private = ChatKind::Private(ChatPrivate {
username: None,
first_name: None,
last_name: None,
});
handle_message(&ctx, &bot, channel).await.unwrap();
assert!(is_group(&group));
assert!(is_group(&supergroup));
assert!(!is_group(&channel));
assert!(!is_group(&private));
assert_eq!(
api.methods(),
vec!["EditMessageCaption", "SendMessage"],
"a channel must not get the group hint"
);
// And an *unsupported* link in a group stays silent too: the hint is
// for links a site adapter claims (the branch's own filter).
let unsupported: Message = serde_json::from_value(serde_json::json!({
"message_id": 4,
"date": 0,
"chat": { "id": -100, "type": "group", "title": "g" },
"text": "https://example.com/x",
"entities": [{ "type": "url", "offset": 0, "length": 19 }],
}))
.expect("a minimal group message deserializes");
handle_message(&ctx, &bot, unsupported).await.unwrap();
assert_eq!(
api.methods(),
vec!["EditMessageCaption", "SendMessage"],
"an unsupported link must not get the hint"
);
}
}
+376
View File
@@ -0,0 +1,376 @@
//! Startup repair, run before any queue worker exists: a queued retry whose
//! local media (a ugoira MP4, a bsky remux, a downloaded temp file) did not
//! survive the restart can never succeed, because the registry that kept those
//! files alive (`send::KEEP_ALIVE`) is in memory. Those rows are re-fetched
//! from their post instead of dead-lettering the user's link.
use super::log_key;
use super::urls::{cached_snapshot, media_to_payload};
use crate::ctx::AppContext;
use crate::link_cache::CachedPost;
use crate::send::{self, Delivery, MediaItemPayload, Task};
// ── Startup repair: queued retries whose local media did not survive ───────
/// A post's fresh media plus the caption and cache snapshot that go with them:
/// what [`refetch`] hands [`apply_refresh`]. Plain data, so the rewrite below
/// can be tested without a network fetch (which cannot be faked here:
/// [`x_media::site::Fetched`] keeps a private field and is not constructible
/// outside its crate).
struct Refetched {
caption: String,
items: Vec<MediaItemPayload>,
cache_data: Option<CachedPost>,
keep_alive: Option<std::sync::Arc<tempfile::TempDir>>,
}
/// Whether a queued task should have its post re-fetched, because it still
/// wants a local file (ugoira MP4, a bsky remux, a downloaded temp file) that is
/// gone. Those files live in the system temp dir and the registry that keeps
/// them alive for the retry (`send::KEEP_ALIVE`) is in memory, so a restart
/// takes all of them — a retry that needs one can only dead-letter.
///
/// A partially delivered album is left alone: its remaining batches cannot be
/// reconciled with a fresh media list without risking a second copy of what the
/// user already received.
fn needs_refetch(task: &Task) -> bool {
if let Task::SendMediaSequence {
batch_index,
sent_message_ids,
..
} = task
&& (*batch_index > 0 || !sent_message_ids.is_empty())
{
return false;
}
task.local_media_paths().iter().any(|path| !path.exists())
}
/// Rebuilds the task from the fresh media, keeping its delivery envelope (chat,
/// reply, forward/edit settings, notify targets): the retry that was queued must
/// still deliver the same way, whoever asked for it.
fn apply_refresh(task: &Task, fresh: &Refetched) -> Option<Task> {
let chat_id = task.chat_id()?;
let (edit_before_forward, forward_channel_id) = match task {
Task::SendMediaSequence {
edit_before_forward,
forward_channel_id,
..
}
| Task::SendAnimation {
edit_before_forward,
forward_channel_id,
..
} => (*edit_before_forward, *forward_channel_id),
Task::ForwardMessages { .. } => return None,
};
let reply_to_message_id = match task {
Task::SendMediaSequence {
reply_to_message_id,
..
}
| Task::SendAnimation {
reply_to_message_id,
..
} => *reply_to_message_id,
Task::ForwardMessages { .. } => return None,
};
let (notify_chat_id, notify_message_id) = task.notify_target();
Some(Task::from_items(
Delivery {
chat_id,
reply_to_message_id,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
},
task.source_url()?.to_string(),
fresh.caption.clone(),
fresh.items.clone(),
fresh.cache_data.clone(),
))
}
/// Fetches the post again and maps it into [`Refetched`]: the same mapping the
/// fresh-fetch path uses (per-site caption format from the chat, render fields
/// for the link-cache snapshot), so a repaired task looks like a first send.
async fn refetch(
ctx: &AppContext<'_>,
chat_id: i64,
url: &str,
) -> Result<Option<Refetched>, x_media::site::FetchError> {
let Some(fetched) = x_media::site::fetch(url).await? else {
return Ok(None);
};
if fetched.media.is_empty() {
return Ok(None);
}
let chat_data = ctx.chat_store.get(chat_id).await;
let format = chat_data.format_for(fetched.site_id);
let caption = fetched.caption_with(&format);
let cache_data = cached_snapshot(&fetched);
let items: Vec<MediaItemPayload> = fetched
.media
.iter()
.filter_map(|media| media_to_payload(media, fetched.sensitive))
.collect();
if items.is_empty() {
return Ok(None);
}
Ok(Some(Refetched {
caption,
items,
cache_data,
keep_alive: fetched.keep_alive(),
}))
}
/// Re-fetches every queued task whose local media did not survive the restart.
/// This runs at startup before queue workers exist, so any SQLite error is
/// returned to the caller and prevents workers from starting on unrepaired
/// state.
pub(crate) async fn repair_lost_local_media(
ctx: &AppContext<'_>,
) -> Result<usize, rusqlite::Error> {
let rows = ctx.task_queue.runnable_rows().await?;
let mut repaired = 0;
for (id, payload) in rows {
let Ok(task) = serde_json::from_str::<Task>(&payload) else {
continue;
};
if !needs_refetch(&task) {
continue;
}
let (Some(url), Some(chat_id)) = (task.source_url().map(str::to_string), task.chat_id())
else {
continue;
};
match refetch(ctx, chat_id, &url).await {
Ok(Some(fresh)) => {
let Some(updated) = apply_refresh(&task, &fresh) else {
continue;
};
let updated = serde_json::to_value(&updated).expect("task serializes");
match ctx.task_queue.replace_payload(&id, &updated).await {
Ok(true) => {
if let Some(dir) = fresh.keep_alive {
send::KEEP_ALIVE.lock().push(dir);
}
repaired += 1;
log::info!(
"startup repair: re-fetched [key={}] for chat={chat_id}",
log_key(&url)
);
}
Ok(false) => {
log::warn!("startup repair: queue row {id} disappeared before rewrite")
}
Err(e) => {
log::error!("startup repair: queue row {id} rewrite failed: {e}");
return Err(e);
}
}
}
Ok(None) | Err(_) => {
let (notify_chat_id, notify_message_id) = task.notify_target();
log::warn!(
"startup repair: [key={}] for chat={chat_id} needed a re-fetch and none was possible",
log_key(&url)
);
send::notify_failure(
ctx.sender,
notify_chat_id,
notify_message_id,
&format!(
"{} — the media held for retry was lost when the bot restarted and the post could not be fetched again. Please send the link again.",
log_key(&url)
),
)
.await;
}
}
}
Ok(repaired)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ctx::test_support::{TestStores, permanent_error, photo_item};
use crate::media_sender::test_support::MockSender;
use crate::send::MediaRef;
fn queued_task(media: &str, batch_index: usize, sent: Vec<i64>) -> Task {
Task::SendMediaSequence {
chat_id: 1,
reply_to_message_id: 2,
caption: "cap".into(),
media_batches: vec![vec![photo_item(media, false, false)]],
batch_index,
sent_message_ids: sent,
source_url: "https://x.com/u/status/1".into(),
edit_before_forward: true,
forward_channel_id: Some(2),
notify_chat_id: Some(1),
notify_message_id: Some(2),
cache_data: None,
}
}
#[test]
fn only_tasks_missing_a_local_file_need_a_refetch() {
// A URL send needs nothing.
assert!(!needs_refetch(&queued_task("https://cdn/1.jpg", 0, vec![])));
// A local path that is still there (a survived temp file) needs nothing.
let dir = tempfile::tempdir().unwrap();
let alive = dir.path().join("ugoira.mp4");
std::fs::write(&alive, b"x").unwrap();
assert!(!needs_refetch(&queued_task(
alive.to_str().unwrap(),
0,
vec![]
)));
// A local path the restart took away does.
assert!(needs_refetch(&queued_task(
"/nonexistent-ugoira.mp4",
0,
vec![]
)));
// A partially delivered album is left to its own retry path.
assert!(!needs_refetch(&queued_task(
"/nonexistent-ugoira.mp4",
1,
vec![7]
)));
assert!(!needs_refetch(&queued_task(
"/nonexistent-ugoira.mp4",
0,
vec![7]
)));
// A channel copy holds no media.
assert!(!needs_refetch(&Task::ForwardMessages {
from_chat_id: 1,
to_chat_id: 2,
message_ids: vec![3],
forward_offset: 0,
notify_chat_id: None,
notify_message_id: None,
}));
}
#[test]
fn apply_refresh_keeps_the_delivery_envelope() {
let task = queued_task("/nonexistent-ugoira.mp4", 0, vec![]);
let fresh = Refetched {
caption: "fresh caption".into(),
items: vec![photo_item("https://cdn/fresh.jpg", true, false)],
cache_data: None,
keep_alive: None,
};
match apply_refresh(&task, &fresh).expect("a repairable task") {
Task::SendMediaSequence {
chat_id,
reply_to_message_id,
caption,
media_batches,
batch_index,
sent_message_ids,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
} => {
// Same delivery: chat, reply, forward/edit settings, notify.
assert_eq!((chat_id, reply_to_message_id), (1, 2));
assert!(edit_before_forward);
assert_eq!(forward_channel_id, Some(2));
assert_eq!((notify_chat_id, notify_message_id), (Some(1), Some(2)));
assert_eq!(source_url, "https://x.com/u/status/1");
// Fresh media, and nothing of it counted as sent yet.
assert_eq!(caption, "fresh caption");
assert!(
matches!(
media_batches[0][0].media_ref(),
MediaRef::Source(media) if media == "https://cdn/fresh.jpg"
),
"fresh media must replace the lost local file"
);
assert!(matches!(
media_batches[0][0],
MediaItemPayload::Photo {
has_spoiler: true,
..
}
));
assert_eq!((batch_index, sent_message_ids.len()), (0, 0));
}
other => panic!("expected a media sequence, got {other:?}"),
}
}
#[tokio::test]
async fn a_queue_scan_error_is_reported_to_the_startup_caller() {
let stores = TestStores::new();
let sender = MockSender::scripted(vec![], permanent_error);
let ctx = stores.ctx(&sender);
let raw = rusqlite::Connection::open(stores.db_path()).unwrap();
raw.execute_batch("DROP TABLE tasks").unwrap();
assert!(repair_lost_local_media(&ctx).await.is_err());
}
/// The whole repair against a real post: a queued row whose media is a local
/// file the restart took away is re-fetched from its `source_url` and
/// rewritten in place, so the retry can still deliver it.
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
async fn live_repair_refetches_a_lost_local_media_row() {
let stores = TestStores::new();
// An empty script: the repair must not need to tell the user anything.
let sender = MockSender::scripted(vec![], permanent_error);
let ctx = stores.ctx(&sender);
let mut task = queued_task("/nonexistent-ugoira.mp4", 0, vec![]);
if let Task::SendMediaSequence { source_url, .. } = &mut task {
*source_url = "https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224".into();
}
stores
.task_queue()
.enqueue(serde_json::to_value(&task).unwrap(), crate::db::now_f64())
.await
.unwrap();
assert_eq!(repair_lost_local_media(&ctx).await, Ok(1));
let updated: Task = serde_json::from_value(stores.queued_payload().await).unwrap();
match updated {
Task::SendMediaSequence {
media_batches,
batch_index,
sent_message_ids,
caption,
..
} => {
let media: Vec<&str> = media_batches
.iter()
.flatten()
.map(|item| match item.media_ref() {
MediaRef::Source(media) | MediaRef::FileId(media) => media.as_str(),
})
.collect();
assert!(!media.is_empty(), "the fresh fetch yielded no media");
assert!(
media.iter().all(|m| m.starts_with("http")),
"the retry must be uploadable from URLs again: {media:?}"
);
assert_eq!((batch_index, sent_message_ids.len()), (0, 0));
assert!(!caption.is_empty());
}
other => panic!("expected a repaired media sequence, got {other:?}"),
}
// The post was re-read, not re-delivered: nothing was sent.
assert!(sender.calls().is_empty(), "{:?}", sender.calls());
}
}
@@ -0,0 +1,102 @@
//! The URL job channel and its worker pool: a bounded queue (backpressure
//! instead of unbounded spawns) drained by [`URL_WORKERS`] supervised workers.
//!
//! teloxide's per-chat workers are sequential, so a batch forward needs its own
//! concurrency: this is where a link handed over by `handlers::mod` actually
//! reaches the pipeline.
use super::urls::{PostSend, url_media};
use crate::ctx::CONTEXT;
use std::sync::LazyLock;
use teloxide::types::Message;
/// One URL job: the message + the extracted URL (the sender and stores come
/// from the shared [`AppContext`], assembled from statics inside the worker).
type UrlJob = (Message, String);
/// Bounded channel of URL jobs drained by [`start_url_workers`]. The bound
/// caps both queued memory and shutdown backlog; a full channel applies
/// backpressure to the per-chat handler instead of spawning unbounded tasks.
pub(crate) static URL_JOBS: LazyLock<
parking_lot::Mutex<Option<tokio::sync::mpsc::Sender<UrlJob>>>,
> = LazyLock::new(|| parking_lot::Mutex::new(None));
/// Set by main's shutdown sequence; workers stop pulling new jobs.
pub(crate) static URL_STOP: std::sync::atomic::AtomicBool =
std::sync::atomic::AtomicBool::new(false);
/// JoinHandles of the URL workers, awaited by [`stop_url_workers`].
static URL_WORKER_HANDLES: LazyLock<parking_lot::Mutex<Option<Vec<tokio::task::JoinHandle<()>>>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
/// Worker count draining URL jobs; keeps the old 8-permit concurrency cap
/// while bounding how many jobs can be queued at all.
const URL_WORKERS: usize = 8;
/// Starts the URL job workers (called once from main after the queue starts).
/// teloxide dispatches updates to a per-chat worker that handles them
/// sequentially, so a batch-forward of many messages would otherwise be
/// processed one at a time (fetch + send each, roughly a second per
/// message); the workers add throughput, and FIFO order preserves per-message
/// URL order.
pub async fn start_url_workers() {
let (tx, rx) = tokio::sync::mpsc::channel::<UrlJob>(256);
*URL_JOBS.lock() = Some(tx);
let rx = std::sync::Arc::new(tokio::sync::Mutex::new(rx));
let mut handles = Vec::with_capacity(URL_WORKERS);
for _ in 0..URL_WORKERS {
let rx = std::sync::Arc::clone(&rx);
handles.push(tokio::spawn(async move {
// Supervised like the queue workers: a panic inside a worker
// (a handler, a poisoned lock) used to kill it for good and
// silently shrink the pool — the remaining workers keep the
// channel drained, so nothing else surfaces the loss. The job the
// panicking worker held is lost; the panic is not.
while !URL_STOP.load(std::sync::atomic::Ordering::Relaxed) {
let rx = std::sync::Arc::clone(&rx);
if let Err(e) = tokio::spawn(async move {
while !URL_STOP.load(std::sync::atomic::Ordering::Relaxed) {
let job = rx.lock().await.recv().await;
match job {
Some((message, url)) => {
url_media(
&CONTEXT,
message.chat.id.0,
message.id.0 as i64,
&url,
PostSend::FromChat,
)
.await;
}
None => break,
}
}
})
.await
{
log::error!("url worker panicked, restarting: {e}");
}
}
}));
}
*URL_WORKER_HANDLES.lock() = Some(handles);
}
/// Stops the URL workers: sets the stop flag, drops the job channel (so
/// workers blocked in \`recv()\` wake with \`None\` and exit) and awaits the
/// worker tasks. Each worker finishes its in-flight job first; jobs still
/// queued in the channel are abandoned (the old implementation neither
/// drained them nor woke blocked workers — it only set a flag checked
/// between jobs).
pub async fn stop_url_workers() {
URL_STOP.store(true, std::sync::atomic::Ordering::Relaxed);
// Dropping the sender makes every worker's recv() return None.
*URL_JOBS.lock() = None;
// Take the handles first so the lock guard drops before the awaits.
let handles = URL_WORKER_HANDLES.lock().take();
if let Some(handles) = handles {
for handle in handles {
if let Err(e) = handle.await {
log::error!("url worker panicked at shutdown: {e}");
}
}
}
}
File diff suppressed because it is too large Load Diff
+81 -83
View File
@@ -9,12 +9,13 @@
//! by the periodic prune in `main`.
use crate::db::now_f64;
use rusqlite::OptionalExtension;
use rusqlite::params;
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use std::time::Duration;
#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)]
#[derive(Serialize, Deserialize, Clone, Copy, Debug, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum CachedMediaKind {
Photo,
@@ -73,113 +74,110 @@ impl LinkCache {
pub async fn get(&self, key: &str, ttl: Duration) -> Option<CachedPost> {
let key = key.to_string();
let ttl = ttl.as_secs_f64();
let result = self
.pool
.with_conn(move |conn| {
let mut stmt =
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
let mut rows = stmt.query(params![key])?;
let Some(row) = rows.next()? else {
return Ok(None);
};
let payload: String = row.get(0)?;
let created_at: f64 = row.get(1)?;
if now_f64() - created_at > ttl {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
return Ok(None);
}
match serde_json::from_str::<CachedPost>(&payload) {
Ok(post) => Ok(Some(post)),
Err(e) => {
// Unreadable payload (e.g. an older schema): drop it
// instead of re-failing the parse on every later hit.
self.pool
.with_conn_or(
log::Level::Warn,
"link cache read failed",
None,
move |conn| {
let Some((payload, created_at)) = conn
.query_row(
"SELECT payload, created_at FROM link_cache WHERE url = ?1",
params![key],
|row| Ok((row.get::<_, String>(0)?, row.get::<_, f64>(1)?)),
)
.optional()?
else {
return Ok(None);
};
if now_f64() - created_at > ttl {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Err(rusqlite::Error::ToSqlConversionFailure(Box::new(e)))
return Ok(None);
}
}
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::warn!("link cache read failed: {e}");
None
}
}
match serde_json::from_str::<CachedPost>(&payload) {
Ok(post) => Ok(Some(post)),
Err(e) => {
// Unreadable payload (e.g. an older schema): drop it
// instead of re-failing the parse on every later hit.
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Err(rusqlite::Error::ToSqlConversionFailure(Box::new(e)))
}
}
},
)
.await
}
pub async fn put(&self, key: &str, post: &CachedPost) {
let key = key.to_string();
let payload = serde_json::to_string(post).expect("cached post serializes");
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"INSERT OR REPLACE INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
params![key, payload, now_f64()],
)?;
Ok(())
})
self.pool
.with_conn_or(
log::Level::Warn,
"link cache write failed",
(),
move |conn| {
conn.execute(
"INSERT OR REPLACE INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
params![key, payload, now_f64()],
)?;
Ok(())
},
)
.await;
if let Err(e) = result {
log::warn!("link cache write failed: {e}");
}
}
/// Drops an entry (e.g. a cached file id that turned out invalid).
pub async fn remove(&self, key: &str) {
let key = key.to_string();
let result = self
.pool
.with_conn(move |conn| {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Ok(())
})
self.pool
.with_conn_or(
log::Level::Warn,
"link cache delete failed",
(),
move |conn| {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Ok(())
},
)
.await;
if let Err(e) = result {
log::warn!("link cache delete failed: {e}");
}
}
/// Removes expired entries; returns how many were deleted.
pub async fn prune(&self, ttl: Duration) -> usize {
let cutoff = now_f64() - ttl.as_secs_f64();
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"DELETE FROM link_cache WHERE created_at < ?1",
params![cutoff],
)
})
.await;
match result {
Ok(n) => n,
Err(e) => {
log::warn!("link cache prune failed: {e}");
0
}
}
self.pool
.with_conn_or(
log::Level::Warn,
"link cache prune failed",
0,
move |conn| {
conn.execute(
"DELETE FROM link_cache WHERE created_at < ?1",
params![cutoff],
)
},
)
.await
}
/// Deletes one entry (by normalized cache key) or the whole cache when
/// `key` is `None`. Returns how many rows were removed.
pub async fn clear(&self, key: Option<&str>) -> usize {
let key = key.map(str::to_string);
let result = self
.pool
.with_conn(move |conn| match &key {
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
None => conn.execute("DELETE FROM link_cache", []),
})
.await;
match result {
Ok(n) => n,
Err(e) => {
log::warn!("link cache clear failed: {e}");
0
}
}
self.pool
.with_conn_or(
log::Level::Warn,
"link cache clear failed",
0,
move |conn| match &key {
Some(key) => {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])
}
None => conn.execute("DELETE FROM link_cache", []),
},
)
.await
}
}
+37 -44
View File
@@ -21,7 +21,7 @@ mod send;
mod state;
use ctx::CONTEXT;
use handlers::{CHAT_STORE, CONFIG, LINK_CACHE, TASK_QUEUE};
use handlers::{CONFIG, TASK_QUEUE};
/// Docker `stop` / `compose down` delivers SIGTERM, which teloxide's ctrlc
/// handler (SIGINT only) never sees — without this the process would die
@@ -157,7 +157,13 @@ async fn main() {
// succeed after a restart — the registry that kept those files alive is in
// memory — so those rows are re-fetched from their post instead of
// dead-lettering the user's link.
let repaired = handlers::repair_lost_local_media(&CONTEXT).await;
let repaired = match handlers::repair_lost_local_media(&CONTEXT).await {
Ok(repaired) => repaired,
Err(e) => {
log::error!("startup repair failed: {e}; refusing to start queue workers");
return;
}
};
if repaired > 0 {
log::info!("startup repair: re-fetched {repaired} queued task(s)");
}
@@ -203,15 +209,7 @@ async fn main() {
{
let bot = bot.clone();
tokio::spawn(async move {
periodic_sweep(
&bot,
&CHAT_STORE,
&LINK_CACHE,
&TASK_QUEUE,
&CONFIG,
stop_rx,
)
.await;
periodic_sweep(crate::ctx::AppContext::from_statics(&bot), stop_rx).await;
});
}
@@ -221,7 +219,6 @@ async fn main() {
.branch(Update::filter_callback_query().branch(endpoint(handlers::callback_query_handler)));
let mut dispatcher = Dispatcher::builder(bot.clone(), handler)
.dependencies(dptree::deps![""])
.enable_ctrlc_handler()
.build();
@@ -232,13 +229,16 @@ async fn main() {
// secret token included) — no explicit registration here.
let listen = CONFIG.webhook_listen.expect("WEBHOOK_LISTEN is not set");
let port = CONFIG.webhook_port.expect("WEBHOOK_PORT is not set");
let mut options = webhooks::Options::new((listen, port).into(), url);
// No secret, no webhook: without one the axum listener accepts any
// POST, and a forged update can impersonate anyone — admins included.
let secret = CONFIG
.webhook_secret_token
.clone()
.expect("WEBHOOK_SECRET_TOKEN is not set (required in webhook mode)");
let mut options = webhooks::Options::new((listen, port).into(), url).secret_token(secret);
if let Some(cert) = &CONFIG.webhook_cert {
options = options.certificate(InputFile::file(cert));
}
if let Some(secret) = &CONFIG.webhook_secret_token {
options = options.secret_token(secret.clone());
}
let mut listener = webhooks::axum(bot.clone(), options)
.await
@@ -269,18 +269,19 @@ async fn main() {
.await;
}
// Graceful stop (Ctrl+C / SIGTERM): stop the sweep, notify the admin,
// drain the queue. Bounded: a worker mid-download (30 s timeout) or a
// long ugoira encode must not hold the shutdown hostage forever.
// Graceful stop (Ctrl+C / SIGTERM): stop the queue first so no new
// persistent task is leased while the URL workers drain. The two drains
// share the bounded shutdown budget; URL work may legitimately outlive it,
// but the queue must not be left running until process exit.
log::info!("Stopping bot");
const SHUTDOWN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
let shutdown = async {
let _ = stop_tx.send(true);
TASK_QUEUE.stop().await;
handlers::stop_url_workers().await;
if let Some(admin) = CONFIG.admin_ids.first() {
let _ = bot.send_message(ChatId(*admin), "Shutting down...").await;
}
TASK_QUEUE.stop().await;
};
if tokio::time::timeout(SHUTDOWN_TIMEOUT, shutdown)
.await
@@ -299,24 +300,21 @@ const SWEEP_INTERVAL: Duration = Duration::from_secs(300);
/// link cache, the idle rate-limit buckets and the idle inline-query entries,
/// and reports the queue only when it is not empty.
///
/// Takes its collaborators instead of reaching for the statics so a test can
/// drive a tick with a paused clock: a sleeping task nothing drives is how the
/// queue's own sweep kept a missing worker wake-up.
async fn periodic_sweep(
sender: &dyn crate::media_sender::MediaSender,
chat_store: &crate::state::ChatStore,
link_cache: &crate::link_cache::LinkCache,
task_queue: &crate::queue::PersistentTaskQueue,
config: &crate::config::Config,
mut stop: watch::Receiver<bool>,
) {
/// Takes the shared [`crate::ctx::AppContext`] — the collaborators as one
/// bundle, production assembling it from the statics and tests from tempdir
/// stores — so a test can drive a tick with a paused clock: a sleeping task
/// nothing drives is how the queue's own sweep kept a missing worker wake-up.
async fn periodic_sweep(ctx: crate::ctx::AppContext<'_>, mut stop: watch::Receiver<bool>) {
loop {
tokio::select! {
_ = stop.changed() => break,
_ = tokio::time::sleep(SWEEP_INTERVAL) => {}
}
let removed = chat_store.prune_expired(config.edit_message_ttl).await;
let pruned = link_cache.prune(config.link_cache_ttl).await;
let removed = ctx
.chat_store
.prune_expired(ctx.config.edit_message_ttl)
.await;
let pruned = ctx.link_cache.prune(ctx.config.link_cache_ttl).await;
if pruned > 0 {
log::info!("link cache: pruned {pruned} expired entr(ies)");
}
@@ -334,7 +332,7 @@ async fn periodic_sweep(
// Only speaks up when the queue is not empty: a healthy bot has nothing
// to report, and a periodic "0 pending" line is noise that hides the
// lines that matter.
if let Some((pending, oldest_run_after)) = task_queue.pending_backlog().await {
if let Some((pending, oldest_run_after)) = ctx.task_queue.pending_backlog().await {
let overdue = crate::db::now_f64() - oldest_run_after;
if overdue >= 0.0 {
log::info!("queue: {pending} pending task(s), oldest {overdue:.0}s overdue");
@@ -351,7 +349,8 @@ async fn periodic_sweep(
// later about a prompt the user already walked away from. The edit
// drops the buttons too. If the prompt was already deleted this
// fails with a 400 "message to edit not found" — log and ignore.
if let Err(e) = sender
if let Err(e) = ctx
.sender
.edit_message_text(
ChatId(chat_id),
MessageId(prompt_message_id as i32),
@@ -455,17 +454,11 @@ mod tests {
},
);
})
.await;
.await
.unwrap();
let (stop_tx, stop_rx) = watch::channel(false);
let sweep = periodic_sweep(
&sender,
stores.chat_store(),
stores.link_cache(),
stores.task_queue(),
&config,
stop_rx,
);
let sweep = periodic_sweep(stores.ctx(&sender), stop_rx);
tokio::pin!(sweep);
// One second short of the interval: nothing has been touched. The
-702
View File
@@ -1,702 +0,0 @@
//! Send abstraction: the message-sending surface [`send`](crate::send)
//! needs, so the send pipeline can be tested with a scripted mock instead of
//! a live teloxide `Bot`.
use std::future::Future;
use std::pin::Pin;
use teloxide::RequestError;
use teloxide::prelude::Requester;
use teloxide::prelude::*;
use teloxide::types::{
CallbackQueryId, ChatAction, ChatId, InlineKeyboardMarkup, InputFile, InputMedia, Message,
MessageId, ParseMode, ReplyParameters,
};
/// Boxed, `Send` future returned by a [`MediaSender`] method (`async fn` in
/// traits is not dyn-compatible).
type BoxFuture<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
/// The message-sending surface the send pipeline uses. The production
/// implementation is teloxide's [`Bot`]; tests inject a scripted mock to
/// cover the fallback and classification logic without touching the
/// Telegram API.
pub trait MediaSender: Send + Sync {
/// Sends a media group, replying to `reply_to`.
fn send_media_group(
&self,
chat_id: ChatId,
reply_to: MessageId,
items: Vec<InputMedia>,
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>>;
/// Sends a lone animation, replying to `reply_to`.
fn send_animation<'a>(
&'a self,
chat_id: ChatId,
reply_to: MessageId,
caption: &'a str,
spoiler: bool,
file: InputFile,
) -> BoxFuture<'a, Result<Message, RequestError>>;
/// Copies messages between chats (forward to channel).
fn copy_messages(
&self,
to: ChatId,
from: ChatId,
ids: Vec<MessageId>,
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>>;
/// Sends a plain text message, optionally replying to `reply_to` and
/// attaching `reply_markup`. Returns the sent message's id: the bot only
/// ever needs that (the edit-before-forward prompt's record is keyed by
/// it), and returning the whole `Message` would force every test mock to
/// construct one.
fn send_message(
&self,
chat_id: ChatId,
text: String,
reply_to: Option<MessageId>,
reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<i64, RequestError>>;
/// Answers a callback query, optionally with a toast `text` shown to the
/// user who pressed the button.
fn answer_callback_query(
&self,
id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Rewrites a message's text and drops its inline keyboard: the
/// edit-expiry sweep rewriting a prompt whose record expired (a button left
/// behind could only answer "Expired").
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Rewrites a message's caption, always with HTML parse mode (every caller
/// in this bot renders escaped HTML: templates and edit-before-forward
/// links).
fn edit_message_caption(
&self,
chat_id: ChatId,
message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Deletes a message (the edit-before-forward prompt after a forward).
fn delete_message(
&self,
chat_id: ChatId,
message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Sets the chat's "typing / uploading …" indicator (cosmetic).
fn send_chat_action(
&self,
chat_id: ChatId,
action: ChatAction,
) -> BoxFuture<'_, Result<(), RequestError>>;
}
impl MediaSender for Bot {
fn send_media_group(
&self,
chat_id: ChatId,
reply_to: MessageId,
items: Vec<InputMedia>,
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>> {
Box::pin(async move {
// Pace media sends per chat (one token per item) so bursts do not
// trip Telegram's flood control.
crate::rate_limit::limiter_for(chat_id.0)
.acquire(items.len() as f64)
.await;
// Same spend against the bot-wide budget: a fan-out over chats is
// invisible to the per-chat buckets.
crate::rate_limit::acquire_global(items.len() as f64).await;
// `<Bot as Requester>::` disambiguates from this trait's same-named
// method (teloxide's API lives in the `Requester` trait).
<Bot as Requester>::send_media_group(self, chat_id, items)
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply())
.await
})
}
fn send_animation<'a>(
&'a self,
chat_id: ChatId,
reply_to: MessageId,
caption: &'a str,
spoiler: bool,
file: InputFile,
) -> BoxFuture<'a, Result<Message, RequestError>> {
Box::pin(async move {
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
crate::rate_limit::acquire_global(1.0).await;
let mut request = <Bot as Requester>::send_animation(self, chat_id, file)
.caption(caption)
.parse_mode(ParseMode::Html)
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply());
if spoiler {
request = request.has_spoiler(true);
}
request.await
})
}
fn copy_messages(
&self,
to: ChatId,
from: ChatId,
ids: Vec<MessageId>,
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>> {
Box::pin(async move {
// Channel forwards are the burstiest path (batch copies); pace
// them per message against the channel's budget.
crate::rate_limit::limiter_for(to.0)
.acquire(ids.len() as f64)
.await;
crate::rate_limit::acquire_global(ids.len() as f64).await;
<Bot as Requester>::copy_messages(self, to, from, ids).await
})
}
fn send_message(
&self,
chat_id: ChatId,
text: String,
reply_to: Option<MessageId>,
reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<i64, RequestError>> {
Box::pin(async move {
let mut request = <Bot as Requester>::send_message(self, chat_id, text);
if let Some(reply_to) = reply_to {
request = request
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply());
}
if let Some(markup) = reply_markup {
request = request.reply_markup(markup);
}
request.await.map(|message| message.id.0 as i64)
})
}
fn answer_callback_query(
&self,
id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
let mut request = <Bot as Requester>::answer_callback_query(self, id);
if let Some(text) = text {
request = request.text(text);
}
request.await.map(|_| ())
})
}
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
<Bot as Requester>::edit_message_text(self, chat_id, message_id, text)
.reply_markup(InlineKeyboardMarkup::default())
.await
.map(|_| ())
})
}
fn edit_message_caption(
&self,
chat_id: ChatId,
message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
<Bot as Requester>::edit_message_caption(self, chat_id, message_id)
.caption(caption)
.parse_mode(ParseMode::Html)
.await
.map(|_| ())
})
}
fn delete_message(
&self,
chat_id: ChatId,
message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
<Bot as Requester>::delete_message(self, chat_id, message_id)
.await
.map(|_| ())
})
}
fn send_chat_action(
&self,
chat_id: ChatId,
action: ChatAction,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
// teloxide's `send_chat_action` returns `Result<True, _>` (its
// unit marker type); map the success to `()`.
<Bot as Requester>::send_chat_action(self, chat_id, action)
.await
.map(|_| ())
})
}
}
/// Test support: a scripted [`MediaSender`] mock (no Telegram API involved).
#[cfg(test)]
pub(crate) mod test_support {
use super::*;
use parking_lot::Mutex;
/// One scripted outcome, consumed front-to-back; the last entry repeats
/// for further calls of the same method kind.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum Outcome {
GroupOk,
GroupErr,
AnimationErr,
CopyOk,
CopyErr,
/// An error from `send_message` (replies are fire-and-forget, so an
/// error is fine for tests).
MessageErr,
/// A successful `send_message`, returning message id [`MockSender::SENT_ID`].
MessageOk,
EditOk,
EditErr,
}
/// A stand-in for `api.telegram.org` for the tests that must drive a real
/// `Bot` — its request building, the per-chat limiter, the bot-wide budget
/// — which the scripted mock bypasses entirely. Records every call and
/// answers the smallest result each method needs.
pub(crate) mod fake_api {
use parking_lot::Mutex;
use std::sync::Arc;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
pub(crate) struct FakeApi {
url: url::Url,
calls: Arc<Mutex<Vec<(String, serde_json::Value)>>>,
server: tokio::task::JoinHandle<()>,
}
impl FakeApi {
/// Binds an ephemeral port and serves until dropped.
pub(crate) async fn start() -> FakeApi {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let calls = Arc::new(Mutex::new(Vec::new()));
let recorded = Arc::clone(&calls);
let server = tokio::spawn(async move {
while let Ok((mut socket, _)) = listener.accept().await {
let recorded = Arc::clone(&recorded);
tokio::spawn(async move {
let Some((method, body)) = read_request(&mut socket).await else {
return;
};
recorded.lock().push((method.clone(), body));
let payload = serde_json::json!({
"ok": true,
"result": canned_result(&method),
})
.to_string();
let response = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\n\
content-length: {}\r\nconnection: close\r\n\r\n{}",
payload.len(),
payload
);
let _ = socket.write_all(response.as_bytes()).await;
let _ = socket.flush().await;
});
}
});
FakeApi {
// Trailing slash: teloxide appends `bot<token>/<method>`.
url: url::Url::parse(&format!("http://{addr}/")).unwrap(),
calls,
server,
}
}
/// Where to point a `Bot`: `Bot::new(token).set_api_url(api.url())`.
pub(crate) fn url(&self) -> url::Url {
self.url.clone()
}
/// Method names in call order.
pub(crate) fn methods(&self) -> Vec<String> {
self.calls.lock().iter().map(|(m, _)| m.clone()).collect()
}
/// The JSON body of the first call to `method` (`Null` for a body
/// that is not JSON, i.e. a multipart upload).
pub(crate) fn body(&self, method: &str) -> serde_json::Value {
self.calls
.lock()
.iter()
.find(|(m, _)| m == method)
.map(|(_, body)| body.clone())
.unwrap_or(serde_json::Value::Null)
}
}
impl Drop for FakeApi {
fn drop(&mut self) {
self.server.abort();
}
}
/// The smallest result teloxide can deserialize for a method. The names
/// arrive as the payload type's own — `SendMediaGroup`, not
/// `sendMediaGroup`: teloxide builds the URL from that, and the Bot API
/// accepts the spelling.
fn canned_result(method: &str) -> serde_json::Value {
match method {
"CopyMessages" => serde_json::json!([{ "message_id": 11 }]),
"SendMediaGroup" => serde_json::json!([minimal_message()]),
"SendMessage" | "SendAnimation" | "EditMessageCaption" => minimal_message(),
_ => serde_json::Value::Bool(true),
}
}
fn minimal_message() -> serde_json::Value {
serde_json::json!({
"message_id": 1,
"date": 0,
"chat": { "id": 1, "type": "private" },
})
}
/// One HTTP/1.1 request: the head up to the blank line, then
/// `content-length` bytes of body — JSON for most methods, multipart
/// for the media ones (teloxide sends `SendMediaGroup` that way).
async fn read_request(socket: &mut TcpStream) -> Option<(String, serde_json::Value)> {
let mut buf = Vec::new();
let mut chunk = [0u8; 4096];
loop {
let n = socket.read(&mut chunk).await.ok()?;
if n == 0 {
return None;
}
buf.extend_from_slice(&chunk[..n]);
let Some(headers_end) = find(&buf, b"\r\n\r\n") else {
continue;
};
let head = String::from_utf8_lossy(&buf[..headers_end]).to_string();
let length: usize = head
.lines()
.find_map(|line| {
line.to_ascii_lowercase()
.strip_prefix("content-length:")
.and_then(|v| v.trim().parse().ok())
})
.unwrap_or(0);
let body_start = headers_end + 4;
while buf.len() < body_start + length {
let n = socket.read(&mut chunk).await.ok()?;
if n == 0 {
break;
}
buf.extend_from_slice(&chunk[..n]);
}
let method = head
.lines()
.next()
// `POST /bot<token>/<method>`
.and_then(|line| line.split(' ').nth(1))
.and_then(|path| path.rsplit('/').next())
.unwrap_or_default()
.to_string();
let body = parse_body(&buf[body_start..], &head);
return Some((method, body));
}
}
/// The request body as JSON: either the JSON body itself, or a
/// multipart form flattened into an object (each part's value parsed as
/// JSON when it is one, so `media` comes back as its array).
fn parse_body(body: &[u8], head: &str) -> serde_json::Value {
let content_type = head
.lines()
.find(|line| line.to_ascii_lowercase().starts_with("content-type:"))
.unwrap_or_default()
.to_ascii_lowercase();
let Some(boundary) = content_type
.split("boundary=")
.nth(1)
.map(|b| b.trim().trim_matches('"').to_string())
else {
return serde_json::from_slice(body).unwrap_or_default();
};
let text = String::from_utf8_lossy(body);
let mut fields = serde_json::Map::new();
for part in text.split(&format!("--{boundary}")).skip(1) {
let Some((part_head, value)) = part.split_once("\r\n\r\n") else {
continue;
};
let Some(name) = part_head
.split("name=\"")
.nth(1)
.and_then(|rest| rest.split('"').next())
else {
continue;
};
let value = value.trim_end_matches("\r\n");
fields.insert(
name.to_string(),
serde_json::from_str(value).unwrap_or_else(|_| value.into()),
);
}
serde_json::Value::Object(fields)
}
fn find(haystack: &[u8], needle: &[u8]) -> Option<usize> {
haystack
.windows(needle.len())
.position(|window| window == needle)
}
}
/// Replays a script and records what was sent, so tests can assert the
/// user-visible text a path produced.
pub(crate) struct MockSender {
script: Mutex<Vec<Outcome>>,
cursor: Mutex<usize>,
calls: Mutex<Vec<&'static str>>,
messages: Mutex<Vec<String>>,
captions: Mutex<Vec<String>>,
answers: Mutex<Vec<Option<String>>>,
/// `(chat, message, text)` of every text rewrite, in order.
edited_texts: Mutex<Vec<(i64, i64, String)>>,
/// Builds the error every `*Err` outcome returns (RequestError is not
/// cloneable, so the factory recreates it per call).
error: Box<dyn Fn() -> RequestError + Send + Sync>,
}
impl MockSender {
/// The message id a successful `send_message` reports.
pub(crate) const SENT_ID: i64 = 1;
pub(crate) fn scripted(
script: Vec<Outcome>,
error: impl Fn() -> RequestError + Send + Sync + 'static,
) -> Self {
MockSender {
script: Mutex::new(script),
cursor: Mutex::new(0),
calls: Mutex::new(Vec::new()),
messages: Mutex::new(Vec::new()),
captions: Mutex::new(Vec::new()),
answers: Mutex::new(Vec::new()),
edited_texts: Mutex::new(Vec::new()),
error: Box::new(error),
}
}
/// Method names in call order (e.g. `["send_media_group",
/// "send_media_group"]` proves the fallback re-sent).
pub(crate) fn calls(&self) -> Vec<&'static str> {
self.calls.lock().clone()
}
/// Texts of the plain messages sent, in order.
pub(crate) fn messages(&self) -> Vec<String> {
self.messages.lock().clone()
}
/// Captions passed to `edit_message_caption`, in order.
pub(crate) fn captions(&self) -> Vec<String> {
self.captions.lock().clone()
}
/// Toast texts of the answered callback queries, in order.
pub(crate) fn answers(&self) -> Vec<Option<String>> {
self.answers.lock().clone()
}
/// `(chat, message, text)` of every `edit_message_text`, in order.
pub(crate) fn edited_texts(&self) -> Vec<(i64, i64, String)> {
self.edited_texts.lock().clone()
}
fn next(&self, kind: &'static str) -> Outcome {
self.calls.lock().push(kind);
let script = self.script.lock();
let mut cursor = self.cursor.lock();
if script.is_empty() {
panic!("mock script exhausted: {kind}");
}
let idx = (*cursor).min(script.len() - 1);
*cursor = idx + 1;
script[idx]
}
fn error(&self) -> RequestError {
(self.error)()
}
}
impl MediaSender for MockSender {
fn send_media_group(
&self,
_chat_id: ChatId,
_reply_to: MessageId,
items: Vec<InputMedia>,
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>> {
Box::pin(async move {
// Record the captions exactly as Telegram receives them (only
// the first item of a group carries one), so tests can assert
// what a recipient sees.
self.captions
.lock()
.extend(items.iter().filter_map(|item| match item {
InputMedia::Photo(photo) => photo.caption.clone(),
InputMedia::Video(video) => video.caption.clone(),
InputMedia::Animation(animation) => animation.caption.clone(),
_ => None,
}));
match self.next("send_media_group") {
Outcome::GroupOk => Ok(Vec::new()),
Outcome::GroupErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for send_media_group"),
}
})
}
fn send_animation<'a>(
&'a self,
_chat_id: ChatId,
_reply_to: MessageId,
_caption: &'a str,
_spoiler: bool,
_file: InputFile,
) -> BoxFuture<'a, Result<Message, RequestError>> {
Box::pin(async move {
match self.next("send_animation") {
Outcome::AnimationErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for send_animation"),
}
})
}
fn copy_messages(
&self,
_to: ChatId,
_from: ChatId,
_ids: Vec<MessageId>,
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>> {
Box::pin(async move {
match self.next("copy_messages") {
Outcome::CopyOk => Ok(vec![MessageId(1)]),
Outcome::CopyErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for copy_messages"),
}
})
}
fn send_message(
&self,
_chat_id: ChatId,
text: String,
_reply_to: Option<MessageId>,
_reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<i64, RequestError>> {
Box::pin(async move {
self.messages.lock().push(text);
match self.next("send_message") {
Outcome::MessageOk => Ok(MockSender::SENT_ID),
Outcome::MessageErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for send_message"),
}
})
}
fn answer_callback_query(
&self,
_id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Always succeeds: the toast is cosmetic, so the script stays
// focused on the outcomes a test cares about.
Box::pin(async move {
self.calls.lock().push("answer_callback_query");
self.answers.lock().push(text);
Ok(())
})
}
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Always succeeds: the only caller is the expiry sweep, which
// tolerates a failure (a prompt the user already deleted), so the
// script stays free for the call the test is about.
Box::pin(async move {
self.calls.lock().push("edit_message_text");
self.edited_texts
.lock()
.push((chat_id.0, message_id.0 as i64, text));
Ok(())
})
}
fn edit_message_caption(
&self,
_chat_id: ChatId,
_message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
self.captions.lock().push(caption);
match self.next("edit_message_caption") {
Outcome::EditOk => Ok(()),
Outcome::EditErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for edit_message_caption"),
}
})
}
fn delete_message(
&self,
_chat_id: ChatId,
_message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Deletion is fire-and-forget in every caller; always succeeds.
Box::pin(async move {
self.calls.lock().push("delete_message");
Ok(())
})
}
fn send_chat_action(
&self,
_chat_id: ChatId,
_action: ChatAction,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
self.calls.lock().push("send_chat_action");
Ok(())
})
}
}
}
+323
View File
@@ -0,0 +1,323 @@
//! Send abstraction: the message-sending surface [`send`](crate::send)
//! needs, so the send pipeline can be tested with a scripted mock instead of
//! a live teloxide `Bot`.
use std::future::Future;
use std::pin::Pin;
use teloxide::RequestError;
use teloxide::prelude::Requester;
use teloxide::prelude::*;
use teloxide::types::{
CallbackQueryId, ChatAction, ChatId, InlineKeyboardMarkup, InlineQueryId, InlineQueryResult,
InputFile, InputMedia, Message, MessageId, ParseMode, ReplyParameters,
};
/// Boxed, `Send` future returned by a [`MediaSender`] method (`async fn` in
/// traits is not dyn-compatible).
type BoxFuture<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
/// The message-sending surface the send pipeline uses. The production
/// implementation is teloxide's [`Bot`]; tests inject a scripted mock to
/// cover the fallback and classification logic without touching the
/// Telegram API.
pub trait MediaSender: Send + Sync {
/// Sends a media group, replying to `reply_to`.
fn send_media_group(
&self,
chat_id: ChatId,
reply_to: MessageId,
items: Vec<InputMedia>,
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>>;
/// Sends a lone animation, replying to `reply_to`.
fn send_animation<'a>(
&'a self,
chat_id: ChatId,
reply_to: MessageId,
caption: &'a str,
spoiler: bool,
file: InputFile,
) -> BoxFuture<'a, Result<Message, RequestError>>;
/// Copies messages between chats (forward to channel).
fn copy_messages(
&self,
to: ChatId,
from: ChatId,
ids: Vec<MessageId>,
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>>;
/// Sends a plain text message, optionally replying to `reply_to` and
/// attaching `reply_markup`. Returns the sent message's id: the bot only
/// ever needs that (the edit-before-forward prompt's record is keyed by
/// it), and returning the whole `Message` would force every test mock to
/// construct one.
fn send_message(
&self,
chat_id: ChatId,
text: String,
reply_to: Option<MessageId>,
reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<i64, RequestError>>;
/// Sends an HTML-formatted plain message.
fn send_html_message(
&self,
chat_id: ChatId,
text: String,
reply_to: Option<MessageId>,
) -> BoxFuture<'_, Result<i64, RequestError>>;
/// Answers an inline query with `results`, cached by Telegram for
/// `cache_time` seconds. An empty `results` answers *empty*, which is a
/// real answer: it stops the client spinning and lets Telegram serve a
/// repeat itself instead of the bot re-running the query.
fn answer_inline_query(
&self,
id: InlineQueryId,
results: Vec<InlineQueryResult>,
cache_time: u32,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Answers a callback query, optionally with a toast `text` shown to the
/// user who pressed the button.
fn answer_callback_query(
&self,
id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Rewrites a message's text and drops its inline keyboard: the
/// edit-expiry sweep rewriting a prompt whose record expired (a button left
/// behind could only answer "Expired").
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Rewrites a message's caption, always with HTML parse mode (every caller
/// in this bot renders escaped HTML: templates and edit-before-forward
/// links).
fn edit_message_caption(
&self,
chat_id: ChatId,
message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Deletes a message (the edit-before-forward prompt after a forward).
fn delete_message(
&self,
chat_id: ChatId,
message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Sets the chat's "typing / uploading …" indicator (cosmetic).
fn send_chat_action(
&self,
chat_id: ChatId,
action: ChatAction,
) -> BoxFuture<'_, Result<(), RequestError>>;
}
impl MediaSender for Bot {
fn send_media_group(
&self,
chat_id: ChatId,
reply_to: MessageId,
items: Vec<InputMedia>,
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>> {
Box::pin(async move {
// Pace media sends per chat (one token per item) so bursts do not
// trip Telegram's flood control.
crate::rate_limit::limiter_for(chat_id.0)
.acquire(items.len() as f64)
.await;
// Same spend against the bot-wide budget: a fan-out over chats is
// invisible to the per-chat buckets.
crate::rate_limit::acquire_global(items.len() as f64).await;
// `<Bot as Requester>::` disambiguates from this trait's same-named
// method (teloxide's API lives in the `Requester` trait).
<Bot as Requester>::send_media_group(self, chat_id, items)
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply())
.await
})
}
fn send_animation<'a>(
&'a self,
chat_id: ChatId,
reply_to: MessageId,
caption: &'a str,
spoiler: bool,
file: InputFile,
) -> BoxFuture<'a, Result<Message, RequestError>> {
Box::pin(async move {
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
crate::rate_limit::acquire_global(1.0).await;
let mut request = <Bot as Requester>::send_animation(self, chat_id, file)
.caption(caption)
.parse_mode(ParseMode::Html)
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply());
if spoiler {
request = request.has_spoiler(true);
}
request.await
})
}
fn copy_messages(
&self,
to: ChatId,
from: ChatId,
ids: Vec<MessageId>,
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>> {
Box::pin(async move {
// Channel forwards are the burstiest path (batch copies); pace
// them per message against the channel's budget.
crate::rate_limit::limiter_for(to.0)
.acquire(ids.len() as f64)
.await;
crate::rate_limit::acquire_global(ids.len() as f64).await;
<Bot as Requester>::copy_messages(self, to, from, ids).await
})
}
fn send_message(
&self,
chat_id: ChatId,
text: String,
reply_to: Option<MessageId>,
reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<i64, RequestError>> {
Box::pin(async move {
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
crate::rate_limit::acquire_global(1.0).await;
let mut request = <Bot as Requester>::send_message(self, chat_id, text);
if let Some(reply_to) = reply_to {
request = request
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply());
}
if let Some(markup) = reply_markup {
request = request.reply_markup(markup);
}
request.await.map(|message| message.id.0 as i64)
})
}
fn send_html_message(
&self,
chat_id: ChatId,
text: String,
reply_to: Option<MessageId>,
) -> BoxFuture<'_, Result<i64, RequestError>> {
Box::pin(async move {
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
crate::rate_limit::acquire_global(1.0).await;
let mut request =
<Bot as Requester>::send_message(self, chat_id, text).parse_mode(ParseMode::Html);
if let Some(reply_to) = reply_to {
request = request
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply());
}
request.await.map(|message| message.id.0 as i64)
})
}
fn answer_inline_query(
&self,
id: InlineQueryId,
results: Vec<InlineQueryResult>,
cache_time: u32,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
<Bot as Requester>::answer_inline_query(self, id, results)
.cache_time(cache_time)
.await
.map(|_| ())
})
}
fn answer_callback_query(
&self,
id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
let mut request = <Bot as Requester>::answer_callback_query(self, id);
if let Some(text) = text {
request = request.text(text);
}
request.await.map(|_| ())
})
}
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
crate::rate_limit::acquire_global(1.0).await;
<Bot as Requester>::edit_message_text(self, chat_id, message_id, text)
.reply_markup(InlineKeyboardMarkup::default())
.await
.map(|_| ())
})
}
fn edit_message_caption(
&self,
chat_id: ChatId,
message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
crate::rate_limit::acquire_global(1.0).await;
<Bot as Requester>::edit_message_caption(self, chat_id, message_id)
.caption(caption)
.parse_mode(ParseMode::Html)
.await
.map(|_| ())
})
}
fn delete_message(
&self,
chat_id: ChatId,
message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
crate::rate_limit::acquire_global(1.0).await;
<Bot as Requester>::delete_message(self, chat_id, message_id)
.await
.map(|_| ())
})
}
fn send_chat_action(
&self,
chat_id: ChatId,
action: ChatAction,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
// Same gap as send_message: actions count against the bot-wide
// budget too (see there); the refresh loop behind
// `run_with_chat_action` makes them frequent enough to matter.
crate::rate_limit::acquire_global(1.0).await;
// teloxide's `send_chat_action` returns `Result<True, _>` (its
// unit marker type); map the success to `()`.
<Bot as Requester>::send_chat_action(self, chat_id, action)
.await
.map(|_| ())
})
}
}
#[cfg(test)]
pub(crate) mod test_support;
@@ -0,0 +1,537 @@
//! Test support: a scripted [`MediaSender`] mock (no Telegram API involved)
//! and the stand-in Telegram API the real-`Bot` tests talk to.
//!
//! [`MediaSender`]: super::MediaSender
use super::*;
use parking_lot::Mutex;
use teloxide::types::InlineQueryResult;
/// One scripted outcome, consumed front-to-back; the last entry repeats
/// for further calls of the same method kind.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum Outcome {
GroupOk,
GroupErr,
AnimationOk,
AnimationErr,
CopyOk,
CopyErr,
/// An error from `send_message` (replies are fire-and-forget, so an
/// error is fine for tests).
MessageErr,
/// A successful `send_message`, returning message id [`MockSender::SENT_ID`].
MessageOk,
EditOk,
EditErr,
}
/// A stand-in for `api.telegram.org` for the tests that must drive a real
/// `Bot` — its request building, the per-chat limiter, the bot-wide budget
/// — which the scripted mock bypasses entirely. Records every call and
/// answers the smallest result each method needs.
pub(crate) mod fake_api {
use parking_lot::Mutex;
use std::sync::Arc;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
pub(crate) struct FakeApi {
url: url::Url,
calls: Arc<Mutex<Vec<(String, serde_json::Value)>>>,
server: tokio::task::JoinHandle<()>,
}
impl FakeApi {
/// Binds an ephemeral port and serves until dropped.
pub(crate) async fn start() -> FakeApi {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let calls = Arc::new(Mutex::new(Vec::new()));
let recorded = Arc::clone(&calls);
let server = tokio::spawn(async move {
while let Ok((mut socket, _)) = listener.accept().await {
let recorded = Arc::clone(&recorded);
tokio::spawn(async move {
let Some((method, body)) = read_request(&mut socket).await else {
return;
};
recorded.lock().push((method.clone(), body));
let payload = serde_json::json!({
"ok": true,
"result": canned_result(&method),
})
.to_string();
let response = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\n\
content-length: {}\r\nconnection: close\r\n\r\n{}",
payload.len(),
payload
);
let _ = socket.write_all(response.as_bytes()).await;
let _ = socket.flush().await;
});
}
});
FakeApi {
// Trailing slash: teloxide appends `bot<token>/<method>`.
url: url::Url::parse(&format!("http://{addr}/")).unwrap(),
calls,
server,
}
}
/// Where to point a `Bot`: `Bot::new(token).set_api_url(api.url())`.
pub(crate) fn url(&self) -> url::Url {
self.url.clone()
}
/// Method names in call order.
pub(crate) fn methods(&self) -> Vec<String> {
self.calls.lock().iter().map(|(m, _)| m.clone()).collect()
}
/// The JSON body of the first call to `method` (`Null` for a body
/// that is not JSON, i.e. a multipart upload).
pub(crate) fn body(&self, method: &str) -> serde_json::Value {
self.calls
.lock()
.iter()
.find(|(m, _)| m == method)
.map(|(_, body)| body.clone())
.unwrap_or(serde_json::Value::Null)
}
}
impl Drop for FakeApi {
fn drop(&mut self) {
self.server.abort();
}
}
/// The smallest result teloxide can deserialize for a method. The names
/// arrive as the payload type's own — `SendMediaGroup`, not
/// `sendMediaGroup`: teloxide builds the URL from that, and the Bot API
/// accepts the spelling.
fn canned_result(method: &str) -> serde_json::Value {
match method {
"CopyMessages" => serde_json::json!([{ "message_id": 11 }]),
"SendMediaGroup" => serde_json::json!([minimal_message()]),
"SendMessage" | "SendAnimation" | "EditMessageCaption" => minimal_message(),
_ => serde_json::Value::Bool(true),
}
}
fn minimal_message() -> serde_json::Value {
serde_json::json!({
"message_id": 1,
"date": 0,
"chat": { "id": 1, "type": "private" },
})
}
/// One HTTP/1.1 request: the head up to the blank line, then
/// `content-length` bytes of body — JSON for most methods, multipart
/// for the media ones (teloxide sends `SendMediaGroup` that way).
async fn read_request(socket: &mut TcpStream) -> Option<(String, serde_json::Value)> {
let mut buf = Vec::new();
let mut chunk = [0u8; 4096];
loop {
let n = socket.read(&mut chunk).await.ok()?;
if n == 0 {
return None;
}
buf.extend_from_slice(&chunk[..n]);
let Some(headers_end) = find(&buf, b"\r\n\r\n") else {
continue;
};
let head = String::from_utf8_lossy(&buf[..headers_end]).to_string();
let length: usize = head
.lines()
.find_map(|line| {
line.to_ascii_lowercase()
.strip_prefix("content-length:")
.and_then(|v| v.trim().parse().ok())
})
.unwrap_or(0);
let body_start = headers_end + 4;
while buf.len() < body_start + length {
let n = socket.read(&mut chunk).await.ok()?;
if n == 0 {
break;
}
buf.extend_from_slice(&chunk[..n]);
}
let method = head
.lines()
.next()
// `POST /bot<token>/<method>`
.and_then(|line| line.split(' ').nth(1))
.and_then(|path| path.rsplit('/').next())
.unwrap_or_default()
.to_string();
let body = parse_body(&buf[body_start..], &head);
return Some((method, body));
}
}
/// The request body as JSON: either the JSON body itself, or a
/// multipart form flattened into an object (each part's value parsed as
/// JSON when it is one, so `media` comes back as its array).
fn parse_body(body: &[u8], head: &str) -> serde_json::Value {
let content_type = head
.lines()
.find(|line| line.to_ascii_lowercase().starts_with("content-type:"))
.unwrap_or_default()
.to_ascii_lowercase();
let Some(boundary) = content_type
.split("boundary=")
.nth(1)
.map(|b| b.trim().trim_matches('"').to_string())
else {
return serde_json::from_slice(body).unwrap_or_default();
};
let text = String::from_utf8_lossy(body);
let mut fields = serde_json::Map::new();
for part in text.split(&format!("--{boundary}")).skip(1) {
let Some((part_head, value)) = part.split_once("\r\n\r\n") else {
continue;
};
let Some(name) = part_head
.split("name=\"")
.nth(1)
.and_then(|rest| rest.split('"').next())
else {
continue;
};
let value = value.trim_end_matches("\r\n");
fields.insert(
name.to_string(),
serde_json::from_str(value).unwrap_or_else(|_| value.into()),
);
}
serde_json::Value::Object(fields)
}
fn find(haystack: &[u8], needle: &[u8]) -> Option<usize> {
haystack
.windows(needle.len())
.position(|window| window == needle)
}
}
/// Replays a script and records what was sent, so tests can assert the
/// user-visible text a path produced.
pub(crate) struct MockSender {
script: Mutex<Vec<Outcome>>,
cursor: Mutex<usize>,
calls: Mutex<Vec<&'static str>>,
messages: Mutex<Vec<String>>,
captions: Mutex<Vec<String>>,
answers: Mutex<Vec<Option<String>>>,
/// `(chat, message, text)` of every text rewrite, in order.
edited_texts: Mutex<Vec<(i64, i64, String)>>,
/// What each `send_animation` handed Telegram: a URL or a file id as that
/// string, an upload as `attach://<id>`.
animation_files: Mutex<Vec<String>>,
/// What every `answer_inline_query` answered with, one entry per result:
/// `cached_photo:<file id>`, `photo:<url>`, and so on. An answer with no
/// results is recorded as an empty inner vec.
inline_answers: Mutex<Vec<Vec<String>>>,
/// Builds the error every `*Err` outcome returns (RequestError is not
/// cloneable, so the factory recreates it per call).
error: Box<dyn Fn() -> RequestError + Send + Sync>,
}
/// A one-string description of an inline result: the kind plus the file id it
/// is served from, or the URL it points Telegram at.
fn inline_result_tag(result: &InlineQueryResult) -> String {
match result {
InlineQueryResult::CachedPhoto(r) => format!("cached_photo:{}", r.photo_file_id.0),
InlineQueryResult::CachedVideo(r) => format!("cached_video:{}", r.video_file_id.0),
InlineQueryResult::CachedMpeg4Gif(r) => format!("cached_gif:{}", r.mpeg4_file_id.0),
InlineQueryResult::Photo(r) => format!("photo:{}", r.photo_url),
InlineQueryResult::Video(r) => format!("video:{}", r.video_url),
InlineQueryResult::Mpeg4Gif(r) => format!("gif:{}", r.mpeg4_url),
other => format!("{other:?}"),
}
}
/// The smallest `Message` the send paths accept, for the outcomes that must
/// report one (`send_animation` reads its id, and its media for the cache).
pub(crate) fn mock_message(id: i64) -> Message {
serde_json::from_value(serde_json::json!({
"message_id": id,
"date": 0,
"chat": { "id": 1, "type": "private" },
}))
.expect("a minimal message deserializes")
}
impl MockSender {
/// The message id a successful `send_message` reports.
pub(crate) const SENT_ID: i64 = 1;
pub(crate) fn scripted(
script: Vec<Outcome>,
error: impl Fn() -> RequestError + Send + Sync + 'static,
) -> Self {
MockSender {
script: Mutex::new(script),
cursor: Mutex::new(0),
calls: Mutex::new(Vec::new()),
messages: Mutex::new(Vec::new()),
captions: Mutex::new(Vec::new()),
answers: Mutex::new(Vec::new()),
edited_texts: Mutex::new(Vec::new()),
animation_files: Mutex::new(Vec::new()),
inline_answers: Mutex::new(Vec::new()),
error: Box::new(error),
}
}
/// Method names in call order (e.g. `["send_media_group",
/// "send_media_group"]` proves the fallback re-sent).
pub(crate) fn calls(&self) -> Vec<&'static str> {
self.calls.lock().clone()
}
/// Texts of the plain messages sent, in order.
pub(crate) fn messages(&self) -> Vec<String> {
self.messages.lock().clone()
}
/// Captions passed to `edit_message_caption`, in order.
pub(crate) fn captions(&self) -> Vec<String> {
self.captions.lock().clone()
}
/// Toast texts of the answered callback queries, in order.
pub(crate) fn answers(&self) -> Vec<Option<String>> {
self.answers.lock().clone()
}
/// `(chat, message, text)` of every `edit_message_text`, in order.
pub(crate) fn edited_texts(&self) -> Vec<(i64, i64, String)> {
self.edited_texts.lock().clone()
}
/// What every `send_animation` handed Telegram, in order.
pub(crate) fn animation_files(&self) -> Vec<String> {
self.animation_files.lock().clone()
}
/// What every `answer_inline_query` answered with, in call order.
pub(crate) fn inline_answers(&self) -> Vec<Vec<String>> {
self.inline_answers.lock().clone()
}
fn next(&self, kind: &'static str) -> Outcome {
self.calls.lock().push(kind);
let script = self.script.lock();
let mut cursor = self.cursor.lock();
if script.is_empty() {
panic!("mock script exhausted: {kind}");
}
let idx = (*cursor).min(script.len() - 1);
*cursor = idx + 1;
script[idx]
}
fn error(&self) -> RequestError {
(self.error)()
}
}
impl MediaSender for MockSender {
fn send_media_group(
&self,
_chat_id: ChatId,
_reply_to: MessageId,
items: Vec<InputMedia>,
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>> {
Box::pin(async move {
// Record the captions exactly as Telegram receives them (only
// the first item of a group carries one), so tests can assert
// what a recipient sees.
self.captions
.lock()
.extend(items.iter().filter_map(|item| match item {
InputMedia::Photo(photo) => photo.caption.clone(),
InputMedia::Video(video) => video.caption.clone(),
InputMedia::Animation(animation) => animation.caption.clone(),
_ => None,
}));
match self.next("send_media_group") {
Outcome::GroupOk => Ok(Vec::new()),
Outcome::GroupErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for send_media_group"),
}
})
}
fn send_animation<'a>(
&'a self,
_chat_id: ChatId,
_reply_to: MessageId,
_caption: &'a str,
_spoiler: bool,
file: InputFile,
) -> BoxFuture<'a, Result<Message, RequestError>> {
// Record what Telegram was handed: a URL or a file id serializes as
// that string, an upload as `attach://<id>`. Enough to tell a cached
// send (which must not re-upload) from a fresh one.
self.animation_files.lock().push(
serde_json::to_value(&file)
.ok()
.and_then(|value| value.as_str().map(str::to_string))
.unwrap_or_default(),
);
Box::pin(async move {
match self.next("send_animation") {
Outcome::AnimationOk => Ok(mock_message(MockSender::SENT_ID)),
Outcome::AnimationErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for send_animation"),
}
})
}
fn copy_messages(
&self,
_to: ChatId,
_from: ChatId,
_ids: Vec<MessageId>,
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>> {
Box::pin(async move {
match self.next("copy_messages") {
Outcome::CopyOk => Ok(vec![MessageId(1)]),
Outcome::CopyErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for copy_messages"),
}
})
}
fn send_message(
&self,
_chat_id: ChatId,
text: String,
_reply_to: Option<MessageId>,
_reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<i64, RequestError>> {
Box::pin(async move {
self.messages.lock().push(text);
match self.next("send_message") {
Outcome::MessageOk => Ok(MockSender::SENT_ID),
Outcome::MessageErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for send_message"),
}
})
}
fn send_html_message(
&self,
_chat_id: ChatId,
text: String,
_reply_to: Option<MessageId>,
) -> BoxFuture<'_, Result<i64, RequestError>> {
Box::pin(async move {
self.messages.lock().push(text);
match self.next("send_html_message") {
Outcome::MessageOk => Ok(MockSender::SENT_ID),
Outcome::MessageErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for send_html_message"),
}
})
}
fn answer_inline_query(
&self,
_id: InlineQueryId,
results: Vec<InlineQueryResult>,
cache_time: u32,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Records what the answer was made of, so a test can tell a cached
// (file-id) result from a URL one. Always succeeds: the debounce's
// release path is covered by `DebounceStates` directly.
assert_eq!(
cache_time, 300,
"the inline cache window is what the tests pin"
);
self.calls.lock().push("answer_inline_query");
self.inline_answers
.lock()
.push(results.iter().map(inline_result_tag).collect());
Box::pin(async move { Ok(()) })
}
fn answer_callback_query(
&self,
_id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Always succeeds: the toast is cosmetic, so the script stays
// focused on the outcomes a test cares about.
Box::pin(async move {
self.calls.lock().push("answer_callback_query");
self.answers.lock().push(text);
Ok(())
})
}
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Always succeeds: the only caller is the expiry sweep, which
// tolerates a failure (a prompt the user already deleted), so the
// script stays free for the call the test is about.
Box::pin(async move {
self.calls.lock().push("edit_message_text");
self.edited_texts
.lock()
.push((chat_id.0, message_id.0 as i64, text));
Ok(())
})
}
fn edit_message_caption(
&self,
_chat_id: ChatId,
_message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
self.captions.lock().push(caption);
match self.next("edit_message_caption") {
Outcome::EditOk => Ok(()),
Outcome::EditErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for edit_message_caption"),
}
})
}
fn delete_message(
&self,
_chat_id: ChatId,
_message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Deletion is fire-and-forget in every caller; always succeeds.
Box::pin(async move {
self.calls.lock().push("delete_message");
Ok(())
})
}
fn send_chat_action(
&self,
_chat_id: ChatId,
_action: ChatAction,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
self.calls.lock().push("send_chat_action");
Ok(())
})
}
}
+139 -79
View File
@@ -23,8 +23,10 @@ use tempfile::NamedTempFile;
pub const PHOTO_MAX_DIMENSION_SUM: u32 = 10000;
/// Resize target with a safety margin so rounding cannot cross the cap.
pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900;
/// Upload cap (bytes): files above this are not uploaded; the bot falls back
/// to a smaller media URL instead.
/// Photo upload cap (bytes): Telegram rejects a larger `sendPhoto`, so the bot
/// falls back to a smaller media URL instead. Videos and animations have their
/// own, larger cap — `send::upload::MAX_MEDIA_UPLOAD_BYTES` — and never become
/// photos.
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024;
/// Decode budget (bytes): a larger intermediate buffer is not worth the peak
/// memory; the photo degrades to the smaller URL instead.
@@ -32,9 +34,9 @@ pub(crate) const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
/// Cap for *downloading* a photo in the send fallback, kept separate from the
/// decode budget above: the whole body is buffered before it is processed, once
/// per download slot in flight, while the decode budget is about a single
/// buffer. Telegram's upload cap is 10 MiB, so a photo this large can only be
/// sent after a downscale that its reduced variant serves just as well — over
/// the cap the item degrades to the smaller URL
/// buffer. Telegram's *photo* upload cap is 10 MiB, so a photo this large can
/// only be sent after a downscale that its reduced variant serves just as
/// well — over the cap the item degrades to the smaller URL
/// (`FallbackError::MediaTooLarge`), it is never an error.
pub(crate) const MAX_PHOTO_DOWNLOAD_BYTES: u64 = 32 * 1024 * 1024;
@@ -63,14 +65,24 @@ fn decode_bytes(w: u32, h: u32, channels: usize) -> u64 {
(w as u64) * (h as u64) * channels as u64
}
/// Units to charge for `bytes`, clamped to the whole budget: an item must never
/// ask for more than exists, or it would wait for itself forever.
fn memory_units(bytes: u64) -> u32 {
bytes
.div_ceil(MEMORY_UNIT_BYTES)
.clamp(1, MEMORY_UNITS as u64) as u32
}
/// Conservative peak estimate for one photo preparation. The source bytes,
/// decoded pixels, any RGBA-to-RGB copy, resize output, and encoded output
/// can coexist briefly; charging only `w*h*channels` under-counts the real
/// process peak.
fn processing_peak_bytes(downloaded: u64, decode: u64) -> u64 {
downloaded
.saturating_add(decode)
.saturating_add(decode / 2)
.saturating_add(decode)
.saturating_add(MAX_UPLOAD_BYTES)
}
/// Reserves `bytes` of the preparation budget until the returned permit drops.
pub(crate) async fn reserve_memory(bytes: u64) -> tokio::sync::OwnedSemaphorePermit {
reserve(std::sync::Arc::clone(&MEMORY_BUDGET), bytes).await
@@ -88,30 +100,51 @@ async fn reserve(
.expect("memory budget semaphore closed")
}
/// The decode buffer a downloaded photo will allocate, from its header alone —
/// zero when it is already within Telegram's limits and is uploaded as-is, zero
/// for a format [`prepare_photo`] does not decode. Mirrors the early return and
/// the guard of the two branches below.
pub(crate) fn decode_budget_bytes(bytes: &[u8]) -> u64 {
if let Some((w, h, _depth, color)) = parse_png_header(bytes) {
if within_limits(w, h, bytes) {
return 0;
}
return decode_bytes(w, h, output_channels(color));
}
if let Some((w, h)) = jpeg_dims(bytes) {
if within_limits(w, h, bytes) {
return 0;
}
return decode_bytes(w, h, 3);
}
0
/// The processing decision for one downloaded photo, taken from its header
/// alone — the one place the within-limits test and the decode-size guard
/// live, so the memory reservation and the branch that acts on it cannot
/// drift.
enum PhotoPlan {
/// Already within Telegram's limits (dimension sum and upload cap): the
/// downloaded file is uploaded untouched, no decode buffer.
AsIs,
/// Needs processing: the decode buffer it will allocate, in bytes.
Decode(u64),
/// Processing would need a decode buffer over [`MAX_DECODE_BYTES`]: the
/// caller falls back to the item's smaller URL.
TooLarge,
}
/// Whether a photo is uploaded untouched (Telegram's dimension sum, and the
/// upload cap its bytes are compared against).
fn within_limits(w: u32, h: u32, bytes: &[u8]) -> bool {
w + h <= PHOTO_MAX_DIMENSION_SUM && bytes.len() as u64 <= MAX_UPLOAD_BYTES
/// [`PhotoPlan`] for a photo whose header said `w`×`h` in `channels` output
/// channels, `len` bytes long.
fn plan_photo(w: u32, h: u32, len: usize, channels: usize) -> PhotoPlan {
if (w as u64) + (h as u64) <= PHOTO_MAX_DIMENSION_SUM as u64 && len as u64 <= MAX_UPLOAD_BYTES {
return PhotoPlan::AsIs;
}
let bytes = decode_bytes(w, h, channels);
if bytes > MAX_DECODE_BYTES {
PhotoPlan::TooLarge
} else {
PhotoPlan::Decode(bytes)
}
}
/// The memory budget for one photo preparation, from its header alone. The
/// result includes the already-buffered download and the conservative decode,
/// transform, resize, and encoding peak; the caller holds that reservation
/// through the whole preparation.
pub(crate) fn prepare_budget_bytes(bytes: &[u8]) -> u64 {
let plan = if let Some((w, h, _depth, color)) = parse_png_header(bytes) {
plan_photo(w, h, bytes.len(), output_channels(color))
} else if let Some((w, h)) = jpeg_dims(bytes) {
plan_photo(w, h, bytes.len(), 3)
} else {
PhotoPlan::AsIs
};
match plan {
PhotoPlan::Decode(decode) => processing_peak_bytes(bytes.len() as u64, decode),
PhotoPlan::AsIs | PhotoPlan::TooLarge => bytes.len() as u64,
}
}
/// JPEG dimensions from the headers, without decoding any pixels.
@@ -172,31 +205,15 @@ pub fn prepare_photo(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, Str
}
}
/// Parses the PNG IHDR (bytes 8..26: signature + length + "IHDR" + width +
/// height + bit depth + color type).
/// The PNG's IHDR as the crate reads it (signature through the first IDAT):
/// width/height/depth/color decide the plan and the decode channels, without
/// decoding any pixels.
fn parse_png_header(bytes: &[u8]) -> Option<(u32, u32, png::BitDepth, png::ColorType)> {
if !bytes.starts_with(b"\x89PNG\r\n\x1a\n") || bytes.len() < 26 {
return None;
}
let w = u32::from_be_bytes(bytes.get(16..20)?.try_into().ok()?);
let h = u32::from_be_bytes(bytes.get(20..24)?.try_into().ok()?);
let depth = match *bytes.get(24)? {
1 => png::BitDepth::One,
2 => png::BitDepth::Two,
4 => png::BitDepth::Four,
8 => png::BitDepth::Eight,
16 => png::BitDepth::Sixteen,
_ => return None,
};
let color = match *bytes.get(25)? {
0 => png::ColorType::Grayscale,
2 => png::ColorType::Rgb,
3 => png::ColorType::Indexed,
4 => png::ColorType::GrayscaleAlpha,
6 => png::ColorType::Rgba,
_ => return None,
};
Some((w, h, depth, color))
let reader = png::Decoder::new(std::io::Cursor::new(bytes))
.read_info()
.ok()?;
let info = reader.info();
Some((info.width, info.height, info.bit_depth, info.color_type))
}
/// Output channels of a decoded frame for the given color type (post
@@ -302,7 +319,7 @@ fn write_temp(bytes: &[u8], ext: &str) -> Result<NamedTempFile, String> {
}
fn target_dims(w: u32, h: u32) -> (u32, u32) {
let scale = PHOTO_TARGET_DIMENSION_SUM as f64 / (w + h) as f64;
let scale = PHOTO_TARGET_DIMENSION_SUM as f64 / ((w as u64) + (h as u64)) as f64;
(
((w as f64 * scale).round() as u32).max(1),
((h as f64 * scale).round() as u32).max(1),
@@ -314,17 +331,16 @@ fn target_dims(w: u32, h: u32) -> (u32, u32) {
/// over the upload cap afterwards becomes JPEG.
fn prepare_png(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
let (w, h, _bit_depth, color_type) = parse_png_header(bytes).ok_or("invalid PNG header")?;
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
let channels = output_channels(color_type);
let plan = plan_photo(w, h, bytes.len(), channels);
if let PhotoPlan::AsIs = plan {
return Ok(PhotoPrep::Upload(file));
}
log::debug!(
"photo {w}x{h} ({_bit_depth:?} {color_type:?}, {} bytes) needs processing",
bytes.len()
);
let channels = output_channels(color_type);
if decode_bytes(w, h, channels) > MAX_DECODE_BYTES {
if let PhotoPlan::TooLarge = plan {
log::warn!("photo decode buffer exceeds the memory budget; falling back to smaller media");
return Ok(PhotoPrep::UseFallback);
}
@@ -360,7 +376,7 @@ fn prepare_png(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
};
let (mut w, mut h) = (out_w, out_h);
if w + h > PHOTO_MAX_DIMENSION_SUM {
if (w as u64) + (h as u64) > PHOTO_MAX_DIMENSION_SUM as u64 {
let (nw, nh) = target_dims(w, h);
pix = resize_pix(pix, w, h, nw, nh)?;
(w, h) = (nw, nh);
@@ -391,18 +407,18 @@ fn prepare_jpeg(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String>
.map_err(|e| format!("jpeg headers: {e}"))?;
let info = decoder.info().ok_or("jpeg info unavailable")?;
let (w, h) = (info.width as u32, info.height as u32);
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
let plan = plan_photo(w, h, bytes.len(), 3);
if let PhotoPlan::AsIs = plan {
return Ok(PhotoPrep::Upload(file));
}
if decode_bytes(w, h, 3) > MAX_DECODE_BYTES {
if let PhotoPlan::TooLarge = plan {
log::warn!("photo decode buffer exceeds the memory budget; falling back to smaller media");
return Ok(PhotoPrep::UseFallback);
}
let pixels = decoder.decode().map_err(|e| format!("jpeg decode: {e}"))?;
let mut pix = PixBuf::Rgb(pixels);
let (mut w, mut h) = (w, h);
if w + h > PHOTO_MAX_DIMENSION_SUM {
if (w as u64) + (h as u64) > PHOTO_MAX_DIMENSION_SUM as u64 {
let (nw, nh) = target_dims(w, h);
pix = resize_pix(pix, w, h, nw, nh)?;
(w, h) = (nw, nh);
@@ -426,16 +442,35 @@ mod tests {
bytes.extend(w.to_be_bytes());
bytes.extend(h.to_be_bytes());
bytes.extend([depth, color, 0, 0, 0]);
// A correct IHDR CRC plus an IDAT chunk header: `png::Decoder` verifies
// the CRC and `read_info` stops at the first IDAT — all the header
// read needs. The hand-rolled parser this fixture used to feed stopped
// four bytes earlier and checked neither.
bytes.extend(crc32(&bytes[12..]).to_be_bytes());
bytes.extend(0u32.to_be_bytes()); // IDAT payload length (never read)
bytes.extend(b"IDAT");
bytes
}
/// CRC-32 as PNG chunks use it (IEEE, reflected).
fn crc32(bytes: &[u8]) -> u32 {
let mut crc = !0u32;
for &b in bytes {
crc ^= b as u32;
for _ in 0..8 {
crc = (crc >> 1) ^ (0xEDB8_8320 & (crc & 1).wrapping_neg());
}
}
!crc
}
/// The budget is a *process-wide* memory bound: `PREP_SLOTS` (6) caps how
/// many photos are prepared at once, but six max-size photos would still
/// hold six decode buffers of up to 512 MiB each.
#[tokio::test]
async fn huge_decodes_cannot_overlap_but_do_run_alone() {
let budget = std::sync::Arc::new(tokio::sync::Semaphore::new(MEMORY_UNITS as usize));
let max_photo = MAX_DECODE_BYTES + MAX_PHOTO_DOWNLOAD_BYTES;
let max_photo = processing_peak_bytes(MAX_PHOTO_DOWNLOAD_BYTES, MAX_DECODE_BYTES);
// One max-size photo fits (clamped to the whole budget), so it can
// never wait for budget that cannot exist.
@@ -467,21 +502,29 @@ mod tests {
);
}
/// A 10-image album of ordinary photos must not serialize: they charge
/// their real (small) buffers, not a fixed heavyweight slot.
/// A 10-image album of ordinary photos must not serialize: the conservative
/// peak still allows several small/medium photos to run together.
#[tokio::test]
async fn ordinary_photos_share_the_budget() {
let budget = std::sync::Arc::new(tokio::sync::Semaphore::new(MEMORY_UNITS as usize));
// A 4 MiB photo that decodes to ~36 MiB (4000x3000 RGB).
let ordinary = 4 * 1024 * 1024 + 36 * 1024 * 1024;
// A 4 MiB photo that decodes to ~36 MiB (4000x3000 RGB): its peak
// costs two 64 MiB units, so four fit in the 512 MiB process budget.
let ordinary = processing_peak_bytes(4 * 1024 * 1024, 36 * 1024 * 1024);
assert_eq!(memory_units(ordinary), 2);
let mut held = Vec::new();
for i in 0..MEMORY_UNITS {
for i in 0..4 {
held.push(
tokio::time::timeout(Duration::from_millis(50), reserve(budget.clone(), ordinary))
.await
.unwrap_or_else(|_| panic!("ordinary photo {i} waited for budget")),
);
}
assert!(
tokio::time::timeout(Duration::from_millis(50), reserve(budget, ordinary))
.await
.is_err(),
"the budget should reject a fifth two-unit photo"
);
}
#[test]
@@ -499,19 +542,33 @@ mod tests {
);
}
/// What the reservation is charged is decided by the header, and it has to
/// agree with what the pipeline does: a photo uploaded as-is costs nothing,
/// one that gets processed costs its decoded buffer.
#[test]
fn decode_budget_follows_the_processing_decision() {
fn a_wrapping_dimension_sum_never_reads_as_within_limits() {
// u32::MAX + 2 wraps to 1: the pre-u64 sum advertised AsIs here and
// handed the absurd dimensions to Telegram untouched.
assert!(matches!(
plan_photo(u32::MAX, 2, 16, 3),
PhotoPlan::TooLarge
));
}
/// What the reservation is charged is decided by the header, and it has to
/// agree with what the pipeline does: a photo uploaded as-is costs only
/// its buffered bytes, while a processed one costs its conservative peak.
#[test]
fn prepare_budget_follows_the_processing_decision() {
// 9999x2 (sum 10001) is over the dimension cap → processed → charged.
let oversized = png_header(9999, 2, 8, 2); // 8-bit RGB
assert_eq!(decode_budget_bytes(&oversized), 9999 * 2 * 3);
assert_eq!(
prepare_budget_bytes(&oversized),
processing_peak_bytes(oversized.len() as u64, 9999 * 2 * 3)
);
// Inside the limits (dimensions *and* bytes) → uploaded as-is.
let small = png_header(100, 100, 8, 2);
assert_eq!(decode_budget_bytes(&small), 0);
// A format the pipeline does not decode costs nothing either.
assert_eq!(decode_budget_bytes(b"GIF89a not a photo"), 0);
assert_eq!(prepare_budget_bytes(&small), small.len() as u64);
// An unsupported format still keeps its already-buffered bytes alive.
let unsupported = b"GIF89a not a photo";
assert_eq!(prepare_budget_bytes(unsupported), unsupported.len() as u64);
// JPEG: 9999x2 is over the cap, so its RGB decode buffer is charged.
let (w, h) = (9999u16, 2u16);
@@ -520,7 +577,10 @@ mod tests {
jpeg_encoder::Encoder::new(&mut bytes, 90)
.encode(&rgb, w, h, jpeg_encoder::ColorType::Rgb)
.unwrap();
assert_eq!(decode_budget_bytes(&bytes), 9999 * 2 * 3);
assert_eq!(
prepare_budget_bytes(&bytes),
processing_peak_bytes(bytes.len() as u64, 9999 * 2 * 3)
);
}
#[test]
+215 -99
View File
@@ -179,6 +179,15 @@ impl PersistentTaskQueue {
loop {
let notified = sweep_notify.notified();
tokio::pin!(notified);
// Register before re-checking stop (same rule as the worker
// loop): a stop() landing between the previous iteration and
// here would wake nobody, and the sweep would sit out a full
// 30s tick — long enough for the shutdown timeout to treat the
// drain as stuck.
notified.as_mut().enable();
if sweep_stop.load(Ordering::Relaxed) {
break;
}
tokio::select! {
_ = &mut notified => {}
_ = interval.tick() => {}
@@ -238,9 +247,8 @@ impl PersistentTaskQueue {
/// `(id, payload)` of every row that can still run (`pending`,
/// `in_progress`). The startup repair reads these before the workers start:
/// with no worker running, no row can be leased while it writes.
pub async fn runnable_rows(&self) -> Vec<(String, String)> {
let result = self
.pool
pub async fn runnable_rows(&self) -> rusqlite::Result<Vec<(String, String)>> {
self.pool
.with_conn(|conn| {
let mut stmt = conn.prepare(
"SELECT id, payload FROM tasks WHERE status IN ('pending', 'in_progress') ORDER BY run_after",
@@ -248,14 +256,7 @@ impl PersistentTaskQueue {
let rows = stmt.query_map([], |row| Ok((row.get(0)?, row.get(1)?)))?;
rows.collect::<rusqlite::Result<Vec<(String, String)>>>()
})
.await;
match result {
Ok(rows) => rows,
Err(e) => {
log::error!("queue row scan failed: {e}");
Vec::new()
}
}
.await
}
/// Replaces a runnable row's payload and restarts its attempt budget: the
@@ -263,7 +264,7 @@ impl PersistentTaskQueue {
/// already spent do not carry over. Startup repair only — a worker's
/// write-back is lease-token guarded instead (`replace_payload` cannot race
/// one: it runs before any worker does).
pub async fn replace_payload(&self, id: &str, payload: &Value) -> bool {
pub async fn replace_payload(&self, id: &str, payload: &Value) -> rusqlite::Result<bool> {
let logged_id = id.to_string();
let (id, payload) = (id.to_string(), payload.to_string());
let result = self
@@ -276,46 +277,32 @@ impl PersistentTaskQueue {
)?;
Ok(affected == 1)
})
.await;
match result {
Ok(true) => true,
Ok(false) => {
log::warn!("queue: row {logged_id} vanished before its payload could be replaced");
false
}
Err(e) => {
log::error!("queue payload replace failed for {logged_id}: {e}");
false
}
.await?;
if !result {
log::warn!("queue: row {logged_id} vanished before its payload could be replaced");
}
Ok(result)
}
pub async fn pending_backlog(&self) -> Option<(i64, f64)> {
let result = self
.pool
.with_conn(|conn| {
let mut stmt = conn
.prepare("SELECT COUNT(*), MIN(run_after) FROM tasks WHERE status='pending'")?;
let mut rows = stmt.query([])?;
match rows.next()? {
Some(row) => {
let count = row.get::<_, i64>(0)?;
match row.get::<_, Option<f64>>(1)? {
Some(oldest) if count > 0 => Ok(Some((count, oldest))),
_ => Ok(None),
}
}
None => Ok(None),
}
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::error!("queue backlog query failed: {e}");
None
}
}
self.pool
.with_conn_or(
log::Level::Error,
"queue backlog query failed",
None,
|conn| {
let (count, oldest) = conn.query_row(
"SELECT COUNT(*), MIN(run_after) FROM tasks WHERE status='pending'",
[],
|row| Ok((row.get::<_, i64>(0)?, row.get::<_, Option<f64>>(1)?)),
)?;
Ok(match oldest {
Some(oldest) if count > 0 => Some((count, oldest)),
_ => None,
})
},
)
.await
}
}
@@ -383,11 +370,25 @@ impl QueueWorker {
async fn run_loop(self) {
while !self.stop.load(Ordering::Relaxed) {
match self.lease_next().await {
Ok(Some(row)) => self.process(row).await,
Ok(Some(row)) => {
let id = row.id.clone();
let token = row.lease_token.clone();
let payload: Value = serde_json::from_str(&row.payload).unwrap_or(Value::Null);
let attempts = row.attempts;
let worker = self.clone();
if let Err(e) = tokio::spawn(async move { worker.process(row).await }).await {
self.recover_panicked_row(&id, &token, payload, attempts, e)
.await;
}
}
Ok(None) => {
let wait_until = self.earliest_run_after().await;
let notified = self.notify.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if self.stop.load(Ordering::Relaxed) {
return;
}
match wait_until {
Some(until) => {
let delay = (until - now_f64()).max(0.0);
@@ -396,13 +397,9 @@ impl QueueWorker {
_ = tokio::time::sleep(Duration::from_secs_f64(delay)) => {}
}
}
None => {
notified.await;
}
None => notified.await,
}
}
// A lease failure while rows are due would otherwise loop
// with sleep(0) and hammer SQLite; back off briefly.
Err(e) => {
log::error!("queue lease failed: {e}");
tokio::time::sleep(Duration::from_secs(1)).await;
@@ -411,6 +408,36 @@ impl QueueWorker {
}
}
/// Turns a handler panic into one normal attempt outcome: retry with the
/// same parsed JSON payload while budget remains, then delete and dead-letter
/// it. Keeping the payload as a JSON value preserves cache/source fields
/// used by the dead-letter callback.
async fn recover_panicked_row(
&self,
id: &str,
lease_token: &str,
payload: Value,
attempts: i32,
error: tokio::task::JoinError,
) {
log::error!("queue task {id} panicked: {error}");
if attempts as u32 >= MAX_RETRIES {
let message = "queue worker panicked".to_string();
if self.delete_row(id, lease_token).await {
(self.dead_letter)(payload, message).await;
}
} else {
self.reschedule(
id,
lease_token,
payload,
scaled_retry_delay(1.0, attempts),
attempts + 1,
)
.await;
}
}
/// Leases the oldest due row (sets it `in_progress` with a lock TTL).
/// Errors are surfaced so the caller can back off instead of spinning.
async fn lease_next(&self) -> Result<Option<LeasedRow>, rusqlite::Error> {
@@ -458,25 +485,20 @@ impl QueueWorker {
}
async fn earliest_run_after(&self) -> Option<f64> {
let result = self
.pool
.with_conn(|conn| {
let mut stmt =
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
let mut rows = stmt.query([])?;
match rows.next()? {
Some(row) => Ok(row.get::<_, Option<f64>>(0)?),
None => Ok(None),
}
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::error!("queue timing query failed: {e}");
None
}
}
self.pool
.with_conn_or(
log::Level::Error,
"queue timing query failed",
None,
|conn| {
conn.query_row(
"SELECT MIN(run_after) FROM tasks WHERE status='pending'",
[],
|row| row.get::<_, Option<f64>>(0),
)
},
)
.await
}
/// Processes one leased row, keeping the lease alive while the handler
@@ -489,8 +511,9 @@ impl QueueWorker {
Ok(value) => value,
Err(e) => {
log::error!("queue: unparseable payload for {}: {e}", row.id);
self.delete_row(&row.id, &row.lease_token).await;
(self.dead_letter)(Value::Null, format!("invalid stored payload: {e}")).await;
if self.delete_row(&row.id, &row.lease_token).await {
(self.dead_letter)(Value::Null, format!("invalid stored payload: {e}")).await;
}
return;
}
};
@@ -539,8 +562,9 @@ impl QueueWorker {
row.id,
row.attempts + 1
);
self.delete_row(&row.id, &row.lease_token).await;
(self.dead_letter)(payload, message).await;
if self.delete_row(&row.id, &row.lease_token).await {
(self.dead_letter)(payload, message).await;
}
} else {
let delay = scaled_retry_delay(delay_seconds, row.attempts);
log::debug!(
@@ -554,8 +578,9 @@ impl QueueWorker {
}
Err(QueueError::Permanent { message, payload }) => {
log::error!("dead-lettering {} {fields}: {message}", row.id);
self.delete_row(&row.id, &row.lease_token).await;
(self.dead_letter)(payload, message).await;
if self.delete_row(&row.id, &row.lease_token).await {
(self.dead_letter)(payload, message).await;
}
}
}
}
@@ -604,7 +629,10 @@ impl QueueWorker {
// future here stops this attempt instead of racing the
// new holder through the same send.
Ok(_) => return Err(LeaseLost),
Err(e) => log::error!("queue lease heartbeat failed: {e}"),
Err(e) => {
log::error!("queue lease heartbeat failed: {e}; abandoning attempt");
return Err(LeaseLost);
}
}
}
}
@@ -618,19 +646,18 @@ impl QueueWorker {
/// (a busy/contended DB is the usual cause and clears), and if the DB still
/// refuses, the row is marked `done` — a status neither the lease query
/// (`pending`) nor the sweep (`in_progress`) looks at — so a task that
/// already ran can never be re-leased. Both writes failing is logged at
/// error level with the row id, since that is the one case where a
/// duplicate send stays possible.
async fn delete_row(&self, id: &str, lease_token: &str) {
/// already ran can never be re-leased. Returns `false` when the token is
/// no longer ours; callers must not run dead-letter side effects then.
async fn delete_row(&self, id: &str, lease_token: &str) -> bool {
for attempt in 0..TERMINAL_WRITE_ATTEMPTS {
match self.try_delete_row(id, lease_token).await {
Ok(true) => return,
Ok(true) => return true,
// The row is not ours any more (re-leased while we worked):
// leaving it alone *is* the clean outcome — retrying or
// tombstoning here would erase the new holder's work.
Ok(false) => {
log::warn!("queue: row {id} was re-leased; not deleting it");
return;
return false;
}
Err(e) => {
log::error!("queue delete failed (attempt {}): {e}", attempt + 1);
@@ -639,12 +666,21 @@ impl QueueWorker {
}
}
match mark_done(&self.pool, id, lease_token).await {
Ok(true) => log::warn!("queue: row {id} marked done instead of deleted"),
Ok(false) => log::warn!("queue: row {id} was re-leased; nothing to tombstone"),
Err(e) => log::error!(
"queue: row {id} could not be deleted or marked done ({e}); \
the expiry sweep may run this finished task again"
),
Ok(true) => {
log::warn!("queue: row {id} marked done instead of deleted");
true
}
Ok(false) => {
log::warn!("queue: row {id} was re-leased; nothing to tombstone");
false
}
Err(e) => {
log::error!(
"queue: row {id} could not be deleted or marked done ({e}); \
the expiry sweep may run this finished task again"
);
false
}
}
}
@@ -864,6 +900,54 @@ mod tests {
queue.stop().await;
}
#[tokio::test]
async fn a_re_leased_row_cannot_dead_letter_the_old_attempt() {
let (queue, _dir) = new_queue().await;
queue
.enqueue(serde_json::json!({"a": 1}), now_f64())
.await
.unwrap();
let id: String = queue
.pool
.with_conn(|conn| conn.query_row("SELECT id FROM tasks", [], |r| r.get(0)))
.await
.unwrap();
set_lease(&queue, &id, "old-holder").await;
let dead_calls = Arc::new(AtomicUsize::new(0));
let worker = QueueWorker {
pool: std::sync::Arc::clone(&queue.pool),
notify: Arc::new(Notify::new()),
stop: Arc::new(AtomicBool::new(false)),
handler: Arc::new(|_payload| {
Box::pin(async {
Err(QueueError::Permanent {
message: "stale failure".into(),
payload: serde_json::json!({"a": 1}),
})
})
}),
dead_letter: Arc::new({
let dead_calls = Arc::clone(&dead_calls);
move |_payload, _message| {
let dead_calls = Arc::clone(&dead_calls);
Box::pin(async move {
dead_calls.fetch_add(1, AtomicOrdering::SeqCst);
})
}
}),
};
set_lease(&queue, &id, "new-holder").await;
worker
.process(LeasedRow {
id,
payload: "{\"a\":1}".into(),
attempts: 0,
lease_token: "old-holder".into(),
})
.await;
assert_eq!(dead_calls.load(AtomicOrdering::SeqCst), 0);
}
#[tokio::test]
async fn pending_backlog_counts_only_unleased_rows() {
let (queue, _dir) = new_queue().await;
@@ -878,9 +962,6 @@ mod tests {
.enqueue(serde_json::json!({"chat_id": 2}), due + 600.0)
.await
.unwrap();
// Hold the first row in the handler so it is leased, not pending: a
// health line that reported work already in flight as backlog would be
// lying about the queue.
let release = Arc::new(tokio::sync::Notify::new());
let held = release.clone();
queue
@@ -904,12 +985,46 @@ mod tests {
let (_, oldest) = queue.pending_backlog().await.unwrap();
assert!(
(oldest - (due + 600.0)).abs() < 1.0,
"oldest is the earliest run_after: {oldest}"
"oldest is earliest: {oldest}"
);
release.notify_one();
queue.stop().await;
}
#[tokio::test]
async fn a_panicking_handler_is_dead_lettered_after_retry_budget() {
let (queue, _dir) = new_queue().await;
let calls = Arc::new(AtomicUsize::new(0));
let dead_calls = Arc::new(AtomicUsize::new(0));
let c = Arc::clone(&calls);
let d = Arc::clone(&dead_calls);
queue
.start(
move |_payload| {
c.fetch_add(1, AtomicOrdering::SeqCst);
Box::pin(async { panic!("handler panic") })
},
move |_payload, message| {
assert!(message.contains("panicked"), "{message}");
d.fetch_add(1, AtomicOrdering::SeqCst);
Box::pin(async {})
},
)
.await;
queue
.enqueue(serde_json::json!({"panic": true}), now_f64())
.await
.unwrap();
tokio::time::sleep(Duration::from_millis(400)).await;
assert_eq!(dead_calls.load(AtomicOrdering::SeqCst), 0);
tokio::time::sleep(Duration::from_millis(3_200)).await;
assert_eq!(calls.load(AtomicOrdering::SeqCst), MAX_RETRIES as usize + 1);
assert_eq!(dead_calls.load(AtomicOrdering::SeqCst), 1);
assert_eq!(queue.pending_backlog().await, None);
queue.stop().await;
}
#[tokio::test]
async fn retryable_reschedules_then_dead_letters() {
let (queue, _dir) = new_queue().await;
@@ -956,7 +1071,7 @@ mod tests {
.enqueue(serde_json::json!({"s": 1}), now_f64())
.await
.unwrap();
let rows = queue.runnable_rows().await;
let rows = queue.runnable_rows().await.unwrap();
assert_eq!(rows.len(), 1);
let (id, payload) = rows[0].clone();
assert_eq!(payload, "{\"s\":1}");
@@ -978,19 +1093,20 @@ mod tests {
queue
.replace_payload(&id, &serde_json::json!({"s": 2}))
.await
.unwrap()
);
let rows = queue.runnable_rows().await;
let rows = queue.runnable_rows().await.unwrap();
assert_eq!(rows[0].1, "{\"s\":2}");
assert_eq!(
queue.pending_backlog().await.map(|(n, _)| n),
Some(1),
"a repaired row is pending work again"
);
// A row that is gone (or done) is not rewritten.
assert!(
!queue
.replace_payload("task_missing", &serde_json::json!({}))
.await
.unwrap()
);
}
+10 -7
View File
@@ -91,21 +91,24 @@ impl TokenBucket {
tokio::time::sleep(Duration::from_secs_f64(wait)).await;
}
/// Current balance, for the tests that assert a call site charged the
/// bucket (a charge is otherwise only observable as a delay).
#[cfg(test)]
pub(crate) fn tokens(&self) -> f64 {
/// Current balance, refilled to now.
fn balance(&self) -> f64 {
let mut state = self.state.lock();
self.refill(&mut state);
state.tokens
}
/// Current balance, for the tests that assert a call site charged the
/// bucket (a charge is otherwise only observable as a delay).
#[cfg(test)]
pub(crate) fn tokens(&self) -> f64 {
self.balance()
}
/// True when the bucket has refilled to capacity: no debt outstanding, so
/// the chat has not sent anything recently.
fn is_idle(&self) -> bool {
let mut state = self.state.lock();
self.refill(&mut state);
state.tokens >= self.capacity
self.balance() >= self.capacity
}
}
+161
View File
@@ -0,0 +1,161 @@
//! The Telegram error policy: which failures the send paths retry, which are
//! permanent, and which the download-and-reupload fallback owns. A status a
//! *site* answers with is classified in `x_media::site`; this is the Bot API's
//! side of the same question.
use super::upload::FallbackError;
use super::{Task, retry_delay_seconds};
use teloxide::{ApiError, RequestError};
/// Telegram's servers failed to fetch a media URL (hotlink protection etc.):
/// these errors are handled by the download-and-reupload fallback, NOT by a
/// queue retry (resending the URL cannot succeed).
pub fn is_media_fetch_failure(e: &ApiError) -> bool {
const MARKERS: [&str; 7] = [
"webpage_media_empty",
"media_empty",
"empty_web_media",
"webpage_curl_failed",
"timeout",
// Oversized photos (width + height > 10000 px) are rejected on URL
// sends too; route them to the download-and-resize fallback.
"photo_invalid_dimensions",
// Telegram refused to fetch the URL it was handed. Single-media URL
// sends answer with this one (the media-group verbs use the
// `webpage_*`/`media_empty` markers above), and it is exactly the
// case the download-and-reupload fallback exists for.
"failed to get http url content",
];
let description = e.to_string().to_lowercase();
MARKERS.iter().any(|marker| description.contains(marker))
}
/// Telegram reported the media file as too large (HTTP 413 on multipart
/// upload, or a "too large" message for URL-fetched media). These errors are
/// handled by the size-check fallback (use a smaller media URL), NOT by a
/// queue retry.
pub fn is_size_error(e: &ApiError) -> bool {
if matches!(e, ApiError::RequestEntityTooLarge) {
return true;
}
let description = e.to_string().to_lowercase();
["too large", "too big"]
.iter()
.any(|marker| description.contains(marker))
}
/// Task-free classification of a Telegram request error. The callers attach
/// the (updated) task when building a [`SendError`].
pub enum Classification {
Retryable {
delay_seconds: f64,
},
Permanent {
message: String,
},
/// Handled by the download fallback, not a queue retry.
MediaFetchFailure,
}
pub fn classify_request_error(e: &RequestError) -> Classification {
match e {
RequestError::RetryAfter(seconds) => Classification::Retryable {
delay_seconds: seconds.seconds() as f64,
},
RequestError::Network(_) => Classification::Retryable {
delay_seconds: retry_delay_seconds(0),
},
// A 5xx from the API — or from a proxy in front of it — is transient.
// teloxide only sleeps 10s on a server error and then parses whatever
// body came back, so by the time we see the error the HTTP status is
// gone: a JSON 5xx body arrives as an unknown description, an HTML
// error page as `InvalidJson`. Both used to be Permanent, which
// dead-lettered a post over a Telegram-side blip.
RequestError::Api(api) if is_server_error_text(&api.to_string()) => {
Classification::Retryable {
delay_seconds: retry_delay_seconds(0),
}
}
RequestError::Api(api) if is_media_fetch_failure(api) => Classification::MediaFetchFailure,
RequestError::Api(api) => Classification::Permanent {
message: api.to_string(),
},
// An unparsable body can only come from something that is not the Bot
// API (which always answers JSON): a 5xx/error page from an
// intermediary, cut off mid-response. A JSON body that merely does not
// match the expected type cannot be fixed by retrying, so that case
// stays permanent.
RequestError::InvalidJson { raw, .. } if !raw.trim_start().starts_with('{') => {
Classification::Retryable {
delay_seconds: retry_delay_seconds(0),
}
}
RequestError::MigrateToChatId(_)
| RequestError::InvalidJson { .. }
| RequestError::Io(_) => Classification::Permanent {
message: e.to_string(),
},
}
}
/// Descriptions a 5xx carries when its body *is* JSON (teloxide keeps only the
/// description text, never the status code). Matched like the media-fetch
/// markers below; anything unmatched stays permanent, so a new permanent API
/// error is not retried just because it is unfamiliar.
fn is_server_error_text(description: &str) -> bool {
const MARKERS: [&str; 4] = [
"server error",
"bad gateway",
"gateway timeout",
"service unavailable",
];
let description = description.to_lowercase();
MARKERS.iter().any(|marker| description.contains(marker))
}
/// Task boxed to keep the error size within `result_large_err` limits.
#[derive(Debug)]
pub enum SendError {
Retryable { delay_seconds: f64, task: Box<Task> },
Permanent { message: String, task: Box<Task> },
}
pub(crate) fn classify_to_send_error(
e: &RequestError,
task: Task,
fetch_failure_label: &str,
) -> SendError {
match classify_request_error(e) {
Classification::Retryable { delay_seconds } => SendError::Retryable {
delay_seconds,
task: Box::new(task),
},
Classification::Permanent { message } => SendError::Permanent {
message,
task: Box::new(task),
},
Classification::MediaFetchFailure => SendError::Permanent {
message: fetch_failure_label.into(),
task: Box::new(task),
},
}
}
impl SendError {
/// Attaches the (updated) task to a task-free [`FallbackError`] from the
/// download/upload pipeline. [`FallbackError::MediaTooLarge`] never
/// escapes the pipeline (it is handled by falling back to the smaller
/// URL), so it is unreachable here.
pub(super) fn from_fallback(f: FallbackError, task: Task) -> SendError {
match f {
FallbackError::Retryable { delay_seconds } => SendError::Retryable {
delay_seconds,
task: Box::new(task),
},
FallbackError::Permanent { message } => SendError::Permanent {
message,
task: Box::new(task),
},
FallbackError::MediaTooLarge => unreachable!("handled inside the upload fallback"),
}
}
}
+61 -77
View File
@@ -2,20 +2,16 @@
//! URL / local path), the per-kind `InputMedia` builders and the media-group
//! assembly with its caption rule.
use super::MediaItemPayload;
use super::{MediaItemPayload, MediaRef};
use teloxide::types::{
InputFile, InputMedia, InputMediaAnimation, InputMediaPhoto, InputMediaVideo, ParseMode,
};
fn parse_media_url(s: &str) -> Result<url::Url, String> {
url::Url::parse(s).map_err(|e| format!("invalid media URL: {e}"))
}
/// The item's media string, whether it is a URL/path or a file id — callers
/// that need the distinction match on [`MediaRef`] themselves.
pub(super) fn item_url(item: &MediaItemPayload) -> &str {
match item {
MediaItemPayload::Photo { media, .. }
| MediaItemPayload::Video { media, .. }
| MediaItemPayload::Animation { media, .. } => media,
match item.media_ref() {
MediaRef::Source(media) | MediaRef::FileId(media) => media,
}
}
@@ -23,7 +19,8 @@ pub(super) fn item_url(item: &MediaItemPayload) -> &str {
/// (e.g. a locally encoded ugoira MP4) is uploaded directly.
pub(super) fn input_file_for(media: &str) -> Result<InputFile, String> {
if media.starts_with("http://") || media.starts_with("https://") {
Ok(InputFile::url(parse_media_url(media)?))
let url = url::Url::parse(media).map_err(|e| format!("invalid media URL: {e}"))?;
Ok(InputFile::url(url))
} else if !std::path::Path::new(media).exists() {
// A retried task may reference a temp file the original send's
// TempDir already cleaned up; fail fast and permanent instead of
@@ -38,59 +35,64 @@ impl MediaItemPayload {
/// The input for a send: a cached file id goes out as `InputFile::file_id`
/// (no fetch, no upload), URLs go to Telegram, anything else is a local
/// path (transient upload fallback).
fn input_file(&self) -> Result<InputFile, String> {
match self {
MediaItemPayload::Photo {
media,
file_id: true,
..
}
| MediaItemPayload::Video {
media,
file_id: true,
..
}
| MediaItemPayload::Animation {
media,
file_id: true,
..
} => Ok(InputFile::file_id(media.clone().into())),
_ => input_file_for(item_url(self)),
pub(super) fn input_file(&self) -> Result<InputFile, String> {
match self.media_ref() {
MediaRef::FileId(id) => Ok(InputFile::file_id(id.clone().into())),
MediaRef::Source(media) => input_file_for(media),
}
}
}
pub(super) fn photo_media(file: InputFile, caption: Option<&str>, spoiler: bool) -> InputMedia {
let mut photo = InputMediaPhoto::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
photo = photo.caption(caption);
/// Builds one media-group item around an already-selected file: the per-kind
/// `InputMedia` (same spoiler/caption handling) plus the video's thumbnail,
/// which Telegram takes as a separate upload/URL. The one place that dispatch
/// is written; callers only choose the `InputFile`.
pub(super) fn media_from(
item: &MediaItemPayload,
file: InputFile,
caption: Option<&str>,
) -> Result<InputMedia, String> {
let media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
let mut media = InputMediaPhoto::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
media = media.caption(caption);
}
if *has_spoiler {
media = media.spoiler();
}
InputMedia::Photo(media)
}
MediaItemPayload::Video { has_spoiler, .. } => {
let mut media = InputMediaVideo::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
media = media.caption(caption);
}
if *has_spoiler {
media = media.spoiler();
}
InputMedia::Video(media)
}
MediaItemPayload::Animation { has_spoiler, .. } => {
let mut media = InputMediaAnimation::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
media = media.caption(caption);
}
if *has_spoiler {
media = media.spoiler();
}
InputMedia::Animation(media)
}
};
// The thumbnail comes off the item itself — every caller passed exactly
// that, and only a video uses it (Telegram takes it as a separate
// upload/URL).
match (item.thumbnail_url(), media) {
(Some(thumb), InputMedia::Video(video)) => {
Ok(InputMedia::Video(video.thumbnail(input_file_for(thumb)?)))
}
(_, media) => Ok(media),
}
if spoiler {
photo = photo.spoiler();
}
InputMedia::Photo(photo)
}
pub(super) fn video_media(file: InputFile, caption: Option<&str>, spoiler: bool) -> InputMedia {
let mut video = InputMediaVideo::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
video = video.caption(caption);
}
if spoiler {
video = video.spoiler();
}
InputMedia::Video(video)
}
pub(super) fn animation_media(file: InputFile, caption: Option<&str>, spoiler: bool) -> InputMedia {
let mut animation = InputMediaAnimation::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
animation = animation.caption(caption);
}
if spoiler {
animation = animation.spoiler();
}
InputMedia::Animation(animation)
}
/// Builds a media group from payloads; only the first item of the batch gets
@@ -104,25 +106,7 @@ pub(super) fn build_media_group(
.enumerate()
.map(|(i, item)| {
let item_caption = if i == 0 { caption } else { None };
Ok(match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(item.input_file()?, item_caption, *has_spoiler)
}
MediaItemPayload::Video {
has_spoiler,
thumbnail,
..
} => {
let mut video = video_media(item.input_file()?, item_caption, *has_spoiler);
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut video) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
video
}
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(item.input_file()?, item_caption, *has_spoiler)
}
})
media_from(item, item.input_file()?, item_caption)
})
.collect()
}
File diff suppressed because it is too large Load Diff
+66 -25
View File
@@ -13,7 +13,10 @@ use crate::queue::{PersistentTaskQueue, QueueError};
use crate::state::EditMessage;
use std::collections::HashMap;
use std::sync::LazyLock;
use teloxide::types::{ChatId, InlineKeyboardButton, InlineKeyboardMarkup, Message, MessageId};
use teloxide::types::{
ChatId, InlineKeyboardButton, InlineKeyboardButtonKind, InlineKeyboardMarkup, Message,
MessageId,
};
/// Persists a successful send under the post's cache key. Skips a send that was
/// served from the cache — its entry already holds the file ids the next repeat
@@ -50,7 +53,7 @@ pub(super) async fn cache_animation_send(
vec![CachedMedia {
kind: CachedMediaKind::Animation,
file_id,
url: source_url.to_string(),
url: super::replayable_cache_url(source_url),
}],
)
.await;
@@ -122,24 +125,29 @@ async fn invalidate_cache(ctx: &AppContext<'_>, task: &Task) {
/// that drop, so without this the local file would be gone by the time the
/// retry sends it. `Arc` because one fetch can serve several tasks (a
/// concurrent duplicate of the same link shares it): each holder keeps the
/// directory alive until its own task settles. Entries are removed when the
/// directory alive until its own task settles. Its entry is removed when that
/// task settles (see [`release_keep_alive`]).
pub(crate) static KEEP_ALIVE: LazyLock<parking_lot::Mutex<Vec<std::sync::Arc<tempfile::TempDir>>>> =
LazyLock::new(|| parking_lot::Mutex::new(Vec::new()));
/// Drops the keep-alive temp dirs holding media referenced by `task` (matched
/// by path prefix). Called once a task settles — sent or permanently failed —
/// so retry-only temp files do not leak; retryable tasks keep them alive.
/// Drops the keep-alive reference this task's pipeline pushed (one entry,
/// matched by path prefix). Called once a task settles — sent or permanently
/// failed — so retry-only temp files do not leak; retryable tasks keep theirs.
/// Exactly one entry goes per call: a shared fetch pushes one per pipeline, so
/// clearing every holder would delete the directory out from under a
/// concurrent duplicate's queued retry.
pub(crate) fn release_keep_alive(task: &Task) {
let paths = task.local_media_paths();
if paths.is_empty() {
return;
}
let mut alive = KEEP_ALIVE.lock();
alive.retain(|dir| {
let dir_path = dir.path();
!paths.iter().any(|p| p.starts_with(dir_path))
});
if let Some(index) = alive
.iter()
.position(|dir| paths.iter().any(|p| p.starts_with(dir.path())))
{
alive.remove(index);
}
}
/// The edit-before-forward prompt's text. It names both controls and the TTL,
@@ -177,13 +185,19 @@ fn coarsest_unit(ttl: std::time::Duration) -> String {
pub(super) const TEMPLATE_BUTTONS_PER_ROW: usize = 3;
/// Hard cap on template buttons; the prompt text names the ones not shown.
pub(super) const MAX_TEMPLATE_BUTTONS: usize = 60;
const MAX_CALLBACK_DATA_BYTES: usize = 64;
const TEMPLATE_CALLBACK_PREFIX: &str = "template|";
/// Template buttons ([`TEMPLATE_BUTTONS_PER_ROW`] per row, at most
/// [`MAX_TEMPLATE_BUTTONS`]), then the confirm/skip pair. Sorted by name: the
/// templates live in a `HashMap`, so an unsorted walk would reshuffle the
/// buttons between prompts.
/// buttons between prompts. A name that cannot fit Telegram's callback-data
/// limit is omitted; legacy/imported state cannot poison the whole prompt.
pub(super) fn build_edit_markup(templates: &HashMap<String, String>) -> InlineKeyboardMarkup {
let mut names: Vec<&String> = templates.keys().collect();
let mut names: Vec<&String> = templates
.keys()
.filter(|name| TEMPLATE_CALLBACK_PREFIX.len() + name.len() <= MAX_CALLBACK_DATA_BYTES)
.collect();
names.sort();
let shown = names.len().min(MAX_TEMPLATE_BUTTONS);
let mut rows = Vec::with_capacity(shown / TEMPLATE_BUTTONS_PER_ROW + 2);
@@ -192,14 +206,14 @@ pub(super) fn build_edit_markup(templates: &HashMap<String, String>) -> InlineKe
chunk
.iter()
.map(|name| {
InlineKeyboardButton::callback(name.as_str(), format!("template|{name}"))
InlineKeyboardButton::callback(
name.as_str(),
format!("{TEMPLATE_CALLBACK_PREFIX}{name}"),
)
})
.collect(),
);
}
// Skip exists because the prompt holds the forward hostage until Confirm:
// without it the only escape was deleting the message and waiting out the
// TTL for a forward that then never happens.
rows.push(vec![
InlineKeyboardButton::callback("↩️ Confirm", "forward"),
InlineKeyboardButton::callback("🛑 Skip", "skip"),
@@ -207,11 +221,6 @@ pub(super) fn build_edit_markup(templates: &HashMap<String, String>) -> InlineKe
InlineKeyboardMarkup::new(rows)
}
/// How many templates the markup could not fit, for the prompt text.
pub(super) fn hidden_template_count(templates: &HashMap<String, String>) -> usize {
templates.len().saturating_sub(MAX_TEMPLATE_BUTTONS)
}
/// Notifies a chat about a dead-lettered task (skips when `notify_chat_id` is
/// absent).
pub(crate) async fn notify_failure(
@@ -277,7 +286,15 @@ pub(crate) async fn post_send_actions(ctx: &AppContext<'_>, task: &Task, message
let templates = ctx.chat_store.get(chat_id).await.template;
let keyboard = build_edit_markup(&templates);
let mut text = edit_prompt_text(ctx.config.edit_message_ttl);
let hidden = hidden_template_count(&templates);
let shown = keyboard
.inline_keyboard
.iter()
.flatten()
.filter(|button| {
matches!(&button.kind, InlineKeyboardButtonKind::CallbackData(data) if data.starts_with(TEMPLATE_CALLBACK_PREFIX))
})
.count();
let hidden = templates.len().saturating_sub(shown);
if hidden > 0 {
// The keyboard is capped; say so instead of silently hiding them.
text.push_str(&format!(
@@ -301,7 +318,8 @@ pub(crate) async fn post_send_actions(ctx: &AppContext<'_>, task: &Task, message
log_key(&source_url)
);
let source_url = source_url.clone();
ctx.chat_store
let saved = match ctx
.chat_store
.update(chat_id, move |data| {
data.edit_message.insert(
prompt_id,
@@ -314,7 +332,25 @@ pub(crate) async fn post_send_actions(ctx: &AppContext<'_>, task: &Task, message
},
);
})
.await
{
Ok((_, true)) => true,
Ok((_, false)) | Err(()) => false,
};
if !saved {
log::error!("edit prompt {prompt_id} could not be persisted; removing it");
let _ = ctx
.sender
.delete_message(ChatId(chat_id), MessageId(prompt_id as i32))
.await;
notify_failure(
ctx.sender,
notify_chat_id,
notify_message_id,
"Could not save the edit-before-forward prompt — nothing was forwarded.",
)
.await;
}
}
Err(e) => {
log::error!("failed to send edit prompt: {e}");
@@ -344,6 +380,7 @@ pub(crate) async fn post_send_actions(ctx: &AppContext<'_>, task: &Task, message
from_chat_id: chat_id,
to_chat_id: channel_id,
message_ids,
forward_offset: 0,
notify_chat_id,
notify_message_id,
};
@@ -426,7 +463,10 @@ pub(crate) async fn handle_task(
});
}
Err(SendError::Permanent { message, task }) => {
settle_task(ctx, &task, Settled::Failed).await;
// The queue dead-letters this payload into
// `dead_letter_notify`, which settles the task — settling
// here as well would release a shared keep-alive
// directory twice.
return Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
@@ -454,7 +494,8 @@ pub(crate) async fn handle_task(
payload: serde_json::to_value(task).expect("task serializes"),
}),
Err(SendError::Permanent { message, task }) => {
settle_task(ctx, &task, Settled::Failed).await;
// Settled by `dead_letter_notify`, which the queue invokes for
// this payload.
Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
+167 -79
View File
@@ -2,10 +2,12 @@
//! itself (hotlink protection), the bot downloads the file, shrinks photos
//! that exceed Telegram's limits and uploads the batch via multipart.
use super::input_media::{animation_media, input_file_for, item_url, photo_media, video_media};
use super::{MediaItemPayload, SendError, Task, classify_to_send_error, retry_delay_seconds};
use super::input_media::{input_file_for, item_url, media_from};
use super::{
MediaItemPayload, MediaRef, SendError, Task, classify_to_send_error, retry_delay_seconds,
};
use crate::media_sender::MediaSender;
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
use crate::photo::{self, PhotoPrep};
use std::sync::LazyLock;
use teloxide::prelude::*;
use teloxide::types::{ChatId, InputFile, InputMedia, MessageId};
@@ -23,6 +25,26 @@ const PREP_CONCURRENCY: usize = 6;
static PREP_SLOTS: LazyLock<tokio::sync::Semaphore> =
LazyLock::new(|| tokio::sync::Semaphore::new(PREP_CONCURRENCY));
/// Telegram's multipart upload limit for everything that is not a photo:
/// its own docs on `sendVideo`/`sendAnimation`/`sendDocument` say 50 MB
/// (`RequestEntityTooLarge` is "larger than 50 MB"), while photos are the
/// 10 MiB [`photo::MAX_UPLOAD_BYTES`] case. Using the photo cap here refused
/// to even download a 10–50 MB video that Telegram itself would have
/// accepted, and a video has no smaller variant to fall back to — so the
/// post was lost.
pub(super) const MAX_MEDIA_UPLOAD_BYTES: u64 = 50 * 1024 * 1024;
/// Whole-transfer budget for one fallback download. The prep slot (and the
/// non-photo memory reservation) is held while this runs, and the idle window
/// alone lets a server drip one byte every 29 s forever — so this path caps
/// its own transfers well below the in-fetch default: 50 MiB in 300 s needs
/// about 1.4 Mbit/s, and a much slower link is better served by the retry
/// path toward the item's smaller fallback URL than by pinning a slot for
/// ten minutes.
/// ponytail: if slow-link reports show up, move the download out of the prep
/// slot (slot = decode/upload only) instead of raising this again.
const FALLBACK_DOWNLOAD_TOTAL: std::time::Duration = std::time::Duration::from_secs(300);
/// Infers a file extension from magic bytes so Telegram detects the mime type
/// on multipart uploads.
pub(super) fn sniff_ext(bytes: &[u8]) -> &'static str {
@@ -67,24 +89,43 @@ pub(super) enum FallbackError {
/// errors are not.
async fn download_to_temp(
item: &MediaItemPayload,
media_url: &str,
) -> Result<(NamedTempFile, bytes::Bytes), FallbackError> {
let media_url = match item {
MediaItemPayload::Photo { media, .. }
| MediaItemPayload::Video { media, .. }
| MediaItemPayload::Animation { media, .. } => media,
};
// The caller narrows the media to a source URL before calling (its entry
// guard rejects a file id), so there is nothing to match on here.
// Photos are downloaded even over the upload cap so `prepare_photo` can
// downscale / transcode them, up to their own download cap; videos and
// animations are refused as soon as the declared size crosses the upload
// cap. The limit is that cap, not `cap + 1`: a file of exactly the cap is
// admitted (`len > max_bytes` is false), and one byte over is not — the
// same boundary the size probe this replaced drew.
let limit = if matches!(item, MediaItemPayload::Photo { .. }) {
// animations are refused as soon as the declared size crosses their own
// (larger) upload cap. The limit is that cap, not `cap + 1`: a file of
// exactly the cap is admitted (`len > max_bytes` is false), and one byte
// over is not — the same boundary the size probe this replaced drew.
let is_photo = matches!(item, MediaItemPayload::Photo { .. });
let limit = if is_photo {
photo::MAX_PHOTO_DOWNLOAD_BYTES
} else {
MAX_UPLOAD_BYTES
MAX_MEDIA_UPLOAD_BYTES
};
let bytes = match x_media::site::download_media_limited(media_url, limit).await {
// A non-photo body is buffered whole and charges the process-wide budget
// for as long as this function holds it (one 64 MiB unit covers the cap):
// `PREP_SLOTS` bounds how many are in flight, this bounds what they add
// up to. Photos charge their own download cap for the same window — their
// real cost (header probe + decode buffer) is charged again by the
// prepare step right after, where both are actually held together.
let _budget = Some(
photo::reserve_memory(if is_photo {
photo::MAX_PHOTO_DOWNLOAD_BYTES
} else {
MAX_MEDIA_UPLOAD_BYTES
})
.await,
);
let bytes = match x_media::site::download_media_limited(
media_url,
limit,
FALLBACK_DOWNLOAD_TOTAL,
)
.await
{
Ok(bytes) => bytes,
Err(e) => return Err(classify_download_error(e)),
};
@@ -96,14 +137,25 @@ async fn download_to_temp(
.map_err(|e| FallbackError::Permanent {
message: format!("temp file failed: {e}"),
})?;
use std::io::Write;
// A write failure is resource exhaustion far more often than a broken temp
// The write runs on a blocking thread: up to 50 MiB of sync disk I/O on
// an executor thread would stall whatever else that worker runs (six prep
// tasks could stall six threads at once on a slow volume). A write
// failure is resource exhaustion far more often than a broken temp
// dir (ENOSPC / EDQUOT), and that clears on its own — worth an attempt
// instead of dropping the post on the first try. Creating the file (above)
// stays permanent: a temp dir that cannot be created at all is a
// deployment fault that should fail loudly and immediately. `Retryable`
// carries no message, so the cause is logged here.
file.as_file_mut().write_all(&bytes).map_err(|e| {
let (written, file, bytes) = tokio::task::spawn_blocking(move || {
use std::io::Write;
let written = file.as_file_mut().write_all(&bytes);
(written, file, bytes)
})
.await
.map_err(|e| FallbackError::Permanent {
message: format!("upload write worker panicked: {e}"),
})?;
written.map_err(|e| {
log::error!("temp file write failed: {e}");
FallbackError::Retryable {
delay_seconds: retry_delay_seconds(0),
@@ -118,6 +170,11 @@ async fn download_to_temp(
/// retry could only ask the same URL again.
fn classify_download_error(err: FetchError) -> FallbackError {
match err {
FetchError::RateLimited {
retry_after_secs, ..
} => FallbackError::Retryable {
delay_seconds: retry_after_secs as f64,
},
FetchError::Http(_) | FetchError::Transient(_) => FallbackError::Retryable {
delay_seconds: retry_delay_seconds(0),
},
@@ -133,23 +190,8 @@ fn media_from_file(
item: &MediaItemPayload,
path: std::path::PathBuf,
caption: Option<&str>,
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(InputFile::file(path), caption, *has_spoiler)
}
MediaItemPayload::Video { has_spoiler, .. } => {
video_media(InputFile::file(path), caption, *has_spoiler)
}
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(InputFile::file(path), caption, *has_spoiler)
}
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
media_from(item, InputFile::file(path), caption)
}
/// Builds the media group item from a (smaller) URL.
@@ -157,23 +199,8 @@ fn media_from_url(
item: &MediaItemPayload,
url: &str,
caption: Option<&str>,
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(input_file_for(url)?, caption, *has_spoiler)
}
MediaItemPayload::Video { has_spoiler, .. } => {
video_media(input_file_for(url)?, caption, *has_spoiler)
}
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(input_file_for(url)?, caption, *has_spoiler)
}
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
media_from(item, input_file_for(url)?, caption)
}
/// One item prepared for the upload fallback: the ready-to-send media plus
@@ -193,19 +220,35 @@ pub(super) async fn prepare_upload_item(
index: usize,
caption: Option<&str>,
) -> Result<PreparedItem, FallbackError> {
// A file id is already Telegram's copy of an uploaded file: there is no
// URL to re-fetch, and without this guard `item_url` presents the id as
// a *path*, which fails at upload time with a confusing open error
// instead of a classification. Re-upload cannot apply to it.
if matches!(item.media_ref(), MediaRef::FileId(_)) {
return Err(FallbackError::Permanent {
message: "file id reached the upload fallback".into(),
});
}
let media_url = item_url(&item);
// Locally produced files (ugoira / bsky remux MP4): nothing to download
// or shrink — upload the file directly. The send is a multipart upload,
// so the only remaining failure is an upload-cap error, which is
// permanent (a video cannot be re-encoded here).
let media_url = item_url(&item);
if !media_url.starts_with("http://") && !media_url.starts_with("https://") {
let media = media_from_file(
&item,
std::path::PathBuf::from(media_url),
caption,
item.thumbnail_url(),
)
.map_err(|message| FallbackError::Permanent { message })?;
let path = std::path::Path::new(media_url);
let size = tokio::fs::metadata(path)
.await
.map_err(|e| FallbackError::Permanent {
message: format!("local media unavailable: {e}"),
})?
.len();
if size > MAX_MEDIA_UPLOAD_BYTES {
return Err(FallbackError::Permanent {
message: "local media exceeds Telegram upload limit".into(),
});
}
let media = media_from_file(&item, path.to_path_buf(), caption)
.map_err(|message| FallbackError::Permanent { message })?;
return Ok(PreparedItem {
index,
media,
@@ -219,7 +262,7 @@ pub(super) async fn prepare_upload_item(
// the match below. A separate size probe used to issue a second GET of the
// same URL for an answer this path already has (and issued it for photos,
// whose answer was discarded one line later).
match download_to_temp(&item).await {
match download_to_temp(&item, media_url).await {
Ok((file, bytes)) => {
if matches!(item, MediaItemPayload::Photo { .. }) {
// Telegram rejects photos wider+taller than 10000 px combined
@@ -228,31 +271,32 @@ pub(super) async fn prepare_upload_item(
// limits degrade to the smaller URL. CPU-heavy work runs off
// the async executor thread.
//
// The header decides what that will cost in memory, so the
// probe travels with the downloaded bytes (both stay alive
// through the decode) and the reservation covers their sum:
// `PREP_SLOTS` bounds how many photos are prepared at once,
// this bounds what they hold between them — 512 MiB, whatever
// the batch looks like.
let (bytes, decode) = tokio::task::spawn_blocking(move || {
let decode = photo::decode_budget_bytes(&bytes);
(bytes, decode)
// Keep the permit inside the blocking closure. If the async
// future is cancelled while `spawn_blocking` is still
// decoding, dropping the permit here would undercount the
// process memory bound until that closure finishes.
let (bytes, budget_bytes) = tokio::task::spawn_blocking(move || {
let budget_bytes = photo::prepare_budget_bytes(&bytes);
(bytes, budget_bytes)
})
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
})?;
let _budget = photo::reserve_memory(bytes.len() as u64 + decode).await;
let prep = tokio::task::spawn_blocking(move || photo::prepare_photo(file, &bytes))
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
})?
.map_err(|message| FallbackError::Permanent { message })?;
let budget = photo::reserve_memory(budget_bytes).await;
let prep = tokio::task::spawn_blocking(move || {
let _budget = budget;
photo::prepare_photo(file, &bytes)
})
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
})?
.map_err(|message| FallbackError::Permanent { message })?;
match prep {
PhotoPrep::Upload(upload) => {
let path = upload.path().to_path_buf();
let media = media_from_file(&item, path, caption, item.thumbnail_url())
let media = media_from_file(&item, path, caption)
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
@@ -265,7 +309,7 @@ pub(super) async fn prepare_upload_item(
message: "photo dimensions exceed Telegram limits and no smaller variant is available"
.into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
let media = media_from_url(&item, url, caption)
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
@@ -276,7 +320,7 @@ pub(super) async fn prepare_upload_item(
}
} else {
let path = file.path().to_path_buf();
let media = media_from_file(&item, path, caption, item.thumbnail_url())
let media = media_from_file(&item, path, caption)
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
@@ -291,7 +335,7 @@ pub(super) async fn prepare_upload_item(
.ok_or_else(|| FallbackError::Permanent {
message: "media too large".into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
let media = media_from_url(&item, url, caption)
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
@@ -398,4 +442,48 @@ mod download_class_tests {
FallbackError::MediaTooLarge
));
}
#[test]
fn rate_limited_media_keeps_the_server_delay() {
match classify_download_error(FetchError::RateLimited {
site: "media",
retry_after_secs: 60,
}) {
FallbackError::Retryable { delay_seconds } => assert_eq!(delay_seconds, 60.0),
_ => panic!("expected retryable rate limit"),
}
}
#[tokio::test]
async fn a_file_id_item_is_refused_before_any_download() {
let item = MediaItemPayload::Photo {
media: MediaRef::FileId("AgACAgIAAx".into()),
has_spoiler: false,
fallback_url: None,
};
match prepare_upload_item(item, 0, None).await {
Err(FallbackError::Permanent { .. }) => {}
Err(_) => panic!("expected a permanent classification, got a different error"),
Ok(_) => panic!("a file id must be refused, not prepared"),
}
}
#[tokio::test]
async fn oversized_local_media_is_refused_before_upload() {
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("video.mp4");
let handle = std::fs::File::create(&file).unwrap();
handle.set_len(MAX_MEDIA_UPLOAD_BYTES + 1).unwrap();
drop(handle);
let item = MediaItemPayload::Video {
media: MediaRef::Source(file.to_string_lossy().into_owned()),
has_spoiler: false,
thumbnail: None,
fallback_url: None,
};
assert!(matches!(
prepare_upload_item(item, 0, None).await,
Err(FallbackError::Permanent { .. })
));
}
}
+260 -46
View File
@@ -3,6 +3,7 @@
use crate::db::unix_now;
use parking_lot::Mutex;
use rusqlite::OptionalExtension;
use rusqlite::params;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
@@ -22,6 +23,14 @@ pub struct ChatData {
pub message_format: HashMap<String, String>,
}
impl ChatData {
/// The chat's caption format for `site`, empty when it has none — the
/// built-in caption then applies (`caption_from_fields`).
pub fn format_for(&self, site: &str) -> String {
self.message_format.get(site).cloned().unwrap_or_default()
}
}
#[derive(Serialize, Deserialize, Clone, Debug, Default)]
pub struct EditMessage {
pub url: String,
@@ -53,53 +62,78 @@ impl ChatStore {
}
}
pub async fn get(&self, chat_id: i64) -> ChatData {
/// Loads persisted state, distinguishing a missing row and a failed read
/// from valid default settings. Only the ordinary read-only `get` path is
/// allowed to degrade to defaults; mutations must not write those defaults
/// back over a real row.
async fn load(&self, chat_id: i64) -> Result<ChatData, ()> {
if let Some(data) = self.cache.lock().get(&chat_id) {
return data.clone();
return Ok(data.clone());
}
let chat_key = chat_id.to_string();
let payload = self
.pool
.with_conn(move |conn| {
// Concurrent handler tasks (batch-forwards) may write chat_state
// while this read runs; the shared busy timeout handles the
// write-lock collision instead of failing the query.
let mut stmt = conn.prepare("SELECT payload FROM chat_state WHERE chat_id = ?1")?;
let mut rows = stmt.query(params![chat_key])?;
match rows.next()? {
Some(row) => Ok(Some(row.get::<_, String>(0)?)),
None => Ok(None),
}
// Concurrent handler tasks (batch-forwards) may write
// chat_state while this read runs; the shared busy timeout
// handles the write-lock collision instead of failing the
// query.
conn.query_row(
"SELECT payload FROM chat_state WHERE chat_id = ?1",
params![chat_key],
|row| row.get::<_, String>(0),
)
.optional()
})
.await
.unwrap_or_else(|e| {
.map_err(|e| {
log::warn!("chat_state read failed: {e}");
None
})
.unwrap_or_default();
let data: ChatData = serde_json::from_str(&payload).unwrap_or_default();
self.cache.lock().insert(chat_id, data.clone());
data
})?;
let payload = payload.unwrap_or_default();
let data = if payload.is_empty() {
ChatData::default()
} else {
serde_json::from_str(&payload).map_err(|e| {
log::warn!("chat_state payload is invalid: {e}");
})?
};
// Only fill a miss: an unconditional insert would let this (possibly
// stale) snapshot overwrite what a concurrent set just wrote.
self.cache
.lock()
.entry(chat_id)
.or_insert_with(|| data.clone());
Ok(data)
}
/// Write-through: update the cache and the DB.
pub async fn set(&self, chat_id: i64, data: &ChatData) {
/// Read-only access may degrade to defaults for display and control flow.
/// Mutating callers use [`Self::update`], which refuses a failed load.
pub async fn get(&self, chat_id: i64) -> ChatData {
self.load(chat_id).await.unwrap_or_default()
}
/// Write-through: update the cache and the DB. Returns whether the DB
/// write landed: the cache is updated either way, so `false` means the
/// change lives only until the next restart and the caller has to say so
/// instead of reporting a save that did not happen.
pub async fn set(&self, chat_id: i64, data: &ChatData) -> bool {
self.cache.lock().insert(chat_id, data.clone());
let payload = serde_json::to_string(data).expect("chat state serializes");
let chat_id = chat_id.to_string();
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
params![chat_id, payload],
)?;
Ok(())
})
.await;
if let Err(e) = result {
log::warn!("chat_state write failed: {e}");
}
self.pool
.with_conn_or(
log::Level::Warn,
"chat_state write failed",
false,
move |conn| {
conn.execute(
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
params![chat_id, payload],
)?;
Ok(true)
},
)
.await
}
/// The per-chat async lock serializing get→mutate→set cycles.
@@ -115,14 +149,19 @@ impl ChatStore {
/// (the batch-forward design spawns several per chat) each snapshot the
/// same `ChatData` and last-writer-wins would silently drop mutations,
/// e.g. a second `edit_message` record. The per-chat lock makes the
/// cycle atomic. Returns the closure's result.
pub async fn update<R>(&self, chat_id: i64, f: impl FnOnce(&mut ChatData) -> R) -> R {
/// cycle atomic. Returns the closure's result plus whether the DB write
/// landed (see [`Self::set`]); callers that do not care ignore the flag.
pub async fn update<R>(
&self,
chat_id: i64,
f: impl FnOnce(&mut ChatData) -> R,
) -> Result<(R, bool), ()> {
let lock = self.lock_for(chat_id);
let _guard = lock.lock().await;
let mut data = self.get(chat_id).await;
let mut data = self.load(chat_id).await?;
let r = f(&mut data);
self.set(chat_id, &data).await;
r
let saved = self.set(chat_id, &data).await;
Ok((r, saved))
}
/// Removes edit-before-forward records whose `created_at + ttl` is in the
@@ -139,7 +178,7 @@ impl ChatStore {
// what the eviction below is for. The pruning itself re-reads and
// writes under the per-chat lock below; taking no lock here means a
// chat appearing later is simply picked up by the next sweep.
let candidates: Vec<i64> = {
let mut candidates: Vec<i64> = {
let cache = self.cache.lock();
cache
.iter()
@@ -153,6 +192,48 @@ impl ChatStore {
.map(|(chat_id, _)| *chat_id)
.collect()
};
let persisted = self
.pool
.with_conn_or(
log::Level::Warn,
"expired prompt scan failed",
Vec::<i64>::new(),
move |conn| {
let mut stmt = conn.prepare("SELECT chat_id, payload FROM chat_state")?;
let rows = stmt.query_map([], |row| {
let id: String = row.get(0)?;
let id: i64 = id.parse().map_err(|e| {
rusqlite::Error::FromSqlConversionFailure(
0,
rusqlite::types::Type::Text,
Box::new(e),
)
})?;
let payload: String = row.get(1)?;
let data: ChatData = serde_json::from_str(&payload).map_err(|e| {
rusqlite::Error::FromSqlConversionFailure(
0,
rusqlite::types::Type::Text,
Box::new(e),
)
})?;
Ok((id, data))
})?;
Ok(rows
.filter_map(Result::ok)
.filter(|(_, data)| {
data.edit_message
.values()
.any(|entry| entry.created_at + ttl_secs <= now)
})
.map(|(id, _)| id)
.collect())
},
)
.await;
candidates.extend(persisted);
candidates.sort_unstable();
candidates.dedup();
let mut removed = Vec::new();
let mut evicted_chats = Vec::new();
for chat_id in candidates {
@@ -179,12 +260,20 @@ impl ChatStore {
}
if !evicted_chats.is_empty() {
let mut cache = self.cache.lock();
let mut locks = self.locks.lock();
for chat_id in &evicted_chats {
cache.remove(chat_id);
locks.remove(chat_id);
}
}
// Per-chat locks go only while uncontended (the same rule as
// rate_limit's prune): pulling a lock out from under an in-flight
// update — between its `lock_for` clone and its `lock().await` —
// would let a second writer `lock_for` a fresh one and enter the
// critical section concurrently. A contended lock stays until a later
// sweep, and dropping the uncontended ones also catches chats an
// earlier sweep had to skip, so the map stays bounded.
self.locks
.lock()
.retain(|_, lock| Arc::strong_count(lock) > 1);
if !removed.is_empty() {
log::info!(
"pruned {} expired edit-before-forward record(s)",
@@ -221,7 +310,8 @@ mod tests {
},
);
})
.await;
.await
.unwrap();
}));
}
for h in handles {
@@ -257,7 +347,8 @@ mod tests {
data.edit_message.insert(1, edit_entry(7, now - 3600));
data.edit_message.insert(2, edit_entry(7, now));
})
.await;
.await
.unwrap();
let removed = store.prune_expired(Duration::from_secs(60)).await;
@@ -271,6 +362,31 @@ mod tests {
);
}
#[tokio::test]
async fn persisted_expired_prompts_are_pruned_after_restart() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("cold.db");
let pool = crate::db::open_store(path.to_str().unwrap()).unwrap();
let raw = rusqlite::Connection::open(&path).unwrap();
let data = ChatData {
edit_message: [(1, edit_entry(7, unix_now() - 3600))]
.into_iter()
.collect(),
..ChatData::default()
};
raw.execute(
"INSERT INTO chat_state (chat_id, payload) VALUES ('7', ?1)",
rusqlite::params![serde_json::to_string(&data).unwrap()],
)
.unwrap();
let store = ChatStore::new(pool);
assert!(store.cache.lock().get(&7).is_none());
assert_eq!(
store.prune_expired(Duration::from_secs(60)).await,
vec![(7, 1)]
);
}
#[tokio::test]
async fn an_idle_chat_is_evicted_and_its_state_reloads() {
let dir = tempfile::tempdir().unwrap();
@@ -284,7 +400,8 @@ mod tests {
data.forward_channel_id = Some(-100);
data.message_format.insert("twitter".into(), "{url}".into());
})
.await;
.await
.unwrap();
assert!(store.cache.lock().contains_key(&9));
let removed = store.prune_expired(Duration::from_secs(60)).await;
@@ -313,7 +430,8 @@ mod tests {
.update(10, |data| {
data.edit_message.insert(1, edit_entry(10, unix_now()));
})
.await;
.await
.unwrap();
store.prune_expired(Duration::from_secs(3600)).await;
@@ -335,7 +453,8 @@ mod tests {
data.template.insert("keep".into(), "[]".into());
data.edit_message.insert(1, edit_entry(8, 0));
})
.await;
.await
.unwrap();
let removed = store.prune_expired(Duration::from_secs(60)).await;
@@ -348,4 +467,99 @@ mod tests {
"eviction dropped state the DB never received"
);
}
#[tokio::test]
async fn a_failed_read_is_not_cached() {
// A read that errors (busy, IO, a missing table) answers the default;
// caching that answer would make the next get return it blind and the
// next update write it back over the chat's real settings.
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("f.db");
let pool = crate::db::open_store(path.to_str().unwrap()).unwrap();
let raw = rusqlite::Connection::open(&path).unwrap();
raw.execute_batch("DROP TABLE chat_state").unwrap();
let store = ChatStore::new(pool);
let first = store.get(7).await;
assert!(first.forward_channel_id.is_none());
assert!(
!store.cache.lock().contains_key(&7),
"a failed read must not poison the cache"
);
// The next get retries the DB and sees the real row.
raw.execute_batch(
"CREATE TABLE chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL)",
)
.unwrap();
let real = ChatData {
forward_channel_id: Some(42),
..ChatData::default()
};
raw.execute(
"INSERT INTO chat_state (chat_id, payload) VALUES ('7', ?1)",
rusqlite::params![serde_json::to_string(&real).unwrap()],
)
.unwrap();
assert_eq!(store.get(7).await.forward_channel_id, Some(42));
}
#[tokio::test]
async fn a_failed_read_does_not_overwrite_existing_state() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("update.db");
let pool = crate::db::open_store(path.to_str().unwrap()).unwrap();
let raw = rusqlite::Connection::open(&path).unwrap();
let stored = "{\"forward_channel_id\":";
raw.execute(
"INSERT INTO chat_state (chat_id, payload) VALUES ('7', ?1)",
rusqlite::params![stored],
)
.unwrap();
let store = ChatStore::new(pool);
let mut called = false;
let result = store
.update(7, |data| {
called = true;
data.message_format.insert("twitter".into(), "{url}".into());
})
.await;
assert!(result.is_err());
assert!(!called, "a failed load must not run a destructive mutation");
assert!(!store.cache.lock().contains_key(&7));
let payload: String = raw
.query_row(
"SELECT payload FROM chat_state WHERE chat_id='7'",
[],
|row| row.get(0),
)
.unwrap();
assert_eq!(payload, stored, "the original row must remain unchanged");
}
#[tokio::test]
async fn prune_spares_a_lock_someone_still_holds() {
// The sweep evicts uncontended locks only: removing one an update
// still holds (its `lock_for` clone alive) would let a second writer
// create a fresh lock and enter the critical section concurrently.
let dir = tempfile::tempdir().unwrap();
let pool = crate::db::open_store(dir.path().join("l.db").to_str().unwrap()).unwrap();
let store = ChatStore::new(pool);
store.set(1, &ChatData::default()).await;
let held = store.lock_for(1); // an update between lock_for and lock().await
store.prune_expired(Duration::from_secs(60)).await;
assert!(
store.locks.lock().contains_key(&1),
"a contended lock must survive the sweep"
);
drop(held);
store.prune_expired(Duration::from_secs(60)).await;
assert!(
!store.locks.lock().contains_key(&1),
"the next sweep drops it once uncontended"
);
}
}
+7 -6
View File
@@ -35,7 +35,7 @@ services:
logging: *default-logging
acme-companion:
image: nginxproxy/acme-companion
image: nginxproxy/acme-companion:2.8.2
restart: always
environment:
DEFAULT_EMAIL: '${DEFAULT_EMAIL:-}'
@@ -64,7 +64,7 @@ services:
WEBHOOK_URL: '${WEBHOOK_URL:-}'
WEBHOOK_SECRET_TOKEN: '${WEBHOOK_SECRET_TOKEN:-}'
# Defaults, listed so they are discoverable; override in .env when needed.
LOCAL_USER_ID: '${LOCAL_USER_ID:-1000}'
LOCAL_USER_ID: '${LOCAL_USER_ID:-9001}'
RUST_LOG: '${RUST_LOG:-info}'
EDIT_MESSAGE_TTL_SECONDS: '${EDIT_MESSAGE_TTL_SECONDS:-86400}'
LINK_CACHE_TTL_SECONDS: '${LINK_CACHE_TTL_SECONDS:-604800}'
@@ -88,11 +88,12 @@ services:
- nginx-proxy
container_name: tgxmb
logging: *default-logging
# Webhook mode only (in polling mode there is no listener, so drop this
# block or set WEBHOOK=true): the bot listens on WEBHOOK_PORT; nginx-proxy
# shows 502s while this is down, so surface it to the orchestrator.
# Probes the listener only when WEBHOOK=true (compose interpolates the
# value from .env); a polling deployment has no listener and must not be
# reported unhealthy. nginx-proxy shows 502s while webhook mode is down,
# so surface that to the orchestrator.
healthcheck:
test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/${WEBHOOK_PORT:-8443}'"]
test: ["CMD-SHELL", "test '${WEBHOOK:-true}' != true || bash -c 'exec 3<>/dev/tcp/127.0.0.1/${WEBHOOK_PORT:-8443}'"]
interval: 30s
timeout: 5s
retries: 3
+20 -4
View File
@@ -5,6 +5,22 @@ if [ "$(id -u)" -eq '0' ]
then
USER_ID=${LOCAL_USER_ID:-9001}
# A non-numeric id breaks useradd/usermod in confusing ways, and uid 0
# would sail straight through the privilege drop below (`setpriv
# --reuid=0` keeps the bot root while looking configured) — refuse both
# up front.
case $USER_ID in
''|*[!0-9]*)
echo "docker-entrypoint: LOCAL_USER_ID must be a numeric uid, got '$USER_ID'" >&2
exit 1
;;
esac
if [ "$USER_ID" -eq 0 ]
then
echo "docker-entrypoint: LOCAL_USER_ID=0 would keep the bot root; refusing" >&2
exit 1
fi
# `docker compose restart` / `docker restart` reuse the same container, so
# the overlay fs keeps the user created on first boot. A second `useradd`
# then fails with exit code 9, which would trip `set -e` and kill the
@@ -12,18 +28,18 @@ then
# otherwise so LOCAL_USER_ID changes still apply.
if ! id user > /dev/null 2>&1
then
useradd --shell /bin/bash -u ${USER_ID} -o -c "" -m user > /dev/null 2>&1 || true
useradd --shell /bin/bash -u "${USER_ID}" -o -c "" -m user > /dev/null 2>&1 || true
else
usermod -u ${USER_ID} -o user > /dev/null 2>&1 || true
usermod -u "${USER_ID}" -o user > /dev/null 2>&1 || true
fi
# Bind-mounted volumes may not support chown; a failure here must not kill
# the container either.
chown -R `id -u user`:`id -u user` /app > /dev/null 2>&1 || true
chown -R "$(id -u user):$(id -g user)" /app > /dev/null 2>&1 || true
export HOME=/home/user
# setpriv (util-linux, present in bookworm-slim) replaces gosu: drop to the
# target user and exec, keeping the process as PID 1.
exec setpriv --reuid=`id -u user` --regid=`id -g user` --init-groups "$@"
exec setpriv --reuid="$(id -u user)" --regid="$(id -g user)" --init-groups "$@"
fi
exec "$@"