Compare commits

..
65 Commits
Author SHA1 Message Date
YoursFunny 8c085bea35 chore: bump version to 1.9.0 2026-09-21 15:05:03 +08:00
YoursFunny 670351d436 fix: let a successful send return a degraded entry to the fast path
Degrading a link-cache entry (previous commit) closed the "user retries right
after a failure" case, but left a new one open: the entry could never regain
file ids. `cache_sent_task` skipped *every* cached send — its rule was "the
entry already holds the ids the next repeat wants" — which is true for a
healthy entry and false for a degraded one. So a degraded entry (pixiv's
hotlink-protected media, say) kept sending by URL forever, and every repeat
paid a download and an upload through the fallback that the file ids would
have avoided. Worse than the re-fetch it replaced.

The rule is now the one it always meant: a send whose cache snapshot still
carries file ids leaves the entry alone, and a send served from a degraded
entry writes back the ids it produced (the URLs in the rewritten entry come
from that send's own items, so they stay correct).

Verified by a test that drives `cache_sent_task` directly with both task
shapes: the degraded one updates the entry to the fresh id, the healthy one
leaves it untouched. 125 bot tests + 91 x-media tests, live suite (14) pass.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 14:38:26 +08:00
YoursFunny 64cf43dc01 perf: degrade a link-cache entry instead of dropping it on a failed send
`link_cache` exists so a repeat link costs nothing: no source request, no
download, no upload. It was written only on a *successful* send, and a send
that failed permanently deleted the entry — so the user's immediate retry, the
one case where they are most likely to try again, re-fetched everything:
site requests, a download, and for a ugoira or a bsky video a full ffmpeg
encode. Invalidation is right about the cause (the cached Telegram file id is
what went stale) and wrong about the cure (the media and its URLs are usually
fine).

Cached media now carries the source URL it was sent from, and a permanent
failure *degrades* the entry: the file ids are cleared, the URLs and the
caption fields stay, and the next request sends from those URLs — Telegram
fetches the media (or the upload fallback does) with no source round trip.
That is the same media a fresh fetch would have produced (site CDN URLs are
stable per post), and it is bounded: an entry that is already degraded, or one
from before this field existed, is removed instead, so a dead post still ends
up re-fetched and reported rather than retried forever.

Verified: a cached send that fails permanently leaves the entry with its URL
and no file id, a second failure drops it, and a degraded entry sends the
media with no fetch at all (the mock records no reply, which is what the
fetch-error path would have produced). 124 bot tests + 91 x-media tests pass,
including a direct test of the two payload shapes.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 14:35:57 +08:00
YoursFunny 62507d3a01 perf: bound the memory photo preparation holds, not just its count
`PREP_SLOTS` caps how many items are prepared at once (6) but says nothing
about what they hold: one photo's decode buffer can be up to
`MAX_DECODE_BYTES` (512 MiB) and that guard is *per photo*, so six of them —
an album of large scans, two chats at once — could peak near 3 GiB on a host
sized for a fraction of it. The upload fallback is the only path that
allocates like this; nothing downstream notices until the kernel does.

Photo preparation now charges a process-wide memory budget
(`MEMORY_UNITS` × 64 MiB = 512 MiB) for what it actually holds: the
downloaded bytes plus the decode buffer the *header* predicts — the same
prediction the per-photo guards apply, now shared (`decode_bytes`,
`decode_budget_bytes`) so the reservation and the guard cannot drift. A photo
that is already within Telegram's limits is billed only its download, so an
ordinary 10-image album still runs several at a time; two photos near the
per-photo cap serialize (each takes the whole budget). The request is clamped
to the budget so a single huge photo runs alone instead of waiting for
permits that cannot exist.

Verified with a throwaway harness against a locally served 9999x9999 PNG
(126 KB on the wire, ~100 MB decoded) driven through the real
`prepare_upload_item`: with the budget held the preparation waits — "after
1516ms the prep is still waiting on the budget" — and finishes in 11.8s the
moment it is released, so the accounting binds in the real path and not just
in the semaphore.

Kept as permanent tests instead: the unit math (rounding, clamp, and that a
max-size photo still gets the whole budget rather than waiting forever), the
budget sharing (huge decodes cannot overlap, ordinary ones do not queue), and
the prediction agreeing with the processing decision (over-sized PNG/JPEG
charged, within-limits and unknown formats free).

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean (122 bot + 91 x-media).
2026-09-21 14:31:50 +08:00
YoursFunny 667f523c8b docs: sync AGENTS with the shared fetch, download budget and inline answer 2026-09-21 13:53:54 +08:00
YoursFunny 507c8ac317 perf: index the link-cache prune, evict chats with no live prompt
Two things the 300 s sweep did the hard way:

- The link cache is pruned by `created_at` (`DELETE FROM link_cache WHERE
  created_at < ?`) and had no index on it, so every sweep scanned the whole
  table — every post sent inside the TTL window, which is up to a week of
  them — while the `url` primary key served none of it. A new migration
  (appended; migration 1 is frozen and already shipped) creates the index, and
  the upgrade test now asserts it exists after an upgrade.
- `ChatStore::prune_expired` only ever *looked* at chats that had an expired
  edit-before-forward record, so a chat with no prompt at all — the common
  case: every chat that ever sent a message or ran a command — stayed in the
  cache and in the per-chat lock map for the process lifetime. The candidate
  set now includes chats holding no records, which is what the eviction below
  was written for; the DB keeps the row, so the next use costs one SELECT
  (pinned by a new test that also shows the durable settings come back).

Deliberately *not* done: skipping the write in `ChatStore::set` when the state
is unchanged. Comparing against the cached copy would skip a serialize plus a
blocking DB round trip for a no-op update — but every one of the 13 `update`
callers mutates something, so the no-op case is a user repeating an identical
command, and the same comparison would also skip the write that repairs a row
whose earlier write failed. A rare saving against a rare repair, and the write
is what makes the cache a cache rather than a source of truth.

Verified: the new eviction test fails without the candidate change (checked by
reverting it) and passes with it; 118 bot tests and 91 x-media tests pass.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 13:52:43 +08:00
YoursFunny 670a7bd02b fix: answer an inline query whose media Telegram cannot fetch
Every item was skipped — `needs_media_headers` for pixiv's pximg.net, or a
local ugoira/bsky MP4 that does not parse as a URL — and the function fell
through to `Ok(false)`, which the debounce reads as "retry this query". The
client got no answer at all (a spinner with nothing behind it) and every
keystroke re-ran the fetch, while an unsatisfiable link is the *permanent*
truth for that query: Telegram fetches inline result URLs itself and never
sends a Referer.

It now answers the empty result set with a 300 s window, so the client stops
spinning and the same query is served from Telegram's cache and from the
debounce (which only releases on a failure). A *failed* fetch still releases,
so a retry is not answered from a stale empty answer.

Verified with a real pixiv link through a real `Bot` against the stand-in
API: `AnswerInlineQuery` with `results: []` and `cache_time: 300` (and
`Ok(true)`, so no release).

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 13:49:21 +08:00
YoursFunny 5ff921222a fix: give a media download a total time budget
`MEDIA_CLIENT` deliberately has no reqwest total timeout (a 30s cap made a
hundreds-of-MB ugoira zip impossible to deliver), and the idle window only
covers *silence*: a server that drips a chunk every 29 s keeps the download
alive indefinitely. On the bot's side each such download holds one of the
process-wide upload-prep slots (`send::upload`'s `PREP_SLOTS`, 6), so a
handful of trickling sources can take the whole fallback path out of service
without ever looking broken.

`DOWNLOAD_TOTAL_TIMEOUT` (600s) bounds the whole transfer, checked between
chunks — a transfer that completes just over the budget is kept rather than
thrown away, and a genuinely slow link (the case the cap was removed for)
stays far inside it. Reported as `Transient` like the idle-window stall: the
transfer may simply have been unlucky, and a retry restarts it.

Verified with a local trickling server (a 1 KiB chunk every 1.2s, chunked so
the client cannot see the total up front): with the budget temporarily
lowered to 2s the download aborted after 2425ms with
`transient: download exceeded 2s` — two chunks in, the server seeing the
client go away — proving the budget and not the 30s idle window ended it.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 13:44:53 +08:00
YoursFunny 0edaef56bd perf: share one fetch between concurrent duplicates of a link
`link_cache` is only written *after* a send succeeds, so two chats posting
the same link at the same moment each ran a full fetch: two sets of source
requests, and for an ugoira or a bsky video two ffmpeg encodes of the same
post — minutes of CPU for the second one. The same applies to a batch
forward racing a queued retry. Within one message `dedupe_urls` already
handled the duplicates; across calls nothing did.

`fetch_shared` keys an in-flight fetch by the post's cache key: the first
caller runs it, the rest subscribe and take its result. The entry is removed
by a guard when the fetch settles (cancellation included), so this dedupes
what is *concurrent* and never answers from an old result — a repeat later
fetches again, and a failure is deliberately not cached: the user is told to
try again, and a cached failure would answer that retry from a stale state.

Two hazards that shape the code, each with a test: a broadcast channel lives
while *any* sender does, so the waiter drops its own clone of the sender
before waiting — otherwise a cancelled sharer would leave it waiting forever
— and a waiter whose sharer vanished fetches for itself instead of failing a
link that is perfectly fetchable.

One fetched post can now serve several sends, which the temp files behind
`Fetched::keep_alive` had to support: the field is `Arc<TempDir>` and the
accessor hands out references (the bot's `KEEP_ALIVE` registry holds the
same), so the ugoira/bsky MP4 stays on disk until the *last* task settles
rather than the first. `take_keep_alive` is gone — a `take` could only ever
serve one of the senders.

Verified: 116 bot tests (three new sharing tests, including the cancelled
sharer) and 91 x-media tests (a new one pinning the keep-alive refcount)
pass, plus a live check that two concurrent `fetch_shared` calls for one
tweet return the very same `Arc` after one fetch's worth of wall time.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 13:42:52 +08:00
YoursFunny 994734b001 perf: gzip and HTTP/2 for the site clients
x-media's reqwest had exactly `["json", "rustls-tls"]` — no decompression, no
HTTP/2 — so every adapter fetched its JSON as identity over HTTP/1.1. The
site APIs compress: twitter's syndication body measures 4469 bytes identity
against 1066 gzipped (4.2x) for a single-tweet response, and the fetch is on
every twitter link. The CDNs all negotiate h2 (the shared client reports
HTTP/2.0 against cdn.syndication.twimg.com, api.bilibili.com and
public.api.bsky.app), which also multiplexes the concurrent media downloads
that used to open a connection each.

Both features are one crate-level switch, so the pages' compression is what
the APIs answer with and requires no adapter change: reqwest adds
`accept-encoding: gzip` and decompresses transparently. `download_media_limited`
keeps a correct size cap either way — it checks the accumulated *body* while
streaming, not the declared Content-Length, which for a compressed response
is the compressed size.

Verified: a local echo server recorded `accept-encoding: gzip` on a request
from the shared client (both clients come from `build_client`), and a probe
over the shared `CLIENT` reported `HTTP/2.0` for the three site hosts above.
`cargo test -p x-media -- --ignored live` (14 passed) covers the real
endpoints with the new transport.

Cargo.lock gains async-compression (+codecs/core), fnv and h2.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 13:09:32 +08:00
YoursFunny a501a17519 perf: stop spending API calls the link pipeline cannot use
Two calls per link went out that could not affect anything:

- `run_with_chat_action` awaited the opening `send_chat_action` to completion
  before the pipeline was polled at all, and again inside the loop on every
  `ACTION_REFRESH`. Telegram round trips are hundreds of ms: the first delay
  came out of the user's wait for every link, and each refresh suspended the
  fetch (an ugoira encode or HLS remux runs for seconds) by the same amount.
- `handle_message` enqueued *every* URL a private chat posted, including
  links no site adapter claims. Those cost a queue slot, a worker wake-up,
  a `Message` clone and (through the action above) one Telegram call, only
  for `url_media_inner` to conclude there was nothing to send. The group
  branch has always made the `cache_key(url).is_some()` test before it acts;
  the private branch now makes it before it enqueues.

The in-flight action is held (`Option<BoxFuture>` — the sender surface is
already type-erased, so it is `Unpin`) and polled as its own `select!`
branch: still polled *before* the pipeline, so the indicator is on screen
before the first send, but a slow Telegram response can no longer delay the
pipeline, and none of the branch bodies ever awaits one. One action is in
flight at a time; a refresh while one is unanswered is skipped rather than
dropping the request mid-flight. Note that `select!` evaluates every branch's
future expression eagerly, so the `None` case is an `async` block whose
`unwrap` only runs when the branch is polled (the eager form panicked).

Behavior pinned by the existing tests, unchanged: the opening action precedes
the first send, a 12s pipeline still sees exactly three actions
(`a_long_pipeline_keeps_the_chat_action_alive`), and an unsupported URL
reaching `url_media` still gets the one indicator before the pipeline settles
— in production it no longer reaches `url_media` at all.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 13:01:45 +08:00
YoursFunny edf4dab26d fix: remember an unavailable bilibili fingerprint
`cookie()` cached the device cookies only on success: the `Option<String>`
it held could not tell "not fetched yet" from "the fetch failed", so a failed
fingerprint meant *every* later post re-asked the SPI endpoint — one extra
round trip per post, forever, exactly the case the fingerprint exists for (a
flagged IP, where that endpoint is the thing answering `-352`/412).

The cache is now `Option<Option<String>>`: the outer level is "an attempt has
been made", the inner one is the cookie it produced, so a failure is
remembered as no-cookie and reaches the request path unchanged. The guard is
also dropped before the request instead of being held across it, which had
serialized every concurrent bilibili fetch behind that one round trip.

A racing first pair still costs a duplicate fingerprint call
(`get_or_insert`, first writer wins) — never a wrong cookie.

Verified live: the 5 bilibili tests (`--ignored`, including
`live_fingerprint_yields_device_cookies` and the four dynamic fetches that
send the cookie) pass. The failure path itself has no unit test: `SPI_URL` is
a const, so there is no seam to make the endpoint fail on demand.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 12:56:20 +08:00
YoursFunny 818a44d697 perf: fetch bsky HLS segments concurrently
A long bluesky video can be 500 segments, and the remux downloaded them
strictly one at a time: the user waited for every round trip in turn, which
is the dominant cost of the whole remux (the ffmpeg concat itself is local).
Each segment's multi-megabyte body also went to disk through a blocking
`std::fs::write` on an executor thread.

Segments now download and write under a small bound (`SEGMENT_CONCURRENCY`,
4 — a segment can be 20 MiB and the playlist is capped at 256 MiB, so this is
also what bounds the remux's peak memory) and the write goes through
`tokio::fs`. Concurrent downloads complete in completion order, and ffmpeg
concatenates the list in whatever order it holds — an out-of-order list is a
*silently* scrambled video, not an error — so `concat_list` sorts by segment
index and carries its own test.

x-media's own tokio features gain `rt` (JoinSet) and `fs`: the library
already used `spawn_blocking` on the strength of the bot crate's features.

Verified against a local HLS fixture — 12 one-second segments of solid
red/green/blue, each served with a 150 ms delay, reached through a name that
resolves to loopback (the guard refuses a literal 127.0.0.1) — with a
temporary in-module test: all 12 sampled frames come back in the right colour
order, and the server recorded a peak of 4 requests in flight, where the
serial version showed 1.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 12:55:12 +08:00
YoursFunny e31bf92df7 perf: parse a twitter syndication body once
`fetch` classified the response with `parse_syndication_body` — which parses
the whole body into a `serde_json::Value` — and then threw that value away
and re-parsed the same text into a `SyndicationTweet`. Two full JSON scans
and two allocations of every string in the body (text, entities, media
details) per tweet.

`Tweet::from_syndication_value` takes the value classification already
built and deserializes from that: `from_value` moves the strings out of the
tree instead of allocating copies, so the response text is scanned once.
The auth fallback had the mirror image of the same waste — it built the
syndication shape as a `Value` and then serialized it back to a string for
a parse — and the 8 test call sites lose their `.to_string()` with it.

Verified live: `cargo test -p x-media -- --ignored live` (14 passed),
including the 5 twitter fetches that exercise this path.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 12:49:26 +08:00
YoursFunny a1c452f19d perf: retry the bsky HLS request, not the whole fetch
A failed bsky video remux was reported as `FetchError::Transient`, so the
fetch loop retried the *whole* adapter — master playlist, variant playlist
and up to 500 segments again (256 MiB of cap each time), for a failure that
happened near the end of the work the retry was about to redo. The message
had to stay honest (returning `Ok` with no media reads as "this post has no
media"), so it needed a class of its own.

`FetchError::MediaPrep` is that class: post fetched, media could not be
prepared locally, not retryable (the per-site `is_retryable` whitelist
excludes it by construction), with its own user-facing text — the generic
"failed to fetch" would have hidden that the download or encode was what
broke. The retry itself is not lost, it moved: `fetch_hls` retries the
request that actually failed, once, for the classes a retry can change
(transport, 429/5xx).

Checked the sibling sites before widening the change: pixiv already degrades
to no media when the ugoira encode fails (`Ok(None)`), and its frame-zip
download failure is the last step so a re-fetch re-does only that; twitter's
auth leg replays two cheap metadata GETs and a GraphQL 5xx *is* worth
retrying. Neither needed the new class.

Verified with a throwaway proxy harness: a 503 answered twice-in-a-row path
costs 2 requests and succeeds, a 404 costs exactly 1 and fails (no pointless
retry). `site::bsky::interface::tests::media_prep_failure_is_not_retried`
pins the classification.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 04:11:39 +08:00
YoursFunny 362eb9e729 perf: bound upload-fallback preparation process-wide
Each batch's items were prepared under their own `Semaphore::new(3)`, which
is not a memory bound: 8 URL workers and 4 queue workers can each be inside
a batch, so a burst could have two dozen downloads in flight at once, each
buffering a whole photo before it is processed. Nothing else on the media
path bounds them — the send itself is paced by the rate limiter, but the
download and the decode happen before it is charged.

One process-wide `PREP_SLOTS` (6) replaces the per-batch semaphore, and the
photo download gets its own cap: `MAX_PHOTO_DOWNLOAD_BYTES` (32 MiB) for the
transfer, with `MAX_DECODE_BYTES` (512 MiB) left as the pre-allocation guard
on a single decoded buffer. A photo over the download cap degrades to its
smaller URL exactly as one over the decode budget does
(`FallbackError::MediaTooLarge` → `fallback_url`) — never an error.

Verified with the same throwaway proxy harness: 4 concurrent 10-item batches
against a server that holds every response 150 ms peak at exactly 6
concurrent downloads (the per-batch three allowed 12) with all 40 items
prepared.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 04:10:28 +08:00
YoursFunny 35074bab67 perf: stop probing a media item's size before downloading it
The upload fallback asked `x_media::site::media_size` for every remote item
before downloading it. That call is a real GET (not a HEAD) on the *un-
guarded* `CLIENT` — so every fallback item cost two requests where one would
do, the response body was never consumed (the connection cannot return to
the pool), and for photos the answer was discarded outright
(`too_large && !matches!(item, Photo { .. })` still fired the request). It
bypassed `media_request`'s private-network guard as well, the one choke
point every other egress goes through.

For videos the probe was redundant twice over: `download_media_limited`
reads the declared Content-Length before any body byte and aborts with
`FetchError::TooLarge`, which the call site already turns into the item's
smaller URL (`FallbackError::MediaTooLarge` → `fallback_url`).

`media_size` is deleted (no other caller) and the download's own cap is the
only size gate. The video cap is now exactly `MAX_UPLOAD_BYTES` instead of
`MAX_UPLOAD_BYTES + 1`, so the boundary the probe drew survives byte for
byte: a file of exactly the cap is admitted (`len > max_bytes` is false),
one byte over degrades to the smaller URL.

Verified with a throwaway harness (a local HTTP server reached through
`TELOXIDE_PROXY`, the one LAN egress the guard allows): a small video, a
photo and an oversized video each cost 1 request where the probe made it 2,
and the oversized one still lands on `/fallback.mp4` without fetching it.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
2026-09-21 04:09:34 +08:00
YoursFunny 1fb7837255 test: name the redirect-guard test so the live filter selects it
The repo runs the ignored network tests with `cargo test -- --ignored
live`, which matches on the test name; the new redirect test had no
`live_` prefix and would have been skipped by it.
2026-09-21 03:03:37 +08:00
YoursFunny 4e723e1657 test: drive a real Bot against a stand-in API
Every test went through `MockSender`, so `media_sender`'s `Bot`
implementation — the URL it builds, the multipart it sends, the per-chat
limiter and the bot-wide budget it charges — was never exercised, and neither
was any handler reached from a real update. The two things that made that hard
are gone:

- `media_sender::test_support::fake_api::FakeApi` is a stand-in for
  `api.telegram.org`: a `tokio` TCP listener that reads one HTTP/1.1 request
  (JSON or multipart), records it and answers the smallest result the method
  needs. No new dependency, and `Bot::new(token).set_api_url(api.url())`
  points a real `Bot` at it. Note for future tests: teloxide keys methods by
  payload type, so the path is `SendMediaGroup`, not `sendMediaGroup`.
- `message_handler` built its own `AppContext::from_statics` internally, so no
  test could reach its branches; its body is now `handle_message(ctx, bot,
  message)` with `message_handler` as the thin `dptree` entry.

Tests: a media group through the real `Bot` (asserting the multipart fields —
chat, media URL, caption — and that the send charged the chat's limiter), the
forward button through the real callback path (`CopyMessages`,
`DeleteMessage`, `AnswerCallbackQuery` with the prompt's ids and the toast
text), and `handle_message` twice (a prompt reply becoming an
`EditMessageCaption`, and a supported link in a group producing the one
explanatory `SendMessage`).

Also closes the redirect-hop gap left open by the download guard: the live
`a_redirect_into_the_hosts_network_is_refused` follows a public redirector to
`169.254.169.254` and asserts the policy refuses the hop (verified against
httpbin.org here, and by mutation — disabling the hop check fails it).

Docs: AGENTS.md's testing conventions and untested-modules list (the Bot
implementation and the handler branches are covered now; `main.rs`'s
startup/shutdown and its `dptree` tree still are not).

`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (201 passed, 16 ignored) clean.
2026-09-21 03:02:43 +08:00
YoursFunny cd8b5ac67b fix: refuse media downloads into the host's own network
The media URLs the bot fetches come from a site's own API response (media
URLs, `fallback_url`, thumbnails), `build_client` left reqwest's default
redirect policy in place (up to 10 hops, any host), and the downloaded bytes
are uploaded to Telegram — so a response pointing at a cloud metadata
endpoint would read it back into a chat.

`media_request` is now the one choke point both download paths go through:
http(s) only, and a host that is no address or name of the host's own network
(`blocked_ip` covers loopback, private, link-local, unspecified, broadcast,
documentation, multicast, IPv6 unique-local/link-local, IPv4-mapped, plus
CGA-NAT and benchmarking ranges; `is_local_name` covers `localhost` and
`*.local`). A refusal is `FetchError::Blocked` — permanent, so the send path
does not retry a URL that would be refused again (a connection error used to
be retryable and burned attempts). The same guard runs on every redirect hop
through a custom redirect policy, keeping reqwest's 10-hop cap.

Deliberate gap, documented at the function: DNS rebinding (a name the site
controls resolving to a private address) needs a `reqwest::dns::Resolve`
wrapper, which would also resolve the operator's own proxy host — and
`TELOXIDE_PROXY` is routinely a LAN address — so it would take down working
deployments to block a much less likely attack.

Verified: three offline tests (the address table, the URL table, and a
refusal that holds with nothing listening at the metadata endpoint), both
mutations confirmed to fail them (guard disabled → the download test fails;
link-local dropped from `blocked_ip` → 169.254.169.254 is accepted), and the
live `download_media_pixiv_original_with_referer` still fetches from
i.pximg.net through the guarded client, so real media downloads are
unaffected. `cargo fmt`, `cargo clippy --workspace --all-targets --locked --
-D warnings`, `cargo test --workspace --locked` (198 passed, 15 ignored)
clean.
2026-09-21 02:51:03 +08:00
YoursFunny bd5a6846e8 test: cover the URL extraction the message entry point runs on
`extract_urls` decides whether a pasted link is seen at all — a bug there is
silence for the user, which is the complaint this bot's UX work keeps coming
back to — and it had no test: it takes a teloxide `Message`, which is not
worth building by hand.

Split it into the two decisions that are ours and keep the offset work
(teloxide's `parse_entities` turning entities into slices of the message
text) in the thin `extract_urls` composition:

- `url_of(kind, text)`: a bare `Url` entity is its own text, a `TextLink`
  keeps its target (not the words the user sees), everything else carries
  none.
- `dedupe_urls`: first occurrence wins, deduped by normalized post id — so
  `/status/1`, `/status/1/photo/1` and a text link to the same post are one
  entry — and by exact text for URLs no site claims.

Verified by mutation, each confirmed to fail the new tests: dropping the
`TextLink` branch, and deduping by raw text (`left: [.../status/1,
.../status/1/photo/1]`).

`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` clean.
2026-09-21 02:47:43 +08:00
YoursFunny da8fde6a4e test: pin the migration chain's append-only rule
`db.rs`'s two rules — `schema_init` is the version-0 baseline and never gains
a column, `MIGRATIONS` is append-only and never edited — were enforced by
comments only. Both have a silent failure mode across releases, and the worst
one (a baseline edit) makes `open_store` fail with `duplicate column name`,
i.e. a fresh deployment that will not start.

Three tests, no production change:

- A pre-migration database (the historical DDL written out literally, so an
  edit to the baseline shows up here instead of being followed) upgrades
  through `open_store`: version at the latest, exactly the migrated column
  set, rows intact.
- The shipped migration text is frozen and compared entry by entry; the
  assertion names the rule when it fires. Appending still passes — that is
  the one allowed change.
- A fresh database lands at the latest version, so a deployment that only
  ever saw fresh databases is on the same schema as an upgraded one, and
  re-opening the same file is a no-op.

Verified by mutation, both confirmed to fail the new tests: editing the
shipped migration (`shipped_migrations_are_frozen`, with the rule in the
message) and adding the column to `schema_init` instead of a migration
(`a_fresh_database_lands_at_the_latest_version`, `duplicate column name:
lease_token`).

Docs synced for this and the previous two items: `main.rs` (startup repair),
`queue.rs` (`runnable_rows`/`replace_payload`), `handlers/` (`urls.rs`'s
repair, `mod.rs`'s `apply_caption_edit`), `db.rs` (the migration tests) and
the untested-modules list (`db.rs` now covered for migrations; the bot-side
live test renamed to match the repo's `--ignored live` filter).

`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (193 passed, 15 ignored) and
`cargo test -p x-media -- --ignored live` (13) clean.
2026-09-21 02:17:14 +08:00
YoursFunny 3b946b1eab fix: never eat a caption silently when the edit fails
Both caption-edit paths logged the error and carried on as if they had worked:
`edit_message_handler` consumed the user's reply (`let _ =`), and the
template button updated the prompt record and dismissed its toast with no
text. Since the edit surface is not rate limited, a 429 or a transient
failure meant the caption never changed and the user got no hint — the text
they sent was simply gone.

`apply_caption_edit` (handlers/mod.rs) is now the one place that applies a
caption edit and reports the outcome:

- A failure the API calls worth retrying (`classify_request_error` →
  `Retryable`) is retried once when the delay is at most 2 s — a reply or a
  button press has already been consumed by then, so a long flood-control
  wait must not stall the chat's update queue behind it.
- Otherwise the caller reports it: the reply path answers the user ("Could
  not update the caption (…). Send it again to retry."), the template path
  puts it in the callback toast and leaves the record alone — a swap that
  never happened must not be recorded as the prompt's template.

Tests: the swallowed-failure test now asserts the notice (it pinned the old
silent behaviour), plus a short `RetryAfter` that is retried and lands, a
60 s one that is not retried and is reported instead, and the template
button's failure toast with the record left unchanged.

`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (190 passed, 15 ignored) clean.
One note: the first full run tripped `download_media_pixiv_original_with_
referer`, the token-gated pixiv CDN download test that AGENTS already
documents as a local-network flake; it passes in isolation and on the rerun.
2026-09-21 02:07:55 +08:00
YoursFunny d540fc31e9 fix: re-fetch queued retries whose local media did not survive a restart
A queued retry that holds a local file — the ugoira MP4, a bsky remux, or a
temp file the reupload fallback downloaded — could never succeed after a
restart: those files live in the system temp dir and `send::KEEP_ALIVE`, the
registry that keeps them alive for the retry, is in memory. The row retried
into an upload error, said nothing about why, and dead-lettered the user's
link even though the payload carries the `source_url`.

`handlers::repair_lost_local_media` now runs in `main` before any worker
starts (so no row can be leased while it writes payloads, which is why it can
replace them without the lease guard a worker's write-back carries):

- `Task::local_media_paths` decides which rows are affected: any local path
  that is gone. A partially delivered album is left alone — its remaining
  batches cannot be reconciled with a fresh media list without risking a
  second copy of what the user already received.
- The post is re-fetched from `source_url` through the ordinary `site::fetch`,
  so a repaired task looks like a first send: fresh media, the chat's caption
  format, a fresh link-cache snapshot, and a new keep-alive entry when the
  re-fetch produced another local file.
- The delivery envelope (chat, reply, forward/edit settings, notify targets) is
  kept, the attempt budget restarts, and nothing counts as sent.
- A post that cannot be fetched again (gone, withheld, site down) notifies the
  user with that reason instead of letting the retry die on a missing file.

New queue plumbing: `runnable_rows()` (pending + in-progress rows, read before
the workers exist) and `replace_payload()` (rewrites the payload, resets
`attempts`, marks the row pending).

Verified: 5 new offline tests (the two decisions above against a real temp
file, the queue scan/replace, and the envelope-preserving rewrite) plus
`a_lost_local_media_row_is_refetched_from_its_post`, a live test that seeds a
row pointing at a missing file with a real bsky post as its source and asserts
the row now carries http(s) media and that nothing was sent — run against the
live API here. `cargo fmt`, `cargo clippy --workspace --all-targets --locked --
-D warnings` and `cargo test --workspace --locked` (187 passed, 15 ignored)
are clean.
2026-09-21 01:48:50 +08:00
YoursFunny 024dfd50b3 fix: bound the inline state map, test the 300s sweep, add a bot-wide send budget
Three gaps the last audit list named, all in the "resource growth, background
timers and limits nobody watches" class.

**Idle inline-query entries are pruned.** `DebounceStates` had no eviction at
all: one entry per user who ever used inline mode, forever, while the rate
limiter's buckets and the chat store both prune in the 300s sweep. Entries
now carry a `last_seen` stamp and `prune_idle_states()` drops the ones idle
past 300s — the window Telegram caches an inline answer for
(`cache_time(300)`), after which a repeat reaches the bot again and has to be
answered fresh, so the entry would only suppress a fetch the user is waiting
for. The boundary is tested through `prune_idle_at(now, idle_for)` so it does
not depend on ageing a monotonic clock.

**The 300s sweep is a function, and tested.** It was an inline `tokio::spawn`
block: the expiry edit (the only part that talks to Telegram) had no test at
all. It is now `periodic_sweep(sender, chat_store, link_cache, task_queue,
config, stop)`, which also prunes the inline entries, driven in a test with
`start_paused` — the loop's own timer fires the tick, exactly one expired
prompt is rewritten in place, a live one keeps its record and buttons. The
interval is pinned as a constant because no assertion on the edits can see it
(a shorter one produces the same single edit; the paused clock can jump past
the boundary while a tick's DB work is in flight). To make the edit reachable
at all, `edit_message_text` joined the `MediaSender` trait (Bot impl + mock
recording), which is also what keeps `main.rs`'s remaining `Bot` calls
unambiguous. `main.rs` leaves the "untested modules" list except for
startup/shutdown and the dispatcher tree.

**The bot-wide send budget exists.** Telegram throttles a bot in total
(~30 msg/s) as well as per chat; only the per-chat bucket existed, so a batch
forward fanned out over many chats was unguarded and earned 429s the queue
then retried. `acquire_global` charges the same spend against a single shared
bucket at the three paced sites (`send_media_group`, `send_animation`,
`copy_messages`). The unpaced ones (`send_message`, the edits, the toasts) stay
unpaced on purpose: they are one call per action, far below the ceiling, and
pacing a user-visible reply would delay it. Not covered: that the send paths
call it (they need a real `Bot`), which is the same structural gap as the
dispatcher tree.

Also: the startup token-exchange decision is now `startup_validation(result)`
instead of living inside the `Site::validate` future, so "a 5xx while the
container comes up must not disable pixiv" is asserted as a decision — the
message the admin gets plus `enabled()` unchanged. The rejected-credential
half is deliberately not exercised: it calls `disable()`, a process-wide flag
with no reset, and a test touching it would order-couple every other pixiv
test.

Verified: `cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (184 passed, 14 ignored) — plus
mutations, each confirmed to fail the relevant test: the sweep not being
driven on its timer, the interval shortened to 60s, and (earlier) the queue
sweep's missing wake-up. Dropped an empty leftover `crates/x-media/tests/`
directory while there (never tracked by git).
2026-09-21 01:03:15 +08:00
YoursFunny 3828d5b483 test: share the handler/cache fixtures from ctx::test_support
The same fixtures were rebuilt in five test modules: a `CachedPost`
literal in `link_cache.rs`, `handlers/urls.rs` and twice in
`send/mod.rs`, the edit-before-forward prompt in `handlers/mod.rs` and
`handlers/callback.rs`, and a scripted API error in both handler
modules. They now live in `ctx::test_support` next to `TestStores`:

- `cached_photo()` — the canonical cached post (photo + file id at
  `https://x.com/u/status/1`, key `twitter:1`); tests mutate the fields
  they care about, as the caption-quote test already did.
- `seed_prompt(template, created_at)` + `PROMPT_ID`/`FORWARDED_ID` —
  the prompt record, the chat template and the bound forward channel.
  The two former copies differed only in which knob the caller set (the
  callback tests backdate it for the expiry cases, the reply tests pick
  the template), so the union is one helper.
- `api_error(message)` — construction only; each test module keeps its
  own message constant, because the wording is what that module's path
  answers with (`chat not found` vs `message not found`).

`send/mod.rs`'s `cached_sequence_cache_data()` (which re-extracted the
post out of the task it had just built) is gone: the two settle tests
seed the cache from the same builder the task uses.

No behaviour change: the values are the ones the tests used except
`file_id` (`AgAC-file-id` everywhere, asserted in the link-cache
round-trip) and `sensitive` (the unasserted `true` in the link-cache
fixture), and every test still passes unchanged.

Verified: `cargo fmt`, `cargo clippy --workspace --all-targets --locked
-- -D warnings` and `cargo test --workspace --locked` (180 passed, 14
ignored).
2026-09-21 00:25:01 +08:00
YoursFunny 39dbd0f3a2 test: drop redundant tests, make the vacuous ones real
Audit of all 205 tests (five read-only passes plus a line-by-line
re-check). Ten test functions were removed or merged and eight
subsumed assertion blocks trimmed; the suite is down to 180 tests with
no loss of mutation coverage, and four tests that were passing for
nothing now fail when the code they name is broken.

Redundant (deleted or merged):
- twitter: `syndication_text_only_has_no_media` (re-asserts its own
  empty fixture), `..._keeps_multibyte_text` (both transforms are
  no-ops for that text), `..._strips_trailing_short_link_without_entities`
  (same branch as `..._media_short_link`, which now also covers the
  real multibyte tweet), `..._regardless_of_index_units` (its
  `display_text_range` rationale outlived the function it described).
- pixiv: `test_fetch` (a bare `is_ok()` on the illustration
  `download_media_pixiv_original_with_referer` already asserts and
  downloads, and the only network touch in a plain `cargo test`),
  `startup_validation_only_disables_on_a_definitive_failure` (four rows
  that are a subset of the retry-policy table; the `validate()` branch
  it was named for is not asserted at all).
- bilibili: `from_item_legacy_draw_shape_still_parses` (its fixture is
  the same legacy `draw` shape `from_item_maps_draw_images_and_topic`
  builds, with a subset of its assertions).
- site/mod.rs: two `Ok(None)` cases merged into one test.
- urls.rs: `cache_hit_success_keeps_the_cache_entry` (the `/test` test
  asserts the same two things under stricter settings), plus a
  `assert_ne!` loop that re-states the mapping assertions above it.
- send/mod.rs: `media_group_success_and_forward_ok` (the forward half is
  covered by `post_send_forwards_immediately_when_configured`; the
  `is_ok()` half cannot see the returned file ids), and two boundary
  rows implied by the constant they sit next to.
- commands.rs: the parse tail that `every_documented_invocation_parses`
  already covers per README form, and three `debug_report` rows the
  escaping test pins with stronger input.

Passing for nothing (now real):
- `truncate_caption_does_not_split_an_html_entity` — the cut lands
  inside the entity, so `!contains("&amp")` never fired; it now asserts
  the exact output in both directions and fails when the guard in
  `truncate_caption` is deleted (verified).
- `pipeline_resizes_oversized_jpeg` — magic bytes and a non-empty buffer
  pass for a copy-through; it now decodes the output's headers and
  fails when the JPEG branch skips the resize (verified).
- `live_validate_with_bogus_token_fails` — expected `PixivError::Api`,
  which the status check before the body read made unreachable; a bogus
  token is a 4xx. Confirmed against the live endpoint: the old
  assertion fails with `got Err(Status(400))`, the new one passes.
- bsky `live_fetch_with_photos` — its URL is a text-only post and it had
  a byte-identical twin, so no live test pinned media; it now points at a
  labelled post with photos and asserts media + the label (live-verified).

Also fixed, found by turning the runtime-sweep test into a real one:
the 30 s lease-expiry sweep recovered crashed rows but never woke a
worker, so a recovered task waited for the next unrelated enqueue (every
worker is parked on `notify` when no row is pending). `recover_update`
now reports its count, `recover_expired` wakes a worker when it changed
something, and `runtime_sweep_recovers_expired_lease` drives the spawned
loop with a paused clock instead of calling the recovery by hand — it
fails on both the missing wake-up and a sweep that recovers nothing.

Verified: `cargo fmt --check`, `cargo clippy --workspace --all-targets
--locked -- -D warnings`, `cargo test --workspace --locked` (180 passed,
14 ignored) and `cargo test -p x-media -- --ignored live` (13 passed).
2026-09-21 00:16:11 +08:00
YoursFunny d3560dca52 docs: add .env.example as the deployment template
`cp .env.example .env` is now the documented starting point: the tracked
template carries every variable (grouped required / sites / bot behaviour /
network / webhook / reverse proxy) with the defaults the code would use
anyway, and the compose comment plus both READMEs point at it. The proxy note
is spelled out where it matters — teloxide panics on a blank `TELOXIDE_PROXY`,
and inside a container the proxy host must be `host.docker.internal`.

Two follow-ups the template exposed:

- `RUST_LOG=` (present but blank, which `.env` makes easy) silenced the log
  again: "unset" was handled, "empty" was not. A blank value now falls back to
  the same default. Verified: blank and unset both produce the full startup
  sequence.
- `TWITTER_AUTH_TOKEN` was in the README prose but missing from the env table
  (both languages).

Verified: `docker compose --env-file .env.example config -q` resolves, and a
script comparing the compose's `${VAR}` references against the template's keys
finds none missing.
2026-09-20 22:28:01 +08:00
YoursFunny 24cbfc2f27 chore(deploy): track docker-compose.yml and read instance values from .env
- `docker-compose.yml.example` becomes `docker-compose.yml`, committed as the
  real deployment file (the log caps came along) and no longer gitignored.
  Every instance value is now `${VAR}` with a `${VAR:-default}` fallback, so
  compose reads it from the gitignored `.env` beside the file and the committed
  composition needs no per-deployment edit; a variable not listed in a
  service's `environment:` never reaches that container. Two knobs the docs
  promised but the file lacked are now wired: `DEFAULT_HOST` on nginx-proxy and
  `BILIBILI_COOKIE`/`CAPTION_QUOTE_TEXT_CHARS` on the bot, and the healthcheck
  follows `WEBHOOK_PORT` instead of a hardcoded 8443. `TELOXIDE_PROXY` is
  deliberately *not* passed — a loopback proxy inside a container is the
  container itself, and teloxide panics on a blank value — so the compose
  comment and both READMEs explain when to add it by hand.
- `BILIBILI_PLAN.md` moves to `docs/BILIBILI_PLAN.md` (nothing referenced it).
- Docs synced: AGENTS.md (the deployment-file row, and `.gitignore` no longer
  lists the compose file) and both READMEs (deployment and webhook steps now
  say "set it in .env" rather than "edit the compose", plus the proxy trap in
  the env table).
2026-09-20 22:07:27 +08:00
YoursFunny 4cdf618c25 fix(retry): fence the queue lease, clean up after a kill, name dead-lettered posts
P2 (hardening) of the retry audit, closing the report's remaining findings.

- Lease fencing. `lease_next` now stamps a random `lease_token`, and every
  write-back a worker makes (the 30s heartbeat, `delete_row`, `reschedule`,
  `mark_done`) is guarded by it. A lease that expired while its holder was
  stalled and was then re-leased used to let *both* holders write the same row:
  one duplicated the send, the other silently discarded the new holder's retry
  (a 0-row update was not even logged). Now a worker that no longer holds the
  lease drops its attempt at the next heartbeat and writes nothing. Reaching
  existing databases needed a migration chain, which `db.rs` had been
  pre-committed to: `MIGRATIONS` + `migrate` track `PRAGMA user_version`, with
  `schema_init` as the version-0 baseline. Verified on a database created
  before this change: user_version 0 -> 1, column added, rows intact.
- Dead-letter notifications no longer mislabel an unparsable payload. A row
  whose payload no longer deserializes as a `Task` (an older version's shape,
  corruption) used to skip the cache invalidation *and* report "Forward failed
  permanently" for a send task, because both were derived from the parsed
  value. The identity now comes off the raw JSON, so the stale link-cache entry
  is dropped and the message names the post.
- Temp files are marked and swept. Every temp file/dir the project creates now
  carries `x_media::TEMP_FILE_PREFIX`, and startup removes entries with that
  prefix older than an hour — a killed process leaves its downloads (up to
  hundreds of MB) behind because no destructor runs, and the age gate keeps the
  sweep away from a second instance's in-flight files. Verified live: the log
  reports the sweep, an aged leftover goes, a fresh prefixed file and an
  unrelated file stay.
2026-09-20 21:34:45 +08:00
YoursFunny 0a82ca5a42 fix(retry): let slow downloads finish, and never re-run a finished task
P1 of the retry audit, from the report's "reliability and diagnosis" batch.

- Media downloads no longer share the 30s *total* timeout of metadata
  fetches. The size caps allowed 10 MiB (reupload fallback) and 512 MiB
  (ugoira frame zip) while the clock allowed 30s, so a slow link made those
  posts impossible: `MEDIA_CLIENT` has no total timeout and instead bounds
  the response head and every chunk with a 30s *idle* window, which keeps the
  stalled-connection protection. Verified against a local probe: the old
  policy aborts a 40s download at 30.0s, the new one completes it (2 MiB,
  40.1s), and a body that stops delivering still fails after exactly 30s.
- A finished row's write-back is no longer best-effort. `delete_row` failing
  left the row `in_progress` with a live lease, so the next sweep flipped it
  back to `pending` and re-ran a completed task — a second album, a second
  prompt, a second channel copy. Both terminal writes are now retried, and a
  delete that still fails falls back to a `done` tombstone that neither the
  lease query nor the sweep looks at; reschedule (no safe tombstone: marking
  it done would drop the retry silently) logs what the sweep will do.
- bsky and pixiv no longer present a *failed* video conversion as a post with
  no media: the remux/ugoira error propagates (pixiv keeps its retry class,
  bsky reports Transient), so the user sees the real cause and `fetch` gets
  its retries. bsky's "no ffmpeg" case stays a degradation — retrying a
  deployment gap cannot help.
- pixiv's token exchange checks the HTTP status before parsing the body, so a
  429/5xx from the OAuth endpoint stays retryable instead of becoming a
  permanent Api/Json error (via the shared `pixiv_error_is_retryable`), and
  startup validation only disables pixiv for a rejected credential — one 503
  while the container came up used to turn every later pixiv link into
  "pixiv support is disabled".
2026-09-20 21:02:46 +08:00
YoursFunny 4cf793cd7e fix(retry): stop losing posts to transient failures and broken promises
P0 of the retry audit. The main finding: a Telegram 5xx was classified
Permanent, so one Telegram-side blip dead-lettered the post.

- `classify_request_error`: a server error is retryable again. teloxide sleeps
  10s on a 5xx and then parses the body, so the HTTP status is gone by the
  time the error arrives; it is recognised by shape instead — a JSON
  server-error description, or an `InvalidJson` whose raw body is not JSON
  (a proxy/error page). A JSON body of the wrong shape stays permanent, since
  retrying a type mismatch cannot help. Reproduced end to end: with the old
  classification a fake 502 (HTML body) logged "failed permanently" and
  dead-lettered; now it logs "queued for retry" and the retry delivers.
- The same class of mistake elsewhere: `is_media_fetch_failure` was missing
  `failed to get HTTP url content`, the description single-media URL sends
  answer with, so hotlink-rejected media failed permanently instead of going
  through the reupload fallback.
- `enqueue_retry` now reports whether the row was written, and the callers
  only promise a retry when it was — a failed enqueue (DB write) used to tell
  the user "retrying in Ns" and then deliver nothing, ever.
- A forward that fails retryably now settles the prompt instead of leaving it
  live: the queued row carries the message ids itself, and a live prompt let
  a second Confirm copy the same messages to the channel twice and let Skip
  answer "nothing was forwarded" while the row still delivered.
- A prompt that could not be sent no longer swallows the gated forward
  silently: the chat is told, since nothing would ever forward.
- `scaled_retry_delay` only scales up, so a server-asked `retry_after` above
  the 300s cap is honoured instead of retried early (which earned another 429
  and then dead-lettered the post).
- Download classification: a 4xx media download is permanent (the media is
  gone or refused) while transport errors and 429/5xx retry — previously every
  download error counted as retryable and burned the whole budget. A temp-file
  *write* failure retries too (resource exhaustion clears; a temp dir that
  cannot be created stays permanent).
- Site status mapping: 401/403 are `Blocked` (permanent) rather than
  `Transient`, so a refusal is reported at once instead of after three
  wasted attempts; and a twitter 200 that is not a tweet is no longer
  reported as withheld content (the empty `{}` withheld shape keeps
  `Sensitive`, which is what triggers the auth fallback).
2026-09-20 20:46:18 +08:00
YoursFunny 36e5e8afe6 chore(log): cap container log growth and echo the resolved config
P2 of the logging plan (the README recipe landed with the code change):

- `docker-compose.yml.example`: one `x-logging` anchor applied to all three
  services. json-file grows without limit by default, so a long-running bot
  and the proxy in front of it fill the disk; capped at 10m × 3 files.
- The startup `config:` line now reports what the process actually resolved —
  the state DB path (a mistyped `DATA_DIR` or a surprising CWD was invisible
  until it bit), both TTLs, the caption-quote setting (`off` rather than a bare
  `0`) and whether a proxy is configured. The proxy URL is never printed (it
  may embed credentials) and admin ids — chat identifiers — stay at `debug`.

Verified: `docker compose config -q` accepts the file, and a scripted fake-API
run shows `caption quote off` / `link cache TTL 3600s` under overrides,
`proxy=yes` with no credential in any line, and the ids at `debug` only.
2026-09-20 19:14:33 +08:00
YoursFunny 3f9821d475 feat(log): survive a bare deployment and name what each line is about
P0 (foundation) + P1 (diagnostic depth) of the logging plan:

- main.rs initializes the timed builder with a default filter of
  `info,hyper_util=warn,reqwest=warn`. Without RUST_LOG nothing was logged at
  all (env_logger falls back to `error`), so `docker run --env-file .env` was
  silent, and the plain `init` had no timestamps.
- info-and-above lines stop printing user URLs (fetch/send failures, inline
  fetch, bsky's remux warnings). The full URL, the message text and the inline
  query move to `trace`, so a `debug` log can be handed to someone else.
- Lifecycle lines name the chat and the post: sent/failed/queued plus the
  total `ms`, the edit prompt, the channel forward, and every queue line
  (`chat=` + `[key=…]` + per-attempt `ms`, dead-letters included).
- Queue work is visible: `x-media`'s fetch line carries its duration (ugoira
  encode and HLS remux included), and the 300s sweep reports the pending count
  and how overdue the oldest task is — only when the queue is non-empty.
- URL workers are supervised like the queue workers: a panicking worker used
  to die silently and shrink the pool for the rest of the process.
- Degradations that still serve the user (cache/state write or read failures,
  a failed chat action) are `warn`, not `error`.

Verified against the scripted fake-API harness: unset RUST_LOG logs info with
timestamps, `debug` carries no user URL, `trace` does, a cache-hit send logs
`chat=111 in 5ms`, a failing send queues and dead-letters with chat+key, and
the sweep reports the pending retry.
2026-09-20 19:07:23 +08:00
YoursFunny 9e873131d4 chore: bump version to 1.8.0 2026-09-20 17:41:20 +08:00
YoursFunny 5b77d14497 fix(commands): make /debug preview the caption a link would actually send
`/debug` reported `Fetched::caption`, the site's built-in caption, so a
chat's `/set_format` override never showed up in the preview — the command
looked like a no-op, and the `/set_format` success reply now tells users to
preview with `/debug`, which made that advice wrong.

`preview_caption` mirrors the send paths instead: the per-site format
override (`caption_from_fields`, empty → built-in) plus the long-post
quoting. Reported from a live run against the test bot.
2026-09-20 17:30:55 +08:00
YoursFunny d4c36feb9a fix(commands): make /set_format and /clear_cache actually parse
teloxide's `split` parser takes exactly one space-separated token per
field, so `/set_format <site> <format>` — a two-token command — never
parsed: `Command::parse` failed, `message_handler` fell through to the URL
flow, and the user got silence. `/clear_cache` without its optional link
failed the same way ("too few arguments"), so clearing everything was
unreachable. Both now use the crate's `parse_arg_remainder` (whole
remainder, trimmed), which is what their executors were already written
against (`split_once(char::is_whitespace)`).

Found by driving the real binary against a scripted fake Bot API: the
`/set_format` replies never appeared while `/set_template` and the other
single-token commands did. `every_documented_invocation_parses` now pins
every documented form, which is what should have caught it.

The placeholder validation and `-` reset added earlier only work now that
the command reaches its executor at all.
2026-09-20 16:42:40 +08:00
YoursFunny 5d0acdac01 feat(ux): onboard users, expose the chat's settings, name failed posts
`/start` was "Hello!" and `/help` was the bare command list teloxide can
render — no argument syntax, no caption placeholders, no mention that
links only work in private chats. Both now carry that guidance, and the
bot's profile description / short description are set at startup so a
shared link says what the bot does.

`/settings` reports what this chat is configured to do (forward channel,
edit-before-forward, per-site formats, saved templates) to anyone in the
chat — `/bot_dict` is a raw admin-only dump. Templates can be removed
(`/remove_template`, listing the live names on a typo) and the prompt's
keyboard folds 3 per row with a cap: Telegram rejects a keyboard over 100
buttons outright, which would silently drop the whole prompt.

Inline results hand URLs to Telegram, which fetches them without any
site headers — pixiv's pximg.net answers 403 to that, so those items are
skipped instead of shipped broken. `needs_media_headers` answers that
question from the same per-site rule the downloader uses.

Dead-letter and retry notices name the failing post and the cause
(`failure_text`), since "Task failed after retries: task failed after 2
retries" said neither which link it was nor what happened.
2026-09-20 15:57:45 +08:00
YoursFunny d6707133cc feat(ux): answer every link, name fetch failures, keep the chat action alive
Four ways a user could get silence are closed: a registered-but-disabled
site (pixiv without a token) now answers instead of being dropped as an
unsupported link, `/test` on such a link replies instead of doing nothing,
a supported link posted in a group gets a one-line hint (channels stay
silent), and fetch failures name their cause — gone / withheld / source
risk control / site disabled / source down — instead of one generic
sentence. `FetchError::Disabled` carries the "matched but switched off"
answer, which `find_site` used to fold into `Ok(None)`.

A withheld tweet no longer degrades to "no media": without
`TWITTER_AUTH_TOKEN` it stays `Sensitive` so the reply says the media is
age-restricted, and a failed authenticated fallback propagates its own
class instead of masquerading as an empty post (`empty_fetched` is gone).

Long jobs stop looking stalled: `run_with_chat_action` re-sends the chat
action every 4s while the pipeline is pending and the hint switches from
typing to send-photo/video once the media kinds are known. Media groups
go from 9 to Telegram's 10.

`/set_format` rejects unknown `{…}` placeholders (a typo used to be
published verbatim in every caption) and resets with `-`. The
edit-before-forward prompt states its TTL and that Confirm is required,
gains a Skip button, and is rewritten in place to "expired" by the sweep
— an edit, never a new message, so a background timer cannot wake a chat.
2026-09-20 15:48:46 +08:00
YoursFunny fb601f4d5d chore: bump version to 1.7.0 2026-09-18 01:17:45 +08:00
YoursFunny d8dd4fa91e test(cache): pin that pre-split entries still parse
A payload written before the title/content split has no `content` field;
`#[serde(default)]` is what keeps it readable, and the cache deletes any
payload it cannot parse — so dropping that default would silently evict
entries rather than degrade them. The test inserts the literal pre-split
JSON and asserts it comes back with its text left in `title` (no
migration: the entry lives one TTL and moving the text would only
reshuffle `/set_format` placeholders until it expires) and its stored
caption untouched.
2026-09-18 01:14:09 +08:00
YoursFunny af96caff40 feat(send): quote a long post's text in an expandable blockquote
A post whose text (the split `title` plus `content`, joined by
`site::compose_text`) reaches `CAPTION_QUOTE_TEXT_CHARS` — default 200,
`0` disables — now has that text wrapped in `<blockquote expandable>`
inside its caption, leaving the URL and author line outside the quote.

Applied at the send boundary (`send_media_sequence`, `send_animation` and
the inline answers), where the caption is already truncated and the same
cache snapshot supplies the text, so a fresh send, a link-cache resend
and a queued retry all decide identically. The text is located as what
follows the author link, with the visible prefix accepted as a match
because `truncate_caption` may cut inside it — that keeps the longest
posts, the ones that most need folding, quoted. Captions whose layout
moves the text elsewhere (pixiv's title-inside-a-link, a `/set_format`
that puts `{title}`/`{content}` first) stay unquoted rather than risking
a blockquote nested in a tag, and a caption that already carries one is
never wrapped again.

Telegram measures a caption *after entities parsing*, so the tags cost no
length and the 1024-character limit cannot be breached; retries replay
the unwrapped caption, so a threshold change takes effect immediately.
The edit-before-forward rewrite stays unquoted by design.

Verified against Telegram: a media-group caption built this way comes
back with `caption_entities` `url` @0, `text_link` @50,
`expandable_blockquote` @56 — the quote starts after the author line.
2026-09-18 00:50:30 +08:00
YoursFunny 52184ba6fb refactor(x-media): split the post title from its content
`Fetched.title` carried whatever text the platform had — a tweet's body,
a bilibili dynamic's body, a pixiv artwork's title — which was enough
while x/twitter (no title at all) set the shape. The platforms actually
disagree: pixiv has a title *and* a description, bilibili has an opus
headline *and* a body. Posts now carry both:

- `title`: the platform's title (a pixiv artwork title, a bilibili opus
  headline or video card title), empty on text-only platforms;
- `content`: the body (tweet / bsky / misskey text, bilibili dynamic
  body, and pixiv's description — fetched for the first time here and
  flattened from the app API's HTML to plain text).

`{content}` joins the caption-format placeholders, so a custom
`/set_format` can include a pixiv description. The built-in captions keep
producing byte-identical output: `compose_text` joins the two fields the
same way the single field already was, and bilibili's forward marker
(`//@author:`) now lands in `content` behind the head line's `title`.

`CachedPost.content` is `#[serde(default)]`, so link-cache entries and
queued task payloads written before the split still parse, their text
living in `title`.
2026-09-18 00:44:49 +08:00
YoursFunny 0eb4e5c78d fix(sites): request the opus serialization so bilibili posts keep their text
An image/text post fetched without `features=itemOpusStyle` comes back in
bilibili's legacy shape, where the post's body and headline are gone
completely — `desc: null`, no `major.opus` — so `title` (and `{title}`)
stayed empty for exactly the posts that do have content
(`opus/1248857553488576532`: legacy `desc: null`, flagged
`major.opus.summary.text = "[doge_金箍]黑白搭配"`). The same flag also
moves the pictures to `major.opus.pics` (key `url`, not `src`).

Text now falls back opus (headline + body) → `desc.text` → archive card
title; media falls back `opus.pics` → `draw.items` → archive cover, so
the legacy shapes keep working if the flag is ever retired.

Verified live: the reported link now yields title "[doge_金箍]黑白搭配"
with its picture; AV dynamics keep their card title; forwards keep
`desc.text` and gain `//@` composition unchanged.
2026-09-17 22:34:11 +08:00
YoursFunny 5c51de217a fix(sites): use the archive title when a bilibili dynamic has no body
A 视频投稿动态 (`DYNAMIC_TYPE_AV`) carries no body at all: the API
answers `desc: null` and the content is the archive card, so `title`
(and `{title}` in caption formats) stayed empty for the most common
dynamic type. Audited 24 live dynamics: every dynamic that *has* text
(a 图文 post, a forward, a text post) keeps it in
`module_dynamic.desc.text` — only the AV card has none, so the video
title now stands in, mirroring pixiv whose `title` is the artwork title
rather than post text.

Also records two API observations in comments/docs: an id that cannot
exist answers `4101105 请求数据发生错误` (kept on the permanent arm), and
the feed endpoints strip `desc.text` so only the detail endpoint shows
whether a post has text.
2026-09-17 22:12:53 +08:00
YoursFunny c1f5d3ca54 feat(sites): add bilibili dynamic support (images and animated images)
Fetch `t.bilibili.com/<id>`, `www.bilibili.com/opus/<id>`,
`t.bilibili.com/h5/dynamic/detail/<id>` and `m.bilibili.com/dynamic/<id>`
through the anonymous `/x/polymer/web-dynamic/v1/detail` endpoint (no
cookie, no WBI signature; the site adds the device cookies
`/x/frontend/finger/spi` hands out, which is what lifts bilibili's
`-352` risk control).

Media: the `major.draw` grid (`.gif` sources become animations, the rest
photos with a downscaled `@518w.jpg` thumbnail used both as preview and
as the oversized fallback), an attached video's cover, and the quoted
dynamic's media for forwards. The video stream itself is not resolved;
`b23.tv` short links stay unmatched (they mostly point at videos, so
matching them would turn a silently ignored link into a failure reply).

`-352`/`-412` map to a retryable error so the queue backs off instead of
dropping the post; a removed dynamic (`500`) is permanent.

Registry-driven, so no bot-side code changes beyond the site lists in the
command replies; found while researching nazurin and
telegram-bili-feed-helper (see BILIBILI_PLAN.md).
2026-09-17 20:48:15 +08:00
YoursFunny 1bb6968108 Merge pull request #3 from TheFunny/dependabot/cargo/rand-0.10.2
build(deps): bump rand from 0.8.8 to 0.10.2
2026-09-17 17:05:30 +08:00
YoursFunny 14b444d109 Merge master into the rand 0.10 migration
Resolve the manifest and lockfile overlap with the rusqlite 0.40 and zip 8.6
bumps that landed on master after this branch was opened:
- crates/xmedia-bot/Cargo.toml: keep rusqlite 0.40 and rand 0.10
- Cargo.lock: re-point x-media/xmedia-bot at rand 0.10.2; teloxide keeps its
  own rand 0.8.8 (it requires ^0.8.5), and cargo pruned the now-unused
  version_check entry

Verified on the merged tree: cargo metadata --locked, cargo fmt --check,
cargo clippy --workspace --all-targets --locked -- -D warnings,
cargo test --workspace --locked (69 + 73 tests).
2026-09-17 16:59:27 +08:00
YoursFunny de3105d4cd Merge pull request #2 from TheFunny/dependabot/cargo/zip-8.6.0
build(deps): bump zip from 2.4.2 to 8.6.0
2026-09-17 16:54:44 +08:00
YoursFunny c46103a23f Merge pull request #1 from TheFunny/dependabot/cargo/rusqlite-0.40.2
build(deps): bump rusqlite from 0.32.1 to 0.40.2
2026-09-17 16:54:00 +08:00
YoursFunny 9529f64b41 fix(send): migrate rand usage to the 0.10 API
rand 0.10 renamed the entry points dependabot's version bump alone cannot
follow: `thread_rng()` -> `rng()`, `gen_range()` -> `random_range()` and the
`Rng` trait -> `RngExt`.

The jitter only needs one float, so use the free function
`rand::random_range(0.2..0.8)` and drop the now-unused trait import instead
of importing `RngExt`.

Verified: cargo fmt --check, cargo clippy --workspace --all-targets -- -D
warnings, cargo test --workspace --locked (retry_delay_seconds_bounds keeps
the 0.2..0.8 jitter window).
2026-09-17 16:43:01 +08:00
dependabot[bot] d0de17329d build(deps): bump rand from 0.8.8 to 0.10.2
Bumps [rand](https://github.com/rust-random/rand) from 0.8.8 to 0.10.2.
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/0.8.8...0.10.2)

---
updated-dependencies:
- dependency-name: rand
  dependency-version: 0.10.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-09-17 03:43:17 +00:00
dependabot[bot] 9af37e92b4 build(deps): bump zip from 2.4.2 to 8.6.0
Bumps [zip](https://github.com/zip-rs/zip2) from 2.4.2 to 8.6.0.
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v2.4.2...v8.6.0)

---
updated-dependencies:
- dependency-name: zip
  dependency-version: 8.6.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-09-17 03:43:10 +00:00
dependabot[bot] e68a1dbd30 build(deps): bump rusqlite from 0.32.1 to 0.40.2
Bumps [rusqlite](https://github.com/rusqlite/rusqlite) from 0.32.1 to 0.40.2.
- [Release notes](https://github.com/rusqlite/rusqlite/releases)
- [Changelog](https://github.com/rusqlite/rusqlite/blob/master/Changelog.md)
- [Commits](https://github.com/rusqlite/rusqlite/compare/v0.32.1...v0.40.2)

---
updated-dependencies:
- dependency-name: rusqlite
  dependency-version: 0.40.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-09-17 03:43:05 +00:00
YoursFunny b9c6d16ff0 docs(ci): note why the docker job skips actions/checkout
build-push-action defaults to the Git context, so BuildKit clones the
repo itself and the job never needs the workspace.
2026-09-17 11:38:49 +08:00
YoursFunny ec65c3ce74 chore: bump version to 1.6.0
Bump both crates (x-media, xmedia-bot) and refresh the lockfile to the
latest semver-compatible releases. No direct dependency or manifest
requirement changed.
2026-09-17 11:21:58 +08:00
YoursFunny 893ab7a1e0 feat(commands): /test sends the media, /debug takes over the parse report
- `/test <url>` now runs the ordinary link pipeline and actually sends the
  media, but with the chat's post-send actions suppressed: no channel forward,
  no edit-before-forward prompt. It is the same code path as a normal link
  (same caption/format handling, link cache, retries, dead-letter
  notification), so "does this link work?" is answered by the send itself.
- `/debug <url>` keeps what `/test` used to do: fetch and reply with the HTML
  parse report, sending/caching/forwarding nothing.
- `urls::url_media` takes a `PostSend` mode (`FromChat` for the URL workers,
  `Suppressed` for `/test`); `build_send_task` maps it to the task's
  `edit_before_forward`/`forward_channel_id`. Notification ids stay set in both
  modes, so a queued retry still reports a dead-letter to the chat.
- `/test` rejects an unsupported URL with the same message the old parse-only
  command used (the URL flow would otherwise ignore it silently).
- Report builder renamed `test_parse_report` -> `debug_report` (with the cap
  constant), `parse_test_arg` -> `parse_arg_remainder` (now shared by both
  commands). README/README.en command tables and AGENTS.md updated; `/help`
  descriptions come from the enum.

Tests: +3 (normal flow still honours the chat's settings, `/test` sends with
them suppressed and keeps the cache entry, `build_send_task` mode mapping). The
suppression test was verified to fail when the mode is ignored.
fmt/clippy clean, 73 + 69 tests pass.
2026-09-17 02:25:42 +08:00
YoursFunny bd032e3d68 ci: lock the dependency set, verify the build inputs on PRs, harden the jobs
- `--locked` on every cargo invocation (ci.yml clippy/test/build, both
  Dockerfile builds). The version bump edits Cargo.lock by hand, so a stale
  lock must fail loudly instead of being silently re-resolved: CI would
  otherwise test a different dependency set than the one committed — and than
  the one the released image is built from.
- docker.yml: build the image (no push, no registry login, read-only build
  cache) on pull requests touching the build inputs. The Dockerfile's
  stub-source machinery, the ffmpeg download and the entrypoint previously
  only ran at release time. Also: a release tag must equal both crate versions
  before anything is built (the binary carries no version, so `v1.5.1` with
  manifests at 1.5.0 used to publish silently wrong tags), `FFMPEG_URL`/
  `FFMPEG_SHA256` are taken from repository variables when set, and the
  unused `setup-qemu-action` step is gone (single-arch build; the comment says
  what arm64 would need).
- ci.yml: `concurrency` cancels superseded runs, `permissions: contents: read`,
  `RUST_BACKTRACE=1`, job timeouts, and a release-profile build of the same
  package the Dockerfile builds (the profile was otherwise never compiled
  before a merge). The `live` job narrows to `-p x-media`: every network- or
  secret-gated test lives there, and the bot crate's offline suite already ran
  in the `test` job. Timeout is 45 min because the release build is cold on
  the first run — a timeout there would kill the job before rust-cache could
  save its cache, leaving every later run cold too.
- Actions pinned to commit SHAs (Dependabot keeps them current);
  `dtolnay/rust-toolchain` stays on its channel ref by design.
- .github/dependabot.yml: crates (patch bumps grouped), action pins, Docker
  base images — the audit gate reports advisories, this is what moves them.
- tokio's `sync` feature is now declared instead of arriving transitively via
  teloxide; `.dockerignore` drops docs and markdown.

Verified locally: `cargo fmt --check`, `cargo clippy --workspace
--all-targets --locked`, `cargo test --workspace --locked` (70 + 69 pass),
`cargo build --release --locked` (6m03s cold, the 15.9 MB stripped binary
starts and registers 10 commands), the tag/version gate against both a
matching and a mismatching tag, and YAML parsing of all three workflow files.
2026-09-17 02:01:43 +08:00
YoursFunny dbda6ec1c2 refactor(send): split the 2100-line module by concern
send.rs had grown back into the shape handlers.rs was split out of: payload
types, error classification, the download-and-reupload fallback, the senders
and the whole post-send/queue shell in one file. Split by concern, leaving
call sites (`crate::send::x`) unchanged:

- `send/input_media.rs`: payload → `InputFile`/`InputMedia` selection and
  `build_media_group` (with its caption-on-first-item rule).
- `send/upload.rs`: the fallback pipeline (download with the upload cap,
  photo downscale handoff, smaller-URL fallback, multipart upload).
- `send/post_send.rs`: link-cache write, the `KEEP_ALIVE` registry for locally
  produced media, `settle_task`, the post-send actions and the queue entry
  points; the parts other modules call are re-exported.
- `send/mod.rs`: payloads, error classification, classification helpers and
  the senders themselves, plus the test module.

No behaviour change: 128 + 1326 + 366 + 345 lines, 70 tests still pass.
AGENTS.md updated for the new layout and for `ctx.rs`.
2026-09-17 01:41:28 +08:00
YoursFunny 0a9ff58a69 refactor: cover the edit/answer surface in MediaSender, test the button flows
docs/architecture-refactor.md §3 sketched the trait with "按需扩展:
edit_message_caption / delete_message / answer_callback_query …", but only the
five send methods landed, so `callback.rs` and the edit-before-forward caption
swap were stuck on the concrete `Bot` and remained untested (AGENTS.md still
lists callback.rs as untestable).

- `MediaSender` gains `answer_callback_query`, `edit_message_caption` (HTML
  parse mode baked in, every caller uses it) and `delete_message`; the mock
  records call order plus the texts, captions and answer toasts, so tests can
  assert what the user saw.
- `send_message` now returns the sent message id instead of the whole
  `Message`: the only consumer of the value is the edit-before-forward prompt
  (which keys its record by it), and returning a `Message` forced every mock
  to build a teloxide type. `reply`/`reply_html` follow.
- `callback.rs`: the dptree entry only unpacks the update; `handle_callback`
  takes plain values + `&AppContext`. `handlers/mod.rs::edit_message_handler`
  likewise takes the values the reply carries. Admin/setup APIs
  (`get_chat`, `get_chat_administrators`, `get_me`, `set_my_commands`) stay on
  the concrete `Bot`: they are not user flows worth a trait.
- The scripted mock moves to `parking_lot::Mutex` (no poisoning unwraps).

Tests: +11 (template button, forward ok/no-channel/retryable, expired+unknown
prompt, caption swap via template, escaping of user text into the caption,
failed swap still consuming the reply, prompt record written by post_send).
fmt/clippy clean, 70 + 69 tests pass.
2026-09-17 01:31:30 +08:00
YoursFunny c2d7c8406e refactor: finish the phase-B seam for the post-send path, funnel settlement
docs/architecture-refactor.md §3 stopped half-done: `url_media` got an injected
`AppContext`, but `send.rs`'s post-send half kept reaching for the process-wide
`CHAT_STORE`/`TASK_QUEUE`/`LINK_CACHE` statics, so the whole shell after a
successful send (edit-before-forward prompt, channel forward, retry enqueue,
cache write) had no test and no way to get one.

- `ctx.rs` now owns `AppContext` (sender + the three stores + config) with
  `from_statics` for production and a `CONTEXT` static for the spawned worker
  closures; `handlers/urls.rs` drops its private copy and the duplicated
  assembler, and the queue handler/dead-letter callbacks take the context
  (main wires them with `CONTEXT`).
- `send_media_sequence`/`send_animation`/`forward_messages`/`post_send_actions`
  take `&AppContext`; the cache write goes through the injected cache.
- New `settle_task(ctx, task, Sent|Failed)` is the single place that ends a
  task: release its keep-alive temp media, and drop the link-cache entry only
  on failure. All five former call sites funnel through it — the earlier
  keep-alive leak existed precisely because one of them had to remember.
  `invalidate_cache`/`invalidate_cache_with` (static + injected pair, the
  latter only existing because of the former) collapse into one private fn.
- `ctx::test_support::TestStores` gives tests a tempdir store set + context;
  `handlers/urls.rs` tests use it instead of hand-rolled setup.

Tests: +5 (post-send forward ok / queued / notified, settle Sent/Failed); the
post-send and settle paths were previously untested. fmt/clippy clean,
60 + 69 tests pass.
2026-09-17 01:27:05 +08:00
YoursFunny abdc27ed5e docs: resync AGENTS.md and the doc comments with the code
AGENTS.md:
- db.rs row claimed a per-store connection pool; there is one shared pool for
  all three tables (statics.rs builds it once).
- retry enqueue moved to send.rs, noted in both handlers rows.
- queue row now names both notifies (workers' + the sweep's).
- Retries bullet documents fetch vs fetch_once.
- test count ~125 -> ~135, the untested-files list no longer claims state.rs
  and handlers.rs are untested, and the live-test inventory mentions the
  token-gated, not-#[ignore]d pixiv download test that makes a local
  `cargo test --workspace` hit the network.
- /bot_dict is admin-only now.

Code docs:
- site/mod.rs: the module doc pointed new sites at `fetch_once` (a name that
  did not exist then and now means a single-attempt fetch) -> `SITES`; the
  cache_key/SITES/Site docs still said "twitter -> bsky -> pixiv" (misskey
  is registered third); RenderData now documents which fields are escaped
  and why url/author_url are not.
- state.rs, callback.rs: drop the pre-misskey site list and the `<name>`
  that rustdoc read as an HTML tag.
- Fixed the remaining rustdoc links/warnings: `cargo doc --workspace
  --no-deps` is now warning-free (was 8).
- docs/site-registry-refactor.md: §1 describes the pre-refactor state; said so.

No behavior change. fmt/clippy clean, 55 + 68 tests pass (the live pixiv
download test flaked on a CDN body timeout, as before).
2026-09-16 23:28:46 +08:00
YoursFunny 3fb8421c3a perf: stop the sweep stealing worker wakeups, retry-free inline fetch
- queue: the lease-expiry sweep waited on the workers' `Notify`. `notify_one`
  stores a permit, so a sweep wakeup could consume the one meant for a worker,
  which then blocked on `notified()` (it only waits when the table looked
  empty, i.e. indefinitely) with a due row sitting there. The sweep now has
  its own notify, woken only by stop.
- x-media: split fetch's retry loop into `fetch` (3 attempts, unchanged) and
  `fetch_once` (1 attempt); inline queries use the latter — the 800ms debounce
  plus 1s/2s backoffs were outlasting the answer window of the query.
- send: chunk_media_items now moves items out of the input Vec instead of
  requiring `T: Clone` and copying every payload.

fmt/clippy clean, 55 + 69 tests pass.
2026-09-16 21:23:53 +08:00
YoursFunny 475cfd18f9 fix: harden the debug command, link cache and rate limiter
- commands: /bot_dict dumped the whole chat state to any member of the chat
  and could exceed Telegram's 4096-char message limit (the send then failed
  and bubbled up as a handler error). It is now admin-only and capped at
  MAX_DEBUG_DUMP_CHARS; README, README.en and the /help description updated.
- send: the edit-before-forward template buttons were built from a HashMap
  walk, so their order changed between prompts. Now sorted by name.
- link_cache: an unparseable payload (older schema) was reported as a miss
  but left in place, re-failing the parse on every later hit; the row is
  dropped on read.
- handlers: a link handed to the URL workers after the channel closed
  (shutdown) was discarded silently; it is now logged.
- rate_limit: LIMITERS kept one bucket per chat that ever sent media,
  forever. The periodic sweep now drops buckets that are idle (refilled to
  capacity) and not held by an in-flight sender; acquire's refill was
  factored into a shared helper used by the idle check.

Tests: +3 (corrupted row dropped, sorted markup, idle-bucket pruning); the
cache one was verified to fail before the fix. fmt/clippy clean, 55 + 69.
2026-09-16 21:16:14 +08:00
YoursFunny 0a577600fd fix: repair four correctness defects in the send/state/handler paths
- send: the PHOTO_INVALID_DIMENSIONS marker never matched (the description is
  lower-cased, the marker was not), so oversized photos sent by URL were
  classified Permanent instead of taking the download-and-downscale fallback.
- inline: the debounce state was one global slot, so a second user's query
  cancelled the first user's pending answer entirely; it is now per user.
- state: prune_expired wrote back a stale snapshot without the per-chat lock,
  clobbering a concurrent update() (lost edit-message record -> "Expired");
  it now re-reads and prunes under the same lock update() uses.
- send/queue: a task dead-lettered on retry exhaustion kept its keep-alive
  temp media (ugoira MP4) alive until process exit; dead_letter_notify now
  releases it, and enqueue_retry releases when the enqueue itself fails.

Also folds the duplicated retry enqueue in post_send_actions into
send::enqueue_retry (single clock source, single place that releases).

Tests: +6 (marker, per-user debounce x3, keep-alive release, prune contract
x2); the marker and keep-alive cases were verified to fail before the fix.
cargo fmt/clippy clean, 52 + 69 tests pass.
2026-09-16 21:11:39 +08:00
49 changed files with 10622 additions and 2707 deletions
+4
View File
@@ -26,6 +26,10 @@
*.db
LICENSE
README.md
# Documentation and scratch files: the build only ever reads the manifests,
# `crates/` and the entrypoint script.
docs/
*.md
data/
cert/
nginx-certs/
+76
View File
@@ -0,0 +1,76 @@
# Copy to `.env` (gitignored) and fill in:
#
# cp .env.example .env
#
# `docker compose` reads it for the `${VAR}` substitutions in
# docker-compose.yml, and `cargo run` reads it through dotenv. Every variable is
# described in README.md ("环境变量说明" / "Environment variables") — this file
# only shows the shape, with the defaults the code would use anyway.
# --- required -------------------------------------------------------------
# Token from @BotFather. Without it the bot exits at startup.
TELOXIDE_TOKEN=
# --- sites (all optional) -------------------------------------------------
# Pixiv: refresh token. Unset = pixiv links answer "support is disabled".
PIXIV_REFRESH_TOKEN=
# Twitter/X: the `auth_token` cookie of a logged-in session, used only for
# NSFW tweets that the public syndication endpoint withholds.
TWITTER_AUTH_TOKEN=
# bilibili: the whole cookie string; only needed when the egress IP stays
# risk-controlled (device cookies are fetched automatically).
BILIBILI_COOKIE=
# --- bot behaviour --------------------------------------------------------
# Admin chat IDs, comma-separated: start/stop notices, admin-only commands.
BOT_ADMIN=
# Log level. Leave the line commented out for the default
# (`info,hyper_util=warn,reqwest=warn`); do not set it to an empty value.
# RUST_LOG=info,xmedia_bot=debug,x_media=debug
# Edit-before-forward record TTL (seconds).
EDIT_MESSAGE_TTL_SECONDS=86400
# Link-result cache TTL (seconds).
LINK_CACHE_TTL_SECONDS=604800
# Wrap a post's text in a collapsible blockquote from this many characters on;
# 0 disables the wrap.
CAPTION_QUOTE_TEXT_CHARS=200
# State directory (local runs only — the container uses /app/data).
DATA_DIR=data
# --- network --------------------------------------------------------------
# HTTP proxy for the Bot API and site fetches. Two traps: teloxide panics on a
# *blank* value, so comment the line out rather than leaving it empty; and
# inside a container the proxy must be reachable from there (use
# host.docker.internal, not 127.0.0.1 — that is the container itself).
# docker-compose.yml does not pass this variable unless you add it to the bot
# service's `environment:` block.
# TELOXIDE_PROXY=http://127.0.0.1:10808
# --- webhook deployment (docker-compose.yml) ------------------------------
# false = long polling (no public URL needed). true = webhook behind the
# bundled nginx-proxy — and then WEBHOOK_LISTEN/PORT/URL are required.
WEBHOOK=false
# WEBHOOK_LISTEN=0.0.0.0
# WEBHOOK_PORT=8443
# WEBHOOK_URL=https://your.domain/
# Validation token Telegram echoes back as X-Telegram-Bot-Api-Secret-Token.
# WEBHOOK_SECRET_TOKEN=
# Self-signed certificate path, used only for Telegram-side validation (TLS is
# terminated by the reverse proxy); unneeded with acme-companion. Not passed by
# docker-compose.yml — add the line there if this deployment needs it.
# WEBHOOK_CERT=/app/cert/cert.pem
# --- reverse proxy (docker-compose.yml) -----------------------------------
# Public domain or IP that nginx-proxy routes for; empty = do not route.
VIRTUAL_HOST=
# Port inside the bot container nginx-proxy forwards to.
VIRTUAL_PORT=8443
# Certificate notification address for acme-companion.
DEFAULT_EMAIL=
# UID the container runs as; it must be able to write ./data on the host.
LOCAL_USER_ID=1000
# Uncomment (here and the matching line in docker-compose.yml) to have
# acme-companion issue the certificate for VIRTUAL_HOST.
# ACME_HOST=
# Send requests with an unknown Host to this vhost (needed for plain-IP access).
# DEFAULT_HOST=
+32
View File
@@ -0,0 +1,32 @@
version: 2
# Pairs with the `actions-rust-lang/audit` gate in ci.yml: the gate reports
# advisories in Cargo.lock, this is what actually moves the dependencies.
# Patch bumps are batched into one PR; minor/major stay separate so they get
# reviewed and tested individually.
updates:
- package-ecosystem: cargo
directory: /
schedule:
interval: weekly
open-pull-requests-limit: 5
groups:
cargo-patch:
applies-to: version-updates
patterns: ['*']
update-types: ['patch']
# The workflow actions are pinned to commit SHAs; that pin is what makes
# bumping them a manual chore, so let the bot do it.
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
open-pull-requests-limit: 5
# The Dockerfile's base images (rust:1-bookworm, debian:bookworm-slim).
- package-ecosystem: docker
directory: /
schedule:
interval: monthly
open-pull-requests-limit: 3
+49 -14
View File
@@ -4,15 +4,19 @@ name: CI
# job that exercises the real source sites and the token-gated pixiv tests.
#
# Layering:
# test — fmt + clippy + the full offline unit suite + cargo-audit
# dependency gate. Runs on every push and PR, including forks
# (it needs no secrets).
# test — fmt + clippy + the full offline unit suite + a release-profile
# build + cargo-audit dependency gate. Runs on every push and PR,
# including forks (it needs no secrets).
# live — the #[ignore]d live-network tests plus the pixiv tests that are
# gated on PIXIV_REFRESH_TOKEN. Runs on schedule / manual dispatch
# / tag pushes only, because pull requests from forks cannot read
# repository secrets. continue-on-error keeps a flaky external site
# from blocking, while the run still records the outcome.
#
# Every action is pinned to a commit SHA (Dependabot keeps the pins current);
# `dtolnay/rust-toolchain` deliberately stays on its channel ref, because the
# ref itself is what selects the toolchain (`@stable` = install stable).
#
# Test gating convention (keep in sync with AGENTS.md "Testing & QA"):
# - pure unit tests: plain #[test] / #[tokio::test], always run.
# - live-network tests: #[ignore = "live network: ..."], only run here.
@@ -28,44 +32,75 @@ on:
- cron: '0 3 * * 1'
workflow_dispatch:
permissions:
contents: read
# A newer push to the same ref supersedes the older run; without this every
# intermediate commit of a PR branch keeps a runner busy to completion.
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
env:
# Panicking tests print their backtrace; free when nothing fails.
RUST_BACKTRACE: 1
jobs:
test:
runs-on: ubuntu-latest
# Generous on purpose: the release-profile build below is cold on the very
# first run (thin LTO + codegen-units = 1 across every dependency), and a
# timeout there would kill the job *before* rust-cache saves its cache —
# leaving every later run cold again.
timeout-minutes: 45
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: dtolnay/rust-toolchain@stable
with:
components: clippy, rustfmt
- uses: Swatinem/rust-cache@v2
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
# `--locked` on every cargo invocation: the version bump edits
# Cargo.lock by hand (AGENTS.md), so a stale lock must fail here instead
# of being silently re-resolved — otherwise CI tests a different
# dependency set than the one committed, and than the one the released
# image is built from.
- name: Check formatting
run: cargo fmt --check
- name: Lint (deny warnings)
run: cargo clippy --workspace --all-targets -- -D warnings
run: cargo clippy --workspace --all-targets --locked -- -D warnings
- name: Run offline tests
run: cargo test --workspace
run: cargo test --workspace --locked
# The release profile (lto/strip/codegen-units=1, overflow checks off)
# was otherwise only exercised by the Docker build on master/tag. Same
# package the Dockerfile builds; the cache keeps it cheap after the
# first run.
- name: Build release profile
run: cargo build --release --locked -p xmedia-bot
# Dependency vulnerability gate: fails the build when a crate in
# Cargo.lock has an unfixed security advisory. Unmaintained/unsound
# *warnings* (dotenv, proc-macro-error2, anyhow transitive) do not fail
# the build by default; the advisory DB is cached across runs.
- name: Audit dependencies
uses: actions-rust-lang/audit@v1
uses: actions-rust-lang/audit@72c09e02f132669d52284a3323acdb503cfc1a24 # v1
live:
needs: test
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/v')
runs-on: ubuntu-latest
timeout-minutes: 30
continue-on-error: true
env:
PIXIV_REFRESH_TOKEN: ${{ secrets.PIXIV_REFRESH_TOKEN }}
TWITTER_AUTH_TOKEN: ${{ secrets.TWITTER_AUTH_TOKEN }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
# Full suite: with the secret present, the pixiv token-gated tests run;
# without it they skip themselves. Live tests stay #[ignore]d here.
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
# Everything network- or secret-gated lives in x-media, and the bot
# crate's suite (MockSender + tempdir stores, no network) already ran in
# the `test` job — rebuilding it here bought nothing.
- name: Run token-gated tests
run: cargo test --workspace
run: cargo test -p x-media --locked
# The live-network tests, by the "live" name filter (all #[ignore]d).
- name: Run live-network tests
run: cargo test --workspace -- --ignored live
run: cargo test -p x-media --locked -- --ignored live
+75 -11
View File
@@ -1,28 +1,75 @@
name: Build Docker Image
# Release builds (master / v* tags) plus a build-only check on pull requests
# that touch anything the image depends on — the Dockerfile's stub-source
# machinery, the ffmpeg download and the entrypoint are exactly the parts that
# would otherwise break only at release time.
#
# Actions are pinned to commit SHAs (Dependabot keeps the pins current).
on:
push:
tags:
- v*
branches:
- master
pull_request:
paths:
- Dockerfile
- docker-entrypoint.sh
- .dockerignore
- Cargo.toml
- Cargo.lock
- .github/workflows/docker.yml
- 'crates/**/Cargo.toml'
env:
APP_NAME: telegram-twitter-media-bot
DOCKERHUB_REPO: yoursfunny/telegram-twitter-media-bot
permissions:
contents: read
# Serialize runs per ref. Never cancel in progress: a killed run would drop a
# half-finished image push.
concurrency:
group: docker-${{ github.ref }}
cancel-in-progress: false
jobs:
# A tag push and a branch push to the same commit fire two workflow runs;
# build only once. Tag runs always build; master runs build only when the
# pushed commit is not already tagged (the tag run covers it).
should-build:
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
build: ${{ steps.check.outputs.build }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
fetch-depth: 0
# A release tag is the version claim: the manifests are bumped by hand,
# so `v1.5.1` with `Cargo.toml` still at 1.5.0 would publish an image
# whose tag lies about what is inside it (the binary carries no version).
- name: Verify the tag matches both crate versions
if: startsWith(github.ref, 'refs/tags/v')
shell: bash
run: |
tag="${GITHUB_REF_NAME#v}"
status=0
for manifest in crates/x-media/Cargo.toml crates/xmedia-bot/Cargo.toml; do
# tr -d '\r': a CRLF checkout (core.autocrlf on Windows) would
# otherwise yield "1.5.0\r" and false-fail every tag.
version="$(sed -n 's/^version = "\(.*\)"/\1/p' "$manifest" | head -1 | tr -d '\r')"
if [ "$version" != "$tag" ]; then
echo "::error file=$manifest::$manifest is at $version but the tag is v$tag"
status=1
else
echo "$manifest: $version matches v$tag"
fi
done
exit "$status"
- id: check
shell: bash
run: |
@@ -33,14 +80,20 @@ jobs:
echo "build=true" >> "$GITHUB_OUTPUT"
fi
# No `actions/checkout` here on purpose: `docker/build-push-action` defaults
# to the Git context (`https://github.com/<owner>/<repo>.git#<ref>`), so
# BuildKit clones the repo itself and authenticates with the automatic
# github.token. Adding `context: .` below without a checkout step would hand
# BuildKit an empty workspace.
docker:
needs: should-build
if: needs.should-build.outputs.build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Docker meta
id: meta
uses: docker/metadata-action@v6
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
with:
images: ${{ env.DOCKERHUB_REPO }}
tags: |
@@ -49,15 +102,15 @@ jobs:
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha
-
name: Set up QEMU
uses: docker/setup-qemu-action@v4
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
# Pull requests build the image to prove the Dockerfile still works, but
# must not read registry credentials (fork PRs have none).
-
name: Login to Docker Hub
uses: docker/login-action@v4
if: github.event_name != 'pull_request'
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
@@ -66,15 +119,26 @@ jobs:
# stage's layers so the cargo-deps and ffmpeg layers are restored
# instead of re-downloaded/recompiled. The scope must be pinned to a
# fixed string: the gha backend defaults to the current git ref, which
# would give every new tag a cold cache on release builds.
# would give every new tag a cold cache on release builds. PR runs only
# read it (cache-to is empty) so they cannot evict the release cache.
#
# FFMPEG_URL/FFMPEG_SHA256 come from repository variables when set, so a
# release can pin an exact ffmpeg build (the Dockerfile default follows
# the project's `/redirect/latest/` URL, which has no sha256 sidecar).
#
# Single-arch (amd64) on purpose: adding arm64 means re-adding
# `docker/setup-qemu-action`, `platforms: linux/amd64,linux/arm64`, and
# parameterizing FFMPEG_URL by $TARGETARCH in the Dockerfile.
-
name: Build and push
uses: docker/build-push-action@v7
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7
with:
push: true
push: ${{ github.event_name != 'pull_request' }}
build-args: |
APP_NAME=${{ env.APP_NAME }}
FFMPEG_URL=${{ vars.FFMPEG_URL || 'https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffmpeg.zip' }}
FFMPEG_SHA256=${{ vars.FFMPEG_SHA256 }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=tgxmb-build
cache-to: type=gha,mode=max,scope=tgxmb-build
cache-to: ${{ github.event_name != 'pull_request' && 'type=gha,mode=max,scope=tgxmb-build' || '' }}
-2
View File
@@ -6,8 +6,6 @@ nginx-certs/
nginx-vhost.d/
nginx-html/
nginx-acme/
docker-compose.yml
.env
+43 -34
View File
@@ -2,9 +2,9 @@
## Project Overview
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, Bluesky, and Misskey (misskey.io) into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README is in Chinese; user-facing bot strings are in English. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, Bluesky, Misskey (misskey.io), and Bilibili dynamics into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README is in Chinese; user-facing bot strings are in English. The project is a Rust port of a Python predecessor (see `queue.rs` comments referencing `utils/task_queue.py`).
Two-crate Cargo workspace (both v1.5.0, edition 2024, resolver 3):
Two-crate Cargo workspace (both v1.9.0, edition 2024, resolver 3):
- **`crates/x-media`** — library that fetches and normalizes media from the four sites. Pure, no Telegram knowledge.
- **`crates/xmedia-bot`** — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.
@@ -18,28 +18,37 @@ Telegram update → Dispatcher (polling or axum webhook) → dptree branches
└─ callback_query → "forward" (copy to channel) / "template|<name>" (apply caption template)
```
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)``Fetched` → builds a `Task``send::send_media_sequence` (media groups ≤ 9, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
Message flow: `message_handler` extracts URLs (from `url`/`text_link` entities, text + caption, deduped) → `x_media::site::fetch(url)``Fetched` → builds a `Task``send::send_media_sequence` (media groups ≤ 10, caption on first item) or `send::send_animation`. On Telegram URL-fetch failure or size error (`send_batch_via_upload`): download via `x_media::site::download_media` to a temp file (≤ 10 MiB), sniff magic bytes (`sniff_ext`), upload via multipart; oversized items fall back to `fallback_url`. On failure: `enqueue_retry` persists resume-state `Task` into the SQLite queue (it reports whether the row was really written, and only then does the user get the "retrying in Ns" notice — an enqueue that fails says so instead) → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s for the bot's own delays, `MAX_RETRIES = 2`) → dead-letter → `notify_failure`. Success → `post_send_actions`: edit-before-forward prompt with inline buttons, or `copy_messages` to the bound forward channel.
Debug command: `/test <url>` runs the same `x_media::site::fetch` and replies with `test_parse_report` (`handlers/commands.rs`) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars and sent with HTML parse mode: raw fields are escaped, and the caption is wrapped in a `<blockquote>` so it renders exactly like the sent media caption (escaped text and links included). It uses a custom `parse_test_arg` parser (whole remainder, trimmed) because teloxide's built-in `split` parser takes exactly one space-separated token.
Debug command: `/debug <url>` runs the same `x_media::site::fetch` and replies with `debug_report` (`handlers/commands.rs`) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars and sent with HTML parse mode: raw fields are escaped, and the caption is wrapped in a `<blockquote>` so it renders exactly like the sent media caption (escaped text and links included). The caption it shows is `preview_caption`'s: the chat's per-site format override plus the long-post quoting, i.e. exactly what the send paths produce — showing the raw built-in caption made `/set_format` look like a no-op, and the `/set_format` success reply points users at `/debug` to preview.
The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → misskey → pixiv) and returns `Ok(None)` for unmatched URLs. `Fetched { source_url, caption, title, media: Vec<Media>, sensitive, site_id, … }`; `caption_with(format)` substitutes `{url} {author} {author_url} {title} {tags}`.
User-facing failure text is a function of the error class, never one generic sentence: `urls::fetch_error_message` maps `FetchError::NotFound` (post gone), `Sensitive` (withheld, needs `TWITTER_AUTH_TOKEN`), `Blocked` (source risk control), `Disabled { site }` (a registered site switched off — pixiv without a token, the one case `fetch` answers `Err` instead of `Ok(None)`) and `Transient`/`Http` (source down) apart. The same distinction drives the group hint: a supported link posted in a group (not a channel) gets one `GROUP_LINK_HINT` reply, because the link pipeline is private-chat only.
The `/test <url>` command runs the ordinary link pipeline (`urls::url_media`) with `PostSend::Suppressed`: the media is sent and cached like any other link, but the chat's `forward_channel_id`/`edit_before_forward` are ignored, so a test never forwards to the channel and never opens the edit prompt (retries and dead-letter notifications behave as usual). `/test`, `/debug`, `/set_format` and `/clear_cache` use the custom `parse_arg_remainder` parser (whole remainder, trimmed) because teloxide's built-in `split` parser takes exactly one space-separated token per field: `/set_format <site> <format>` never parsed with it (and `/clear_cache` without an argument did not either), and a command that fails to parse falls through to the URL flow in silence. `commands::tests::every_documented_invocation_parses` pins every documented form against exactly that.
The inline path (`handlers/inline.rs`) hands media URLs straight to Telegram, which fetches them itself and cannot send site-specific headers — so `x_media::site::needs_media_headers(url)` (true exactly where a site's `media_headers` is non-empty, i.e. pixiv's pximg.net) marks the media that must be skipped instead of shipped broken; locally produced media (ugoira MP4, bsky remux) fails `Url::parse` and is skipped the same way. Inline results are therefore URL-only by construction, and a query whose every item was skipped is answered *empty* (with a cache window) rather than left unanswered — an unanswered query keeps the client spinning and, through the debounce's release, re-runs the fetch on every keystroke.
`url_media` is a thin wrapper over `url_media_inner`: `run_with_chat_action` sends the chat action, then re-sends it every `ACTION_REFRESH` (4 s) while the pipeline future is pending, because Telegram drops an action after ~5 s and a fetch (ugoira encode, HLS remux) plus an upload routinely outlasts that. The pipeline flips the shared `ActionHint` from `Typing` to `UploadPhoto`/`UploadVideo` once the media kinds are known. The `select!` is `biased` on the pipeline branch so a finished pipeline never emits a stray action.
The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registry (per-site `impl Site`, in order twitter → bsky → misskey → pixiv → bilibili) and returns `Ok(None)` for unmatched URLs (`Err(FetchError::Disabled { site })` when the URL matches a registered site whose `enabled()` is false — see `disabled_site`). `Fetched { source_url, caption, title, content, media: Vec<Media>, sensitive, site_id, … }` (title and content are split per platform: a pixiv artwork's title and description, a bilibili headline and body, and text-only posts whose text is all `content`); `caption_with(format)` substitutes `{url} {author} {author_url} {title} {content} {tags}`.
## Key Directories
| Path | Purpose |
|---|---|
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky\|misskey>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Twitter's `from_syndication_json` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`&gt;` `&lt;` `&amp;` `&#39;`) — so the stored text is raw and the caption escapes exactly once |
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands`), shared `send::BOT` force-init, queue worker start, site login validation (`site::validate_all`), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media*` (the streaming `download_media_to_file` and the capped `download_media_limited`, which is where a download's size and its total time budget are enforced); `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
| `crates/x-media/src/site/<twitter\|pixiv\|bsky\|misskey\|bilibili>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, `cache_key`/`is_retryable`/`media_headers`, unit struct `<Name>Site` implementing `site::Site`, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`; without the token a withheld tweet stays `FetchError::Sensitive` and the bot reports it as age-restricted instead of "no media"). Misskey targets misskey.io only (`POST /api/notes/show`, 400+`NO_SUCH_NOTE` → NotFound). Bilibili fetches dynamics (images/animated images only — an attached video degrades to its cover, and its title stands in for the post text, which AV dynamics do not have) from `/x/polymer/web-dynamic/v1/detail` sent with `features=itemOpusStyle` (without that flag the legacy serialization drops an image/text post's body and headline entirely — `desc` comes back `null`; the adapter still parses the legacy `major.draw`/`desc`/`archive` shapes as a fallback). No WBI signature is involved; device cookies `buvid3`/`buvid4` are fetched automatically from `/x/frontend/finger/spi` because bilibili's `-352` risk control starts rejecting plain requests, `BILIBILI_COOKIE` is the escalation when an IP stays blocked; `b23.tv` short links are deliberately unmatched. Twitter's `from_syndication_value` HTML-decodes the API text — syndication and GraphQL `full_text` both arrive pre-escaped (`&gt;` `&lt;` `&amp;` `&#39;`) — so the stored text is raw and the caption escap
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands``setMyCommands` plus the profile description texts), shared `send::BOT` force-init, startup sweep of this project's leftover temp files (`x_media::TEMP_FILE_PREFIX` + an age gate, since a killed process runs no destructors), startup repair of queued retries whose local media did not survive a restart (`handlers::repair_lost_local_media`, before any worker can lease: those rows are re-fetched from their `source_url`), queue worker start, site login validation (`site::validate_all`), `periodic_sweep` (`SWEEP_INTERVAL` 300 s): expired prompts are rewritten in place to `EDIT_PROMPT_EXPIRED_TEXT` with an empty keyboard — an edit, never a new message, so a background timer cannot wake a chat — plus the link-cache prune, the idle rate-limit buckets and the idle inline-query entries, and the queue backlog line (only when non-empty). Takes its collaborators rather than the statics so its loop is testable with a paused clock, dptree handler tree, webhook vs polling dispatch |
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
| `crates/xmedia-bot/src/db.rs` | `DbPool`: per-store SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) over `$DATA_DIR/task_queue.db` (default `data/`); `open_store` creates file + schema, `with_conn` runs all rusqlite I/O in `spawn_blocking` |
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. the `/test <url>` parse-only debug command), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons), `statics.rs` (global statics) |
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table) |
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections |
| `crates/xmedia-bot/src/send.rs` | Media senders, upload fallback, error classification, queue task handlers |
| `crates/xmedia-bot/src/media_sender.rs` | `MediaSender` trait: the send surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`send_chat_action`) implemented by teloxide `Bot` (per-chat rate-limited) and by a scripted `MockSender` in tests |
| `crates/xmedia-bot/src/rate_limit.rs` | Per-chat token bucket (`CAPACITY = 20`, ~20 msg/min refill) paced before sends reach the API so batch forwards don't trip flood control |
| `crates/xmedia-bot/src/db.rs` | `DbPool`: one shared SQLite connection pool (`POOL_SIZE = 4`, WAL, busy_timeout) for all three tables over `$DATA_DIR/task_queue.db` (default `data/`) — the three stores share it; `open_store` creates file + schema and then applies the `PRAGMA user_version` migration chain (`MIGRATIONS` + `migrate` — append-only; `schema_init` is the version-0 baseline and must not gain columns an existing database would never receive — `db.rs`'s tests pin a pre-migration database upgrading intact, the shipped migration text frozen (appending is the only allowed change) and a fresh database landing at the latest version), `with_conn` runs all rusqlite I/O in `spawn_blocking` |
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`, the group-only `GROUP_LINK_HINT` for a supported link posted outside a private chat), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. `/test <url>` (send-only) / `/debug <url>` (parse-only) and the admin-only `/bot_dict` state dump; `/set_format` rejects unknown `{…}` placeholders and resets with `-`), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency; one *shared* in-flight fetch per cache key (`fetch_shared`: a second chat, a batch forward or a retry asking for the same post meanwhile waits for the first caller's result, the entry is dropped the moment the fetch settles so nothing is ever answered from an old fetch, and a waiter whose sharer was cancelled fetches for itself); plus the startup repair `repair_lost_local_media`, whose decision (`needs_refetch`) and rewrite (`apply_refresh`) are pure and tested while the fetch itself is a live test), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons, incl. `skip`; a forward that fails retryably is both queued *and* settles the prompt — the queued row carries the message ids itself, and a prompt left live let a second Confirm copy the same messages twice and let Skip answer "nothing was forwarded" while the row still delivered), `statics.rs` (global statics) |
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table); the 300 s sweep's `prune_expired` evicts any chat with no live edit-before-forward prompt, so the cache (and the per-chat lock map) stays bounded to active prompts — durable settings reload from the DB on next use |
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + the source media URLs + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune; a permanent send failure *degrades* the entry instead of dropping it (the file ids go, the URLs stay, so the next request re-sends from those without a fetch), and a degraded entry that fails again is removed |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, a `lease_token` fence: `lease_next` stamps a random token and every write-back (heartbeat, `delete`, `reschedule`, `mark_done`) is guarded by it, so a lease that expired and was re-leased cannot be written by its former holder — a lost lease stops the attempt instead; a finished row's `DELETE`/reschedule retried and a failed delete falling back to a `done` tombstone (the lease query and the sweep only look at `pending`/`in_progress`, so a task that already ran cannot be resurrected and re-run), `runnable_rows`/`replace_payload` (the startup repair's read/rewrite path: it runs before the workers exist, which is why it needs no lease token), `notify_one` worker wakeup plus a separate `Notify` for the 30 s lease-expiry sweep (a shared one let the sweep steal the workers' wakeup permit; the sweep does notify the workers after it actually recovered a row, since a recovered task is due immediately while every worker may be parked on `notify` with no pending row to sleep on), `busy_timeout` on all connections |
| `crates/xmedia-bot/src/ctx.rs` | `AppContext`: the injected collaborators (`sender` + `ChatStore`/`PersistentTaskQueue`/`LinkCache`/`Config`), `from_statics` for production and the `CONTEXT` static the worker closures hold. `test_support::TestStores` backs handler tests with a tempdir store set, and the module also carries the fixtures those tests share — the canonical cached post (`cached_photo`), the edit-before-forward prompt (`seed_prompt` with its `PROMPT_ID`/`FORWARDED_ID`) and a scripted API error (`api_error`) — so no two test modules keep their own copies |
| `crates/xmedia-bot/src/send/` | `send/mod.rs`: `Task`/`MediaItemPayload` payloads, `SendError`/`Classification`, `send_media_sequence`/`send_animation`/`forward_messages`; `send/input_media.rs`: payload → `InputFile`/`InputMedia` + `build_media_group` (caption on the first item only); `send/upload.rs`: the download-and-reupload fallback (`prepare_upload_item`/`send_batch_via_upload`, photo downscale handoff); `send/post_send.rs`: link-cache write, `KEEP_ALIVE` registry, `settle_task`, `post_send_actions`, `handle_task`/`dead_letter_notify` |
| `crates/xmedia-bot/src/media_sender.rs` | `MediaSender` trait: the user-flow surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`answer_callback_query`/`edit_message_text`/`edit_message_caption`/`delete_message`/`send_chat_action`) implemented by teloxide `Bot` (per-chat rate-limited) and by a recording `MockSender` in tests. Admin/setup APIs (`get_chat`, `set_my_commands`, …) stay on the concrete `Bot`. `test_support` holds the scripted `MockSender` and `fake_api` (the stand-in API the real-`Bot` tests drive) |
| `crates/xmedia-bot/src/rate_limit.rs` | Two token buckets paced before sends reach the API so batch forwards don't trip flood control: one per chat (`CAPACITY = 20`, ~20 msg/min refill) and one bot-wide (`acquire_global`, 30/s — Telegram's per-bot ceiling, invisible to any per-chat bucket and only binding when a batch fans out over many chats). `prune_idle` drops the per-chat buckets that refilled while unheld |
## Development Commands
@@ -57,49 +66,49 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
## Code Conventions & Common Patterns
- **Errors via `thiserror` derive** (no anyhow): the public, stringified errors — `FetchError` (`Http`/`Json`/`Pixiv`/`Site`/`NotFound`/`Blocked`) and `PixivError` — derive `thiserror::Error` with `#[from]` conversions; `Display`/`source()` come from the derive. The internal control-flow enums — `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `Classification`, `FallbackError` — carry no `Display` and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive `thiserror`, internal flow enums stay plain.
- **Errors via `thiserror` derive** (no anyhow): the public, stringified errors — `FetchError` (`Http`/`Json`/`Pixiv`/`Site`/`NotFound`/`Blocked`/`Disabled`/`Sensitive`/`TooLarge`/`MediaPrep`/`Transient`/`Io`) and `PixivError` — derive `thiserror::Error` with `#[from]` conversions; `Display`/`source()` come from the derive. The internal control-flow enums — `QueueError` (`Retryable { delay_seconds, payload }` / `Permanent`), `SendError` (Retryable/Permanent), `Classification`, `FallbackError` — carry no `Display` and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive `thiserror`, internal flow enums stay plain.
- **Global state via `std::sync::LazyLock` statics**, not DI: `CONFIG`, `CHAT_STORE`, `TASK_QUEUE` in `handlers/statics.rs`; shared reqwest `CLIENT` in `x-media/src/site/mod.rs`. `Bot` is passed/cloned into handlers; queue workers share the process-wide `send::BOT` (`LazyLock<Bot>`, force-initialized in `main` so a missing token fails at startup).
- **Async**: tokio multi-thread runtime (`#[tokio::main]` default). All rusqlite I/O inside `tokio::task::spawn_blocking`. Long loops use `tokio::select!` with `tokio::sync::{watch, Notify}` stop/wake channels. No streams.
- **Blocking sync primitives**: `parking_lot::Mutex` for hot caches, `tokio::sync::Mutex` for async-shared state (pixiv token cache), `AtomicBool` for feature gates.
- **Site adapter convention**: each site module exports `PATTERN: LazyLock<Regex>`, `enabled() -> bool`, `fetch_from_url(url) -> Result<Fetched, FetchError>`, plus `cache_key`/`is_retryable`/`media_headers`, and a unit struct `<Name>Site` implementing `site::Site`; the central dispatcher (`site/mod.rs`) only iterates the `SITES` registry. Adding a site = new `site/<name>/{mod.rs,interface.rs,model.rs}` + one `Box::new(...)` entry in `SITES` — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return `SiteFuture` (a boxed `Pin<Box<dyn Future + Send>>`) because `async fn` in traits is not dyn-compatible.
- **Serde**: per-site `model.rs` are pure `Deserialize` DTOs mirroring API JSON; site structs in `interface.rs` have private fields, a `caption()` builder, and `impl From<SiteStruct> for Fetched`. Persisted payloads use internally-tagged enums (`#[serde(tag = "kind")]` / `type`).
- **Naming**: module-per-concern, snake_case files, `CamelCase` types, `snake_case` fns. `//!` module docs and `///` docs on non-obvious logic (syndication token, ugoira encoding, `display_text_range`).
- **Retries**: only `x-media::site::fetch` retries (3 attempts, `1 << attempt` backoff, HTTP errors only). Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`).
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`). Level convention: `info` = lifecycle + per-post business results (`sent`/`forwarded`/`copied`), admin/operator actions and anomalies (fallback, retry enqueue, dead-letter is `error`); `debug` = per-request detail (message/command/URL extraction, `fetching`/`fetched`, batch sends, queue processing, photo processing, inline queries). Full user-submitted URLs and message text only appear at `debug`; at `info` and above links are printed via the normalized cache key (`handlers::log_key`, e.g. `[key=twitter:123...]`) so logs stay short and do not echo user data.
- **Retries**: only `x-media::site::fetch` retries (3 attempts, `1 << attempt` backoff, HTTP errors only); `site::fetch_once` is the same code path with a single attempt, used by inline queries whose answer window is shorter than the backoff. A status a site answers with is classified by what a *retry* can change: 404/410 are `NotFound` and 401/403 are `Blocked` (permanent, reported at once), 429/5xx are `Transient` and retried. Queue retries are explicit `QueueError::Retryable` with computed delay (`retry_delay_seconds`), scaled per attempt by `scaled_retry_delay` — which only ever scales **up**, so a delay the server asked for (Telegram `retry_after`) is never shortened. `send::classify_request_error` is the send-side counterpart: `RetryAfter` and `Network` are retryable, and so is a 5xx — teloxide sleeps 10 s on a server error and then parses the body, so by then the HTTP status is gone and the condition is recognised by shape instead (a JSON server-error description, or an `InvalidJson` whose raw body is not JSON, i.e. a proxy/error page).
- Logging via `log` macros (`pretty_env_logger`, level from `RUST_LOG`). `main.rs` initializes the **timed** builder with a default filter of `info,hyper_util=warn,reqwest=warn` when `RUST_LOG` is unset: the plain `init` had no timestamps and fell back to `error`, so a deployment that forgot the variable logged nothing at all, and at `debug` the HTTP client's own lines outnumbered the bot's two to one. An explicit `RUST_LOG` overrides the default wholesale. Level convention: `info` = lifecycle + per-post business results (`sent`/`forwarded`/`copied`, with `chat=` and the total `ms`), admin/operator actions and anomalies (fallback, retry enqueue, dead-letter is `error`); `debug` = per-request detail (URL extraction, `fetching`/`fetched` with the fetch duration, batch sends, queue processing with the row's `chat=`/`key=` and per-attempt `ms`, photo processing, inline queries); `trace` = user data (the full URL, the message text, the inline query). At `debug` and above links are printed via the normalized cache key (`handlers::log_key`, e.g. `[key=twitter:123...]`), so a `debug` log can be shared without echoing what users pasted, and degradations that leave the user served (a failed cache read/write, a failed chat action) are `warn`, not `error`. The only queue/sweep aggregate is the 300 s sweep's queue line, and it speaks only when the queue is non-empty.
## Important Files
| File | Why it matters |
|---|---|
| `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) |
| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `commands.rs` = command dispatch (incl. the `/test <url>` parse-only debug command); `urls.rs` = URL extraction + retry enqueue; `inline.rs`/`callback.rs` = inline queries / edit-before-forward buttons |
| `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`; fallback chain; `classify_request_error`; download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`) |
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL |
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) |
| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `mod.rs` also holds `apply_caption_edit`, the one place a caption edit is applied and its failure classified: a short retryable delay is retried once, anything else is reported to the user instead of being swallowed (`callback.rs`'s template button answers its toast with the failure and leaves the record alone); `commands.rs` = command dispatch (incl. `/test <url>` send-only, `/debug <url>` parse-only, the read-only `/settings` every chat member can read — unlike the admin-only `/bot_dict` raw dump — and template removal; `/start`/`/help` carry the guidance teloxide's `descriptions()` cannot render, and `/set_format` rejects unknown `{…}` placeholders, resetting with `-`); `urls.rs` = URL extraction + the per-URL pipeline (`url_media` takes a `PostSend` mode: chat settings vs `/test`'s suppressed actions); `inline.rs` = debounced inline queries (hotlink-protected and local media skipped); `callback.rs` = edit-before-forward buttons (dptree entry + testable `handle_callback` core, incl. `skip`) |
| `crates/xmedia-bot/src/send/` | `mod.rs`: constants `MAX_MEDIA_GROUP = 10`; `classify_request_error` (5xx/non-JSON bodies retry, see the Retries bullet) and the media-fetch markers that route a URL send into the reupload fallback — including `failed to get HTTP url content`, the description single-media URL sends answer with; the senders. `upload.rs`: download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`), with a download's class from `classify_download_error` (transport/429/5xx retry; 4xx is permanent — the media itself is gone or refused — and a temp-file *write* failure retries, being resource exhaustion far more often than a broken temp dir). Item preparation is bounded **process-wide** (`PREP_SLOTS` in `upload.rs`: URL workers and queue workers can each be inside a batch, so a per-batch bound is not a memory bound), and the check that routes an oversized item to `fallback_url` is the download's own declared-Content-Length abort (`FetchError::TooLarge``MediaTooLarge`) — there is no separate size probe, which used to cost a second request per item. `post_send.rs`: settlement (`settle_task`), cache write, post-send actions (dead-letter text via `failure_text`: post key + cause, since the raw error alone does not say which link died), queue handlers. `input_media.rs`: payload → `InputMedia` |
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL. Two budgets, not one: `MAX_PHOTO_DOWNLOAD_BYTES` (32 MiB) caps the *download* in the send fallback — the whole body is buffered, once per prep slot — while `MAX_DECODE_BYTES` (512 MiB) stays the pre-allocation guard that decides whether a decoded photo can be processed at all; over either one the item degrades to its smaller URL |
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection), `needs_media_headers` (the same per-site rule, asked by the inline path to skip what Telegram cannot fetch) |
| `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` |
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime bump (cargo's freshness is mtime-based and `cargo clean -p` removes 0 files — the touch is what forces the real sources to rebuild while deps stay cached), static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, optional `FFMPEG_SHA256` checksum, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint. Runtime ships **no libssl/libcrypto/CA bundle** — rustls webpki-roots handles all TLS, and the static ffmpeg only processes local files (downloads go through reqwest) |
| `docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim) |
| `docker-compose.yml.example` | Deployment env reference (real `docker-compose.yml` is gitignored). Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) |
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master; **no test step**; buildx gha cache (`cache-from`/`cache-to`, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs |
| `docker-compose.yml` | The deployment composition, committed as-is: every instance value (token, admins, site credentials, domain) is a `${VAR}` substitution read from the gitignored `.env` beside it, so the file needs no per-deployment edit — and a variable not listed in a service's `environment:` never reaches that container. Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) |
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master, plus a build-only check on PRs touching the build inputs; **no test step**; verifies a release tag matches both crate versions; buildx gha cache (`cache-from` always, `cache-to` except on PRs, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs; `FFMPEG_URL`/`FFMPEG_SHA256` come from repo variables when set |
| `README.md` | Feature docs + command table (Chinese) |
## Runtime/Tooling Preferences
- **Rust, stable, edition 2024**, workspace resolver 3. No `rust-version`/MSRV pin, no `rust-toolchain.toml` — recent stable is assumed. No nightly features.
- Package manager: **Cargo** (workspace with path dep `x-media``xmedia-bot`). No `[workspace.package]`/shared deps — each crate lists deps independently.
- **TLS is rustls end-to-end** (no native-tls/openssl in the tree, no libssl in the Docker runtime image): `teloxide` is declared `default-features = false` with `["webhooks-axum", "macros", "rustls", "ctrlc_handler"]` (the removed `default` also carried `native-tls` and `ctrlc_handler` — the latter must stay); x-media's reqwest is `default-features = false` with `["json", "rustls-tls"]` (webpki-roots baked in, so the image ships no CA bundle). One reqwest 0.12.28 in the lock.
- **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md`, `README.en.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag (the tag push triggers the Docker Hub build).
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `DATA_DIR` (default `data`, CWD-relative; the SQLite dir, auto-created), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
- **TLS is rustls end-to-end** (no native-tls/openssl in the tree, no libssl in the Docker runtime image): `teloxide` is declared `default-features = false` with `["webhooks-axum", "macros", "rustls", "ctrlc_handler"]` (the removed `default` also carried `native-tls` and `ctrlc_handler` — the latter must stay); x-media's reqwest is `default-features = false` with `["json", "rustls-tls", "gzip", "http2"]` (webpki-roots baked in, so the image ships no CA bundle; `gzip` because the site APIs answer their JSON compressed — twitter's syndication body is 4469 bytes identity vs 1066 gzipped — and `http2` because every site CDN here negotiates h2). One reqwest 0.12.28 in the lock.
- **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md`, `README.en.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag (the tag push triggers the Docker Hub build). The tag must equal both crate versions: `.github/workflows/docker.yml` verifies that before building, and `--locked` verifies the lock file.
- Config is **environment-variable driven** (dotenv loads `.env`, which is gitignored; `.env.example` is the tracked template — `cp .env.example .env` — and is also the file `docker compose` substitutes `${VAR}` from, so every variable the compose passes must be documented there). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BILIBILI_COOKIE` (optional; whole bilibili cookie string — bilibili dynamics fetch anonymously and add their own device cookies, this only rescues an egress IP that bilibili has hard-flagged with `-352`/412), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `CAPTION_QUOTE_TEXT_CHARS` (default 200; a post whose text — the `title` plus `content` joined, see `site::compose_text` — reaches this length gets that text wrapped in an expandable blockquote inside its caption, the URL and author line staying outside; `0` disables it. Applied at the send boundary in `send::quote_long_caption`, which locates the text as what follows the author link, so a `/set_format` that moves `{title}`/`{content}` elsewhere and pixiv's title-inside-a-link layout opt out; `copy_messages` forwards and queued retries inherit the wrap, while the edit-before-forward rewrite stays unquoted by design), `DATA_DIR` (default `data`, CWD-relative; the SQLite dir, auto-created), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `$DATA_DIR/task_queue.db` (default `data/task_queue.db`, CWD-relative — run from the workspace root, or `/app` in Docker; set `DATA_DIR` to pin state anywhere). Mount `./data` and `./cert` volumes.
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `docker-compose.yml`, `/target`, `.idea/`.
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `nginx-*` (proxy state), `/target`, `.idea/` (the compose file is tracked; only `.env` carries the deployment's own values).
- Docs are in Chinese (README, AGENTS.md); user-facing bot strings are in English. Keep that split when editing user-facing strings and docs.
## Testing & QA
- **~125 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches.
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (2), `site/misskey/interface.rs` (1), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""``is_err()` alone would run them tokenless and fail). Run the full offline suite with `cargo test --workspace`.
- **~180 tests, all inline `#[cfg(test)] mod tests`** — no `tests/` integration directories. Framework: built-in Rust test + `#[tokio::test]` (dev-deps only in `x-media`: tokio macros/rt-multi-thread, dotenv).
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches. Tests that must go through a **real `Bot`** (its URL/multipart building, the per-chat limiter and the bot-wide budget) talk to a stand-in API instead (`media_sender::test_support::fake_api::FakeApi`, a `tokio` TCP listener that records every call and answers the smallest result each method needs — teloxide keys methods by payload type, so the recorded name is `SendMediaGroup`, not `sendMediaGroup`): a media group, the edit-before-forward prompt through the real callback path, and `handlers::handle_message` (the context-taking body of `message_handler`, split out for exactly this).
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (1), `site/misskey/interface.rs` (1), `site/bilibili/interface.rs` (5), `site/pixiv/api.rs` (1); `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/mod.rs` also has a **token-gated but not `#[ignore]`d** pixiv download test (`download_media_pixiv_original_with_referer`): it hits `i.pximg.net` whenever `PIXIV_REFRESH_TOKEN` is set, so a local `cargo test --workspace` is not fully offline and can flake on a pixiv CDN body timeout. `disabled_site_is_reported_not_ignored` (same file) is gated the other way round: it asserts `fetch` answers `FetchError::Disabled { site: "pixiv" }` for a pixiv link and early-returns when `PIXIV_REFRESH_TOKEN` **is** set (the site is then enabled). Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""``is_err()` alone would run them tokenless and fail), and the bilibili live tests early-return when the API answers risk control (`-352`, which bilibili applies per IP by request volume). Run the full offline suite with `cargo test --workspace`.
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
- **CI** — `.github/workflows/ci.yml` runs `cargo fmt --check` + `cargo clippy --workspace --all-targets -- -D warnings` + `cargo test --workspace` + a `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN` from secrets, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds/pushes the image only.
- Untested and hard to test without a mock seam: `handlers.rs` (depends directly on teloxide `Bot`); `main.rs`, `config.rs`, `state.rs`; `media.rs`, `lib.rs`, all `model.rs`.
- **CI** — `.github/workflows/ci.yml` (actions pinned to commit SHAs, `--locked` on every cargo invocation, `concurrency` cancels superseded runs, `RUST_BACKTRACE=1`) runs `cargo fmt --check` + `cargo clippy --workspace --all-targets --locked -- -D warnings` + `cargo test --workspace --locked` + a release-profile `cargo build --release --locked` + an `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `-p x-media` since every network/secret-gated test lives there, `continue-on-error`) for the `#[ignore]`d live + token tests. `.github/workflows/docker.yml` builds and pushes the image on master/tag and runs a **build-only check on pull requests touching the build inputs** (`Dockerfile`, entrypoint, manifests, `.dockerignore`); a release tag must match both crate versions or the build stops, and `FFMPEG_URL`/`FFMPEG_SHA256` are taken from repository variables when set (a release can pin an exact ffmpeg build). `.github/dependabot.yml` keeps crates, the pinned actions and the Docker base images current.
- Untested and hard to test without a mock seam: `config.rs`, `handlers/statics.rs`; `db.rs` is covered for the migration chain but not for pool behaviour under contention; `main.rs` is covered where it was split out (`periodic_sweep`, `sweep_temp_dir`) but not for startup/shutdown or its `dptree` branch tree (the handlers themselves are, through the stand-in API); in `x-media`: `media.rs`, `lib.rs`, all `model.rs`. The `commands.rs` *executor* needs a real `Bot` (only its pure report builder is tested). Everything else — `handlers/{mod,callback,inline,urls}.rs`, `send/*`, `ctx.rs`, `state.rs`, `queue.rs`, `link_cache.rs`, `rate_limit.rs` — is driven through `TestStores`/`ctx::test_support` and the scripted `MockSender`.
- No coverage tracking.
Generated
+640 -706
View File
File diff suppressed because it is too large Load Diff
+4 -2
View File
@@ -11,6 +11,8 @@ ARG APP_NAME=telegram-twitter-media-bot
# runners. `/redirect/latest/` floats to the newest release build; each build
# also ships a .sha256. Swap `amd64` for `arm64` when building arm64 images.
ARG FFMPEG_URL=https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffmpeg.zip
# Arm64 images need this URL swapped for the `linux/arm64` build (currently
# hardcoded amd64; the workflow builds amd64 only — see docker.yml).
# Optional sha256 of ffmpeg.zip (pinned releases only): set to verify the
# download. The mirror publishes .sha256 sidecars next to pinned builds, e.g.
# https://ffmpeg.martin-riedl.de/download/linux/amd64/<id>_9.0/ffmpeg.zip.sha256
@@ -28,7 +30,7 @@ COPY crates/xmedia-bot/Cargo.toml crates/xmedia-bot/Cargo.toml
RUN mkdir -p crates/x-media/src crates/xmedia-bot/src \
&& printf 'fn main() {}\n' > crates/xmedia-bot/src/main.rs \
&& : > crates/x-media/src/lib.rs \
&& cargo build --release -p xmedia-bot
&& cargo build --release --locked -p xmedia-bot
# 2. Static ffmpeg next (cached unless FFMPEG_URL changes), so source edits
# never re-download it. The zip contains a single `ffmpeg` binary at the
@@ -51,7 +53,7 @@ RUN wget -q -O /tmp/ffmpeg.zip "$FFMPEG_URL" \
# removes 0 files and the stub binary silently ships.)
COPY crates/ ./crates/
RUN find crates -type f -name '*.rs' -exec touch {} + \
&& cargo build --release -p xmedia-bot
&& cargo build --release --locked -p xmedia-bot
# ---------- runtime stage ----------
FROM debian:bookworm-slim
+37 -22
View File
@@ -1,14 +1,18 @@
# TelegramXMediaBot
A Telegram bot that turns post links from X / Twitter, Pixiv, Bluesky, and Misskey (misskey.io) into media messages (images, video, GIF) with the post's title, author, and tags.
A Telegram bot that turns post links from X / Twitter, Pixiv, Bluesky, Misskey (misskey.io), and Bilibili dynamics into media messages (images, video, GIF) with the post's title, author, and tags.
## Features
- Sending a link in a private chat fetches and sends the images, videos and GIFs automatically; oversized media is split into batches
- Text-only posts report "no media"; unsupported links are silently ignored
- Inline queries (`@bot <link>`)
- Bind a forward channel for automatic forwarding; edit the caption before forwarding and apply custom templates
- Failed sends are retried automatically with persistence; the user is notified after retries are exhausted
- Sending a link in a private chat fetches and sends the images, videos and GIFs automatically; oversized media is split into batches (10 items per group)
- Text-only posts report "no media"; unsupported links are silently ignored. Fetch failures name the reason (post gone / content withheld / source risk control / site not enabled)
- Long posts (text ≥ `CAPTION_QUOTE_TEXT_CHARS`, default 200) show **the text part** of their caption inside a collapsible blockquote, with the link and author line left outside it
- Inline queries (`@bot <link>`) — except Pixiv images and locally transcoded animations, which Telegram cannot fetch (no Referer) and would show broken, so they are skipped (such a query answers empty rather than spinning or re-fetching); a supported link posted in a group gets a one-line hint to use the private chat or inline mode (channels stay silent)
- `/start` explains the supported sites and how to use it; `/help` lists the commands plus argument syntax, the caption placeholders and the private-chat rule; the bot's profile description texts are set at startup
- `/settings` shows this chat's configuration (forward channel, edit-before-forward, per-site caption formats, saved templates); templates are added with `/set_template` and removed with `/remove_template`
- Bind a forward channel for automatic forwarding; edit the caption before forwarding and apply custom templates (the prompt carries Confirm / Skip buttons, states its expiry, and is marked expired in place once it lapses)
- Failed sends are retried automatically with persistence; the notice names which link failed, how long the retry waits, or the final cause
- The chat action stays on screen for the whole fetch, so long jobs (ugoira transcode, large uploads) do not look stalled
- Pixiv ugoira animations are transcoded to MP4; Bluesky videos are remuxed (HLS stream → MP4)
- Photos exceeding Telegram's size/dimension limits are compressed automatically (original format kept, JPEG fallback only when needed)
- Link-result cache: after a successful send the Telegram file ids and caption fields are cached locally, so a repeated link is re-sent from local state — no source-site request, no media file stored (expiry controlled by `LINK_CACHE_TTL_SECONDS`, default 7 days)
@@ -23,24 +27,29 @@ export PIXIV_REFRESH_TOKEN=<token>
cargo run -p xmedia-bot
```
Docker deployment (see `docker-compose.yml.example`):
Docker deployment (`docker-compose.yml` in this repo is the orchestration; instance values live in the `.env` next to it, and compose substitutes every `${VAR}` from there):
```bash
cp .env.example .env # fill in TELOXIDE_TOKEN and the rest; every line is commented
docker build -t tgxmb .
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
# or use the bundled orchestration (nginx-proxy + acme-companion):
docker compose up -d
```
Environment variables: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `BOT_ADMIN`, `EDIT_MESSAGE_TTL_SECONDS`, `LINK_CACHE_TTL_SECONDS`, `RUST_LOG`, `TELOXIDE_PROXY`, `WEBHOOK*`, `TWITTER_AUTH_TOKEN` (optional).
Environment variables: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `BOT_ADMIN`, `EDIT_MESSAGE_TTL_SECONDS`, `LINK_CACHE_TTL_SECONDS`, `RUST_LOG`, `TELOXIDE_PROXY`, `WEBHOOK*`, `TWITTER_AUTH_TOKEN` (optional), `BILIBILI_COOKIE` (optional).
NSFW tweets: the public syndication endpoint does not return sensitive content. Setting `TWITTER_AUTH_TOKEN` (the `auth_token` cookie value of a logged-in x.com session) lets the bot fetch NSFW media in the logged-in state only when it hits a withheld tweet; without it, the bot reports no media.
NSFW tweets: the public syndication endpoint does not return sensitive content. Setting `TWITTER_AUTH_TOKEN` (the `auth_token` cookie value of a logged-in x.com session) lets the bot fetch NSFW media in the logged-in state only when it hits a withheld tweet; without it the bot answers that the post's media is withheld and needs `TWITTER_AUTH_TOKEN`.
Bilibili dynamics are fetched anonymously by default (no login; the bot fetches bilibili's anonymous `buvid3`/`buvid4` device cookies itself to raise the success rate). If the server's egress IP gets hard-flagged by bilibili (persistent `risk control (-352)` log lines or HTTP 412), set `BILIBILI_COOKIE` (the whole cookie string from a logged-in browser, e.g. `SESSDATA=…; bili_jct=…`) to restore access. Only a dynamic's images and animations are sent; an attached video degrades to its cover image.
### Webhook deployment (needs a reverse proxy)
`docker-compose.yml.example` ships an [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) reverse-proxy orchestration. Pick one deployment shape:
`docker-compose.yml` ships an [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) reverse-proxy orchestration. The committed file needs **no editing**: domain, tokens and admins are instance values and live in the `.env` beside it (compose reads and substitutes `${VAR}` at startup). Pick one deployment shape:
**With a domain**
1. Point a DNS A record at the server
2. In compose set `VIRTUAL_HOST` and `WEBHOOK_URL` to the domain, and uncomment `ACME_HOST` (set it to the domain)
2. In `.env` set `VIRTUAL_HOST` and `WEBHOOK_URL` to the domain; to have acme-companion issue the certificate, also uncomment the `ACME_HOST` line in `docker-compose.yml` and set `ACME_HOST` in `.env`
3. acme-companion issues and renews certificates automatically — nothing manual
**IP only**
@@ -68,7 +77,7 @@ Let's Encrypt can issue certificates for public IPs (available since 2026, valid
--key-file /acme.sh/<SERVER_IP>.key \
--reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP"
```
3. In compose set `VIRTUAL_HOST: '<SERVER_IP>'` and `WEBHOOK_URL: 'https://<SERVER_IP>/'`; no `WEBHOOK_CERT` needed. Renewal is handled by the acme.sh daemon (`--days 3` = renew every 3 days, buffer against the 7-day validity), and a successful renewal HUP-notifies nginx-proxy to load the new certificate.
3. In `.env` set `VIRTUAL_HOST=<SERVER_IP>` and `WEBHOOK_URL=https://<SERVER_IP>/`; no `WEBHOOK_CERT` needed. Renewal is handled by the acme.sh daemon (`--days 3` = renew every 3 days, buffer against the 7-day validity), and a successful renewal HUP-notifies nginx-proxy to load the new certificate.
Limitations: certificate validity ~7 days; only http-01/tls-alpn-01 validation (port 80 must be publicly reachable); no DNS-01, private IPs or IP ranges; at most 5 certificates per 168 hours for the same IP set. It is recommended to trial-issue with `--server letsencrypt_test` first, then switch to the production server.
@@ -80,15 +89,18 @@ Telegram only accepts ports 443/80/88/8443.
| Variable | Description |
|---|---|
| `TELOXIDE_TOKEN` | Bot token (required) |
| `PIXIV_REFRESH_TOKEN` | Pixiv refresh token; Pixiv is disabled without it |
| `PIXIV_REFRESH_TOKEN` | Pixiv refresh token; Pixiv is disabled without it (a pixiv link then gets an explicit "site not enabled" reply instead of silence) |
| `TWITTER_AUTH_TOKEN` | Optional; the `auth_token` cookie of a logged-in x.com session, used only to fetch NSFW tweets' media |
| `BILIBILI_COOKIE` | Optional bilibili cookie string (`SESSDATA=…; bili_jct=…`); only needed when the egress IP stays risk-controlled (device cookies are fetched automatically) |
| `BOT_ADMIN` | Admin chat IDs, comma-separated; receives start/stop notifications |
| `EDIT_MESSAGE_TTL_SECONDS` | Edit-before-forward record expiry in seconds, default 86400 |
| `EDIT_MESSAGE_TTL_SECONDS` | Edit-before-forward record expiry in seconds, default 86400; once lapsed the prompt is rewritten in place to "expired — nothing was forwarded" (no extra message) |
| `LINK_CACHE_TTL_SECONDS` | Link-result cache expiry in seconds, default 604800 (7 days) |
| `CAPTION_QUOTE_TEXT_CHARS` | **The text part** of the caption (the joined `{title}` + `{content}`) is wrapped in a collapsible blockquote once it reaches this many characters, default 200; `0` disables |
| `DATA_DIR` | Data directory (where the SQLite `task_queue.db` lives), default `data` (relative to the working directory, created automatically) |
| `RUST_LOG` | Log level |
| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW) |
| `RUST_LOG` | Log level, default `info,hyper_util=warn,reqwest=warn` (an unset variable no longer silences the log). Recipes: `info,xmedia_bot=debug,x_media=debug` (app detail, no dependency noise) / `debug,hyper_util=off` (everything) / `trace` (also prints full links and message text — **user data**) |
| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW). **Never leave it blank** (`TELOXIDE_PROXY=`) — teloxide panics on a value it cannot parse; omit the line when unused. `docker-compose.yml` deliberately does not pass it to the container (a `127.0.0.1` proxy there is the container itself): add the line and use `host.docker.internal:<port>` when a deployment needs one |
| `LOCAL_USER_ID` | UID the container runs as, default 9001 |
| `VIRTUAL_HOST` | Public domain or IP; nginx-proxy routes by this |
| `VIRTUAL_HOST` | Public domain or IP; nginx-proxy routes by this (set it in `.env`, which compose reads) |
| `VIRTUAL_PORT` | Port the bot listens on inside the container; nginx-proxy's forwarding target |
| `ACME_HOST` | Domain deployment: when set to the domain, acme-companion issues/renews certificates automatically |
| `DEFAULT_HOST` | nginx-proxy routes requests with unknown Host headers to this vhost (needed for IP access) |
@@ -109,14 +121,17 @@ Telegram only accepts ports 443/80/88/8443.
| `/help` | List all commands and usage (this command table) |
| `/set_forward_channel <channel>` | Set the forward channel: `@channel` or channel ID; media messages are forwarded to it automatically afterwards |
| `/remove_forward_channel` | Remove the forward channel |
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or taps a template button to apply one) |
| `/edit_before_forward` | Toggle "edit before forward": when enabled, the bot posts a prompt after forwarding; replying to it edits the first forwarded message's caption (or tapping a template button applies one), then `↩️ Confirm` forwards and `🛑 Skip` drops this forward; the prompt states its expiry and is marked expired in place when it lapses (nothing is forwarded) |
| `/set_template <name>` | Reply to a message containing `[]` to save it as a named template; `[]` is replaced by the original post link when forwarding (used with "edit before forward") |
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv` / `misskey`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/remove_template <name>` | Remove a template (names are listed by `/settings`; the prompt's keyboard shows at most 60) |
| `/settings` | Show this chat's configuration: forward channel, edit-before-forward, per-site caption formats, saved templates |
| `/set_format <site> <format>` | Customize the caption format for one site. Sites: `twitter` / `bsky` / `pixiv` / `misskey` / `bilibili`. Placeholders: `{url}` `{author}` `{author_url}` `{title}` `{content}` `{tags}`; unknown placeholders are rejected with the list of valid ones, and `-` restores the site's built-in format (preview with `/debug <link>`) |
| `/clear_cache [link]` | Clear the link cache (admin only); with a link only that entry, otherwise everything |
| `/bot_dict` | Show the current chat state (debugging) |
| `/test <link>` | Debug: parse a link and report the parse result only (site, title, author, tags, media list) — no media is sent |
| `/bot_dict` | Show the current chat state (debugging; admin only) |
| `/test <link>` | Parse a link and send its media; no channel forward, no edit-before-forward prompt (send only) |
| `/debug <link>` | Debug: parse a link and report the parse result only (site, title, author, tags, media list) — no media is sent |
Link processing works only in private chats; commands work in any chat.
Link processing works only in private chats; commands work in any chat. A supported link posted in a group gets a one-line hint to use the private chat or inline mode; channels stay silent.
## Notes
+37 -22
View File
@@ -1,14 +1,18 @@
# TelegramXMediaBot
Telegram 机器人,将 X / Twitter、Pixiv、Bluesky、Misskey (misskey.io) 的帖子链接转换为媒体消息发送,附带帖子标题、作者与标签。
Telegram 机器人,将 X / Twitter、Pixiv、Bluesky、Misskey (misskey.io)、Bilibili 动态的帖子链接转换为媒体消息发送,附带帖子标题、作者与标签。
## 功能
- 私聊发送链接后自动抓取并发送图片、视频与 GIF,超量图片自动分批
- 纯文字帖提示无媒体;不支持的链接静默忽略
- 支持内联查询(`@机器人 <链接>`
- 可绑定转发频道自动转发;支持转发前编辑 caption 与自定义模板
- 发送失败自动重试并持久化,重试耗尽后通知用户
- 私聊发送链接后自动抓取并发送图片、视频与 GIF,超量图片自动分批(每批 10 张)
- 纯文字帖提示无媒体;不支持的链接静默忽略。抓取失败会按原因分别提示(帖子已删除 / 内容受限 / 源站风控 / 站点未启用)
- 长帖(正文 ≥ `CAPTION_QUOTE_TEXT_CHARS`,默认 200)的**正文部分**用可折叠引用块展示,链接与作者行留在引用块外
- 支持内联查询(`@机器人 <链接>`;Pixiv 图片与本地转码的动图不支持内联 —— Telegram 取图时无法携带 Referer,会显示破图,因此跳过;这类查询直接返回空结果,不会一直转圈或反复请求);在群聊里发链接会提示改用私聊或内联查询(频道内保持静默)
- `/start` 说明支持的站点与用法,`/help` 列出命令、参数格式、caption 占位符与私聊限制;bot 资料页(description / short description)启动时一并设置
- `/settings` 查看本聊天配置(转发频道、转发前编辑开关、各站点 caption 格式、模板列表);模板可用 `/set_template` 增、`/remove_template`
- 可绑定转发频道自动转发;支持转发前编辑 caption 与自定义模板(提示消息带 Confirm / Skip 按钮并写明过期时间,过期后就地标记为已过期)
- 发送失败自动重试并持久化,重试耗尽后通知用户;提示会写明是哪条链接、重试等待多久、或最终失败的原因
- 抓取期间持续显示"正在输入 / 正在发送"状态,长任务(ugoira 转码、大图上传)不会看起来卡死
- Pixiv ugoira 动图自动转码为 MP4Bluesky 视频自动转码(HLS 流 → MP4)
- 超过 Telegram 尺寸/大小限制的图片自动压缩(保持原格式,必要时转 JPEG)
- 链接结果本地缓存:成功发送后缓存 Telegram file id 与 caption 等,再次收到相同链接直接本地重发,不再请求源站、不保存媒体文件(`LINK_CACHE_TTL_SECONDS` 控制过期,默认 7 天)
@@ -23,24 +27,29 @@ export PIXIV_REFRESH_TOKEN=<token>
cargo run -p xmedia-bot
```
Docker 部署(参考 `docker-compose.yml.example`):
Docker 部署(编排见仓库里的 `docker-compose.yml`,实例相关的值写在同目录的 `.env`compose 会自动替换其中的 `${VAR}`):
```bash
cp .env.example .env # 填 TELOXIDE_TOKEN 等,逐项都有注释
docker build -t tgxmb .
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
# 或者用仓库里的编排(含 nginx-proxy + acme-companion):
docker compose up -d
```
环境变量:`TELOXIDE_TOKEN`(必填)、`PIXIV_REFRESH_TOKEN``BOT_ADMIN``EDIT_MESSAGE_TTL_SECONDS``LINK_CACHE_TTL_SECONDS``RUST_LOG``TELOXIDE_PROXY``WEBHOOK*``TWITTER_AUTH_TOKEN`(可选)。
环境变量:`TELOXIDE_TOKEN`(必填)、`PIXIV_REFRESH_TOKEN``BOT_ADMIN``EDIT_MESSAGE_TTL_SECONDS``LINK_CACHE_TTL_SECONDS``RUST_LOG``TELOXIDE_PROXY``WEBHOOK*``TWITTER_AUTH_TOKEN`(可选)`BILIBILI_COOKIE`(可选)
NSFW 推文:公开的 syndication 接口不返回敏感内容。设置 `TWITTER_AUTH_TOKEN`(登录 x.com 后浏览器 Cookie 里的 `auth_token` 值)后,bot 会仅在遇到 NSFW 推文时以登录态获取媒体;未设置则提示无媒体
NSFW 推文:公开的 syndication 接口不返回敏感内容。设置 `TWITTER_AUTH_TOKEN`(登录 x.com 后浏览器 Cookie 里的 `auth_token` 值)后,bot 会仅在遇到 NSFW 推文时以登录态获取媒体;未设置则回复该推文内容受限(需要配置 `TWITTER_AUTH_TOKEN`
Bilibili 动态默认匿名抓取(无需登录,bot 会自动从 B 站的匿名指纹接口取 `buvid3`/`buvid4` 设备 cookie 以提高成功率)。若服务器出口 IP 被 B 站重度风控(日志里的 `risk control (-352)` 或 HTTP 412,且持续出现),设置 `BILIBILI_COOKIE`(登录后浏览器里整条 Cookie 串,如 `SESSDATA=…; bili_jct=…`)可恢复访问。当前只发送动态里的图片与动图,动态内嵌视频发送其封面。
### Webhook 部署(需要反向代理)
`docker-compose.yml.example` 内置了 [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) 反向代理编排,按部署环境二选一:
`docker-compose.yml` 内置了 [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) 反向代理编排,仓库里的这份文件**不需要改动**:域名、令牌、管理员等实例相关的值都写在同目录的 `.env` 里(compose 启动时自动读取并替换 `${VAR}`)。按部署环境二选一:
**有域名**
1. DNS A 记录指向服务器
2. compose 里设 `VIRTUAL_HOST``WEBHOOK_URL` 为域名,并取消注释 `ACME_HOST`设为域名
2. `.env` 里设 `VIRTUAL_HOST``WEBHOOK_URL` 为域名;要由 acme-companion 自动签发证书时,再取消 `docker-compose.yml``ACME_HOST` 那行的注释,并在 `.env` 里把 `ACME_HOST` 设为域名
3. acme-companion 自动签发与续期证书,无需手动处理
**只有 IP**
@@ -68,7 +77,7 @@ Let's Encrypt 支持为公网 IP 签发证书(2026 年起可用,有效期约
--key-file /acme.sh/<SERVER_IP>.key \
--reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP"
```
3. compose 里设 `VIRTUAL_HOST: '<SERVER_IP>'`、`WEBHOOK_URL: 'https://<SERVER_IP>/'`,无需 `WEBHOOK_CERT`。续期由 acme.sh daemon 自动完成(`--days 3` = 每 3 天续一次,证书 7 天有效有缓冲),续期成功后自动 HUP 通知 nginx-proxy 加载新证书。
3. `.env` 里设 `VIRTUAL_HOST=<SERVER_IP>`、`WEBHOOK_URL=https://<SERVER_IP>/`,无需 `WEBHOOK_CERT`。续期由 acme.sh daemon 自动完成(`--days 3` = 每 3 天续一次,证书 7 天有效有缓冲),续期成功后自动 HUP 通知 nginx-proxy 加载新证书。
限制:证书约 7 天有效;验证仅支持 http-01/tls-alpn-0180 端口必须公网可达);不支持 DNS-01、私有 IP 与 IP 段;同一 IP 集合每 168 小时限签发 5 张。建议先用 `--server letsencrypt_test` 试签,成功后再切正式服务器。
@@ -80,15 +89,18 @@ Telegram 只接受 443/80/88/8443 端口。
| 变量 | 说明 |
|---|---|
| `TELOXIDE_TOKEN` | Bot token(必填) |
| `PIXIV_REFRESH_TOKEN` | Pixiv 刷新令牌;未设置则禁用 Pixiv |
| `PIXIV_REFRESH_TOKEN` | Pixiv 刷新令牌;未设置则禁用 Pixiv(此时收到 pixiv 链接会明确回复「站点未启用」,不会静默忽略) |
| `TWITTER_AUTH_TOKEN` | 可选;登录 x.com 后浏览器 Cookie 里的 `auth_token`,仅在遇到 NSFW 推文时以登录态获取媒体 |
| `BILIBILI_COOKIE` | 可选的 B 站 Cookie 串(`SESSDATA=…; bili_jct=…`),仅在出口 IP 被持续风控时才需要(设备 cookie 由 bot 自动获取) |
| `BOT_ADMIN` | 管理员聊天 ID,逗号分隔;接收启动/停止通知 |
| `EDIT_MESSAGE_TTL_SECONDS` | 转发前编辑记录过期秒数,默认 86400 |
| `EDIT_MESSAGE_TTL_SECONDS` | 转发前编辑记录过期秒数,默认 86400;过期后提示消息会被就地改写为「已过期,未转发」(不额外发消息打扰) |
| `LINK_CACHE_TTL_SECONDS` | 链接结果缓存过期秒数,默认 604800(7 天) |
| `CAPTION_QUOTE_TEXT_CHARS` | 正文(`{title}` + `{content}` 合计)达到该长度(字符)时,caption 的**正文部分**用可折叠引用块包裹,默认 200;`0` 关闭 |
| `DATA_DIR` | 数据目录(SQLite 数据库 `task_queue.db` 所在目录),默认 `data`(相对工作目录,会自动创建) |
| `RUST_LOG` | 日志级别 |
| `TELOXIDE_PROXY` | HTTP 代理(如 `http://127.0.0.1:10808`);同时作用于 Telegram Bot API 与站点抓取请求,网络受限环境(如 GFW)必需 |
| `RUST_LOG` | 日志级别,默认 `info,hyper_util=warn,reqwest=warn`(未设置也**不会**哑掉)。排障配方:`info,xmedia_bot=debug,x_media=debug`(应用细节,无依赖噪音)/ `debug,hyper_util=off`(全量)/ `trace`(额外打印完整链接与消息原文,**含用户数据**) |
| `TELOXIDE_PROXY` | HTTP 代理(如 `http://127.0.0.1:10808`);同时作用于 Telegram Bot API 与站点抓取请求,网络受限环境(如 GFW)必需。**不要留空值**`TELOXIDE_PROXY=`)——teloxide 对无法解析的值会直接 panic;不用代理就别写这一行。容器里要用代理时,`docker-compose.yml` 的 `environment` 里默认没有它(容器内的 `127.0.0.1` 是容器自己),需要时手动加上并把地址换成 `host.docker.internal:<port>` |
| `LOCAL_USER_ID` | 容器内运行用户 UID,默认 9001 |
| `VIRTUAL_HOST` | 对外域名或 IPnginx-proxy 按此路由 |
| `VIRTUAL_HOST` | 对外域名或 IPnginx-proxy 按此路由(写在 `.env`compose 读取) |
| `VIRTUAL_PORT` | bot 容器内监听端口,nginx-proxy 的转发目标 |
| `ACME_HOST` | 域名部署:设为域名时由 acme-companion 自动签发/续期证书 |
| `DEFAULT_HOST` | nginx-proxy 将未知 Host 的请求路由到该 vhost(IP 访问时需要) |
@@ -109,14 +121,17 @@ Telegram 只接受 443/80/88/8443 端口。
| `/help` | 查看全部命令及用法(即本文档的命令表) |
| `/set_forward_channel <频道>` | 设置转发频道,参数为 `@频道名` 或频道 ID;设置后发送的媒体消息会自动转发到该频道 |
| `/remove_forward_channel` | 取消转发频道 |
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板),再点 `↩️ Confirm` 才会真正转发,`🛑 Skip` 放弃本次转发;提示消息写明过期时间,过期后原地标记为已过期且不会转发 |
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv` / `misskey`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
| `/remove_template <名称>` | 删除某个模板(名称见 `/settings`;提示消息的模板按钮最多显示 60 个) |
| `/settings` | 查看本聊天配置:转发频道、转发前编辑开关、各站点 caption 格式、模板列表 |
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv` / `misskey` / `bilibili`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{content}` `{tags}`;未识别的占位符会被拒绝并列出可用项,格式填 `-` 恢复站点默认格式(可用 `/debug <链接>` 预览效果) |
| `/clear_cache [链接]` | 清空链接缓存(仅管理员);带链接只清该条,否则清空全部 |
| `/bot_dict` | 查看当前聊天状态(调试用) |
| `/test <链接>` | 调试:只解析链接并返回解析结果(站点、标题、作者、标签、媒体列表),不发送任何媒体 |
| `/bot_dict` | 查看当前聊天状态(调试用;仅管理员 |
| `/test <链接>` | 解析链接并发送媒体;不转发到频道、不弹转发前编辑提示(仅发送) |
| `/debug <链接>` | 调试:只解析链接并返回解析结果(站点、标题、作者、标签、媒体列表),不发送任何媒体 |
链接处理仅限私聊;命令在任意聊天可用。
链接处理仅限私聊;命令在任意聊天可用。在群聊里发受支持的链接会回复一条提示(改用私聊或内联查询),频道内保持静默。
## 备注
+5 -5
View File
@@ -1,22 +1,22 @@
[package]
name = "x-media"
version = "1.5.0"
version = "1.9.0"
edition = "2024"
[dependencies]
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "gzip", "http2"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
regex = "1.12"
html-escape = "0.2"
url = "2.5.2"
bytes = "1"
zip = "2"
zip = "8"
tempfile = "3"
thiserror = "2"
rand = "0.8"
rand = "0.10"
log = "0.4"
tokio = { version = "1.40", features = ["time"] }
tokio = { version = "1.40", features = ["time", "rt", "fs"] }
[dev-dependencies]
tokio = { version = "1.40", features = ["macros", "rt-multi-thread"] }
+7
View File
@@ -1,2 +1,9 @@
pub mod media;
pub mod site;
/// Prefix every temp file and temp dir this project creates, so a startup
/// sweep can recognise its own leftovers: a killed process leaves them behind
/// (`TempDir`/`NamedTempFile` clean up on drop, and a killed process runs no
/// destructors), and without a marker the only safe assumption about the OS
/// temp directory is "not mine".
pub const TEMP_FILE_PREFIX: &str = "tgxmb-";
File diff suppressed because it is too large Load Diff
+6
View File
@@ -0,0 +1,6 @@
mod interface;
mod model;
pub use interface::{
BilibiliSite, PATTERN, cache_key, enabled, fetch_from_url, is_retryable, media_headers,
};
+158
View File
@@ -0,0 +1,158 @@
//! Serde DTOs for the Bilibili dynamic detail endpoint
//! (`/x/polymer/web-dynamic/v1/detail`), mirroring live responses
//! (field paths verified 2026-09-17). Every field is optional so an API
//! shape change degrades to "no media" instead of a parse failure.
use serde::Deserialize;
#[derive(Deserialize, Debug)]
pub(crate) struct Detail {
/// Business code: `0` = OK, `-352`/`-412` = risk control, `500`/`4101147`
/// = gone.
pub(crate) code: i64,
#[serde(default)]
pub(crate) message: Option<String>,
#[serde(default)]
pub(crate) data: Option<Data>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Data {
#[serde(default)]
pub(crate) item: Option<Box<Item>>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Item {
/// The dynamic id, same numeric id as in the URL.
#[serde(default)]
pub(crate) id_str: String,
#[serde(default)]
pub(crate) modules: Option<Modules>,
/// The quoted dynamic when this item is a forward. A forward shell often
/// carries no media of its own — the original holds it.
#[serde(default)]
pub(crate) orig: Option<Box<Item>>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Modules {
#[serde(default)]
pub(crate) module_author: Option<Author>,
#[serde(default)]
pub(crate) module_dynamic: Option<Dynamic>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Author {
#[serde(default)]
pub(crate) name: String,
#[serde(default)]
pub(crate) mid: Option<i64>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Dynamic {
#[serde(default)]
pub(crate) desc: Option<Desc>,
#[serde(default)]
pub(crate) major: Option<Major>,
/// A single topic (`{"id":…,"name":…}`), the dynamic's only tag source.
#[serde(default)]
pub(crate) topic: Option<Topic>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Desc {
#[serde(default)]
pub(crate) text: String,
}
/// `major` is a tagged union: `type` (`MAJOR_TYPE_DRAW` / `_OPUS` /
/// `_ARCHIVE` / …) plus one payload object per type. Only the three payloads
/// this adapter reads are modeled; an unknown major simply yields no media.
#[derive(Deserialize, Debug)]
pub(crate) struct Major {
#[serde(default)]
pub(crate) draw: Option<Draw>,
#[serde(default)]
pub(crate) opus: Option<Opus>,
#[serde(default)]
pub(crate) archive: Option<Archive>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Draw {
#[serde(default)]
pub(crate) items: Vec<Pic>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Pic {
/// `major.draw` image URL.
#[serde(default)]
pub(crate) src: Option<String>,
/// `major.opus.pics` image URL — the opus shape names the field
/// differently while carrying the same image.
#[serde(default)]
pub(crate) url: Option<String>,
}
impl Pic {
/// The image URL, whichever key this serialization put it under.
pub(crate) fn url(&self) -> Option<&str> {
self.src.as_deref().or(self.url.as_deref())
}
}
/// `major.opus`: the serialization of an image/text post the web client asks
/// for (`features=itemOpusStyle`). It carries the parts the legacy shape drops
/// entirely — the document title and body of an opus post, whose
/// `module_dynamic.desc` comes back `null`.
#[derive(Deserialize, Debug)]
pub(crate) struct Opus {
/// Document headline; often absent.
#[serde(default)]
pub(crate) title: Option<String>,
/// Document body (untruncated: a 307-char sample came back whole).
#[serde(default)]
pub(crate) summary: Option<Desc>,
#[serde(default)]
pub(crate) pics: Vec<Pic>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Archive {
/// The attached video's cover — the only image an AV dynamic has (the
/// video itself is deliberately not resolved, see the module docs).
#[serde(default)]
pub(crate) cover: Option<String>,
/// The video's title. An AV dynamic has no body of its own (`desc` comes
/// back `null`), so this card title is the post's content.
#[serde(default)]
pub(crate) title: Option<String>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct Topic {
#[serde(default)]
pub(crate) name: String,
}
/// Response of the anonymous fingerprint endpoint (`/x/frontend/finger/spi`),
/// the source of the adapter's device cookies.
#[derive(Deserialize, Debug)]
pub(crate) struct Fingerprint {
#[serde(default)]
pub(crate) data: Option<FingerprintData>,
}
#[derive(Deserialize, Debug)]
pub(crate) struct FingerprintData {
/// Sent as the `buvid3` cookie.
#[serde(default, rename = "b_3")]
pub(crate) buvid3: String,
/// Sent as the `buvid4` cookie.
#[serde(default, rename = "b_4")]
pub(crate) buvid4: String,
}
+152 -42
View File
@@ -51,6 +51,17 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
// encode path — the temp file stays alive via `_keep_alive`). On any
// failure the video item is dropped and the post degrades to its text.
let mut media = Vec::with_capacity(fetched.media.len());
// The remux warnings below name the post, not the CDN URL they were
// working on: the media URL is derived from what the user pasted, and
// `warn` is a level operators share.
let key = cache_key(url).unwrap_or_else(|| "?".into());
// A failed remux is remembered: if it leaves the post with no media at
// all, returning `Ok` would read as "this post has no media". It is
// reported as `FetchError::MediaPrep` rather than a transient failure —
// the download legs already got their own retry in place ([`fetch_hls`]),
// and the fetch loop's retry would only download every segment again to
// fail the same way.
let mut remux_failure: Option<String> = None;
for item in fetched.media {
let is_hls = matches!(&item, Media::Video { url, .. }
if url.contains("playlist") || url.ends_with(".m3u8"));
@@ -70,12 +81,25 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
url: mp4_path.to_string_lossy().into_owned(),
thumbnail_url,
});
fetched._keep_alive = Some(keep_alive);
fetched._keep_alive = Some(std::sync::Arc::new(keep_alive));
}
// No ffmpeg: a deployment gap, not a bad moment — retrying it
// would only waste the fetch budget, so the post degrades (and an
// all-video post reports the media type as unsupported).
Ok(None) => log::warn!("bsky video remux unavailable for [key={key}]"),
Err(e) => {
log::warn!("bsky video remux failed for [key={key}]: {e}");
remux_failure = Some(e);
}
Ok(None) => log::warn!("bsky video remux unavailable for {url}"),
Err(e) => log::warn!("bsky video remux failed for {url}: {e}"),
}
}
if media.is_empty()
&& let Some(reason) = remux_failure
{
return Err(FetchError::MediaPrep(format!(
"bsky video remux failed: {reason}"
)));
}
fetched.media = media;
Ok(fetched)
}
@@ -99,6 +123,41 @@ pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// Segments fetched (and written) at once while remuxing an HLS video. Small
/// on purpose: a segment can be up to 20 MiB and the whole playlist is capped
/// at 256 MiB, so this is also what bounds the remux's peak memory.
const SEGMENT_CONCURRENCY: usize = 4;
/// The ffmpeg concat list for the downloaded segments, **in segment order**.
/// The downloads complete in completion order (`JoinSet`), and ffmpeg would
/// happily concatenate them in whatever order the list holds: an out-of-order
/// list produces a silently scrambled video, not an error.
fn concat_list(files: &mut [(usize, std::path::PathBuf)]) -> String {
files.sort_by_key(|(i, _)| *i);
files
.iter()
.map(|(_, path)| format!("file '{}'\n", path.to_string_lossy()))
.collect()
}
/// One HLS fetch (a playlist or a segment) with an in-place retry for a
/// retryable class (transport, 429/5xx). These used to get their retry from the
/// outer fetch loop, which pays for it by replaying the whole post: master
/// playlist, variant playlist and every segment again. A segment failing near
/// the end of a 500-segment video meant downloading the entire thing twice
/// more, so the second attempt belongs on the request that actually failed.
async fn fetch_hls(url: &str, cap: u64) -> Result<bytes::Bytes, String> {
match crate::site::download_media_limited(url, cap).await {
Err(FetchError::Http(_) | FetchError::Transient(_)) => {
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
crate::site::download_media_limited(url, cap)
.await
.map_err(|e| e.to_string())
}
other => other.map_err(|e| e.to_string()),
}
}
/// Downloads an HLS playlist (master or media) and remuxes its segments to a
/// single MP4 via ffmpeg. Returns the MP4 path plus the temp dir that must
/// stay alive until the file is uploaded. `Ok(None)` when ffmpeg is missing.
@@ -114,7 +173,7 @@ async fn resolve_bsky_video(
crate::site::log_once_ffmpeg_missing();
return Ok(None);
}
let master = crate::site::download_media_limited(playlist_url, 1_048_576)
let master = fetch_hls(playlist_url, 1_048_576)
.await
.map_err(|e| format!("bsky video master playlist: {e}"))?;
let master = String::from_utf8_lossy(&master);
@@ -149,7 +208,7 @@ async fn resolve_bsky_video(
playlist_url.to_string()
};
let variant = crate::site::download_media_limited(&playlist_url, 1_048_576)
let variant = fetch_hls(&playlist_url, 1_048_576)
.await
.map_err(|e| format!("bsky video media playlist: {e}"))?;
let variant = String::from_utf8_lossy(&variant);
@@ -169,24 +228,53 @@ async fn resolve_bsky_video(
return Err("bsky video has too many segments".to_string());
}
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let frames_dir = tempfile::Builder::new()
.prefix(crate::TEMP_FILE_PREFIX)
.tempdir()
.map_err(|e| e.to_string())?;
let out_dir = tempfile::Builder::new()
.prefix(crate::TEMP_FILE_PREFIX)
.tempdir()
.map_err(|e| e.to_string())?;
// Segments are fetched concurrently under a small bound, and written with
// `tokio::fs` (a multi-megabyte `std::fs::write` blocks the executor
// thread). Serially, a several-hundred-segment video made the user wait
// for every round trip in turn — the dominant cost of a remux.
let mut total: u64 = 0;
let mut list = String::new();
for (i, seg) in segments.iter().enumerate() {
let bytes = crate::site::download_media_limited(seg, 20 * 1024 * 1024)
.await
.map_err(|e| format!("bsky segment {i}: {e}"))?;
total += bytes.len() as u64;
let mut written: Vec<(usize, std::path::PathBuf)> = Vec::with_capacity(segments.len());
let mut next = 0;
let mut set = tokio::task::JoinSet::new();
loop {
while set.len() < SEGMENT_CONCURRENCY && next < segments.len() {
let i = next;
next += 1;
let seg = segments[i].clone();
let path = frames_dir.path().join(format!("seg_{i:04}.ts"));
set.spawn(async move {
let bytes = fetch_hls(&seg, 20 * 1024 * 1024)
.await
.map_err(|e| format!("bsky segment {i}: {e}"))?;
tokio::fs::write(&path, &bytes)
.await
.map_err(|e| format!("bsky segment {i}: {e}"))?;
Ok::<_, String>((i, bytes.len() as u64, path))
});
}
let Some(joined) = set.join_next().await else {
break;
};
let (i, len, path) = joined.map_err(|e| format!("bsky segment task panicked: {e}"))??;
total += len;
if total > 256 * 1024 * 1024 {
return Err("bsky video exceeds total size cap".to_string());
}
let path = frames_dir.path().join(format!("seg_{i:04}.ts"));
std::fs::write(&path, &bytes).map_err(|e| e.to_string())?;
list.push_str(&format!("file '{}'\n", path.to_string_lossy()));
written.push((i, path));
}
let list = concat_list(&mut written);
let list_path = frames_dir.path().join("list.txt");
std::fs::write(&list_path, &list).map_err(|e| e.to_string())?;
tokio::fs::write(&list_path, &list)
.await
.map_err(|e| e.to_string())?;
let output = out_dir.path().join("video.mp4");
let list_str = list_path.to_string_lossy().into_owned();
@@ -235,6 +323,9 @@ pub async fn fetch(handle: &str, rkey: &str) -> Result<Post, FetchError> {
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
// A refusal or an auth demand is not a bad moment: retrying it
// three times only delays an error the user has to see.
401 | 403 => Err(FetchError::Blocked),
_ => Err(FetchError::Transient(format!("bsky status {status}"))),
};
}
@@ -330,13 +421,16 @@ impl From<Post> for Fetched {
url: url.clone(),
author: encode_text(&post.author).into_owned(),
author_url: author_url.clone(),
title: encode_text(&post.text).into_owned(),
// A post has no title: its text is all content.
title: String::new(),
content: encode_text(&post.text).into_owned(),
tags: String::new(),
});
Fetched {
source_url: url,
caption: post.caption(),
title: post.text.clone(),
title: String::new(),
content: post.text.clone(),
media: post.media,
sensitive: post.sensitive,
site_id: "bsky",
@@ -357,6 +451,22 @@ mod tests {
serde_json::json!({ "thread": post_json })
}
/// The downloads finish in completion order; ffmpeg concatenates whatever
/// order `list.txt` holds, so an unsorted list is a scrambled video rather
/// than an error.
#[test]
fn concat_list_is_in_segment_order() {
let mut files = vec![
(2, std::path::PathBuf::from("/t/seg_0002.ts")),
(0, std::path::PathBuf::from("/t/seg_0000.ts")),
(1, std::path::PathBuf::from("/t/seg_0001.ts")),
];
assert_eq!(
concat_list(&mut files),
"file '/t/seg_0000.ts'\nfile '/t/seg_0001.ts'\nfile '/t/seg_0002.ts'\n"
);
}
#[test]
fn pattern_matches_handle_and_did() {
let cases = [
@@ -389,6 +499,18 @@ mod tests {
}
}
/// A remux failure is a `MediaPrep`, which the fetch loop does not retry:
/// replaying the post means downloading every HLS segment again, when the
/// request that failed already got its second attempt in place
/// ([`fetch_hls`]). The classes below are the ones still retried there.
#[test]
fn media_prep_failure_is_not_retried() {
assert!(!is_retryable(&FetchError::MediaPrep(
"bsky video remux failed: segment 400: 503".into()
)));
assert!(is_retryable(&FetchError::Transient("429".into())));
}
#[test]
fn from_json_images_with_missing_defaults() {
let raw = thread_json(serde_json::json!({
@@ -410,7 +532,8 @@ mod tests {
fetched.source_url,
"https://bsky.app/profile/user.bsky.social/post/3xxxx"
);
assert_eq!(fetched.title, "hello <world>");
assert_eq!(fetched.title, "");
assert_eq!(fetched.content, "hello <world>");
assert_eq!(fetched.media.len(), 1);
assert!(!fetched.sensitive);
// display_name absent -> empty fallback
@@ -456,31 +579,18 @@ mod tests {
));
}
/// The one live bsky check: a labelled post with photos — source URL,
/// caption, media and the sensitive label all survive the parse. This
/// replaced a second byte-identical live test whose URL is a *text-only*
/// post, so neither copy pinned any media.
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
async fn live_fetch_with_photos() {
let fetched =
fetch_from_url("https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m")
.await
.unwrap();
assert_eq!(
fetched.source_url,
"https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m"
);
assert!(!fetched.caption.is_empty());
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to public.api.bsky.app"]
async fn live_fetch_smoke() {
let fetched =
fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
.await
.unwrap();
assert_eq!(
fetched.source_url,
"https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224"
);
let url = "https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224";
let fetched = fetch_from_url(url).await.unwrap();
assert_eq!(fetched.source_url, url);
assert!(!fetched.caption.is_empty());
assert!(!fetched.media.is_empty(), "expected photos in {url}");
assert!(fetched.sensitive, "expected a label on {url}");
}
}
+19 -11
View File
@@ -77,6 +77,8 @@ pub async fn fetch(note_id: &str) -> Result<model::Note, FetchError> {
if !status.is_success() {
return Err(match status.as_u16() {
400 => not_found_or_invalid(response).await,
// A refusal or an auth demand is not a bad moment.
401 | 403 => FetchError::Blocked,
_ => FetchError::Transient(format!("misskey status {status}")),
});
}
@@ -123,21 +125,23 @@ impl From<model::Note> for Fetched {
let cw = content.cw.as_deref().unwrap_or_default();
// Notes carry hashtags inline in the text (no structured tags array);
// a CW note gets the marker prefixed so recipients see the spoiler.
let mut title = cw.to_string();
if !cw.is_empty() && !title.ends_with(' ') {
title.push(' ');
let mut text = cw.to_string();
if !cw.is_empty() && !text.ends_with(' ') {
text.push(' ');
}
title.push_str(content.text.as_deref().unwrap_or_default().trim());
let title = title.trim().to_string();
text.push_str(content.text.as_deref().unwrap_or_default().trim());
let text = text.trim().to_string();
let caption = caption(&url, &author_url, &author, &title);
let caption = caption(&url, &author_url, &author, &text);
let sensitive = content.cw.is_some() || content.files.iter().any(|f| f.is_sensitive);
let media: Vec<Media> = content.files.iter().filter_map(media_from_file).collect();
Fetched {
source_url: url.clone(),
caption,
title: title.clone(),
// A note has no title: its text (CW marker included) is content.
title: String::new(),
content: text.clone(),
media,
sensitive,
site_id: "misskey",
@@ -145,7 +149,8 @@ impl From<model::Note> for Fetched {
url,
author: encode_text(&author).into_owned(),
author_url: author_url.clone(),
title: encode_text(&title).into_owned(),
title: String::new(),
content: encode_text(&text).into_owned(),
tags: String::new(),
}),
_keep_alive: None,
@@ -257,7 +262,8 @@ mod tests {
"https://misskey.io/notes/aotihl10lqrs015s"
);
assert_eq!(fetched.site_id, "misskey");
assert_eq!(fetched.title, "hello");
assert_eq!(fetched.title, "");
assert_eq!(fetched.content, "hello");
assert!(fetched.sensitive);
assert_eq!(fetched.media.len(), 1);
match &fetched.media[0] {
@@ -308,7 +314,8 @@ mod tests {
note["text"] = serde_json::json!("body");
let fetched: Fetched = note_json(note).into();
assert!(fetched.sensitive);
assert_eq!(fetched.title, "spoiler body");
assert_eq!(fetched.title, "");
assert_eq!(fetched.content, "spoiler body");
}
#[test]
@@ -341,7 +348,8 @@ mod tests {
}
});
let fetched: Fetched = note_json(note).into();
assert_eq!(fetched.title, "inner text");
assert_eq!(fetched.title, "");
assert_eq!(fetched.content, "inner text");
assert_eq!(fetched.media.len(), 1);
// The source URL still points at the renote shell the user posted.
assert_eq!(
+560 -83
View File
@@ -1,8 +1,8 @@
//! Site fetching dispatcher and unified result types.
//!
//! Dispatch order: twitter → bsky → misskey → pixiv. Each site module
//! exports a `PATTERN`, `enabled()` and `fetch_from_url()`; a future site
//! plugs in by adding one guarded entry in [`fetch_once`].
//! Dispatch order: twitter → bsky → misskey → pixiv → bilibili. Each site
//! module exports a `PATTERN`, `enabled()` and `fetch_from_url()`; a future
//! site plugs in by adding one guarded entry in `SITES`.
use std::future::Future;
use std::pin::Pin;
@@ -13,6 +13,7 @@ use std::time::Duration;
use regex::Regex;
use thiserror::Error;
pub mod bilibili;
pub mod bsky;
pub mod misskey;
pub mod pixiv;
@@ -20,43 +21,76 @@ pub mod twitter;
pub use pixiv::PixivError;
/// The result of fetching a post: canonical URL, HTML caption, raw text,
/// media list and spoiler flag. Produced by [`fetch`].
/// The result of fetching a post: canonical URL, HTML caption, the post's
/// title and body, media list and spoiler flag. Produced by [`fetch`].
#[derive(Debug)]
pub struct Fetched {
/// Canonical URL: x.com/{author}/status/{id} |
/// https://www.pixiv.net/artworks/{id} |
/// https://bsky.app/profile/{handle}/post/{rkey}
/// Canonical URL: `x.com/{author}/status/{id}` |
/// `https://www.pixiv.net/artworks/{id}` |
/// `https://bsky.app/profile/{handle}/post/{rkey}` |
/// `https://www.bilibili.com/opus/{id}`
pub source_url: String,
/// The exact HTML produced by the site's caption().
pub caption: String,
/// Raw post text (tweet text / bsky text / pixiv title).
/// The post's own title, where the platform has one: a pixiv artwork's
/// title, the headline of a bilibili opus post or the title of the video
/// an AV dynamic attaches. Empty on the platforms whose posts are text
/// only (x/twitter, bsky, misskey) and on bilibili posts without a
/// headline.
pub title: String,
/// The post's body text, as the platform exposes it: a tweet, a bsky or
/// misskey post, a bilibili dynamic's text, a pixiv artwork's description
/// (HTML flattened). Empty when the post has no text at all.
pub content: String,
pub media: Vec<crate::media::Media>,
/// Spoiler flag for all media of this post.
pub sensitive: bool,
/// Site id (`"twitter"` / `"bsky"` / `"pixiv"`): the single source of
/// Site id (`"twitter"` / `"bsky"` / `"pixiv"` / `"bilibili"`): the single source of
/// truth for site identity — caption-format lookup, cache-key prefix and
/// the SetFormat whitelist all derive from it. Set by the producing site.
pub site_id: &'static str,
/// Raw values (pre-escaped) for user-customizable caption formats.
pub(crate) render_data: Option<RenderData>,
/// Keeps temp files (e.g. an encoded ugoira MP4) alive until the caller
/// finishes uploading; not part of the public contract.
pub(crate) _keep_alive: Option<tempfile::TempDir>,
/// finishes uploading; not part of the public contract. Shared rather than
/// owned because one fetched post can serve several sends — the bot shares
/// one in-flight fetch between concurrent duplicates of the same link — and
/// the files have to outlive every one of them.
pub(crate) _keep_alive: Option<std::sync::Arc<tempfile::TempDir>>,
}
/// Pre-escaped values for `{url} {author} {author_url} {title} {tags}`
/// placeholders in user-supplied caption formats.
/// Values for the `{url} {author} {author_url} {title} {content} {tags}`
/// placeholders in user-supplied caption formats, substituted by
/// [`caption_from_fields`] as HTML text (never as an attribute value).
///
/// `author`, `title`, `content` and `tags` come from the site API (post
/// text, display names, descriptions) and are HTML-escaped at construction.
/// `url` and `author_url` stay raw: they are canonical URLs the adapter
/// builds from numeric ids and API-constrained handles/DIDs, so they carry
/// no escapable character — the bot's `/test` report relies on that when it
/// embeds them.
#[derive(Debug)]
pub(crate) struct RenderData {
pub url: String,
pub author: String,
pub author_url: String,
pub title: String,
pub content: String,
pub tags: String,
}
/// The post's text as one string: title and content joined by a line break,
/// each only when it is non-empty. This is what the sites' built-in captions
/// show after the author line, and what the bot quotes when it is long.
pub fn compose_text(title: &str, content: &str) -> String {
match (title.is_empty(), content.is_empty()) {
(false, false) => format!("{title}\n{content}"),
(false, true) => title.to_string(),
(true, false) => content.to_string(),
(true, true) => String::new(),
}
}
impl Fetched {
/// The site this post came from (used for per-site format overrides).
/// A thin alias over [`Fetched::site_id`] kept for callers that read the
@@ -80,32 +114,36 @@ impl Fetched {
&data.author,
&data.author_url,
&data.title,
&data.content,
&data.tags,
),
_ => truncate_caption(&self.caption),
}
}
/// The pre-escaped placeholder values (author, author_url, title, tags)
/// a caller needs to rebuild a caption later, e.g. for a cached post
/// where the [`Fetched`] is no longer available.
pub fn render_fields(&self) -> Option<(&str, &str, &str, &str)> {
/// The pre-escaped placeholder values (author, author_url, title,
/// content, tags) a caller needs to rebuild a caption later, e.g. for a
/// cached post where the [`Fetched`] is no longer available.
pub fn render_fields(&self) -> Option<(&str, &str, &str, &str, &str)> {
self.render_data.as_ref().map(|d| {
(
d.author.as_str(),
d.author_url.as_str(),
d.title.as_str(),
d.content.as_str(),
d.tags.as_str(),
)
})
}
/// Hands over the temp dir keeping locally produced media (ugoira MP4,
/// A reference to the temp dir keeping locally produced media (ugoira MP4,
/// bsky remux MP4) alive. The bot keeps it while its task may still be
/// retried by the queue, which runs after this [`Fetched`] is dropped and
/// its temp files would otherwise be gone. `None` when no such dir exists.
pub fn take_keep_alive(&mut self) -> Option<tempfile::TempDir> {
self._keep_alive.take()
/// its temp files would otherwise be gone. `None` when no such dir exists;
/// each clone keeps the directory alive for as long as it lives, so two
/// sends of one post can each hold the same files.
pub fn keep_alive(&self) -> Option<std::sync::Arc<tempfile::TempDir>> {
self._keep_alive.clone()
}
}
@@ -140,6 +178,11 @@ pub fn truncate_caption(caption: &str) -> String {
/// [`Fetched::caption_with`]. An empty format returns `built_in` unchanged.
/// The result is truncated to [`MAX_CAPTION_CHARS`] (Telegram's caption
/// limit for HTML parse mode).
///
/// One flat argument per placeholder keeps the two callers (the fresh and the
/// cached caption path) mirroring each other; the same shape as the bot's
/// `debug_report`.
#[allow(clippy::too_many_arguments)]
pub fn caption_from_fields(
format: &str,
built_in: &str,
@@ -147,6 +190,7 @@ pub fn caption_from_fields(
author: &str,
author_url: &str,
title: &str,
content: &str,
tags: &str,
) -> String {
if format.is_empty() {
@@ -159,6 +203,7 @@ pub fn caption_from_fields(
.replace("{author}", author)
.replace("{author_url}", author_url)
.replace("{title}", title)
.replace("{content}", content)
.replace("{tags}", tags),
)
}
@@ -166,7 +211,7 @@ pub fn caption_from_fields(
/// Stable per-post cache key derived from any supported URL, so variant
/// domains (x.com / twitter.com / fxtwitter.com, mobile, `/photo/N`
/// suffixes) map to the same post. Delegates to each registered site's
/// `cache_key` (dispatch order twitter → bsky → pixiv).
/// `cache_key` (in registry order).
pub fn cache_key(url: &str) -> Option<String> {
SITES.iter().find_map(|site| site.cache_key(url))
}
@@ -207,6 +252,12 @@ pub enum FetchError {
NotFound,
#[error("blocked")]
Blocked,
/// The URL matches a registered site that is disabled right now (pixiv
/// without `PIXIV_REFRESH_TOKEN`, or after a failed login). Distinct from
/// `Ok(None)` — an unsupported link — so the bot can tell the user why
/// the link was not handled instead of silently ignoring it.
#[error("{site} support is disabled")]
Disabled { site: &'static str },
/// The post exists but its content is withheld (twitter NSFW /
/// age-restricted tweets come back as an empty `{}` from syndication).
#[error("content withheld (sensitive)")]
@@ -214,6 +265,13 @@ pub enum FetchError {
/// A download exceeded the caller's size cap (see [`download_media_limited`]).
#[error("media too large")]
TooLarge,
/// The post was fetched, but its media could not be prepared locally — a
/// download or encode step that runs *after* the site's own response
/// (bsky's HLS remux, say). Deliberately not retryable: the retry would
/// replay the whole fetch, redoing the download work that just failed
/// instead of the request that failed.
#[error("media could not be prepared: {0}")]
MediaPrep(String),
/// A transient server-side failure (429 / 5xx); [`fetch`] retries these.
#[error("transient: {0}")]
Transient(String),
@@ -223,15 +281,57 @@ pub enum FetchError {
Io(std::io::Error),
}
/// Shared HTTP client (browser User-Agent) for twitter/bsky fetches and
/// [`download_media`].
pub(crate) static CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
/// How long a download may make no progress: the response head, and then each
/// individual chunk, must arrive within this window. Not a total timeout — see
/// [`DOWNLOAD_TOTAL_TIMEOUT`].
const DOWNLOAD_IDLE_TIMEOUT: Duration = Duration::from_secs(30);
/// Absolute ceiling for one media download, on top of the idle window. A server
/// that drips a byte every 29 s keeps [`next_chunk`] satisfied indefinitely, and
/// on the bot's side each such download holds one of the process-wide upload-prep
/// slots (`send::upload`'s `PREP_SLOTS`) for as long as it lasts. Generous on
/// purpose: the legitimate cases are big — an ugoira frame zip runs to hundreds
/// of MB and an HLS remux pulls a whole video — and a slow link is not an error.
/// Checked between chunks, so a transfer that completes just over the budget is
/// kept rather than thrown away.
const DOWNLOAD_TOTAL_TIMEOUT: Duration = Duration::from_secs(600);
/// The error a download reports when it spends its whole budget without
/// finishing. Retryable: the transfer may simply have been unlucky, and a retry
/// of the post restarts the download.
fn download_too_slow() -> FetchError {
FetchError::Transient(format!(
"download exceeded {}s",
DOWNLOAD_TOTAL_TIMEOUT.as_secs()
))
}
/// Builds a client with the shared configuration (browser User-Agent, the
/// Bot API's proxy, per-runtime pools under test). `total_timeout` is what
/// differs between the two clients below.
fn build_client(total_timeout: Option<Duration>) -> reqwest::Client {
let mut builder = reqwest::Client::builder()
.user_agent("Mozilla/5.0")
.connect_timeout(Duration::from_secs(10));
// Redirects stay allowed (site CDNs use them), but every hop goes through
// the same guard as the initial URL, and the cap stays reqwest's default:
// a third-party response must not be able to walk the bot into the host's
// own network.
builder = builder.redirect(reqwest::redirect::Policy::custom(|attempt| {
if !media_url_allowed(attempt.url()) {
log::warn!("refusing a media redirect into the host's own network");
return attempt.error(FetchError::Blocked);
}
if attempt.previous().len() >= 10 {
return attempt.stop();
}
attempt.follow()
}));
if let Some(total) = total_timeout {
// reqwest has no total timeout by default; a stalled connection
// would otherwise pin a fetch/handler forever.
.timeout(Duration::from_secs(30))
.connect_timeout(Duration::from_secs(10));
builder = builder.timeout(total);
}
// Route site fetches through the same proxy the Bot API uses, so a
// network that needs TELOXIDE_PROXY (e.g. behind the GFW) does not
// leave site fetches dead while the bot itself works.
@@ -249,7 +349,55 @@ pub(crate) static CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
#[cfg(test)]
let builder = builder.pool_max_idle_per_host(0);
builder.build().expect("failed to build HTTP client")
});
}
/// Shared HTTP client (browser User-Agent) for the site fetches — metadata
/// requests, where 30s is generous.
pub(crate) static CLIENT: LazyLock<reqwest::Client> =
LazyLock::new(|| build_client(Some(Duration::from_secs(30))));
/// Client for media *downloads*, with no reqwest-level total timeout: a 10 MiB
/// fallback download, or an ugoira frame zip that may be hundreds of MB,
/// legitimately takes minutes on a slow link — a 30s total cap made those posts
/// impossible to deliver at all (the size cap said 512 MiB, the clock said 30s).
/// What a stalled connection cannot do is hang a worker: the head and every
/// chunk are bounded by [`DOWNLOAD_IDLE_TIMEOUT`] (see [`next_chunk`]), and a
/// transfer that keeps trickling but never finishes is bounded by
/// [`DOWNLOAD_TOTAL_TIMEOUT`].
static MEDIA_CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| build_client(None));
/// The error a download reports when it stops making progress.
fn download_stalled() -> FetchError {
FetchError::Transient(format!(
"download stalled for {}s",
DOWNLOAD_IDLE_TIMEOUT.as_secs()
))
}
/// Sends a media-download request: the response head must arrive within the
/// idle window, and a non-2xx status is classified by [`download_status_error`].
async fn send_download(request: reqwest::RequestBuilder) -> Result<reqwest::Response, FetchError> {
let response = match tokio::time::timeout(DOWNLOAD_IDLE_TIMEOUT, request.send()).await {
Ok(Ok(response)) => response,
Ok(Err(e)) => return Err(e.into()),
Err(_) => return Err(download_stalled()),
};
if response.status().is_success() {
Ok(response)
} else {
Err(download_status_error(response.status()))
}
}
/// One body chunk, or `None` at the end. A body that stops delivering is a
/// transient download error rather than a hang.
async fn next_chunk(response: &mut reqwest::Response) -> Result<Option<bytes::Bytes>, FetchError> {
match tokio::time::timeout(DOWNLOAD_IDLE_TIMEOUT, response.chunk()).await {
Ok(Ok(chunk)) => Ok(chunk),
Ok(Err(e)) => Err(e.into()),
Err(_) => Err(download_stalled()),
}
}
/// Whether a usable `ffmpeg` binary is on PATH (probed once). Shared by the
/// pixiv ugoira encoder and the bsky HLS remuxer.
@@ -275,20 +423,21 @@ pub(crate) fn log_once_ffmpeg_missing() {
}
}
/// Site adapter: one impl per supported site (twitter / bsky / pixiv),
/// registered in [`SITES`]. All site-specific knowledge — URL pattern,
/// Site adapter: one impl per supported site (twitter / bsky / misskey /
/// pixiv / bilibili), registered in `SITES`. All site-specific knowledge — URL pattern,
/// cache-key format, fetch, retry policy, media-host headers, startup
/// validation — lives in the site module; the central dispatcher only
/// iterates the registry.
///
/// Async methods return a boxed future (see [`SiteFuture`]): `async fn` /
/// Async methods return a boxed future (see `SiteFuture`): `async fn` /
/// RPITIT in traits are not dyn-compatible (verified on rustc 1.95), and
/// `+ Send` is required since URL/queue workers spawn these futures. The
/// site structs are stateless unit structs, so the boxed futures never
/// borrow from `self` beyond the call's scope.
pub trait Site: Send + Sync {
/// Stable site id (`"twitter"` / `"bsky"` / `"pixiv"`): caption-format
/// lookup, cache-key prefixes and the SetFormat whitelist derive from it.
/// Stable site id (`"twitter"` / `"bsky"` / `"misskey"` / `"pixiv"` /
/// `"bilibili"`): caption-format lookup, cache-key prefixes and the
/// SetFormat whitelist derive from it.
fn id(&self) -> &'static str;
/// URL pattern; the dispatcher's first match wins (dispatch order).
fn pattern(&self) -> &'static Regex;
@@ -324,14 +473,15 @@ pub trait Site: Send + Sync {
type SiteFuture<'a, T, E = FetchError> = Pin<Box<dyn Future<Output = Result<T, E>> + Send + 'a>>;
/// The one registry of supported sites, in dispatch order (twitter → bsky →
/// pixiv). Adding a site = new module + one `Box::new(...)` entry here; the
/// bot crate never lists sites itself.
/// misskey → pixiv → bilibili). Adding a site = new module + one
/// `Box::new(...)` entry here; the bot crate never lists sites itself.
static SITES: LazyLock<Vec<Box<dyn Site>>> = LazyLock::new(|| {
vec![
Box::new(twitter::TwitterSite),
Box::new(bsky::BskySite),
Box::new(misskey::MisskeySite),
Box::new(pixiv::PixivSite),
Box::new(bilibili::BilibiliSite),
]
});
@@ -343,6 +493,17 @@ fn find_site(url: &str) -> Option<&'static dyn Site> {
.map(|site| site.as_ref())
}
/// The site whose pattern matches `url` but which is disabled right now.
/// `None` when no site matches the URL at all, or when the matching site is
/// enabled. Lets the dispatcher tell "unsupported link" (silently ignored)
/// apart from "this bot has that site switched off" (reported to the user).
fn disabled_site(url: &str) -> Option<&'static str> {
SITES
.iter()
.find(|site| !site.enabled() && site.pattern().is_match(url))
.map(|site| site.id())
}
/// Every supported site id, in dispatch order. The bot's SetFormat whitelist
/// derives from this list.
pub fn site_ids() -> Vec<&'static str> {
@@ -366,7 +527,9 @@ pub async fn validate_all() -> Vec<(&'static str, String)> {
}
/// Fetches a post from its URL. Returns `Ok(None)` when no site pattern
/// matches (unsupported links are silently ignored by the bot).
/// matches (unsupported links are silently ignored by the bot) and
/// [`FetchError::Disabled`] when the URL belongs to a registered site that is
/// switched off right now — the two are different answers for the user.
///
/// Transient failures are retried: 3 total attempts with 1s then 2s delays.
/// What counts as transient is the matched site's own policy (`is_retryable`
@@ -375,23 +538,47 @@ pub async fn validate_all() -> Vec<(&'static str, String)> {
/// are returned immediately; retrying them only wastes attempts against the
/// source site.
pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
fetch_with_attempts(url, MAX_FETCH_ATTEMPTS).await
}
/// [`fetch`] without the retry backoff (one attempt). For callers with a
/// short deadline: an inline query's answer window is measured in seconds, so
/// the 1s + 2s retry sleeps would outlast the query the answer belongs to.
pub async fn fetch_once(url: &str) -> Result<Option<Fetched>, FetchError> {
fetch_with_attempts(url, 1).await
}
/// Total attempts of the retried [`fetch`] (3: the initial try plus two).
const MAX_FETCH_ATTEMPTS: u32 = 3;
async fn fetch_with_attempts(url: &str, attempts: u32) -> Result<Option<Fetched>, FetchError> {
// Wall time of the whole fetch, retry backoff included: the ugoira encode
// and the HLS remux live inside it, so this is where a slow fetch shows.
let started = std::time::Instant::now();
let Some(site) = find_site(url) else {
return Ok(None);
// A registered-but-disabled site (pixiv without a token) is not an
// unsupported link: report it, so the bot answers the user instead of
// ignoring the message.
return match disabled_site(url) {
Some(site) => Err(FetchError::Disabled { site }),
None => Ok(None),
};
};
for attempt in 0..3u32 {
for attempt in 0..attempts.max(1) {
match site.fetch_from_url(url).await {
Ok(fetched) => {
// Per-request detail: debug only, keyed by the post id.
log::debug!(
"fetched [key={}]: site {} returned {} media",
"fetched [key={}]: site {} returned {} media in {}ms",
cache_key(url).unwrap_or_else(|| "?".into()),
fetched.site_name(),
fetched.media.len()
fetched.media.len(),
started.elapsed().as_millis()
);
return Ok(Some(fetched));
}
Err(err) => {
if site.is_retryable(&err) && attempt < 2 {
if site.is_retryable(&err) && attempt + 1 < attempts {
tokio::time::sleep(Duration::from_secs(1 << attempt)).await;
} else {
return Err(err);
@@ -402,9 +589,96 @@ pub async fn fetch(url: &str) -> Result<Option<Fetched>, FetchError> {
unreachable!("retry loop always returns")
}
/// Whether fetching `url` requires site-specific headers (pixiv's `Referer`
/// for `pximg.net` hotlink protection, see [`Site::media_headers`]). Telegram's
/// own fetch of a media URL sends none of them, so a URL that needs them fails
/// there — callers that hand a URL to Telegram (inline query results) must
/// skip such media instead of shipping a broken item.
pub fn needs_media_headers(url: &str) -> bool {
SITES.iter().any(|site| site.media_headers(url).is_some())
}
/// Applies every site's media-header rule to a download request (pixiv's
/// `Referer` for pximg.net hotlink protection). Sites contribute via their
/// `media_headers(url)` — the central download code carries no per-site logic.
/// Whether an address must never be fetched. Media URLs come from a site's own
/// API response and the bytes are uploaded to Telegram, so following one into
/// the host's own network would turn the bot into a proxy for it: a cloud
/// metadata endpoint read back into a chat.
fn blocked_ip(addr: std::net::IpAddr) -> bool {
use std::net::IpAddr;
match addr {
IpAddr::V4(v4) => {
let [a, b, ..] = v4.octets();
v4.is_private() // 10/8, 172.16/12, 192.168/16
|| v4.is_loopback() // 127/8
|| v4.is_link_local() // 169.254/16 — the cloud metadata range
|| v4.is_unspecified()
|| v4.is_broadcast()
|| v4.is_documentation()
|| v4.is_multicast()
// Ranges the std helpers do not cover: carrier-grade NAT and
// benchmarking.
|| (a == 100 && (64..=127).contains(&b))
|| (a == 198 && (18..=19).contains(&b))
}
IpAddr::V6(v6) => {
let [first, ..] = v6.segments();
v6.is_loopback()
|| v6.is_unspecified()
|| v6.is_multicast()
|| (first & 0xfe00) == 0xfc00 // unique local fc00::/7
|| (first & 0xffc0) == 0xfe80 // link local fe80::/10
|| v6.to_ipv4_mapped().is_some_and(|v4| blocked_ip(IpAddr::V4(v4)))
}
}
}
/// `localhost` (and anything under it) plus the mDNS `.local` suffix: names that
/// only ever mean this machine.
fn is_local_name(name: &str) -> bool {
let name = name.trim_end_matches('.').to_ascii_lowercase();
name == "localhost" || name.ends_with(".localhost") || name.ends_with(".local")
}
/// Whether a media URL may be requested at all: http(s), and a host that is no
/// address or name of the host's own network. Applied to the URL a download
/// starts from *and* to every redirect hop.
///
/// The residual gap is DNS rebinding — a name the site controls that resolves to
/// a private address. Closing it needs a `reqwest::dns::Resolve` wrapper
/// filtering resolved addresses; it is deliberately not installed, because the
/// same resolver also resolves the operator's proxy host and `TELOXIDE_PROXY`
/// is routinely a LAN address, so the guard would take down a working
/// deployment to block a far less likely attack.
fn media_url_allowed(url: &url::Url) -> bool {
if !matches!(url.scheme(), "http" | "https") {
return false;
}
match url.host() {
Some(url::Host::Ipv4(v4)) => !blocked_ip(v4.into()),
Some(url::Host::Ipv6(v6)) => !blocked_ip(v6.into()),
Some(url::Host::Domain(name)) => !is_local_name(name),
None => false,
}
}
/// Prepares a media download: refuses a URL pointing inside the host's own
/// network ([`FetchError::Blocked`], permanent — the same URL would be refused
/// again), then applies the site's media headers. One choke point so every
/// download path gets the guard.
fn media_request(url: &str) -> Result<reqwest::RequestBuilder, FetchError> {
let parsed = url::Url::parse(url).map_err(|e| {
log::warn!("media url is not a url: {e}");
FetchError::Blocked
})?;
if !media_url_allowed(&parsed) {
log::warn!("refusing to fetch media from the host's own network");
return Err(FetchError::Blocked);
}
Ok(apply_media_headers(MEDIA_CLIENT.get(parsed), url))
}
fn apply_media_headers(mut request: reqwest::RequestBuilder, url: &str) -> reqwest::RequestBuilder {
for site in SITES.iter() {
if let Some(headers) = site.media_headers(url) {
@@ -416,30 +690,31 @@ fn apply_media_headers(mut request: reqwest::RequestBuilder, url: &str) -> reqwe
request
}
/// Downloads media bytes for the bot's upload fallback: when Telegram's own
/// fetch of a media URL is blocked (hotlink protection), the bot downloads
/// the file itself and uploads it via multipart. Site-appropriate headers
/// come from each site's `media_headers` (pixiv image hosts need `Referer`).
/// Returns the Content-Length of a media URL, or `None` when the server does
/// not report one. Used to check whether a file fits Telegram's size limits
/// before downloading/uploading it.
pub async fn media_size(url: &str) -> Result<Option<u64>, FetchError> {
let response = apply_media_headers(CLIENT.get(url), url)
.send()
.await?
.error_for_status()?;
Ok(response.content_length())
/// Maps a media download's HTTP status onto the same classes the site
/// adapters use, so callers can tell "try again" from "this URL is dead":
/// 4xx is a property of the media (gone, refused by the host), while 429/5xx
/// is a property of the moment. A transport error never reaches this — it
/// fails in `send()` and stays [`FetchError::Http`].
fn download_status_error(status: reqwest::StatusCode) -> FetchError {
match status.as_u16() {
401 | 403 => FetchError::Blocked,
404 | 410 => FetchError::NotFound,
_ => FetchError::Transient(format!("media status {status}")),
}
}
/// Downloads a media file with a hard size cap: the body is streamed and the
/// download aborts with [`FetchError::TooLarge`] the moment the cap is
/// crossed (or when a declared Content-Length already exceeds it). Keeps the
/// bot from buffering arbitrarily large bodies into memory.
/// bot from buffering arbitrarily large bodies into memory — the size check
/// the bot's upload fallback needs is the one here, not a probe of its own.
///
/// This is the bot's download path for the upload fallback: when Telegram
/// cannot fetch a media URL itself (hotlink protection), the bot downloads
/// the file and uploads it via multipart. Site-appropriate headers come from
/// each site's `media_headers` (pixiv image hosts need `Referer`).
pub async fn download_media_limited(url: &str, max_bytes: u64) -> Result<bytes::Bytes, FetchError> {
let response = apply_media_headers(CLIENT.get(url), url)
.send()
.await?
.error_for_status()?;
let response = send_download(media_request(url)?).await?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
@@ -447,7 +722,11 @@ pub async fn download_media_limited(url: &str, max_bytes: u64) -> Result<bytes::
}
let mut response = response;
let mut buf = Vec::new();
while let Some(chunk) = response.chunk().await? {
let started = std::time::Instant::now();
while let Some(chunk) = next_chunk(&mut response).await? {
if started.elapsed() > DOWNLOAD_TOTAL_TIMEOUT {
return Err(download_too_slow());
}
buf.extend_from_slice(&chunk);
if buf.len() as u64 > max_bytes {
return Err(FetchError::TooLarge);
@@ -472,10 +751,7 @@ pub async fn download_media_to_file(
out: &mut std::fs::File,
) -> Result<u64, FetchError> {
use std::io::Write;
let response = apply_media_headers(CLIENT.get(url), url)
.send()
.await?
.error_for_status()?;
let response = send_download(media_request(url)?).await?;
if let Some(len) = response.content_length()
&& len > max_bytes
{
@@ -483,7 +759,11 @@ pub async fn download_media_to_file(
}
let mut response = response;
let mut total: u64 = 0;
while let Some(chunk) = response.chunk().await? {
let started = std::time::Instant::now();
while let Some(chunk) = next_chunk(&mut response).await? {
if started.elapsed() > DOWNLOAD_TOTAL_TIMEOUT {
return Err(download_too_slow());
}
total += chunk.len() as u64;
if total > max_bytes {
return Err(FetchError::TooLarge);
@@ -497,6 +777,38 @@ pub async fn download_media_to_file(
mod tests {
use super::*;
/// Locally produced media (ugoira MP4, bsky remux MP4) lives in a temp dir
/// whose lifetime is refcounted: one fetch result can serve several sends
/// (the bot shares one in-flight fetch between concurrent duplicates), and
/// the files must outlive all of them — but no longer than the last one.
#[test]
fn a_keep_alive_clone_outlives_the_fetched() {
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("media.mp4");
std::fs::write(&file, b"mp4").unwrap();
let fetched = Fetched {
source_url: "https://x.com/u/status/1".into(),
caption: String::new(),
title: String::new(),
content: String::new(),
media: Vec::new(),
sensitive: false,
site_id: "twitter",
render_data: None,
_keep_alive: Some(std::sync::Arc::new(dir)),
};
let shared = fetched.keep_alive().expect("a temp dir to share");
drop(fetched);
assert!(file.exists(), "the file must survive the fetched post");
let second = shared.clone();
drop(shared);
assert!(file.exists(), "another holder keeps it alive");
drop(second);
assert!(!file.exists(), "the last holder releases the directory");
}
#[test]
fn cache_key_normalizes_domain_variants() {
assert_eq!(
@@ -519,6 +831,10 @@ mod tests {
cache_key("https://bsky.app/profile/handle.example/post/3lorem"),
Some("bsky:handle.example/3lorem".into())
);
assert_eq!(
cache_key("https://t.bilibili.com/1245284537985925159"),
Some("bilibili:1245284537985925159".into())
);
assert_eq!(cache_key("https://example.com/not-a-post"), None);
}
@@ -527,16 +843,21 @@ mod tests {
assert_eq!(site_id_from_key("twitter:123"), "twitter");
assert_eq!(site_id_from_key("pixiv:123"), "pixiv");
assert_eq!(site_id_from_key("bsky:handle.example/3lorem"), "bsky");
assert_eq!(site_id_from_key("bilibili:123"), "bilibili");
assert_eq!(site_id_from_key("unknown:1"), "unknown");
assert_eq!(site_id_from_key("no-colon"), "unknown");
}
#[test]
fn registry_lists_all_sites_in_dispatch_order() {
assert_eq!(site_ids(), vec!["twitter", "bsky", "misskey", "pixiv"]);
assert_eq!(
site_ids(),
vec!["twitter", "bsky", "misskey", "pixiv", "bilibili"]
);
// Enabled sites dispatch; unsupported URLs never match.
assert!(find_site("https://x.com/u/status/1").is_some());
assert!(find_site("https://misskey.io/notes/abc").is_some());
assert!(find_site("https://t.bilibili.com/1245284537985925159").is_some());
assert!(find_site("https://example.com/x").is_none());
// Cache keys are pattern-driven, independent of the enabled() gate
// (pixiv is disabled in tests without PIXIV_REFRESH_TOKEN).
@@ -564,25 +885,34 @@ mod tests {
// The format string is escaped, the field values are substituted
// verbatim (callers pass the already-escaped render data).
let out = caption_from_fields(
"see {author} at {url} — {title}",
"see {author} at {url} — {title}: {content}",
"",
"https://x.com/u/status/1",
"A &amp; B",
"https://x.com/u",
"hello <world>",
"the body",
"",
);
assert_eq!(
out,
"see A &amp; B at https://x.com/u/status/1 — hello <world>"
"see A &amp; B at https://x.com/u/status/1 — hello <world>: the body"
);
// Empty format keeps the built-in caption untouched.
assert_eq!(
caption_from_fields("", "built-in", "u", "a", "au", "t", "g"),
caption_from_fields("", "built-in", "u", "a", "au", "t", "c", "g"),
"built-in"
);
}
#[test]
fn compose_text_joins_title_and_content() {
assert_eq!(compose_text("标题", "正文"), "标题\n正文");
assert_eq!(compose_text("标题", ""), "标题");
assert_eq!(compose_text("", "正文"), "正文");
assert_eq!(compose_text("", ""), "");
}
#[test]
fn truncate_caption_keeps_short_text() {
assert_eq!(truncate_caption("short"), "short");
@@ -605,13 +935,25 @@ mod tests {
#[test]
fn truncate_caption_does_not_split_an_html_entity() {
// An entity crossing the cut must not be left half-open (&amp without ;).
let mut long = "a".repeat(MAX_CAPTION_CHARS - 4);
long.push_str("&amp;bbbb");
let out = truncate_caption(&long);
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
assert!(!out.contains("&amp"), "half entity left: {out:?}");
assert!(!out.ends_with('&'));
// The exact output is what pins the guard: a cut that keeps `&am` (no
// `;`) leaves a half-open entity that `!contains("&amp")` cannot see,
// so the old assertions stayed green with the guard deleted. Both
// directions matter — an entity the cut falls inside is dropped whole,
// one the cut falls after is kept whole.
for (long, expected) in [
(
"a".repeat(MAX_CAPTION_CHARS - 4) + "&amp;bbbb",
"a".repeat(MAX_CAPTION_CHARS - 4) + "",
),
(
"a".repeat(MAX_CAPTION_CHARS - 6) + "&amp;bbbb",
"a".repeat(MAX_CAPTION_CHARS - 6) + "&amp;…",
),
] {
let out = truncate_caption(&long);
assert_eq!(out, expected);
assert!(out.chars().count() <= MAX_CAPTION_CHARS, "{out:?}");
}
}
#[test]
@@ -622,16 +964,151 @@ mod tests {
assert!(out.chars().count() <= MAX_CAPTION_CHARS);
}
#[test]
fn blocked_addresses_are_the_hosts_own_network() {
for addr in [
"127.0.0.1",
"10.0.0.1",
"172.16.0.1",
"192.168.1.1",
"169.254.169.254", // cloud metadata
"0.0.0.0",
"255.255.255.255",
"100.64.0.1", // carrier-grade NAT
"198.18.0.1", // benchmarking
"::1",
"::",
"fc00::1",
"fe80::1",
"::ffff:127.0.0.1",
] {
assert!(blocked_ip(addr.parse().unwrap()), "{addr}");
}
for addr in [
"1.1.1.1",
"93.184.216.34",
"2606:4700::1111",
"::ffff:1.1.1.1",
] {
assert!(!blocked_ip(addr.parse().unwrap()), "{addr}");
}
}
#[test]
fn media_urls_inside_the_host_are_refused() {
for url in [
"http://127.0.0.1:9/x",
"http://169.254.169.254/latest/meta-data/",
"http://[::1]:9/x",
"https://localhost/",
"https://prompt.localhost/x",
"https://printer.local/x",
"file:///etc/passwd",
"gopher://example.com/1",
] {
let parsed = url::Url::parse(url).unwrap();
assert!(!media_url_allowed(&parsed), "{url}");
}
// Real media hosts and any public address stay fetchable.
for url in [
"https://i.pximg.net/img-original/img/1.jpg",
"https://cdn.bsky.app/img/feed_thumbnail/plain/x",
"http://example.com/a",
"https://93.184.216.34/a",
] {
let parsed = url::Url::parse(url).unwrap();
assert!(media_url_allowed(&parsed), "{url}");
}
}
/// The redirect-hop guard, against a public redirector: the initial URL is
/// checked by [`media_request`], but a redirect is the part of the path a
/// third-party response actually controls.
#[tokio::test]
async fn unsupported_url_returns_none() {
let result = fetch("https://example.com/some/article").await;
assert!(matches!(result, Ok(None)), "got {result:?}");
#[ignore = "live network: requires outbound HTTPS to httpbin.org"]
async fn live_redirect_into_the_hosts_network_is_refused() {
let url = "https://httpbin.org/redirect-to?url=http://169.254.169.254/latest/meta-data/";
match download_media(url).await.unwrap_err() {
// A policy refusal reaches the caller wrapped by reqwest.
FetchError::Http(e) => assert!(e.is_redirect(), "got {e}"),
FetchError::Blocked => {}
other => panic!("expected a refusal, got {other:?}"),
}
}
#[tokio::test]
async fn unknown_scheme_returns_none() {
let result = fetch("not a url at all").await;
assert!(matches!(result, Ok(None)), "got {result:?}");
async fn a_download_into_the_hosts_network_is_refused() {
// Refused on the URL alone: nothing has to be listening (or leaking) at
// the metadata endpoint for this to hold, and the class is permanent so
// the send path does not retry it.
for url in [
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:9/secret",
] {
let err = download_media(url).await.unwrap_err();
assert!(matches!(err, FetchError::Blocked), "{url}: got {err:?}");
}
// A malformed URL is refused the same way instead of becoming a
// retryable transport error.
assert!(matches!(
download_media("not a url").await.unwrap_err(),
FetchError::Blocked
));
}
#[tokio::test]
async fn unsupported_urls_return_none() {
// Neither a URL no site pattern matches nor a string that is no URL at
// all is an error: both answer `Ok(None)`, which is what keeps the bot
// silent on links it cannot handle (only a registered-but-disabled site
// gets a reply).
for url in ["https://example.com/some/article", "not a url at all"] {
let result = fetch(url).await;
assert!(matches!(result, Ok(None)), "{url}: got {result:?}");
}
}
#[test]
fn media_headers_are_reported_only_where_telegram_would_fail() {
// pixiv's CDN needs a Referer, which only the bot can send: an inline
// result pointing at it renders broken, so callers skip it.
assert!(needs_media_headers(
"https://i.pximg.net/img-original/img/2024/01/01/00/00/00/1_p0.jpg"
));
// The rest serve direct requests (verified per site in their modules).
for url in [
"https://pbs.twimg.com/media/1.jpg",
"https://cdn.bsky.app/img/1.jpg",
"https://media.misskeyusercontent.jp/io/1.webp",
"https://i0.hdslb.com/bfs/1.jpg",
] {
assert!(!needs_media_headers(url), "{url}");
}
}
#[tokio::test]
async fn disabled_site_is_reported_not_ignored() {
// pixiv is the only token-gated site; with PIXIV_REFRESH_TOKEN set it
// is enabled and this link would hit the network, so skip then.
if std::env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|s| !s.is_empty())
.is_some()
{
eprintln!("skipping: PIXIV_REFRESH_TOKEN is set");
return;
}
let result = fetch("https://www.pixiv.net/artworks/1").await;
assert!(
matches!(result, Err(FetchError::Disabled { site: "pixiv" })),
"got {result:?}"
);
// The cache key still resolves: the bot keys the reply and the link
// cache off it even when the site is off.
assert_eq!(
cache_key("https://www.pixiv.net/artworks/1"),
Some("pixiv:1".into())
);
}
#[tokio::test]
+35 -30
View File
@@ -76,6 +76,13 @@ impl PixivAPI {
.header("User-Agent", AUTH_USER_AGENT)
.send()
.await?;
// Check the status *before* reading the body: a 429/5xx from the
// token endpoint is worth retrying (the class comes from
// `is_retryable`), while parsing a maintenance page as JSON turned it
// into a permanent `Api`/`Json` error with no retry at all.
if !response.status().is_success() {
return Err(PixivError::Status(response.status().as_u16()));
}
let json: serde_json::Value = serde_json::from_str(&response.text().await?)?;
let access_token = json
.get("access_token")
@@ -134,8 +141,11 @@ impl PixivAPI {
let mut illustration = Illustration::from_model(&model);
if matches!(&model.r#type, TypeModel::Ugoira) {
// Real ugoira support: download the frame zip and encode an MP4.
// Without ffmpeg (or on encode failure) the post stays
// unsupported (empty media, like Python).
// Without ffmpeg the post stays unsupported (empty media, like
// Python) — but a *failed* download/encode is reported instead:
// a ugoira post has no static image to fall back to, so
// swallowing it would present a transient zip-download error as
// "this post has no media", with the retries skipped.
match self.ugoira_video(illust_id).await {
Ok(Some((mp4_path, _keep_alive))) => {
illustration.media.push(Media::Video {
@@ -143,10 +153,13 @@ impl PixivAPI {
url: mp4_path,
thumbnail_url: model.image_urls.medium.clone(),
});
illustration._keep_alive = Some(_keep_alive);
illustration._keep_alive = Some(std::sync::Arc::new(_keep_alive));
}
Ok(None) => {}
Err(e) => log::error!("ugoira encode failed for {illust_id}: {e}"),
Err(e) => {
log::error!("ugoira encode failed for {illust_id}: {e}");
return Err(FetchError::Pixiv(e));
}
}
}
Ok(illustration)
@@ -208,6 +221,7 @@ impl PixivAPI {
// memory: ugoira zips can be hundreds of MB, and the old
// download_media_limited path spiked RAM up to the size cap.
let mut zip_file = tempfile::Builder::new()
.prefix(crate::TEMP_FILE_PREFIX)
.suffix(".zip")
.tempfile()
.map_err(|e| PixivError::Api(format!("temp zip failed: {e}")))?;
@@ -220,8 +234,14 @@ impl PixivAPI {
let frame_delays = metadata.frames.iter().map(|f| f.delay).collect::<Vec<_>>();
let result =
tokio::task::spawn_blocking(move || -> Result<(String, tempfile::TempDir), String> {
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
let frames_dir = tempfile::Builder::new()
.prefix(crate::TEMP_FILE_PREFIX)
.tempdir()
.map_err(|e| e.to_string())?;
let out_dir = tempfile::Builder::new()
.prefix(crate::TEMP_FILE_PREFIX)
.tempdir()
.map_err(|e| e.to_string())?;
// Extract frames to canonical zero-padded names; pixiv ugoira
// frames are uniformly jpg or png per artwork. The zip is read
@@ -378,35 +398,20 @@ mod tests {
use super::*;
use dotenv::dotenv;
/// Skips when `PIXIV_REFRESH_TOKEN` is absent or empty (CI without the
/// secret must stay green; GitHub Actions exposes an unset secret as an
/// empty string, so `is_err()` alone is not enough).
fn require_pixiv_token() -> bool {
std::env::var("PIXIV_REFRESH_TOKEN")
.ok()
.filter(|s| !s.is_empty())
.is_some()
}
#[tokio::test]
async fn test_fetch() {
dotenv().ok();
if !require_pixiv_token() {
eprintln!("skipping: no PIXIV_REFRESH_TOKEN");
return;
}
let result = fetch(126839080).await;
assert!(result.is_ok());
println!("{:#?}", result);
}
#[tokio::test]
#[ignore = "live network: requires outbound HTTPS to oauth.secure.pixiv.net"]
async fn live_validate_with_bogus_token_fails() {
dotenv().ok();
// A bogus token must surface as Api error (invalid_grant), not panic.
// A rejected credential must surface as a permanent status, not a panic
// and not a retryable class: the exchange answers 4xx and the status is
// checked before the body is read (api.rs, `get_access_token`). This
// used to assert `Api`, which that check made unreachable — `Api` is
// only reached from a 2xx body without an `access_token`.
let client = PixivAPI::new("bogus_token_for_testing".to_string());
let result = client.get_access_token().await;
assert!(matches!(result, Err(PixivError::Api(_))), "got {result:?}");
assert!(
matches!(result, Err(PixivError::Status(code)) if (400..500).contains(&code)),
"got {result:?}"
);
}
}
+150 -20
View File
@@ -46,17 +46,25 @@ impl Site for PixivSite {
}
fn validate(&self) -> SiteFuture<'static, (), String> {
Box::pin(async {
match super::api::validate().await {
Ok(()) => Ok(()),
Err(e) => {
// Keep the old behavior: a failed login disables pixiv
// for the rest of this process.
super::api::disable();
Err(format!("{e}"))
}
}
})
Box::pin(async { startup_validation(super::api::validate().await) })
}
}
/// Turns the startup token exchange's outcome into what the bot reports, and
/// disables pixiv only for a rejected credential. A bad *moment* — a 5xx or a
/// network error while the container comes up — must not disable it: disabling
/// on any error turned every later pixiv link into "support is disabled".
/// Separate from the network call so the decision is testable.
fn startup_validation(result: Result<(), PixivError>) -> Result<(), String> {
match result {
Ok(()) => Ok(()),
Err(e) if pixiv_error_is_retryable(&e) => {
Err(format!("{e} (transient — pixiv stays enabled)"))
}
Err(e) => {
super::api::disable();
Err(format!("{e}"))
}
}
}
@@ -84,18 +92,24 @@ pub fn cache_key(url: &str) -> Option<String> {
pub fn is_retryable(err: &FetchError) -> bool {
match err {
FetchError::Http(_) | FetchError::Transient(_) => true,
FetchError::Pixiv(e) => match e {
PixivError::Http(_) => true,
PixivError::Status(code) if *code == 429 || *code >= 500 => true,
PixivError::Status(_)
| PixivError::Api(_)
| PixivError::Json(_)
| PixivError::NoAuth => false,
},
FetchError::Pixiv(e) => pixiv_error_is_retryable(e),
_ => false,
}
}
/// The pixiv-specific half of the retry policy, shared with startup
/// validation: a bad moment (429/5xx, a network error) is retryable, a
/// rejected credential is not.
fn pixiv_error_is_retryable(err: &PixivError) -> bool {
match err {
PixivError::Http(_) => true,
PixivError::Status(code) if *code == 429 || *code >= 500 => true,
PixivError::Status(_) | PixivError::Api(_) | PixivError::Json(_) | PixivError::NoAuth => {
false
}
}
}
/// pximg.net is hotlink-protected: downloads must carry the pixiv Referer.
/// The match is on the media host, not the site PATTERN — pixiv's PATTERN
/// only matches `pixiv.net/artworks/...`, never `i.pximg.net`.
@@ -107,17 +121,67 @@ pub fn media_headers(url: &str) -> Option<Vec<(&'static str, String)>> {
}
}
/// Flattens the app API's HTML description into plain text: `<br>` (and `<p>`)
/// become line breaks, other tags are dropped, entities decoded, the ends
/// trimmed. A caption shows text, not markup, so the author's `<a href>` links
/// contribute their link text only.
fn flatten_html(raw: &str) -> String {
let mut out = String::with_capacity(raw.len());
let mut chars = raw.chars().peekable();
while let Some(c) = chars.next() {
// Only `<` followed by `/` or a letter opens a tag — a bare `<` in
// prose ("2 < 3") is text.
let opens_tag = c == '<'
&& chars
.peek()
.is_some_and(|next| *next == '/' || next.is_ascii_alphabetic());
if !opens_tag {
out.push(c);
continue;
}
let mut tag = String::new();
let mut closed = false;
for c in chars.by_ref() {
if c == '>' {
closed = true;
break;
}
tag.push(c);
}
if !closed {
// Unclosed `<…`: keep it as text rather than dropping the tail.
out.push('<');
out.push_str(&tag);
break;
}
// `<br>`, `<br/>`, `<br />` with or without attributes, and both
// halves of a paragraph break the line; everything else is dropped.
let tag = tag
.trim()
.trim_start_matches('/')
.trim_end_matches('/')
.trim()
.to_ascii_lowercase();
if tag == "p" || tag.starts_with("br") {
out.push('\n');
}
}
html_escape::decode_html_entities(&out).trim().to_string()
}
#[derive(Debug)]
pub struct Illustration {
id: String,
title: String,
/// The artwork's description, HTML flattened to plain text.
content: String,
author: String,
author_id: String,
tags: Vec<String>,
pub(crate) media: Vec<Media>,
nsfw: bool,
/// Keeps a temp dir (ugoira MP4) alive until the send completes.
pub(crate) _keep_alive: Option<tempfile::TempDir>,
pub(crate) _keep_alive: Option<std::sync::Arc<tempfile::TempDir>>,
}
impl Illustration {
@@ -150,6 +214,7 @@ impl Illustration {
pub fn from_model(model: &IllustrationModel) -> Self {
let id = model.id.to_string();
let title = model.title.clone();
let content = flatten_html(&model.caption);
let author = model.user.name.clone();
let author_id = model.user.id.to_string();
let mut tags: Vec<String> = model.tags.iter().map(|tag| tag.name.clone()).collect();
@@ -193,6 +258,7 @@ impl Illustration {
Self {
id,
title,
content,
author,
author_id,
tags,
@@ -218,12 +284,14 @@ impl From<Illustration> for Fetched {
author: encode_text(&illustration.author).into_owned(),
author_url: author_url.clone(),
title: encode_text(&illustration.title).into_owned(),
content: encode_text(&illustration.content).into_owned(),
tags: encode_text(&tags).into_owned(),
});
Fetched {
source_url: url,
caption: illustration.caption(),
title: illustration.title.clone(),
content: illustration.content.clone(),
media: illustration.media,
sensitive: illustration.nsfw,
site_id: "pixiv",
@@ -262,6 +330,7 @@ mod tests {
"illust": {
"id": 123,
"title": "Art <title>",
"caption": "一行说明<br />二行 <a href=\"https://x.example/\">链接</a> &amp; 结尾",
"type": type_,
"image_urls": {
"medium": "medium.jpg",
@@ -284,6 +353,44 @@ mod tests {
Illustration::from_model(&model)
}
/// The description arrives as HTML and becomes plain-text content: breaks
/// kept, tags dropped (links keep their text), entities decoded.
#[test]
fn from_json_maps_description_to_content() {
let v = illust_json("illust", 1, None, Some("o.jpg"), vec![], 0);
let illustration = parse(v);
assert_eq!(illustration.content, "一行说明\n二行 链接 & 结尾");
let fetched: Fetched = illustration.into();
assert_eq!(fetched.title, "Art <title>");
assert_eq!(fetched.content, "一行说明\n二行 链接 & 结尾");
// The built-in caption keeps its layout: the description stays out of
// it and is available through `{content}`.
assert!(!fetched.caption.contains("一行说明"), "{}", fetched.caption);
assert_eq!(
fetched.render_fields().unwrap().3,
"一行说明\n二行 链接 &amp; 结尾"
);
assert!(
fetched
.caption_with("{title}: {content}")
.ends_with("一行说明\n二行 链接 &amp; 结尾")
);
}
#[test]
fn flatten_html_handles_common_markup() {
assert_eq!(flatten_html(""), "");
assert_eq!(flatten_html("plain"), "plain");
assert_eq!(flatten_html("a<br />b<br/>c<br>d"), "a\nb\nc\nd");
// A paragraph break is a blank line, exactly like `<br /><br />` —
// writing it as one newline would flatten the author's paragraphs.
assert_eq!(flatten_html("<p>one</p><p>two</p>"), "one\n\ntwo");
assert_eq!(flatten_html("a &amp; b &lt;c&gt;"), "a & b <c>");
// Nothing to strip: angle brackets that are not a tag survive.
assert_eq!(flatten_html("2 < 3"), "2 < 3");
}
#[test]
fn pattern_matches_all_forms() {
let cases = [
@@ -317,6 +424,29 @@ mod tests {
}
}
#[test]
fn startup_validation_keeps_the_site_enabled_on_a_bad_moment() {
use super::super::api;
// The startup decision, not the retry policy: a 5xx/429 while the
// container comes up must leave pixiv enabled and say so in the message
// the admin gets. The rejected-credential half is not exercised here —
// it calls `disable()`, a process-wide flag with no reset, so a test
// touching it would order-couple every other pixiv test (the predicate
// it keys on is covered by the table below).
for err in [PixivError::Status(429), PixivError::Status(503)] {
let enabled_before = api::enabled();
let message = startup_validation(Err(err)).unwrap_err();
assert!(message.contains("stays enabled"), "{message}");
assert_eq!(
api::enabled(),
enabled_before,
"a bad moment must not disable the site"
);
}
assert!(startup_validation(Ok(())).is_ok());
}
#[test]
fn is_retryable_classifies_transient_and_permanent() {
// Transient: network errors, explicit transient, pixiv 429/5xx.
+4
View File
@@ -6,6 +6,10 @@ use serde::Deserialize;
pub struct IllustrationModel {
pub id: u64,
pub title: String,
/// The artwork's description as the app API returns it — HTML in most
/// works (`<br />`, `<a href>`, sometimes `<p>`), empty for many.
#[serde(default)]
pub caption: String,
pub r#type: TypeModel,
pub image_urls: ImageUrlsModel,
pub user: UserInfoModel,
+8 -4
View File
@@ -132,6 +132,9 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
log::warn!("twitter auth fetch {id}: HTTP {status}");
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
// A stale/refused `auth_token` is not a bad moment: retrying it
// three times only delays the report.
401 | 403 => Err(FetchError::Blocked),
_ => Err(FetchError::Transient(format!(
"twitter auth status {status}"
))),
@@ -146,7 +149,7 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
"missing tweet fields in GraphQL response",
)))
})?;
Tweet::from_syndication_json(&syndication_shape.to_string()).map_err(FetchError::Json)
Tweet::from_syndication_value(syndication_shape).map_err(FetchError::Json)
}
/// Locates the tweet for `id` in a `TweetDetail` response and unwraps
@@ -220,7 +223,7 @@ fn normalize_tweet_result(result: &Value) -> Result<Value, FetchError> {
}
/// Maps a GraphQL `{core, legacy, ...}` tweet onto the syndication JSON
/// shape [`Tweet::from_syndication_json`] parses, so the existing text /
/// shape [`Tweet::from_syndication_value`] parses, so the existing text /
/// media handling (t.co expansion, `name=orig`, mp4 variant) is reused.
fn to_syndication_shape(tweet: &Value) -> Option<Value> {
let legacy = tweet.get("legacy")?;
@@ -305,7 +308,7 @@ mod tests {
let json = conversation(tweet_result());
let result = parse_tweet_result(&json, "2083868672721039569").unwrap();
let shape = to_syndication_shape(&result).unwrap();
let tweet = Tweet::from_syndication_json(&shape.to_string()).unwrap();
let tweet = Tweet::from_syndication_value(shape).unwrap();
let fetched: crate::site::Fetched = tweet.into();
assert!(fetched.sensitive);
@@ -327,7 +330,8 @@ mod tests {
"https://x.com/nsfw_author/status/2083868672721039569"
);
// The appended media short link (no URL-entity mapping) is stripped.
assert_eq!(fetched.title, "nsfw content");
assert_eq!(fetched.title, "");
assert_eq!(fetched.content, "nsfw content");
}
#[test]
+97 -122
View File
@@ -43,27 +43,25 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
match fetch(id).await {
Ok(tweet) => Ok(tweet.into()),
// Syndication withholds NSFW/age-restricted tweets (empty `{}`).
// Retry as the logged-in user when TWITTER_AUTH_TOKEN is set;
// otherwise degrade to an empty result (the bot replies
// "No media found").
// Retry as the logged-in user when TWITTER_AUTH_TOKEN is set; without
// the token the withholding is reported as `Sensitive`, so the bot can
// answer "age-restricted / needs TWITTER_AUTH_TOKEN" instead of the
// misleading "No media found".
Err(FetchError::Sensitive) => {
if super::auth::enabled() {
match super::auth::fetch(id).await {
Ok(tweet) => Ok(tweet.into()),
// The tweet is genuinely gone (deleted / suspended /
// tombstoned): report it instead of degrading to an
// empty result ("No media found"). Only unexpected
// fallback failures (network, parse) keep the NSFW
// placeholder.
Err(FetchError::NotFound) => Err(FetchError::NotFound),
// Deleted/suspended (tombstoned) and unexpected fallback
// failures keep their own class: the bot reports what
// actually happened rather than "No media found".
Err(e) => {
log::warn!("twitter auth fallback failed for {id}: {e}");
Ok(empty_fetched(url))
Err(e)
}
}
} else {
log::debug!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
Ok(empty_fetched(url))
Err(FetchError::Sensitive)
}
}
Err(e) => Err(e),
@@ -90,23 +88,6 @@ pub fn media_headers(_url: &str) -> Option<Vec<(&'static str, String)>> {
None
}
/// A Fetched with no media for withheld tweets: the bot replies
/// "No media found" and moves on instead of erroring.
fn empty_fetched(url: &str) -> Fetched {
Fetched {
source_url: url.to_string(),
// The raw user-supplied URL goes into an HTML caption; escape it so
// crafted links cannot break the parse (Telegram 400).
caption: encode_text(url).into_owned(),
title: String::new(),
media: vec![],
sensitive: true,
site_id: "twitter",
render_data: None,
_keep_alive: None,
}
}
/// Fetches a tweet from the syndication endpoint. Deleted/blocked tweets
/// surface as `FetchError::NotFound`; withheld content (empty tombstone,
/// age-restricted) as `FetchError::Sensitive`.
@@ -124,17 +105,23 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
if !status.is_success() {
return match status.as_u16() {
404 | 410 => Err(FetchError::NotFound),
// A refusal or an auth demand is not a bad moment: retrying it
// three times only delays an error the user has to see.
401 | 403 => Err(FetchError::Blocked),
_ => Err(FetchError::Transient(format!("twitter status {status}"))),
};
}
let text = response.text().await?;
// Classify before parsing the tweet (see [`parse_syndication_body`]).
parse_syndication_body(&text)?;
Tweet::from_syndication_json(&text).map_err(FetchError::Json)
// Classify before building the tweet (see [`parse_syndication_body`]), and
// build it from the value that classification already parsed: this used to
// scan and allocate the whole body twice.
let body = parse_syndication_body(&text)?;
Tweet::from_syndication_value(body).map_err(FetchError::Json)
}
/// Parses and classifies a syndication response body. `Ok` means the body is
/// a real tweet payload; `Err` carries the permanent error class:
/// Parses and classifies a syndication response body. `Ok` carries the parsed
/// body on for the caller to build the tweet from — the same value, so the
/// text is never parsed twice; `Err` carries the permanent error class:
/// - `NotFound`: an `errors` array (deleted/blocked) or a `TweetTombstone`
/// **with a reason** — "This Post was deleted by the Post author." /
/// "This Post is from a suspended account." (the tweet is gone).
@@ -165,7 +152,18 @@ fn parse_syndication_body(text: &str) -> Result<serde_json::Value, FetchError> {
return Err(FetchError::NotFound);
}
if body.get("id_str").is_none() {
return Err(FetchError::Sensitive);
// Syndication answers an empty `{}` for withheld (NSFW /
// age-restricted) tweets: the documented case, kept as `Sensitive`
// because it is what triggers the logged-in auth fallback.
if body.as_object().is_some_and(|object| object.is_empty()) {
return Err(FetchError::Sensitive);
}
// Any other shape is not a tweet: an interstitial, a truncated body,
// a change on their side. Reporting that as withheld content told the
// user to set TWITTER_AUTH_TOKEN for something auth cannot fix.
return Err(FetchError::Transient(
"unexpected syndication body".to_string(),
));
}
Ok(body)
}
@@ -232,8 +230,13 @@ impl Tweet {
)
}
pub fn from_syndication_json(raw_json: &str) -> Result<Self, serde_json::Error> {
let json: model::SyndicationTweet = serde_json::from_str(raw_json)?;
/// Builds a tweet from an already-parsed syndication body. Takes the value
/// rather than JSON text so a caller that had to parse it anyway (the
/// fetch path classifies the raw shape; the auth fallback builds the shape
/// itself) does not pay for a second scan — `from_value` moves the strings
/// out instead.
pub fn from_syndication_value(body: serde_json::Value) -> Result<Self, serde_json::Error> {
let json: model::SyndicationTweet = serde_json::from_value(body)?;
let id = json.id_str;
// Expand the user's t.co short links to their real destinations and
// strip the appended media short link, mirroring FxEmbed's linkFixer
@@ -352,17 +355,20 @@ impl From<Tweet> for Fetched {
fn from(tweet: Tweet) -> Self {
let url = tweet.url();
let author_url = tweet.author_url();
// A tweet has no title: its text is all content.
let render_data = Some(crate::site::RenderData {
url: url.clone(),
author: encode_text(&tweet.author).into_owned(),
author_url: author_url.clone(),
title: encode_text(&tweet.text).into_owned(),
title: String::new(),
content: encode_text(&tweet.text).into_owned(),
tags: String::new(),
});
Fetched {
source_url: url,
caption: tweet.caption(),
title: tweet.text.clone(),
title: String::new(),
content: tweet.text.clone(),
media: tweet.media,
sensitive: tweet.sensitive,
site_id: "twitter",
@@ -432,12 +438,13 @@ mod tests {
"entities": { "urls": [] },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
let tweet = Tweet::from_syndication_value(raw).unwrap();
// The appended media short link is stripped, then entities decoded.
assert_eq!(tweet.text, ">^ω^< & more 'quoted'");
assert_eq!(tweet.author, "O'Brien");
let fetched: Fetched = tweet.into();
assert_eq!(fetched.title, ">^ω^< & more 'quoted'");
assert_eq!(fetched.title, "");
assert_eq!(fetched.content, ">^ω^< & more 'quoted'");
// The caption escapes the raw text exactly once (encode_text covers
// & < >; apostrophes stay literal — they are harmless in text).
assert!(
@@ -490,13 +497,14 @@ mod tests {
}
}
]));
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
let tweet = Tweet::from_syndication_value(raw).unwrap();
let fetched: Fetched = tweet.into();
assert_eq!(
fetched.source_url,
"https://x.com/author_handle/status/861627479294746624"
);
assert_eq!(fetched.title, "a & b <c>");
assert_eq!(fetched.title, "");
assert_eq!(fetched.content, "a & b <c>");
assert!(fetched.sensitive);
assert_eq!(fetched.media.len(), 2);
match &fetched.media[0] {
@@ -524,14 +532,6 @@ mod tests {
);
}
#[test]
fn syndication_text_only_has_no_media() {
let raw = fixture(serde_json::json!([]));
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
let fetched: Fetched = tweet.into();
assert!(fetched.media.is_empty());
}
#[test]
fn syndication_gif_maps_to_animated() {
let raw = fixture(serde_json::json!([
@@ -543,61 +543,37 @@ mod tests {
}
}
]));
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
let tweet = Tweet::from_syndication_value(raw).unwrap();
assert!(matches!(&tweet.media[0], Media::Animated { .. }));
}
#[test]
fn syndication_text_strips_trailing_media_short_link() {
// Real syndication shape: the appended media short link sits after the
// visible text; the unmapped t.co link is stripped by content.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": "hello world https://t.co/abc123",
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, "hello world");
assert!(!tweet.caption().contains("t.co"));
}
#[test]
fn syndication_text_strips_trailing_link_regardless_of_index_units() {
// Real tweet 2084567054481571919: the visible text is 30 code points
// but 41 UTF-16 units, and the two endpoints historically reported
// display_text_range in different units (UTF-16 on syndication, code
// points on GraphQL). The FxEmbed-style content-based strip ignores
// the range entirely, so the appended media link is removed for any
// response shape.
let text = "妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero https://t.co/XnIi83EkEB";
let visible = "妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero";
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "2084567054481571919",
"text": text,
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, visible, "left a partial link");
assert!(!tweet.caption().contains("t.co"));
}
#[test]
fn syndication_text_strips_trailing_short_link_without_entities() {
// No URL entities at all: the leftover t.co link is stripped by the
// content regex.
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": "hello https://t.co/abc123",
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, "hello");
// visible text and there are no URL entities, so the unmapped t.co link
// is stripped by content alone. The second row is real tweet
// 2084567054481571919 (30 code points but 41 UTF-16 units, and the two
// endpoints historically reported `display_text_range` in different
// units): a content-based strip cannot leave a partial link behind for
// either unit system.
for (text, visible) in [
("hello world https://t.co/abc123", "hello world"),
(
"妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero https://t.co/XnIi83EkEB",
"妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero",
),
] {
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": text,
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_value(raw).unwrap();
assert_eq!(tweet.text, visible, "left a partial link in {text:?}");
assert!(!tweet.caption().contains("t.co"), "{text:?}");
}
}
#[test]
@@ -619,7 +595,7 @@ mod tests {
},
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
let tweet = Tweet::from_syndication_value(raw).unwrap();
assert_eq!(
tweet.text,
"Test Tweet with @mentionThis $twtr http://bit.ly/2pUk4be #hashtag"
@@ -638,7 +614,7 @@ mod tests {
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
let tweet = Tweet::from_syndication_value(raw).unwrap();
assert_eq!(tweet.text, "check #tag");
}
@@ -661,28 +637,11 @@ mod tests {
},
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
let tweet = Tweet::from_syndication_value(raw).unwrap();
assert_eq!(tweet.text, "see for context");
assert!(!tweet.caption().contains("t.co"));
}
#[test]
fn syndication_text_keeps_multibyte_text() {
// Text-only tweet: no short links, the multibyte text is untouched.
let text = "コミティア落ちたので、明日は行きません。🙏ごめんなさい";
let units: Vec<u16> = text.encode_utf16().collect();
assert_eq!(units.len(), 28);
let raw = serde_json::json!({
"__typename": "Tweet",
"id_str": "1",
"text": text,
"user": { "name": "N", "screen_name": "h" },
"mediaDetails": []
});
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
assert_eq!(tweet.text, text, "full text kept intact");
}
#[test]
fn original_twimg_url_rewrites_photo_urls() {
assert_eq!(
@@ -706,9 +665,14 @@ mod tests {
#[test]
fn syndication_token_matches_js_formula() {
// JS: ((861627479294746624 / 1e15) * PI).toString(36) == "236.vrsocvda"
let token = syndication_token(861627479294746624);
assert!(token.starts_with("236.v"), "got {token}");
// JS: ((861627479294746624 / 1e15) * PI).toString(36) == "236.vrsocvda".
// This loop truncates ten base-36 fraction digits instead of rendering
// the shortest round-tripping one, so it agrees with JS on the stem and
// diverges in the tail (`…d9ui` vs `…da`). Pinned exactly, because the
// token is a fixed function of the id: a stub or a wrong constant must
// not pass. The endpoint currently serves public tweets regardless of
// the token, which is why the tail is left as is.
assert_eq!(syndication_token(861627479294746624), "236.vrsocvd9ui");
}
#[test]
@@ -776,6 +740,17 @@ mod tests {
));
}
#[test]
fn syndication_unexpected_shape_is_transient_not_withheld() {
// A 200 that is not a tweet at all (an interstitial, a truncated
// body) must not be reported as withheld content: that message tells
// the user to set TWITTER_AUTH_TOKEN, which cannot fix it.
match parse_syndication_body("{\"foo\":1}") {
Err(FetchError::Transient(_)) => {}
other => panic!("expected Transient, got {other:?}"),
}
}
#[test]
fn syndication_tweet_body_passes() {
let raw = fixture(serde_json::json!([]));
+4 -4
View File
@@ -1,11 +1,11 @@
[package]
name = "xmedia-bot"
version = "1.5.0"
version = "1.9.0"
edition = "2024"
[dependencies]
teloxide = { version = "0.17", default-features = false, features = ["webhooks-axum", "macros", "rustls", "ctrlc_handler"] }
tokio = { version = "1.40", features = ["rt-multi-thread", "macros", "time"] }
tokio = { version = "1.40", features = ["rt-multi-thread", "macros", "time", "sync"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
log = "0.4"
@@ -13,8 +13,8 @@ pretty_env_logger = "0.5"
dotenv = "0.15"
url = "2.5.2"
html-escape = "0.2"
rusqlite = { version = "0.32", features = ["bundled"] }
rand = "0.8"
rusqlite = { version = "0.40", features = ["bundled"] }
rand = "0.10"
tempfile = "3"
parking_lot = "0.12"
bytes = "1"
+8 -1
View File
@@ -1,5 +1,6 @@
//! Central env handling. The only other places that read env are
//! `Bot::from_env` (TELOXIDE_TOKEN) and x-media (PIXIV_REFRESH_TOKEN).
//! `Bot::from_env` (TELOXIDE_TOKEN) and x-media (PIXIV_REFRESH_TOKEN,
//! TWITTER_AUTH_TOKEN, BILIBILI_COOKIE).
use std::env;
use std::net::IpAddr;
@@ -12,6 +13,10 @@ pub struct Config {
pub edit_message_ttl: Duration,
/// LINK_CACHE_TTL_SECONDS, default 604800 (7 days).
pub link_cache_ttl: Duration,
/// CAPTION_QUOTE_TEXT_CHARS, default 200: a post whose text (title plus
/// content) is at least this many characters gets that text wrapped in an
/// expandable blockquote inside its caption. `0` disables the wrap.
pub caption_quote_text_chars: usize,
// Webhook settings (moved out of main; names/defaults unchanged).
pub webhook_enabled: bool,
pub webhook_url: Option<url::Url>,
@@ -57,6 +62,7 @@ impl Config {
Duration::from_secs(parse_u64("EDIT_MESSAGE_TTL_SECONDS", 24 * 3600));
let link_cache_ttl =
Duration::from_secs(parse_u64("LINK_CACHE_TTL_SECONDS", 7 * 24 * 3600));
let caption_quote_text_chars = parse_u64("CAPTION_QUOTE_TEXT_CHARS", 200) as usize;
let webhook_enabled = env::var("WEBHOOK")
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
@@ -92,6 +98,7 @@ impl Config {
admin_ids,
edit_message_ttl,
link_cache_ttl,
caption_quote_text_chars,
webhook_enabled,
webhook_url,
webhook_listen,
+188
View File
@@ -0,0 +1,188 @@
//! Runtime context: the collaborators a handler needs, injected as one struct
//! so tests can substitute a scripted sender and tempdir-backed stores.
//!
//! The production context is assembled from the process-wide statics
//! ([`AppContext::from_statics`]); the spawned worker closures hold
//! [`CONTEXT`], which is `'static` for that reason.
use crate::config::Config;
use crate::handlers::{CHAT_STORE, CONFIG, LINK_CACHE, TASK_QUEUE};
use crate::link_cache::LinkCache;
use crate::media_sender::MediaSender;
use crate::queue::PersistentTaskQueue;
use crate::send::BOT;
use crate::state::ChatStore;
use std::sync::LazyLock;
pub struct AppContext<'a> {
pub sender: &'a dyn MediaSender,
pub chat_store: &'a ChatStore,
pub task_queue: &'a PersistentTaskQueue,
pub link_cache: &'a LinkCache,
pub config: &'a Config,
}
impl<'a> AppContext<'a> {
/// The stores are the process-wide statics; `sender` is whatever the caller
/// was handed (the dispatcher's `Bot` clone for update handlers, the shared
/// queue `Bot` for the worker loops). Update handlers build their own
/// context from the `Bot` they received so the same code path works with an
/// injected mock in tests.
pub fn from_statics(sender: &'a dyn MediaSender) -> AppContext<'a> {
AppContext {
sender,
chat_store: &CHAT_STORE,
task_queue: &TASK_QUEUE,
link_cache: &LINK_CACHE,
config: &CONFIG,
}
}
}
/// The URL/queue workers' context: `'static` because `tokio::spawn`ed closures
/// and the queue's handler type require it.
pub static CONTEXT: LazyLock<AppContext<'static>> =
LazyLock::new(|| AppContext::from_statics(&*BOT));
/// Test support: a tempdir-backed set of stores plus the context borrowing
/// them, so a handler test needs one line of setup.
#[cfg(test)]
pub(crate) mod test_support {
use super::*;
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost};
use crate::state::EditMessage;
use std::sync::Arc;
use teloxide::{ApiError, RequestError};
/// The edit-before-forward prompt's message id, and the message the prompt
/// refers to (the one whose caption a reply swaps).
pub(crate) const PROMPT_ID: i64 = 7;
pub(crate) const FORWARDED_ID: i64 = 9;
/// A Telegram API error, for the tests that script a failure.
pub(crate) fn api_error(message: &str) -> RequestError {
RequestError::Api(ApiError::Unknown(message.to_string()))
}
/// The cached post every test that touches the link cache starts from: one
/// photo with a Telegram file id at the canonical URL (key `twitter:1`).
/// Tests that need another field mutate the returned value.
pub(crate) fn cached_photo() -> CachedPost {
CachedPost {
url: "https://x.com/u/status/1".into(),
caption: "cap".into(),
title: "t".into(),
content: "c".into(),
author: "a".into(),
author_url: "au".into(),
tags: String::new(),
sensitive: false,
media: vec![CachedMedia {
kind: CachedMediaKind::Photo,
file_id: "AgAC-file-id".into(),
url: "https://pbs.twimg.com/media/photo.jpg".into(),
}],
}
}
/// Seeds the live prompt a post-send leaves behind in chat 1: the chat's
/// template, a bound forward channel (the prompt's "forward" button
/// branches on it) and the record for [`PROMPT_ID`] pointing at
/// [`FORWARDED_ID`]. `template` is the record's template — what a reply
/// swaps the caption through, `""` for none — and `created_at` backdates
/// the record for the expiry cases.
pub(crate) async fn seed_prompt(ctx: &AppContext<'_>, template: &str, created_at: i64) {
ctx.chat_store
.update(1, |data| {
data.forward_channel_id = Some(2);
data.template
.insert("tpl".to_string(), "<b>[]</b>".to_string());
data.edit_message.insert(
PROMPT_ID,
EditMessage {
url: "https://x.com/u/status/1".into(),
chat_id: 1,
forward_message_ids: vec![FORWARDED_ID],
template: template.to_string(),
created_at,
},
);
})
.await;
}
pub(crate) struct TestStores {
_dir: tempfile::TempDir,
pool: Arc<crate::db::DbPool>,
chat_store: ChatStore,
task_queue: PersistentTaskQueue,
link_cache: LinkCache,
config: Config,
}
impl TestStores {
pub(crate) fn new() -> Self {
let dir = tempfile::tempdir().unwrap();
let pool = crate::db::open_store(dir.path().join("ctx.db").to_str().unwrap()).unwrap();
TestStores {
_dir: dir,
chat_store: ChatStore::new(Arc::clone(&pool)),
task_queue: PersistentTaskQueue::new(Arc::clone(&pool)),
link_cache: LinkCache::new(Arc::clone(&pool)),
config: Config::load(),
pool,
}
}
pub(crate) fn ctx<'a>(&'a self, sender: &'a dyn MediaSender) -> AppContext<'a> {
AppContext {
sender,
chat_store: &self.chat_store,
task_queue: &self.task_queue,
link_cache: &self.link_cache,
config: &self.config,
}
}
pub(crate) fn chat_store(&self) -> &ChatStore {
&self.chat_store
}
/// The parsed config, mutable so a test can pin a knob (e.g. the
/// caption-quote threshold) instead of depending on the environment.
pub(crate) fn config_mut(&mut self) -> &mut Config {
&mut self.config
}
pub(crate) fn link_cache(&self) -> &LinkCache {
&self.link_cache
}
pub(crate) fn task_queue(&self) -> &PersistentTaskQueue {
&self.task_queue
}
/// Rows persisted in the task queue: what "queued for retry" looks like
/// from the outside.
pub(crate) async fn queued_tasks(&self) -> i64 {
let pool = Arc::clone(&self.pool);
pool.with_conn(|conn| {
conn.query_row("SELECT COUNT(*) FROM tasks", [], |row| row.get(0))
})
.await
.unwrap()
}
/// The single queued task payload, for asserting what was rescheduled.
pub(crate) async fn queued_payload(&self) -> serde_json::Value {
let pool = Arc::clone(&self.pool);
let payload: String = pool
.with_conn(|conn| {
conn.query_row("SELECT payload FROM tasks LIMIT 1", [], |row| row.get(0))
})
.await
.unwrap();
serde_json::from_str(&payload).unwrap()
}
}
}
+180 -5
View File
@@ -124,9 +124,44 @@ pub fn open_store(path: &str) -> rusqlite::Result<Arc<DbPool>> {
}
let conn = open_db(path)?;
schema_init(&conn)?;
migrate(&conn)?;
Ok(Arc::new(DbPool::new(path)))
}
/// Schema migrations, applied in order and tracked by `PRAGMA user_version`
/// (the index in this array + 1 is the version a statement brings the
/// database to). Append only — never edit or reorder an entry, or databases
/// already past it would skip or repeat work.
const MIGRATIONS: &[&str] = &[
// 1: lease fencing. A worker's write-backs (`delete`/`reschedule`/the
// lease heartbeat) are guarded by the token it was leased with, so a
// lease that expired and was re-leased by another worker can no longer be
// written by its former holder — which used to duplicate a send or drop
// the new holder's retry state, silently.
"ALTER TABLE tasks ADD COLUMN lease_token TEXT",
// 2: the 300 s sweep prunes the link cache by `created_at`
// (`DELETE FROM link_cache WHERE created_at < ?`). Without an index that
// is a full scan of every post sent inside the TTL window — up to a week
// of them — on every sweep; the `url` primary key cannot serve it.
"CREATE INDEX IF NOT EXISTS idx_link_cache_created_at ON link_cache(created_at)",
];
/// Brings an existing database up to [`MIGRATIONS`]. Idempotent: a database
/// already at the latest version does no work.
fn migrate(conn: &Connection) -> rusqlite::Result<()> {
let version: i64 = conn.query_row("PRAGMA user_version", [], |row| row.get(0))?;
for (index, statement) in MIGRATIONS.iter().enumerate() {
let target = index as i64 + 1;
if version >= target {
continue;
}
conn.execute_batch(statement)?;
// `PRAGMA` does not take bind parameters; the value is our own index.
conn.execute_batch(&format!("PRAGMA user_version = {target}"))?;
}
Ok(())
}
fn rusqlite_error(e: std::io::Error) -> rusqlite::Error {
rusqlite::Error::ToSqlConversionFailure(Box::new(e))
}
@@ -135,11 +170,11 @@ fn rusqlite_error(e: std::io::Error) -> rusqlite::Error {
/// The three stores used to own their own schema; keeping it in one place
/// means one initialization for the whole database file.
///
/// ⚠️ Schema-change reminder (deferred, see `docs/architecture-refactor.md`
/// §5): this is a plain `CREATE TABLE IF NOT EXISTS` with no versioning.
/// Before any column/table change that must migrate existing databases, land
/// the `PRAGMA user_version` migration chain first (`MIGRATIONS: &[&str]` +
/// `migrate(conn)`), then restructure this function.
/// This is the **baseline** schema (version 0): a fresh database is created
/// exactly like this, and anything that must *change* an existing one is
/// appended to [`MIGRATIONS`] instead of being edited in here — otherwise a
/// database created before the change would never gain the new column and a
/// freshly created one would try to apply the migration a second time.
pub fn schema_init(conn: &Connection) -> rusqlite::Result<()> {
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
@@ -166,3 +201,143 @@ pub fn now_f64() -> f64 {
pub fn unix_now() -> i64 {
now_f64() as i64
}
#[cfg(test)]
mod tests {
use super::*;
use rusqlite::Connection;
/// The schema as it shipped *before* the first migration: what an existing
/// deployment has on disk when it starts on the new binary. Written out
/// literally rather than derived from `schema_init`, so an edit to the
/// baseline shows up here instead of being followed silently.
const V0_SCHEMA: &str = "CREATE TABLE tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
CREATE INDEX idx_tasks_pending ON tasks(status, run_after); \
CREATE TABLE chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL); \
CREATE TABLE link_cache (url TEXT PRIMARY KEY, payload TEXT NOT NULL, \
created_at REAL NOT NULL);";
/// The migrations that have already shipped, verbatim. Appending is the only
/// allowed change: editing one that a database has already applied leaves
/// deployments on different schemas with nothing to notice it — the version
/// counter says "done" and skips the new text.
const SHIPPED_MIGRATIONS: &[&str] = &["ALTER TABLE tasks ADD COLUMN lease_token TEXT"];
fn columns(conn: &Connection, table: &str) -> Vec<String> {
let mut stmt = conn
.prepare(&format!("PRAGMA table_info({table})"))
.unwrap();
let mut names: Vec<String> = stmt
.query_map([], |row| row.get::<_, String>(1))
.unwrap()
.map(Result::unwrap)
.collect();
names.sort();
names
}
fn user_version(conn: &Connection) -> i64 {
conn.query_row("PRAGMA user_version", [], |row| row.get(0))
.unwrap()
}
#[tokio::test]
async fn a_pre_migration_database_upgrades_and_keeps_its_rows() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("old.db");
{
let conn = Connection::open(&path).unwrap();
conn.execute_batch(V0_SCHEMA).unwrap();
conn.execute(
"INSERT INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
VALUES ('task_old', '{\"chat_id\":1}', 0, 0, 'pending', 0, 0)",
[],
)
.unwrap();
assert_eq!(user_version(&conn), 0, "the fixture starts un-migrated");
assert!(
!columns(&conn, "tasks").contains(&"lease_token".to_string()),
"the fixture is the pre-migration shape"
);
}
let pool = open_store(path.to_str().unwrap()).unwrap();
pool.with_conn(|conn| {
assert_eq!(user_version(conn), MIGRATIONS.len() as i64);
let mut expected = vec![
"id",
"payload",
"run_after",
"attempts",
"status",
"locked_until",
"created_at",
"lease_token",
];
expected.sort();
assert_eq!(
columns(conn, "tasks"),
expected,
"an upgrade must add the migration's column and nothing else"
);
let payload: String = conn
.query_row(
"SELECT payload FROM tasks WHERE id = 'task_old'",
[],
|row| row.get(0),
)
.unwrap();
assert_eq!(payload, "{\"chat_id\":1}", "rows survive the upgrade");
// The link-cache prune's index arrives with the migrations (the
// baseline schema has none): without it every sweep scans the
// whole table.
let index: i64 = conn
.query_row(
"SELECT COUNT(*) FROM sqlite_master \
WHERE type = 'index' AND name = 'idx_link_cache_created_at'",
[],
|row| row.get(0),
)
.unwrap();
assert_eq!(index, 1, "the migration's index must exist");
Ok(())
})
.await
.unwrap();
}
#[test]
fn shipped_migrations_are_frozen() {
assert!(
MIGRATIONS.len() >= SHIPPED_MIGRATIONS.len(),
"migrations were removed or reordered, not appended"
);
for (index, (shipped, current)) in SHIPPED_MIGRATIONS.iter().zip(MIGRATIONS).enumerate() {
assert_eq!(
shipped,
current,
"migration {} already shipped: append a new one instead of editing it",
index + 1
);
}
}
#[tokio::test]
async fn a_fresh_database_lands_at_the_latest_version() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("fresh.db");
let pool = open_store(path.to_str().unwrap()).unwrap();
// Every migration is applied on creation, so a deployment that only ever
// saw fresh databases is on the same schema as an upgraded one.
pool.with_conn(|conn| {
assert_eq!(user_version(conn), MIGRATIONS.len() as i64);
Ok(())
})
.await
.unwrap();
// Opening the same file again is a no-op (the version gate skips it).
open_store(path.to_str().unwrap()).unwrap();
}
}
+383 -72
View File
@@ -1,56 +1,101 @@
//! Callback query handling: the edit-before-forward prompt's "forward" and
//! "template|<name>" buttons.
//! Callback query handling: the edit-before-forward prompt's `"forward"` and
//! `"template|<name>"` buttons.
//!
//! [`callback_query_handler`] is the dptree entry; it only pulls the plain
//! values out of the teloxide update and hands them to [`handle_callback`],
//! which holds the button logic and is driven directly by tests.
use super::urls::enqueue_retry;
use super::{CHAT_STORE, CONFIG, TASK_QUEUE};
use crate::ctx::AppContext;
use crate::db::unix_now;
use crate::send::{self, Task};
use teloxide::RequestError;
use teloxide::prelude::*;
use teloxide::types::{CallbackQuery, ChatId, MessageId, ParseMode};
use teloxide::types::{CallbackQuery, CallbackQueryId, MessageId};
/// The `"forward"` button's data.
const FORWARD: &str = "forward";
/// The `"skip"` button's data: drop the prompt without forwarding.
const SKIP: &str = "skip";
/// Prefix of a template button's data: `"template|<name>"`.
const TEMPLATE_PREFIX: &str = "template|";
pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<(), RequestError> {
let callback_query_id = query.id;
let data = query.data.clone();
let Some(message) = &query.message else {
return respond(());
};
let chat_id = message.chat().id.0;
let prompt_message_id = message.id().0 as i64;
let ttl_secs = CONFIG.edit_message_ttl.as_secs() as i64;
let chat_data = CHAT_STORE.get(chat_id).await;
let Some(data) = query.data.clone() else {
return respond(());
};
let ctx = AppContext::from_statics(&bot);
handle_callback(
&ctx,
query.id.clone(),
message.chat().id.0,
message.id().0 as i64,
&data,
)
.await;
respond(())
}
/// Handles one button press on the edit-before-forward prompt.
async fn handle_callback(
ctx: &AppContext<'_>,
callback_query_id: CallbackQueryId,
chat_id: i64,
prompt_message_id: i64,
data: &str,
) {
let ttl_secs = ctx.config.edit_message_ttl.as_secs() as i64;
let chat_data = ctx.chat_store.get(chat_id).await;
let edit = chat_data.edit_message.get(&prompt_message_id).cloned();
let Some(edit) = edit else {
log::debug!(
"callback from {}: no edit record for prompt {prompt_message_id}",
chat_id
);
bot.answer_callback_query(callback_query_id)
.text("Expired")
.await?;
return respond(());
log::debug!("callback from {chat_id}: no edit record for prompt {prompt_message_id}");
let _ = ctx
.sender
.answer_callback_query(callback_query_id, Some("Expired".to_string()))
.await;
return;
};
// Lazy expiry: a stale record (past the TTL, not yet swept) is dropped.
if edit.created_at + ttl_secs <= unix_now() {
CHAT_STORE
ctx.chat_store
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
bot.answer_callback_query(callback_query_id)
.text("Expired")
.await?;
return respond(());
let _ = ctx
.sender
.answer_callback_query(callback_query_id, Some("Expired".to_string()))
.await;
return;
}
let Some(data) = data else {
return respond(());
};
log::info!(
"callback from {} on prompt {prompt_message_id}: {data}",
chat_id
);
if data == "forward" {
log::info!("callback from {chat_id} on prompt {prompt_message_id}: {data}");
if data == SKIP {
// Skip works with or without a forward channel: it is the explicit
// "do not forward this" answer, and it drops the record so the forward
// can never happen later.
log::info!("edit-before-forward prompt {prompt_message_id} skipped");
ctx.chat_store
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
let _ = ctx
.sender
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
.await;
let _ = ctx
.sender
.answer_callback_query(
callback_query_id,
Some("Skipped — nothing was forwarded.".to_string()),
)
.await;
return;
}
if data == FORWARD {
match chat_data.forward_channel_id {
Some(channel_id) => {
let forward_task = Task::ForwardMessages {
@@ -60,71 +105,337 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
notify_chat_id: Some(chat_id),
notify_message_id: Some(prompt_message_id),
};
match send::forward_messages(&bot, &forward_task).await {
let (answer, settled) = match send::forward_messages(ctx, &forward_task).await {
Ok(()) => {
log::info!(
"forwarded {} message(s) to channel {channel_id}",
edit.forward_message_ids.len()
);
bot.answer_callback_query(callback_query_id)
.text("✅ Forwarded")
.await?;
let _ = bot
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
.await;
CHAT_STORE
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
("✅ Forwarded".to_string(), true)
}
Err(send::SendError::Retryable {
delay_seconds,
task,
}) => {
log::info!("forward queued for retry in {delay_seconds:.1}s");
enqueue_retry(&TASK_QUEUE, *task, delay_seconds).await;
bot.answer_callback_query(callback_query_id)
.text("Forward queued for retry.")
.await?;
// The queued row owns the forward from here (it carries
// the message ids itself), so the prompt is settled
// either way: leaving it live let a second Confirm copy
// the same messages to the channel twice, and let Skip
// answer "nothing was forwarded" while the row still
// delivered it.
let queued =
send::enqueue_retry(ctx.task_queue, &task, delay_seconds).await;
if queued {
log::info!("forward queued for retry in {delay_seconds:.1}s");
("Forward queued for retry.".to_string(), true)
} else {
log::error!("forward retry could not be queued");
(
"Forward failed and the retry could not be queued.".to_string(),
true,
)
}
}
Err(send::SendError::Permanent { message, .. }) => {
log::error!("forward failed permanently: {message}");
bot.answer_callback_query(callback_query_id)
.text(format!("Forward failed: {message}"))
.await?;
(format!("Forward failed: {message}"), false)
}
};
if settled {
// The prompt is done: drop it and its record.
let _ = ctx
.sender
.delete_message(ChatId(chat_id), MessageId(prompt_message_id as i32))
.await;
ctx.chat_store
.update(chat_id, |data| {
data.edit_message.remove(&prompt_message_id);
})
.await;
}
let _ = ctx
.sender
.answer_callback_query(callback_query_id, Some(answer))
.await;
}
None => {
log::debug!("forward callback without a forward channel set");
bot.answer_callback_query(callback_query_id)
.text("No forward channel set.")
.await?;
let _ = ctx
.sender
.answer_callback_query(
callback_query_id,
Some("No forward channel set.".to_string()),
)
.await;
}
}
return respond(());
return;
}
if let Some(name) = data.strip_prefix("template|") {
if let Some(name) = data.strip_prefix(TEMPLATE_PREFIX) {
let mut answer = None;
if let Some(template_html) = chat_data.template.get(name).cloned()
&& let Some(first_forward_id) = edit.forward_message_ids.first().copied()
{
// Raw template including the [] placeholder (Python parity).
let _ = bot
.edit_message_caption(ChatId(chat_id), MessageId(first_forward_id as i32))
.caption(template_html)
.parse_mode(ParseMode::Html)
.await;
CHAT_STORE
.update(chat_id, |data| {
if let Some(entry) = data.edit_message.get_mut(&prompt_message_id) {
entry.template = name.to_string();
}
})
.await;
log::info!("template '{name}' applied to prompt {prompt_message_id}");
match super::apply_caption_edit(
ctx.sender,
ChatId(chat_id),
MessageId(first_forward_id as i32),
template_html,
)
.await
{
super::EditOutcome::Applied => {
ctx.chat_store
.update(chat_id, |data| {
if let Some(entry) = data.edit_message.get_mut(&prompt_message_id) {
entry.template = name.to_string();
}
})
.await;
log::info!("template '{name}' applied to prompt {prompt_message_id}");
}
// Nothing was applied, so nothing is recorded either: the
// prompt keeps rendering through whatever it used before, and
// the toast says why (a silently "successful" press left the
// caption unchanged).
super::EditOutcome::Failed(reason) => {
log::error!("template '{name}' could not be applied: {reason}");
answer = Some(format!("Could not apply the template: {reason}"));
}
}
}
bot.answer_callback_query(callback_query_id).await?;
let _ = ctx
.sender
.answer_callback_query(callback_query_id, answer)
.await;
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ctx::test_support::{FORWARDED_ID, PROMPT_ID, TestStores, api_error, seed_prompt};
use crate::media_sender::test_support::{MockSender, Outcome};
/// The Telegram wording the mocks answer with: a chat the bot cannot reach.
const API_ERROR: &str = "Bad Request: chat not found";
fn callback_id() -> CallbackQueryId {
CallbackQueryId("cb-1".to_string())
}
#[tokio::test]
async fn template_button_swaps_the_caption_and_records_the_choice() {
let sender = MockSender::scripted(vec![Outcome::EditOk], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "template|tpl").await;
assert_eq!(
sender.calls(),
vec!["edit_message_caption", "answer_callback_query"]
);
// The raw template, including the [] the user edits into.
assert_eq!(sender.captions(), vec!["<b>[]</b>"]);
assert_eq!(sender.answers(), vec![None]);
let data = ctx.chat_store.get(1).await;
assert_eq!(data.edit_message[&PROMPT_ID].template, "tpl");
}
#[tokio::test]
async fn a_failed_template_swap_is_reported_in_the_toast() {
let sender = MockSender::scripted(vec![Outcome::EditErr], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "template|tpl").await;
// The caption never changed, so the toast says so and the record does
// not claim the template was applied.
let toast = sender.answers().last().cloned().flatten();
assert!(
toast
.as_deref()
.is_some_and(|t| t.contains("Could not apply the template")),
"{toast:?}"
);
assert_eq!(
ctx.chat_store.get(1).await.edit_message[&PROMPT_ID].template,
""
);
}
#[tokio::test]
async fn forward_button_copies_then_clears_the_prompt() {
let sender = MockSender::scripted(vec![Outcome::CopyOk], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(
sender.calls(),
vec!["copy_messages", "delete_message", "answer_callback_query"]
);
assert_eq!(sender.answers(), vec![Some("✅ Forwarded".to_string())]);
assert!(
ctx.chat_store.get(1).await.edit_message.is_empty(),
"a settled prompt must drop its record"
);
}
#[tokio::test]
async fn skip_drops_the_prompt_without_forwarding() {
// "skip" needs no forward channel and no scripted outcomes: it deletes
// the prompt and drops the record, so no forward can ever happen.
let sender = MockSender::scripted(vec![], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "skip").await;
assert_eq!(
sender.calls(),
vec!["delete_message", "answer_callback_query"]
);
assert_eq!(
sender.answers(),
vec![Some("Skipped — nothing was forwarded.".to_string())]
);
assert!(
ctx.chat_store.get(1).await.edit_message.is_empty(),
"a skipped prompt must drop its record"
);
}
/// The whole callback path against a stand-in API through a real `Bot`:
/// copy, delete, toast, carrying the ids the prompt held. The scripted
/// mock records that a call happened; this records what the API received.
#[tokio::test]
async fn the_forward_button_talks_to_the_api_through_a_real_bot() {
use crate::media_sender::test_support::fake_api::FakeApi;
use teloxide::Bot;
let api = FakeApi::start().await;
let bot = Bot::new("42:TEST").set_api_url(api.url());
let stores = TestStores::new();
let ctx = stores.ctx(&bot);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(
api.methods(),
vec!["CopyMessages", "DeleteMessage", "AnswerCallbackQuery"]
);
let copy = api.body("CopyMessages");
assert_eq!(copy["chat_id"], 2, "the prompt's channel");
assert_eq!(copy["from_chat_id"], 1);
assert_eq!(copy["message_ids"], serde_json::json!([FORWARDED_ID]));
assert_eq!(api.body("AnswerCallbackQuery")["text"], "✅ Forwarded");
}
#[tokio::test]
async fn forward_without_a_channel_is_reported() {
let sender = MockSender::scripted(vec![], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
ctx.chat_store
.update(1, |data| data.forward_channel_id = None)
.await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(sender.calls(), vec!["answer_callback_query"]);
assert_eq!(
sender.answers(),
vec![Some("No forward channel set.".to_string())]
);
}
#[tokio::test]
async fn retryable_forward_is_queued_and_settles_the_prompt() {
use teloxide::types::Seconds;
let sender = MockSender::scripted(vec![Outcome::CopyErr], || {
RequestError::RetryAfter(Seconds::from_seconds(7))
});
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
// The queued row carries the message ids itself, so it owns the
// forward from here and the prompt is closed with it. Keeping it live
// (the old behaviour) let a second Confirm copy the same messages to
// the channel twice, and let Skip answer "nothing was forwarded" while
// the row still delivered it.
assert_eq!(
sender.calls(),
vec!["copy_messages", "delete_message", "answer_callback_query"]
);
assert_eq!(
sender.answers(),
vec![Some("Forward queued for retry.".to_string())]
);
assert_eq!(stores.queued_tasks().await, 1);
assert!(
!ctx.chat_store
.get(1)
.await
.edit_message
.contains_key(&PROMPT_ID),
"the record must be dropped so the prompt cannot be used again"
);
// A second tap finds no record: it cannot enqueue a duplicate copy.
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(
sender.calls(),
vec![
"copy_messages",
"delete_message",
"answer_callback_query",
"answer_callback_query"
]
);
assert_eq!(
sender.answers().last().map(|a| a.as_deref()),
Some(Some("Expired"))
);
assert_eq!(stores.queued_tasks().await, 1, "no second forward row");
}
#[tokio::test]
async fn unknown_and_expired_prompts_answer_expired() {
let sender = MockSender::scripted(vec![], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
// No record at all.
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(sender.answers(), vec![Some("Expired".to_string())]);
// A record past its TTL (nothing swept it yet) is dropped on use.
let stale = crate::db::unix_now() - ctx.config.edit_message_ttl.as_secs() as i64 - 1;
seed_prompt(&ctx, "", stale).await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(
sender.answers(),
vec![Some("Expired".to_string()), Some("Expired".to_string())]
);
assert!(
ctx.chat_store.get(1).await.edit_message.is_empty(),
"the expired record must be dropped"
);
assert_eq!(sender.calls(), vec!["answer_callback_query"; 2]);
}
respond(())
}
+637 -46
View File
@@ -1,7 +1,10 @@
//! Bot command parsing, the `/`-command executor and `setMyCommands`
//! registration. URL/inline/callback flows live in their own modules.
use super::urls::{PostSend, url_media};
use super::{CHAT_STORE, CONFIG, LINK_CACHE, log_key, reply, reply_html};
use crate::ctx::AppContext;
use crate::state::ChatData;
use teloxide::RequestError;
use teloxide::prelude::*;
use teloxide::types::{ChatId, Message, Recipient};
@@ -31,30 +34,154 @@ pub(crate) enum Command {
parse_with = "split"
)]
SetTemplate(String),
#[command(description = "Show chat state (debug)")]
#[command(description = "Remove a saved template", parse_with = "split")]
RemoveTemplate(String),
#[command(description = "Show this chat's settings")]
Settings,
#[command(description = "Show chat state (debug; admin only)")]
BotDict,
#[command(description = "Set site caption format", parse_with = "split")]
#[command(
description = "Set site caption format (- to reset)",
parse_with = parse_arg_remainder
)]
SetFormat(String),
#[command(
description = "Clear link cache (admin; optional URL, else all)",
parse_with = "split"
parse_with = parse_arg_remainder
)]
ClearCache(String),
#[command(
description = "Test link parsing (debug; no media sent)",
parse_with = parse_test_arg
description = "Send a link's media (no forwarding)",
parse_with = parse_arg_remainder
)]
Test(String),
#[command(
description = "Parse a link and report it (debug; nothing sent)",
parse_with = parse_arg_remainder
)]
Debug(String),
}
/// `/test` argument parser: the whole remainder after the command name,
/// trimmed. The built-in `split` parser takes exactly one space-separated
/// `/test` and `/debug` argument parser: the whole remainder after the command
/// name, trimmed. The built-in `split` parser takes exactly one space-separated
/// token and rejects the rest, so a URL followed by a trailing space (or
/// pasted text) would silently fall through to the URL flow instead.
fn parse_test_arg(s: String) -> Result<(String,), ParseError> {
fn parse_arg_remainder(s: String) -> Result<(String,), ParseError> {
Ok((s.trim().to_string(),))
}
/// Placeholders `/set_format` accepts, mirroring what
/// `x_media::site::caption_from_fields` substitutes.
const FORMAT_PLACEHOLDERS: [&str; 6] = ["url", "author", "author_url", "title", "content", "tags"];
/// `/start`'s welcome: what the bot is for, where links work, where to look
/// next. The old "Hello!" left a first-time user with nothing.
const START_TEXT: &str = "\
Send me a post link and I'll send back its images, videos and GIFs with the title, author and tags.
Supported: X/Twitter, Pixiv, Bluesky, Misskey (misskey.io), Bilibili.
In a private chat just paste the link. In a group, use inline mode (type @, pick me, then the link).
/help lists every command.";
/// Appended to `/help`'s command list: argument syntax, caption
/// placeholders and the private-chat rule — none of which teloxide's
/// `descriptions()` renders (it prints `/command — description` only).
const HELP_FOOTER: &str = "\
Arguments
/set_forward_channel <@channel or channel id>
/set_template <name> — reply to a message containing [] to save it
/remove_template <name> — see /settings for the saved names
/set_format <site> <format> — '-' restores the built-in format
/test <link> / /debug <link>
Caption placeholders (for /set_format)
{url} {author} {author_url} {title} {content} {tags}
A template's [] is replaced by the post link when forwarding.
Links are handled in private chats only; in a group use inline mode.";
/// Cap on template names echoed by `/settings`: a chat with hundreds of
/// templates must not produce a message Telegram rejects for length.
const MAX_SETTINGS_TEMPLATE_NAMES: usize = 30;
/// Sorted template names: the order `/settings`, `/remove_template` and the
/// prompt's buttons all show.
fn sorted_template_names(data: &ChatData) -> Vec<String> {
let mut names: Vec<String> = data.template.keys().cloned().collect();
names.sort();
names
}
/// `/settings`: what this chat is configured to do, readable by anyone in it
/// (unlike `/bot_dict`, which dumps the raw state and is admin-only).
fn settings_text(data: &ChatData) -> String {
let mut lines = Vec::new();
match data.forward_channel_id {
Some(id) => lines.push(format!("Forward channel: {id}")),
None => lines.push(
"Forward channel: not set (use /set_forward_channel <@channel or id>)".to_string(),
),
}
lines.push(format!(
"Edit before forward: {}",
if data.edit_before_forward {
"on"
} else {
"off"
}
));
let mut formats: Vec<String> = data
.message_format
.iter()
.map(|(site, format)| format!("{site} => {format}"))
.collect();
formats.sort();
lines.push(if formats.is_empty() {
"Caption formats: built-in for every site".to_string()
} else {
format!("Caption formats:\n {}", formats.join("\n "))
});
let names = sorted_template_names(data);
lines.push(match names.len() {
0 => "Templates: none".to_string(),
n => format!(
"Templates ({n}): {}{}",
names
.iter()
.take(MAX_SETTINGS_TEMPLATE_NAMES)
.cloned()
.collect::<Vec<_>>()
.join(", "),
if n > MAX_SETTINGS_TEMPLATE_NAMES {
format!(", +{} more", n - MAX_SETTINGS_TEMPLATE_NAMES)
} else {
String::new()
}
),
});
lines.join("\n")
}
/// The first `{…}` token in a caption format that is not a known placeholder
/// (`None` when all of them are). The renderer replaces exact keys only, so an
/// unknown token would be published verbatim in every caption of that site —
/// caught here instead.
fn unknown_placeholder(format: &str) -> Option<&str> {
let mut rest = format;
while let Some(open) = rest.find('{') {
let after = &rest[open + 1..];
// An unclosed `{` is not a placeholder token at all.
let close = after.find('}')?;
let token = &after[..close];
if !FORMAT_PLACEHOLDERS.contains(&token) {
return Some(token);
}
rest = &after[close + 1..];
}
None
}
enum SetForwardChannelError {
EmptyParameter,
NotChannel,
@@ -133,11 +260,17 @@ pub(crate) async fn execute_command(
) -> Result<(), RequestError> {
match command {
Command::Start => {
bot.send_message(message.chat.id, "Hello!").await?;
bot.send_message(message.chat.id, START_TEXT).await?;
}
Command::Help => {
bot.send_message(message.chat.id, Command::descriptions().to_string())
.await?;
// The command list plus the parts teloxide's `descriptions()`
// cannot show: argument syntax, caption placeholders, and where a
// link actually works.
bot.send_message(
message.chat.id,
format!("{}\n\n{}", Command::descriptions(), HELP_FOOTER),
)
.await?;
}
Command::SetForwardChannel(channel) => {
let result = match set_forward_channel_handler(bot, message, channel).await {
@@ -225,10 +358,64 @@ pub(crate) async fn execute_command(
};
reply(bot, message.chat.id.0, message.id, text).await?;
}
Command::RemoveTemplate(name) => {
let chat_id = message.chat.id.0;
let name = name.trim().to_string();
if name.is_empty() {
reply(
bot,
chat_id,
message.id,
"Usage: /remove_template <name> (see /settings for the saved names)",
)
.await?;
return Ok(());
}
let removed = CHAT_STORE
.update(chat_id, |data| data.template.remove(&name).is_some())
.await;
let text = if removed {
format!("Template '{name}' removed.")
} else {
// Name the live templates: a typo would otherwise look like a
// successful delete.
let names = sorted_template_names(&CHAT_STORE.get(chat_id).await);
if names.is_empty() {
format!("No template named '{name}'. None are saved yet.")
} else {
format!("No template named '{name}'. Saved: {}", names.join(", "))
}
};
reply(bot, chat_id, message.id, text).await?;
}
Command::Settings => {
let chat_id = message.chat.id.0;
let data = CHAT_STORE.get(chat_id).await;
reply(bot, chat_id, message.id, settings_text(&data)).await?;
}
Command::BotDict => {
// Debug dump of the chat's persisted state: admin only (it echoes
// forward-channel ids and templates to whoever asks).
let sender_id = message
.from
.as_ref()
.map(|user| user.id.0 as i64)
.unwrap_or(-1);
if !CONFIG.admin_ids.contains(&sender_id) {
reply(bot, message.chat.id.0, message.id, "Admin only.").await?;
return Ok(());
}
let chat_data = CHAT_STORE.get(message.chat.id.0).await;
let debug = format!("{chat_data:?}");
let text = html_escape::encode_text(&debug).into_owned();
let debug = html_escape::encode_text(&format!("{chat_data:?}")).into_owned();
// A chat with many templates/edit records exceeds Telegram's 4096
// char message limit; the dump is plain text (no parse mode), so a
// plain byte-boundary cut is safe.
let end = debug.floor_char_boundary(MAX_DEBUG_DUMP_CHARS.min(debug.len()));
let text = if end < debug.len() {
format!("{}", &debug[..end])
} else {
debug
};
reply(bot, message.chat.id.0, message.id, text).await?;
}
Command::SetFormat(arg) => {
@@ -253,7 +440,45 @@ pub(crate) async fn execute_command(
bot,
message.chat.id.0,
message.id,
"Unknown site. Use twitter, bsky, pixiv or misskey.",
"Unknown site. Use twitter, bsky, pixiv, misskey or bilibili.",
)
.await?;
return Ok(());
}
// `-` resets to the site's built-in caption: without it a chat that
// set a format once could never get back to the default (the
// built-in format string is not something a user can retype).
if format == "-" {
CHAT_STORE
.update(chat_id, |data| {
data.message_format.remove(site);
})
.await;
reply(
bot,
message.chat.id.0,
message.id,
"Format reset to the built-in one.",
)
.await?;
return Ok(());
}
// A typo like {titel} would otherwise be rendered literally into
// every caption of that site (the renderer only substitutes the
// exact keys), which is invisible until a post arrives.
if let Some(token) = unknown_placeholder(&format) {
reply(
bot,
message.chat.id.0,
message.id,
format!(
"Unknown placeholder {{{token}}}. Available: {}",
FORMAT_PLACEHOLDERS
.iter()
.map(|name| format!("{{{name}}}"))
.collect::<Vec<_>>()
.join(" ")
),
)
.await?;
return Ok(());
@@ -263,7 +488,13 @@ pub(crate) async fn execute_command(
data.message_format.insert(site.to_string(), format);
})
.await;
reply(bot, message.chat.id.0, message.id, "Format set.").await?;
reply(
bot,
message.chat.id.0,
message.id,
"Format set. Use /debug <link> to preview the caption.",
)
.await?;
}
Command::ClearCache(arg) => {
let sender_id = message
@@ -294,7 +525,7 @@ pub(crate) async fn execute_command(
bot,
message.chat.id.0,
message.id,
"Unrecognized link. Use a twitter/x, pixiv, bsky or misskey post URL.",
"Unrecognized link. Use a twitter/x, pixiv, bsky, misskey or bilibili post URL.",
)
.await?;
return Ok(());
@@ -327,17 +558,54 @@ pub(crate) async fn execute_command(
.await?;
return Ok(());
}
if x_media::site::cache_key(url).is_none() {
reply(
bot,
message.chat.id.0,
message.id,
"No enabled site matches this link (twitter/x, pixiv, bsky, misskey or bilibili).",
)
.await?;
return Ok(());
}
// The ordinary link pipeline with the chat's post-send actions
// suppressed: the media is sent (and cached) like a normal link,
// but nothing is forwarded to the channel and no
// edit-before-forward prompt opens. Info level echoes the
// normalized key (never the raw URL) per the logging convention.
log::info!("test: sending [key={}]", log_key(url));
let ctx = AppContext::from_statics(bot);
url_media(
&ctx,
message.chat.id.0,
message.id.0 as i64,
url,
PostSend::Suppressed,
)
.await;
}
Command::Debug(arg) => {
let url = arg.trim();
if url.is_empty() {
reply(
bot,
message.chat.id.0,
message.id,
"Usage: /debug <post url>",
)
.await?;
return Ok(());
}
// Debug tool: report the parse result only — nothing is sent,
// cached or forwarded. Info level echoes the normalized key
// (never the raw URL) per the logging convention.
log::info!("test: parsing [key={}]", log_key(url));
// cached or forwarded.
log::info!("debug: parsing [key={}]", log_key(url));
match x_media::site::fetch(url).await {
Ok(None) => {
reply(
bot,
message.chat.id.0,
message.id,
"No enabled site matches this link (twitter/x, pixiv, bsky or misskey).",
"No enabled site matches this link (twitter/x, pixiv, bsky, misskey or bilibili).",
)
.await?;
}
@@ -351,14 +619,33 @@ pub(crate) async fn execute_command(
.await?;
}
Ok(Some(fetched)) => {
let report = test_parse_report(
// The preview must show what a link would actually send:
// the chat's per-site format override plus the long-post
// quoting. Rendering the raw built-in caption here made
// `/set_format` look like it did nothing.
let format = CHAT_STORE
.get(message.chat.id.0)
.await
.message_format
.get(fetched.site_name())
.cloned()
.unwrap_or_default();
let caption = preview_caption(
&format,
&fetched.caption,
&fetched.source_url,
fetched.render_fields(),
CONFIG.caption_quote_text_chars,
);
let report = debug_report(
url,
fetched.site_name(),
&fetched.source_url,
&fetched.title,
&fetched.content,
fetched.render_fields(),
fetched.sensitive,
&fetched.caption,
&caption,
&fetched.media,
);
// HTML report: the caption renders inside a <blockquote>
@@ -376,20 +663,69 @@ fn plural(n: usize) -> &'static str {
if n == 1 { "y" } else { "ies" }
}
/// Bot profile texts (Bot API `setMyDescription` / `setMyShortDescription`):
/// shown on the bot's profile page and in the share sheet. Without them a
/// shared link says nothing about what the bot does.
const BOT_DESCRIPTION: &str = "\
Send a post link from X/Twitter, Pixiv, Bluesky, Misskey (misskey.io) or Bilibili and get its images, videos and GIFs back with the title, author and tags.
Links are handled in private chats; a group can use inline mode. /help lists every command.";
const BOT_SHORT_DESCRIPTION: &str =
"Post links (X, Pixiv, Bluesky, Misskey, Bilibili) -> media messages";
/// Registers the bot's command list with Telegram so clients show it in the
/// `/` menu (Bot API `setMyCommands`).
/// `/` menu (Bot API `setMyCommands`), plus its profile description texts.
pub async fn register_commands(bot: &Bot) -> Result<(), RequestError> {
let commands = Command::bot_commands();
bot.set_my_commands(commands.clone()).await?;
log::info!("registered {} commands", commands.len());
// Profile texts are cosmetic: a failure (rare) must not abort startup.
if let Err(e) = bot.set_my_description().description(BOT_DESCRIPTION).await {
log::warn!("failed to set the bot description: {e}");
}
if let Err(e) = bot
.set_my_short_description()
.short_description(BOT_SHORT_DESCRIPTION)
.await
{
log::warn!("failed to set the bot short description: {e}");
}
Ok(())
}
/// Telegram's plain-text message limit is 4096 chars; the report stays under
/// it even for very large threads (many media lines + a long caption).
const MAX_TEST_REPORT_CHARS: usize = 4000;
const MAX_DEBUG_REPORT_CHARS: usize = 4000;
/// Builds the HTML report for the `/test` command: what the parser produced
/// Cap for the `/bot_dict` debug dump: the state is echoed as one plain-text
/// message, so it must stay under Telegram's 4096-char limit.
const MAX_DEBUG_DUMP_CHARS: usize = 3500;
/// The caption a link would actually send for this chat: the per-site format
/// override (empty = the site's built-in caption) and, on a long post, the
/// same text quoting the send paths apply. `/debug` shows this so the preview
/// cannot drift from what the send paths produce.
fn preview_caption(
format: &str,
built_in: &str,
url: &str,
fields: Option<(&str, &str, &str, &str, &str)>,
quote_chars: usize,
) -> String {
let caption = match fields {
// Same call the send paths make through `Fetched::caption_with`: an
// empty format falls back to the built-in caption.
Some((author, author_url, title, content, tags)) => x_media::site::caption_from_fields(
format, built_in, url, author, author_url, title, content, tags,
),
None => x_media::site::truncate_caption(built_in),
};
let text = fields
.map(|(_, _, title, content, _)| x_media::site::compose_text(title, content))
.unwrap_or_default();
crate::send::quote_long_caption(&caption, &text, quote_chars).into_owned()
}
/// Builds the HTML report for the `/debug` command: what the parser produced
/// for a link (site, canonical URL, title/author/tags, caption and the media
/// list) — no media is sent and nothing is cached or forwarded. Sent with
/// HTML parse mode: raw fields are escaped, the pre-escaped render fields are
@@ -399,12 +735,13 @@ const MAX_TEST_REPORT_CHARS: usize = 4000;
/// constructing a `Fetched` (its render fields are `pub(crate)` to the
/// x-media crate).
#[allow(clippy::too_many_arguments)]
fn test_parse_report(
fn debug_report(
url: &str,
site_id: &str,
source_url: &str,
title: &str,
render: Option<(&str, &str, &str, &str)>,
content: &str,
render: Option<(&str, &str, &str, &str, &str)>,
sensitive: bool,
caption: &str,
media: &[x_media::media::Media],
@@ -424,7 +761,8 @@ fn test_parse_report(
html_escape::encode_text(source_url)
));
lines.push(format!("title: {}", html_escape::encode_text(title)));
if let Some((author, author_url, _title, tags)) = render {
lines.push(format!("content: {}", html_escape::encode_text(content)));
if let Some((author, author_url, _title, _content, tags)) = render {
// The render fields are already pre-escaped for HTML captions; embed
// them as-is so the report renders them exactly like the final
// caption. `author_url` is raw and gets escaped here.
@@ -460,8 +798,8 @@ fn test_parse_report(
"
",
);
if out.chars().count() > MAX_TEST_REPORT_CHARS {
let end = out.floor_char_boundary(MAX_TEST_REPORT_CHARS - 1);
if out.chars().count() > MAX_DEBUG_REPORT_CHARS {
let end = out.floor_char_boundary(MAX_DEBUG_REPORT_CHARS - 1);
out = format!("{}", &out[..end]);
}
out
@@ -469,11 +807,13 @@ fn test_parse_report(
#[cfg(test)]
mod tests {
use super::{MAX_TEST_REPORT_CHARS, test_parse_report};
use super::{
MAX_DEBUG_REPORT_CHARS, debug_report, preview_caption, settings_text, unknown_placeholder,
};
use x_media::media::Media;
#[test]
fn test_parse_report_renders_fields_and_media() {
fn debug_report_renders_fields_and_media() {
let media = vec![
Media::Illustration {
title: None,
@@ -487,22 +827,27 @@ mod tests {
thumbnail_url: "https://cdn.example/2.jpg".into(),
},
];
let report = test_parse_report(
let report = debug_report(
"https://x.com/u/status/1",
"twitter",
"https://x.com/u/status/1",
"My title",
Some(("Author", "https://x.com/u", "My title", "tag1 tag2")),
"My content",
Some((
"Author",
"https://x.com/u",
"My title",
"My content",
"tag1 tag2",
)),
false,
"<a href=\"https://x.com/u\">Author</a> · My title",
&media,
);
assert!(report.contains("site: twitter"), "{report}");
assert!(report.contains("key: twitter:1"), "{report}");
assert!(report.contains("title: My title"), "{report}");
assert!(report.contains("author: Author"), "{report}");
assert!(report.contains("content: My content"), "{report}");
assert!(report.contains("author_url: https://x.com/u"), "{report}");
assert!(report.contains("tags: tag1 tag2"), "{report}");
assert!(report.contains("sensitive: false"), "{report}");
assert!(report.contains("media (2):"), "{report}");
assert!(
@@ -516,28 +861,31 @@ mod tests {
}
#[test]
fn test_parse_report_without_render_data_and_no_media() {
let report = test_parse_report("u", "pixiv", "s", "t", None, true, "c", &[]);
fn debug_report_without_render_data_has_no_author_line() {
let report = debug_report("u", "pixiv", "s", "t", "c", None, true, "p", &[]);
// The `None` branch above is the point: with no render fields there is
// no author line to print. The `sensitive`/`media` lines are the same
// format sites the escaping test already pins with values.
assert!(!report.contains("author:"), "{report}");
assert!(report.contains("sensitive: true"), "{report}");
assert!(report.contains("media (0):"), "{report}");
}
#[test]
fn test_parse_report_wraps_caption_in_blockquote() {
fn debug_report_wraps_caption_in_blockquote() {
// The report is an HTML message: raw fields are escaped, pre-escaped
// render fields are embedded as-is, and the caption is wrapped in a
// <blockquote> so it shows exactly as it will render in the sent
// media caption (escaped text and links included).
let report = test_parse_report(
let report = debug_report(
"https://x.com/u/status/1",
"twitter",
"https://x.com/u/status/1",
"A & B <C>",
"body & <more>",
Some((
"A &amp; B",
"https://x.com/u",
"A &amp; B &lt;C&gt;",
"body &amp; &lt;more&gt;",
"#a &amp; #b",
)),
false,
@@ -563,7 +911,7 @@ mod tests {
}
#[test]
fn test_parse_report_is_capped() {
fn debug_report_is_capped() {
// 200 media lines ≈ 8 KB, comfortably over the cap.
let media: Vec<Media> = (0..200)
.map(|i| Media::Illustration {
@@ -573,8 +921,251 @@ mod tests {
fallback_url: None,
})
.collect();
let report = test_parse_report("u", "twitter", "s", "t", None, false, "c", &media);
assert!(report.chars().count() <= MAX_TEST_REPORT_CHARS, "{report}");
let report = debug_report("u", "twitter", "s", "t", "c", None, false, "p", &media);
assert!(report.chars().count() <= MAX_DEBUG_REPORT_CHARS, "{report}");
assert!(report.ends_with('…'), "{report}");
}
#[test]
fn settings_text_reports_the_chat_configuration() {
use crate::state::ChatData;
// A fresh chat: the defaults must be spelled out, including how to set
// the channel (an empty field is not a status).
let empty = settings_text(&ChatData::default());
assert!(empty.contains("Forward channel: not set"), "{empty}");
assert!(empty.contains("/set_forward_channel"), "{empty}");
assert!(empty.contains("Edit before forward: off"), "{empty}");
assert!(empty.contains("built-in for every site"), "{empty}");
assert!(empty.contains("Templates: none"), "{empty}");
let configured = ChatData {
forward_channel_id: Some(-100123),
edit_before_forward: true,
template: [("b", "[]"), ("a", "[]")]
.into_iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect(),
message_format: [("twitter", "{author}: {content}")]
.into_iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect(),
..ChatData::default()
};
let text = settings_text(&configured);
assert!(text.contains("Forward channel: -100123"), "{text}");
assert!(text.contains("Edit before forward: on"), "{text}");
assert!(text.contains("twitter => {author}: {content}"), "{text}");
// Sorted, so the same chat always reports the same thing.
assert!(text.contains("Templates (2): a, b"), "{text}");
}
#[test]
fn help_and_start_cover_what_the_command_list_cannot() {
// The placeholders the renderer substitutes must be the ones the help
// lists: a stale list is worse than none.
for placeholder in super::FORMAT_PLACEHOLDERS {
assert!(
super::HELP_FOOTER.contains(&format!("{{{placeholder}}}")),
"help does not document {{{placeholder}}}"
);
}
// The private-chat rule and the template placeholder semantics are the
// two things users got wrong most often.
assert!(super::HELP_FOOTER.contains("private chats only"));
assert!(super::HELP_FOOTER.contains("[]"));
assert!(super::START_TEXT.contains("inline mode"));
assert!(super::START_TEXT.contains("/help"));
// Both must stay inside Telegram's message limit.
assert!(super::HELP_FOOTER.chars().count() < 2000);
assert!(super::START_TEXT.chars().count() < 2000);
}
#[test]
fn every_command_is_registered_and_parses() {
use teloxide::utils::command::BotCommands;
use super::Command;
let registered: Vec<String> = Command::bot_commands()
.into_iter()
.map(|command| command.command.trim_start_matches('/').to_string())
.collect();
for expected in [
"start",
"help",
"settings",
"set_forward_channel",
"remove_template",
"set_format",
"test",
"debug",
] {
assert!(
registered.iter().any(|name| name == expected),
"{expected} missing from {registered:?}"
);
}
// Telegram caps a command description at 256 chars.
for command in Command::bot_commands() {
assert!(
command.description.chars().count() <= 256,
"{}: description too long",
command.command
);
}
}
#[test]
fn every_documented_invocation_parses() {
use teloxide::utils::command::BotCommands;
use super::Command;
// The README's forms, verbatim. teloxide's `split` parser accepts
// EXACTLY one token per `String` field, so a command documented with
// two arguments (or an optional one) silently stops parsing — and a
// command that does not parse falls through to the URL flow in
// silence.
type Check = fn(&Command) -> bool;
let cases: Vec<(&str, Check)> = vec![
("/start", |c| matches!(c, Command::Start)),
("/help", |c| matches!(c, Command::Help)),
("/settings", |c| matches!(c, Command::Settings)),
("/edit_before_forward", |c| {
matches!(c, Command::EditBeforeForward)
}),
("/remove_forward_channel", |c| {
matches!(c, Command::RemoveForwardChannel)
}),
("/bot_dict", |c| matches!(c, Command::BotDict)),
(
"/set_forward_channel @a_channel",
|c| matches!(c, Command::SetForwardChannel(a) if a == "@a_channel"),
),
(
"/set_template tpl",
|c| matches!(c, Command::SetTemplate(a) if a == "tpl"),
),
(
"/remove_template tpl",
|c| matches!(c, Command::RemoveTemplate(a) if a == "tpl"),
),
(
"/set_format twitter {author}: {title}",
|c| matches!(c, Command::SetFormat(a) if a == "twitter {author}: {title}"),
),
(
"/set_format twitter -",
|c| matches!(c, Command::SetFormat(a) if a == "twitter -"),
),
// Documented as "clear everything" when called without a link.
(
"/clear_cache",
|c| matches!(c, Command::ClearCache(a) if a.is_empty()),
),
(
"/clear_cache https://x.com/u/status/1",
|c| matches!(c, Command::ClearCache(a) if a == "https://x.com/u/status/1"),
),
(
"/test https://x.com/u/status/1",
|c| matches!(c, Command::Test(a) if a == "https://x.com/u/status/1"),
),
(
"/debug https://x.com/u/status/1",
|c| matches!(c, Command::Debug(a) if a == "https://x.com/u/status/1"),
),
];
for (text, ok) in cases {
match Command::parse(text, "") {
Ok(parsed) => assert!(ok(&parsed), "{text} parsed as the wrong variant"),
Err(e) => panic!("{text} did not parse: {e}"),
}
}
}
#[test]
fn preview_caption_applies_the_chat_format_and_the_long_post_quote() {
let fields = Some((
"Author",
"https://x.com/u",
"Pinned title",
"Pinned body",
"#tag",
));
// No format override → the site's built-in caption, untouched.
assert_eq!(
preview_caption(
"",
"built-in caption",
"https://x.com/u/status/1",
fields,
200
),
"built-in caption"
);
// The bug this pins: `/debug` used to print the built-in caption even
// with a format set, so `/set_format` looked like it did nothing.
let formatted = preview_caption(
"{author} · {title}",
"built-in caption",
"https://x.com/u/status/1",
fields,
200,
);
assert_eq!(formatted, "Author · Pinned title");
// `{url}` comes from the canonical post URL, as in the send paths.
assert_eq!(
preview_caption(
"{url} {title}",
"built-in",
"https://x.com/u/status/1",
fields,
200
),
"https://x.com/u/status/1 Pinned title"
);
// A long post's text is quoted exactly like the send paths quote it.
let long = "".repeat(300);
let fields = Some(("Author", "https://x.com/u", "", long.as_str(), ""));
let quoted = preview_caption(
"",
"https://x.com/u/status/1\n<a href=\"https://x.com/u\">Author</a>: 正…",
"https://x.com/u/status/1",
fields,
200,
);
assert!(quoted.contains("<blockquote expandable>"), "{quoted}");
// Without render fields (a site that does not expose them) the
// built-in caption is all there is.
assert_eq!(
preview_caption("", "built-in", "https://x.com/u/status/1", None, 200),
"built-in"
);
}
#[test]
fn unknown_placeholder_finds_typos_only() {
assert_eq!(unknown_placeholder("{author} — {title}"), None);
// Every key the renderer substitutes must pass, in any combination.
assert_eq!(
unknown_placeholder("{url}{author}{author_url}{title}{content}{tags}"),
None
);
// Plain text and braces Telegram renders literally are not tokens.
assert_eq!(unknown_placeholder("no placeholders here"), None);
assert_eq!(unknown_placeholder("{unclosed"), None);
assert_eq!(unknown_placeholder("{titel}"), Some("titel"));
assert_eq!(unknown_placeholder("{title} {Content}"), Some("Content"));
// A typo after a valid token is still found.
assert_eq!(unknown_placeholder("{url} {tag}"), Some("tag"));
}
}
+212 -49
View File
@@ -3,6 +3,7 @@
//! inline cache instead of re-fetching.
use super::log_key;
use std::collections::HashMap;
use std::sync::LazyLock;
use teloxide::RequestError;
use teloxide::prelude::*;
@@ -19,16 +20,95 @@ use x_media::media::Media;
/// post id. Only answer once the query has been stable for this long.
const INLINE_DEBOUNCE: std::time::Duration = std::time::Duration::from_millis(800);
/// Last seen inline query and whether it was already answered. Guards the
/// debounce timer: a repeat of an answered query is served by Telegram's
/// inline cache (see `cache_time`), not by another fetch.
/// How long a debounce entry is worth keeping: the window Telegram caches an
/// inline answer for (`answer_inline_query` asks for `cache_time(300)`). Past
/// it a repeat is sent to the bot again and has to be answered fresh, so the
/// entry would only suppress a fetch the user is waiting for.
const INLINE_STATE_TTL: std::time::Duration = std::time::Duration::from_secs(300);
/// Last seen inline query per user and whether it was already answered.
/// Guards the debounce timer: a repeat of an answered query is served by
/// Telegram's inline cache (see `cache_time`), not by another fetch. Keyed by
/// user id — a single shared slot would let one user's typing burst (or a
/// different user's query) cancel another user's pending answer.
struct InlineDebounceState {
query: String,
answered: bool,
/// When a query last touched this entry, so the periodic sweep can drop
/// one per user who ever used inline mode (the map had no eviction at all,
/// unlike the rate limiter's buckets and the chat store).
last_seen: std::time::Instant,
}
static INLINE_DEBOUNCE_STATE: LazyLock<parking_lot::Mutex<Option<InlineDebounceState>>> =
LazyLock::new(|| parking_lot::Mutex::new(None));
#[derive(Default)]
struct DebounceStates(HashMap<u64, InlineDebounceState>);
impl DebounceStates {
/// Records `query` as the user's newest query. Returns false when it is a
/// repeat whose answer already went out (Telegram's inline cache serves
/// it; re-fetching would only hit the source site again).
fn note(&mut self, user_id: u64, query: &str) -> bool {
if let Some(prev) = self.0.get(&user_id)
&& prev.query == query
&& prev.answered
{
return false;
}
self.0.insert(
user_id,
InlineDebounceState {
query: query.to_string(),
answered: false,
last_seen: std::time::Instant::now(),
},
);
true
}
/// Drops entries no query has touched for `idle_for`. Split from the clock
/// so the boundary is testable without ageing a monotonic instant.
fn prune_idle_at(&mut self, now: std::time::Instant, idle_for: std::time::Duration) -> usize {
let before = self.0.len();
self.0
.retain(|_, state| now.saturating_duration_since(state.last_seen) < idle_for);
before - self.0.len()
}
/// Claims the answer for the user's newest query; false when a newer query
/// superseded it or the answer was already claimed.
fn claim(&mut self, user_id: u64, query: &str) -> bool {
let Some(state) = self.0.get_mut(&user_id) else {
return false;
};
if state.query != query || state.answered {
return false;
}
state.answered = true;
state.last_seen = std::time::Instant::now();
true
}
/// Releases a claimed-but-unsent answer so a repeat can retry the fetch.
fn release(&mut self, user_id: u64, query: &str) {
if let Some(state) = self.0.get_mut(&user_id)
&& state.query == query
{
state.answered = false;
state.last_seen = std::time::Instant::now();
}
}
}
/// Drops debounce entries idle for [`INLINE_STATE_TTL`]; the 300 s sweep calls
/// this next to the rate limiter's prune. Returns how many were dropped.
pub(crate) fn prune_idle_states() -> usize {
INLINE_DEBOUNCE_STATE
.lock()
.prune_idle_at(std::time::Instant::now(), INLINE_STATE_TTL)
}
static INLINE_DEBOUNCE_STATE: LazyLock<parking_lot::Mutex<DebounceStates>> =
LazyLock::new(|| parking_lot::Mutex::new(DebounceStates::default()));
pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), RequestError> {
if query.query.is_empty() {
@@ -41,48 +121,25 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
// Debounce: record the query and answer only after it has been stable for
// INLINE_DEBOUNCE (the timer below). An already-answered repeat of the
// same query is left to Telegram's inline cache instead of re-fetching.
{
let mut state = INLINE_DEBOUNCE_STATE.lock();
if let Some(prev) = state.as_ref()
&& prev.query == query.query
&& prev.answered
{
return respond(());
}
*state = Some(InlineDebounceState {
query: query.query.clone(),
answered: false,
});
let user_id = query.from.id.0;
if !INLINE_DEBOUNCE_STATE.lock().note(user_id, &query.query) {
return respond(());
}
let query_text = query.query.clone();
tokio::spawn(async move {
tokio::time::sleep(INLINE_DEBOUNCE).await;
// Only the last query of a typing burst survives: earlier timers see
// the query changed and give up without answering.
{
let mut state = INLINE_DEBOUNCE_STATE.lock();
let Some(state) = state.as_mut() else {
return;
};
if state.query != query_text || state.answered {
return;
}
// Claim the answer so a repeat of the same query cannot start a
// second fetch; reset below when no answer was produced.
state.answered = true;
// Only the user's last query of a typing burst survives: earlier
// timers see the query changed and give up without answering.
if !INLINE_DEBOUNCE_STATE.lock().claim(user_id, &query_text) {
return;
}
match answer_inline_query(bot, query).await {
Ok(true) => {}
// No results produced (or nothing to answer): let a repeat of the
// same query retry the fetch.
Ok(false) | Err(_) => {
let mut state = INLINE_DEBOUNCE_STATE.lock();
if let Some(state) = state.as_mut()
&& state.query == query_text
{
state.answered = false;
}
}
// The fetch or the answer call failed: release so a repeat of the
// same query may retry it. An *empty* answer is a real answer
// (`Ok(true)`), so a link whose media Telegram cannot fetch is not
// re-fetched on every keystroke.
Ok(false) | Err(_) => INLINE_DEBOUNCE_STATE.lock().release(user_id, &query_text),
}
});
respond(())
@@ -91,19 +148,44 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
/// Fetches the post behind an inline query and answers it. The caller has
/// already applied the debounce. Returns `true` when an answer was sent.
async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, RequestError> {
log::debug!(
"inline query: {} [key={}]",
query.query,
log_key(&query.query)
);
match x_media::site::fetch(&query.query).await {
// The query is user input: `debug` keeps only its normalized key, the
// text itself is `trace` (same split as the message handler).
log::debug!("inline query [key={}]", log_key(&query.query));
log::trace!("inline query: {}", query.query);
// No retries: the debounce plus a 1s/2s backoff would outlast the inline
// query the answer belongs to.
match x_media::site::fetch_once(&query.query).await {
Ok(Some(fetched)) => {
let mut results: Vec<InlineQueryResult> = Vec::new();
// Inline results have the same 1024-char caption limit as regular
// messages; truncate once here for all items.
// messages; truncate once here for all items, then apply the same
// long-post quoting as the send paths. `answer_inline_query` has no
// `AppContext` (the debounce spawns it), so the parsed config comes
// from the process-wide static, and the text is the *escaped*
// title/content the built-in caption embeds (the raw
// `Fetched.title`/`content` differ whenever the post contains
// `<`/`&`).
let caption = x_media::site::truncate_caption(&fetched.caption);
let text = fetched
.render_fields()
.map(|(_, _, title, content, _)| x_media::site::compose_text(title, content))
.unwrap_or_default();
let caption = crate::send::quote_long_caption(
&caption,
&text,
super::CONFIG.caption_quote_text_chars,
);
for (i, media) in fetched.media.iter().enumerate() {
let id = format!("{i}");
// Telegram fetches an inline result's URL itself and cannot
// send site-specific headers, so hotlink-protected media
// (pixiv's pximg.net) would render as a broken file there.
// Locally produced media (ugoira MP4, bsky remux) is a local
// path and does not parse as a URL at all — same skip.
if x_media::site::needs_media_headers(media.url()) {
log::debug!("inline: skipping hotlink-protected media {id}");
continue;
}
let Some(url) = url::Url::parse(media.url()).ok() else {
continue;
};
@@ -111,7 +193,7 @@ async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, Reque
.thumbnail_url()
.and_then(|t| url::Url::parse(t).ok())
.unwrap_or_else(|| url.clone());
let caption = caption.clone();
let caption = caption.clone().into_owned();
let result = match media {
Media::Illustration { .. } => {
// Inline photo results have their own (smaller) size
@@ -153,9 +235,90 @@ async fn answer_inline_query(bot: Bot, query: InlineQuery) -> Result<bool, Reque
.await?;
return Ok(true);
}
// Every item was skipped: Telegram fetches an inline result's URL
// itself, so pixiv's hotlink-protected media (and a local ugoira /
// bsky MP4) can never be one. Answer *empty* — the client stops
// spinning, and the same query is not re-fetched on every
// keystroke: an unanswered query releases the debounce below
// (`Ok(false)`), which is what made this re-run the fetch each
// time, and the window lets Telegram serve the repeats itself.
log::debug!("inline: nothing Telegram can fetch for the query; answering empty");
bot.answer_inline_query(query.id, Vec::new())
.cache_time(300)
.await?;
return Ok(true);
}
Ok(None) => {}
Err(e) => log::error!("inline fetch {}: {e}", query.query),
Err(e) => log::error!("inline fetch [key={}]: {e}", log_key(&query.query)),
}
Ok(false)
}
#[cfg(test)]
mod tests {
use super::{DebounceStates, INLINE_STATE_TTL};
const URL_A: &str = "https://x.com/a/status/1";
const URL_B: &str = "https://x.com/b/status/2";
#[test]
fn debounce_state_is_per_user() {
let mut states = DebounceStates::default();
// Two users query different links: both proceed, and neither timer
// cancels the other (a single shared slot dropped one of them).
assert!(states.note(1, URL_A));
assert!(states.note(2, URL_B));
assert!(states.claim(1, URL_A), "user 1's answer was cancelled");
assert!(states.claim(2, URL_B), "user 2's answer was cancelled");
}
#[test]
fn answered_query_is_suppressed_per_user_only() {
let mut states = DebounceStates::default();
assert!(states.note(1, URL_A));
assert!(states.claim(1, URL_A));
// A repeat of the answered query by the same user is left to
// Telegram's inline cache.
assert!(!states.note(1, URL_A));
// Another user pasting the same link still gets an answer.
assert!(states.note(2, URL_A));
assert!(states.claim(2, URL_A));
}
#[test]
fn idle_states_are_pruned_and_live_ones_kept() {
let mut states = DebounceStates::default();
assert!(states.note(1, URL_A));
let first = states.0[&1].last_seen;
// Entry 2 is strictly newer, so one timestamp can sit exactly on the
// window's edge for one and comfortably inside it for the other.
std::thread::sleep(std::time::Duration::from_millis(2));
assert!(states.note(2, URL_B));
assert_eq!(
states.prune_idle_at(first + INLINE_STATE_TTL, INLINE_STATE_TTL),
1
);
assert!(
!states.0.contains_key(&1),
"the entry past the window must go"
);
assert!(states.0.contains_key(&2), "the live entry must stay");
// A pruned user's repeat is answered fresh instead of suppressed.
assert!(states.note(1, URL_A));
}
#[test]
fn newer_query_supersedes_and_failed_answer_is_released() {
let mut states = DebounceStates::default();
assert!(states.note(1, URL_A));
assert!(states.note(1, URL_B));
// The stale timer for the half-typed query gives up…
assert!(!states.claim(1, URL_A));
// …and the newest one answers.
assert!(states.claim(1, URL_B));
// No results → release so a repeat may retry the fetch.
states.release(1, URL_B);
assert!(states.claim(1, URL_B));
}
}
+376 -39
View File
@@ -15,28 +15,32 @@ mod urls;
pub use callback::callback_query_handler;
pub use commands::register_commands;
pub use inline::inline_query_handler;
pub(crate) use inline::prune_idle_states;
/// The resolved `$DATA_DIR/task_queue.db` path, for the startup config line.
pub(crate) use statics::db_path;
pub use statics::{CHAT_STORE, CONFIG, LINK_CACHE, TASK_QUEUE};
pub(crate) use urls::repair_lost_local_media;
pub use urls::{start_url_workers, stop_url_workers};
use crate::ctx::AppContext;
use crate::media_sender::MediaSender;
use commands::{Command, execute_command};
use teloxide::RequestError;
use teloxide::prelude::*;
use teloxide::types::{ChatId, ChatKind, Message, MessageId, ParseMode, ReplyParameters};
use teloxide::types::{
ChatId, ChatKind, Message, MessageId, ParseMode, PublicChatKind, ReplyParameters,
};
use teloxide::utils::command::BotCommands;
use urls::{URL_JOBS, extract_urls};
/// Reply to a message by id, keeping the reply decoration even if the
/// original was already deleted.
pub(crate) async fn reply<T>(
/// original was already deleted. Returns the reply's message id.
pub(crate) async fn reply(
sender: &dyn MediaSender,
chat_id: i64,
reply_to: MessageId,
text: T,
) -> Result<Message, RequestError>
where
T: Into<String>,
{
text: impl Into<String>,
) -> Result<i64, RequestError> {
sender
.send_message(ChatId(chat_id), text.into(), Some(reply_to), None)
.await
@@ -50,35 +54,87 @@ pub(crate) async fn reply_html(
chat_id: i64,
reply_to: MessageId,
text: String,
) -> Result<Message, RequestError> {
) -> Result<i64, RequestError> {
// `<Bot as Requester>::` disambiguates from the MediaSender trait's
// same-named method (see media_sender.rs).
<Bot as Requester>::send_message(bot, ChatId(chat_id), text)
.parse_mode(ParseMode::Html)
.reply_parameters(ReplyParameters::new(reply_to).allow_sending_without_reply())
.await
.map(|message| message.id.0 as i64)
}
/// Log prefix tying the whole lifecycle of one link (fetch → send → cache →
/// forward) together: the normalized cache key (`twitter:123…`, `pixiv:123`,
/// `bsky:handle/rkey`) instead of the raw URL, so logs stay short and do not
/// echo full user-submitted URLs at info level.
/// `bsky:handle/rkey`, `bilibili:123…`) instead of the raw URL, so logs stay
/// short and do not echo full user-submitted URLs at info level.
pub fn log_key(url: &str) -> String {
x_media::site::cache_key(url).unwrap_or_else(|| "<unsupported>".to_string())
}
/// How long a caption edit may sleep before it gives up on retrying: the reply
/// (or button press) that carried the text is already consumed, so the update
/// must not stall the chat's queue behind a long flood-control wait — the user
/// is told to send it again instead.
const CAPTION_EDIT_MAX_RETRY_WAIT: std::time::Duration = std::time::Duration::from_secs(2);
/// Whether a caption edit landed.
enum EditOutcome {
Applied,
/// The API's reason, for the message the user gets.
Failed(String),
}
/// Applies a caption edit, retrying once when the API names a short retryable
/// delay (`RetryAfter`/network/5xx). A failed edit used to be logged and
/// swallowed while the record was updated anyway: the user saw nothing, the
/// caption never changed, and the text they typed was gone. Callers report
/// [`EditOutcome::Failed`] instead.
async fn apply_caption_edit(
sender: &dyn MediaSender,
chat_id: ChatId,
message_id: MessageId,
caption: String,
) -> EditOutcome {
let mut attempt = 0;
loop {
match sender
.edit_message_caption(chat_id, message_id, caption.clone())
.await
{
Ok(()) => return EditOutcome::Applied,
Err(e) => {
let reason = e.to_string();
if attempt == 0
&& let crate::send::Classification::Retryable { delay_seconds } =
crate::send::classify_request_error(&e)
&& std::time::Duration::from_secs_f64(delay_seconds)
<= CAPTION_EDIT_MAX_RETRY_WAIT
{
attempt = 1;
log::debug!("caption edit failed ({reason}), retrying once");
tokio::time::sleep(std::time::Duration::from_secs_f64(delay_seconds)).await;
continue;
}
log::error!("edit_message_caption failed: {reason}");
return EditOutcome::Failed(reason);
}
}
}
}
/// Edit-before-forward: a reply to the prompt swaps the caption of the first
/// forwarded message. Returns true when the message was consumed as an edit.
async fn edit_message_handler(bot: &Bot, message: &Message) -> bool {
let Some(reply) = message.reply_to_message() else {
return false;
};
let chat_id = message.chat.id.0;
let Some(text) = message.text() else {
return false;
};
let chat_data = CHAT_STORE.get(chat_id).await;
let Some(edit) = chat_data.edit_message.get(&(reply.id.0 as i64)) else {
/// Body of [`message_handler`]'s edit branch, without teloxide update types so
/// it can be driven by tests.
async fn edit_message_handler(
ctx: &AppContext<'_>,
chat_id: i64,
reply_to_message_id: i64,
text: &str,
) -> bool {
let chat_data = ctx.chat_store.get(chat_id).await;
let Some(edit) = chat_data.edit_message.get(&reply_to_message_id) else {
return false;
};
let Some(first_forward_id) = edit.forward_message_ids.first() else {
@@ -98,22 +154,48 @@ async fn edit_message_handler(bot: &Bot, message: &Message) -> bool {
.map(|template| template.replace("[]", &link))
.unwrap_or(link)
};
let result = bot
.edit_message_caption(ChatId(chat_id), MessageId(*first_forward_id as i32))
.caption(new_text)
.parse_mode(ParseMode::Html)
.await;
match result {
Ok(_) => log::info!(
"edit-before-forward: caption swapped on message {first_forward_id} for prompt {}",
reply.id.0
match apply_caption_edit(
ctx.sender,
ChatId(chat_id),
MessageId(*first_forward_id as i32),
new_text,
)
.await
{
EditOutcome::Applied => log::info!(
"edit-before-forward: caption swapped on message {first_forward_id} for prompt {reply_to_message_id}"
),
Err(e) => log::error!("edit_message_caption failed: {e}"),
// The reply was a caption for this prompt, so it stays consumed either
// way — but the user is told the swap failed instead of losing it
// silently (and can send it again).
EditOutcome::Failed(reason) => {
let _ = reply(
ctx.sender,
chat_id,
MessageId(reply_to_message_id as i32),
format!("Could not update the caption ({reason}). Send it again to retry."),
)
.await;
}
}
true
}
/// The `dptree` entry point: the process-wide context, plus the bot the
/// dispatcher handed us (used for the replies this module sends itself).
pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestError> {
handle_message(&AppContext::from_statics(&bot), &bot, message).await
}
/// Body of [`message_handler`], taking its context. Every branch here — the
/// edit-reply interception, the command path, the private-chat link enqueue and
/// the group hint — is otherwise reachable only through the process-wide
/// statics, which is why none of them had a test.
pub(crate) async fn handle_message(
ctx: &AppContext<'_>,
bot: &Bot,
message: Message,
) -> Result<(), RequestError> {
let is_private = matches!(message.chat.kind, ChatKind::Private(_));
let sender = message
.from
@@ -127,28 +209,49 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
&t[..end]
})
.unwrap_or("<no text>");
// Per-request detail: debug only (message text is user data).
// Per-request detail: who and where at `debug`; the message text itself is
// user data and only ever appears at `trace`, so a `debug` log can be
// shared without leaking what people pasted.
log::debug!(
"message from {sender} in {} (private={is_private}): {text_preview}",
"message from {sender} in {} (private={is_private})",
message.chat.id
);
log::trace!("message text: {text_preview}");
// URL/edit flows only run in private chats; commands run in any chat.
if is_private && edit_message_handler(&bot, &message).await {
if is_private
&& let Some(reply) = message.reply_to_message()
&& let Some(text) = message.text()
&& edit_message_handler(ctx, message.chat.id.0, reply.id.0 as i64, text).await
{
return respond(());
}
if let Some(text) = message.text()
&& let Ok(command) = Command::parse(text, "")
{
log::debug!("command from {}: {text_preview}", message.chat.id);
execute_command(&bot, &message, command).await?;
// The command name is what the operator needs at `debug`; its argument
// may be a user-supplied URL, which stays at `trace`.
log::debug!(
"command from {}: {}",
message.chat.id,
text.split_whitespace().next().unwrap_or("<empty>")
);
log::trace!("command text: {text_preview}");
execute_command(bot, &message, command).await?;
return respond(());
}
if is_private {
let urls = extract_urls(&message);
// Only links a site adapter claims: an unsupported URL never gets a
// media message, so enqueuing it would spend a queue slot, a worker
// wake-up and (through `run_with_chat_action`) a Telegram call on
// nothing. Same test the group branch below makes for its hint.
let urls: Vec<String> = extract_urls(&message)
.into_iter()
.filter(|url| x_media::site::cache_key(url).is_some())
.collect();
if !urls.is_empty() {
// Debug only, and echo the normalized keys instead of the raw URLs.
let keys: Vec<String> = urls.iter().map(|u| log_key(u)).collect();
log::debug!("extracted {} URL(s): {keys:?}", urls.len());
log::debug!("queuing {} supported URL(s): {keys:?}", urls.len());
}
for url in urls {
// Clone out of the lock: the parking_lot guard is !Send and must
@@ -157,8 +260,242 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
log::warn!("url workers not started; dropping link");
break;
};
let _ = tx.send((message.clone(), url)).await;
// A closed channel means the workers are stopping (shutdown):
// report the dropped link instead of losing it silently.
if tx.send((message.clone(), url)).await.is_err() {
log::warn!("url workers stopped; dropping link");
break;
}
}
} else if is_group(&message.chat.kind)
&& extract_urls(&message)
.iter()
.any(|url| x_media::site::cache_key(url).is_some())
{
// A supported link in a group used to be dropped in silence, which
// reads as a broken bot (the command menu is registered globally, so
// the expectation is there). Unsupported links stay ignored; the hint
// names the two paths that do work. Channels are excluded — the reply
// would be posted into the channel itself.
let _ = reply(ctx.sender, message.chat.id.0, message.id, GROUP_LINK_HINT).await;
}
respond(())
}
/// Answer for a link posted where the pipeline does not run (a group): links
/// are private-chat only, inline mode is the group path.
const GROUP_LINK_HINT: &str =
"Links are handled in private chat only — send me this link there, or use inline mode here.";
/// Groups and supergroups, as opposed to private chats and channels.
fn is_group(kind: &ChatKind) -> bool {
matches!(
kind,
ChatKind::Public(chat)
if matches!(
chat.kind,
PublicChatKind::Group | PublicChatKind::Supergroup(_)
)
)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ctx::test_support::{FORWARDED_ID, PROMPT_ID, TestStores, api_error, seed_prompt};
use crate::media_sender::test_support::{MockSender, Outcome};
use teloxide::RequestError;
/// The Telegram wording the mocks answer with: a message the bot cannot
/// edit (the prompt was deleted).
const API_ERROR: &str = "Bad Request: message not found";
#[tokio::test]
async fn reply_to_a_prompt_swaps_the_caption_through_its_template() {
let sender = MockSender::scripted(vec![Outcome::EditOk], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "tpl", crate::db::unix_now()).await;
let consumed = edit_message_handler(&ctx, 1, PROMPT_ID, "new caption").await;
assert!(consumed, "a reply to the prompt must be consumed");
assert_eq!(
sender.captions(),
vec!["<b><a href=\"https://x.com/u/status/1\">new caption</a></b>"]
);
}
#[tokio::test]
async fn reply_text_and_url_are_escaped_into_the_caption() {
let sender = MockSender::scripted(vec![Outcome::EditOk], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
edit_message_handler(&ctx, 1, PROMPT_ID, "<script>alert(1)</script>").await;
// No raw markup from user text may reach the HTML caption.
assert_eq!(
sender.captions(),
vec!["<a href=\"https://x.com/u/status/1\">&lt;script&gt;alert(1)&lt;/script&gt;</a>"]
);
}
#[tokio::test]
async fn a_failed_caption_swap_is_reported_and_consumed() {
// The script is per call, in order: the edit fails, the notice follows.
let sender = MockSender::scripted(vec![Outcome::EditErr, Outcome::MessageOk], || {
api_error(API_ERROR)
});
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "tpl", crate::db::unix_now()).await;
// The edit failed (message deleted etc.); the reply must still be
// swallowed instead of being treated as a link to fetch — and the user
// must be told, because the text they sent is gone either way.
assert!(edit_message_handler(&ctx, 1, PROMPT_ID, "new caption").await);
assert_eq!(sender.calls(), vec!["edit_message_caption", "send_message"]);
let notice = sender.messages().join(" ");
assert!(notice.contains("Could not update the caption"), "{notice}");
}
#[tokio::test(start_paused = true)]
async fn a_short_retryable_caption_failure_is_retried_once() {
use teloxide::types::Seconds;
// A one-second flood-control wait is worth honouring: the retry lands
// and the user never hears about it.
let sender = MockSender::scripted(vec![Outcome::EditErr, Outcome::EditOk], || {
RequestError::RetryAfter(Seconds::from_seconds(1))
});
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "tpl", crate::db::unix_now()).await;
assert!(edit_message_handler(&ctx, 1, PROMPT_ID, "new caption").await);
assert_eq!(
sender.calls(),
vec!["edit_message_caption", "edit_message_caption"]
);
}
#[tokio::test(start_paused = true)]
async fn a_long_retryable_caption_failure_is_not_retried() {
use teloxide::types::Seconds;
// A minute-long wait must not stall the chat's update queue behind it:
// the user is told to send the caption again instead.
let sender = MockSender::scripted(vec![Outcome::EditErr, Outcome::MessageOk], || {
RequestError::RetryAfter(Seconds::from_seconds(60))
});
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "tpl", crate::db::unix_now()).await;
assert!(edit_message_handler(&ctx, 1, PROMPT_ID, "new caption").await);
assert_eq!(sender.calls(), vec!["edit_message_caption", "send_message"]);
}
#[tokio::test]
async fn reply_to_an_unrelated_message_is_not_consumed() {
let sender = MockSender::scripted(vec![], || api_error(API_ERROR));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
// No prompt record for that message id → the reply runs the normal
// (URL/command) path instead.
assert!(!edit_message_handler(&ctx, 1, PROMPT_ID, "hello").await);
assert!(sender.calls().is_empty());
}
/// A reply driven through the real message entry point into a real `Bot`:
/// the routing (reply-to-prompt → caption swap, before the command and URL
/// branches) and the request teloxide builds.
#[tokio::test]
async fn a_prompt_reply_reaches_the_api_as_a_caption_edit() {
use crate::media_sender::test_support::fake_api::FakeApi;
use teloxide::Bot;
let api = FakeApi::start().await;
let bot = Bot::new("42:TEST").set_api_url(api.url());
let stores = TestStores::new();
let ctx = stores.ctx(&bot);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
let message: Message = serde_json::from_value(serde_json::json!({
"message_id": PROMPT_ID + 1,
"date": 0,
"chat": { "id": 1, "type": "private" },
"from": { "id": 5, "is_bot": false, "first_name": "u" },
"reply_to_message": {
"message_id": PROMPT_ID,
"date": 0,
"chat": { "id": 1, "type": "private" },
"text": "prompt",
},
"text": "new caption",
}))
.expect("a minimal message deserializes");
handle_message(&ctx, &bot, message).await.unwrap();
assert_eq!(api.methods(), vec!["EditMessageCaption"]);
let body = api.body("EditMessageCaption");
assert_eq!(body["chat_id"], 1);
assert_eq!(body["message_id"], FORWARDED_ID);
assert_eq!(
body["caption"],
"<a href=\"https://x.com/u/status/1\">new caption</a>"
);
// The other branch of the same entry point: a supported link in a group
// gets the one explanatory reply (the link pipeline is private-chat only,
// and dropping it in silence reads as a broken bot).
let group: Message = serde_json::from_value(serde_json::json!({
"message_id": 2,
"date": 0,
"chat": { "id": -100, "type": "group", "title": "g" },
"from": { "id": 5, "is_bot": false, "first_name": "u" },
"text": "https://x.com/u/status/1",
"entities": [{ "type": "url", "offset": 0, "length": 24 }],
}))
.expect("a minimal group message deserializes");
handle_message(&ctx, &bot, group).await.unwrap();
assert_eq!(api.methods(), vec!["EditMessageCaption", "SendMessage"]);
assert_eq!(api.body("SendMessage")["text"], GROUP_LINK_HINT);
}
#[test]
fn the_link_hint_is_for_groups_only() {
use teloxide::types::{ChatPrivate, ChatPublic, PublicChatChannel, PublicChatSupergroup};
let group = ChatKind::Public(ChatPublic {
title: None,
kind: PublicChatKind::Group,
});
let supergroup = ChatKind::Public(ChatPublic {
title: None,
kind: PublicChatKind::Supergroup(PublicChatSupergroup {
username: None,
is_forum: false,
}),
});
// A channel must stay silent: the hint reply would be posted into the
// channel itself.
let channel = ChatKind::Public(ChatPublic {
title: None,
kind: PublicChatKind::Channel(PublicChatChannel { username: None }),
});
let private = ChatKind::Private(ChatPrivate {
username: None,
first_name: None,
last_name: None,
});
assert!(is_group(&group));
assert!(is_group(&supergroup));
assert!(!is_group(&channel));
assert!(!is_group(&private));
}
}
+3 -2
View File
@@ -23,8 +23,9 @@ static DB: LazyLock<Arc<db::DbPool>> = LazyLock::new(|| {
/// create parent dirs, so the old hardcoded `data/task_queue.db` failed with
/// a confusing error when started from a directory without `data/`, and a
/// CWD-relative path is a footgun for systemd / cron deployments — `DATA_DIR`
/// lets them pin the state anywhere.
fn db_path() -> std::path::PathBuf {
/// lets them pin the state anywhere. Also read by the startup config line, so
/// the log says where the state actually landed.
pub(crate) fn db_path() -> std::path::PathBuf {
let dir = std::env::var("DATA_DIR").unwrap_or_else(|_| "data".to_string());
let dir_path = std::path::Path::new(&dir);
std::fs::create_dir_all(dir_path).expect("failed to create data directory");
File diff suppressed because it is too large Load Diff
+101 -29
View File
@@ -5,7 +5,7 @@
//! key]. A repeated link is then answered entirely from local state — no
//! re-fetch of the source site, no re-upload — and no media file is stored
//! on disk (the file ids point at Telegram's servers). Entries expire after
//! [`Config::link_cache_ttl`]; a stale entry is dropped lazily on read and
//! `Config::link_cache_ttl`; a stale entry is dropped lazily on read and
//! by the periodic prune in `main`.
use crate::db::now_f64;
@@ -26,6 +26,12 @@ pub enum CachedMediaKind {
pub struct CachedMedia {
pub kind: CachedMediaKind,
pub file_id: String,
/// The media URL the send used, kept so an entry whose file ids stopped
/// working can still be re-sent without touching the source site (see the
/// bot's `invalidate_cache`). Empty for entries written before this field
/// existed — those can only be dropped and re-fetched.
#[serde(default)]
pub url: String,
}
/// Everything needed to re-send a post without touching the source site:
@@ -38,6 +44,10 @@ pub struct CachedPost {
/// override).
pub caption: String,
pub title: String,
/// The post's body text. Defaulted on read: entries written before the
/// title/content split carry it inside `title`.
#[serde(default)]
pub content: String,
pub author: String,
pub author_url: String,
pub tags: String,
@@ -78,15 +88,21 @@ impl LinkCache {
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
return Ok(None);
}
Ok(Some(serde_json::from_str::<CachedPost>(&payload).map_err(
|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)),
)?))
match serde_json::from_str::<CachedPost>(&payload) {
Ok(post) => Ok(Some(post)),
Err(e) => {
// Unreadable payload (e.g. an older schema): drop it
// instead of re-failing the parse on every later hit.
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
Err(rusqlite::Error::ToSqlConversionFailure(Box::new(e)))
}
}
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::error!("link cache read failed: {e}");
log::warn!("link cache read failed: {e}");
None
}
}
@@ -106,7 +122,7 @@ impl LinkCache {
})
.await;
if let Err(e) = result {
log::error!("link cache write failed: {e}");
log::warn!("link cache write failed: {e}");
}
}
@@ -121,7 +137,7 @@ impl LinkCache {
})
.await;
if let Err(e) = result {
log::error!("link cache delete failed: {e}");
log::warn!("link cache delete failed: {e}");
}
}
@@ -140,7 +156,7 @@ impl LinkCache {
match result {
Ok(n) => n,
Err(e) => {
log::error!("link cache prune failed: {e}");
log::warn!("link cache prune failed: {e}");
0
}
}
@@ -160,7 +176,7 @@ impl LinkCache {
match result {
Ok(n) => n,
Err(e) => {
log::error!("link cache clear failed: {e}");
log::warn!("link cache clear failed: {e}");
0
}
}
@@ -170,21 +186,49 @@ impl LinkCache {
#[cfg(test)]
mod tests {
use super::*;
use crate::ctx::test_support::cached_photo;
fn entry() -> CachedPost {
CachedPost {
url: "https://x.com/u/status/1".into(),
caption: "cap".into(),
title: "t".into(),
author: "a".into(),
author_url: "au".into(),
tags: "".into(),
sensitive: true,
media: vec![CachedMedia {
kind: CachedMediaKind::Photo,
file_id: "AgAC...".into(),
}],
/// A payload written before the title/content split has no `content`
/// field. It must still read back — the cache deletes what it cannot
/// parse — with its text left where it was stored (`title`) and the
/// caption it replays untouched. No migration: a self-hosted cache entry
/// lives one TTL, and moving the text would only reshuffle `/set_format`
/// placeholders until it expires.
#[tokio::test]
async fn pre_split_entry_still_parses() {
let dir = tempfile::tempdir().unwrap();
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
let legacy = serde_json::json!({
"url": "https://x.com/u/status/1",
"caption": "https://x.com/u/status/1\n<a href=\"au\">a</a>: old text",
"title": "old text",
"author": "a",
"author_url": "au",
"tags": "",
"sensitive": false,
"media": [{"kind": "photo", "file_id": "AgAC..."}]
});
{
let conn = rusqlite::Connection::open(dir.path().join("c.db")).unwrap();
conn.execute(
"INSERT INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
params!["twitter:1", legacy.to_string(), now_f64()],
)
.unwrap();
}
let got = cache
.get("twitter:1", Duration::from_secs(3600))
.await
.expect("a pre-split payload must not be dropped");
assert_eq!(got.title, "old text");
assert_eq!(got.content, "");
assert_eq!(
got.caption,
"https://x.com/u/status/1\n<a href=\"au\">a</a>: old text"
);
}
#[tokio::test]
@@ -193,12 +237,14 @@ mod tests {
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
cache.put("twitter:1", &entry()).await;
cache.put("twitter:1", &cached_photo()).await;
let got = cache.get("twitter:1", Duration::from_secs(3600)).await;
assert!(got.is_some());
let got = got.unwrap();
assert_eq!(got.url, "https://x.com/u/status/1");
assert_eq!(got.media[0].file_id, "AgAC...");
assert_eq!(got.media[0].file_id, "AgAC-file-id");
// The source URL rides along: it is what a degraded entry falls back to.
assert_eq!(got.media[0].url, "https://pbs.twimg.com/media/photo.jpg");
}
#[tokio::test]
@@ -207,7 +253,7 @@ mod tests {
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
cache.put("twitter:1", &entry()).await;
cache.put("twitter:1", &cached_photo()).await;
// Force the row into the past so a 1s TTL expires it.
{
let conn = rusqlite::Connection::open(dir.path().join("c.db")).unwrap();
@@ -228,14 +274,40 @@ mod tests {
);
}
#[tokio::test]
async fn unreadable_entry_is_dropped_on_read() {
// A payload from an older schema must not be re-parsed on every hit:
// the row is removed and the read reports a miss.
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("c.db");
let cache = LinkCache::new(crate::db::open_store(db_path.to_str().unwrap()).unwrap());
{
let conn = rusqlite::Connection::open(&db_path).unwrap();
conn.execute(
"INSERT INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
params!["twitter:1", "{not json", now_f64()],
)
.unwrap();
}
assert!(
cache
.get("twitter:1", Duration::from_secs(3600))
.await
.is_none()
);
// Dropped, not left behind for the next hit.
assert_eq!(cache.clear(None).await, 0, "corrupted row still present");
}
#[tokio::test]
async fn remove_and_prune() {
let dir = tempfile::tempdir().unwrap();
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
cache.put("twitter:1", &entry()).await;
cache.put("pixiv:2", &entry()).await;
cache.put("twitter:1", &cached_photo()).await;
cache.put("pixiv:2", &cached_photo()).await;
cache.remove("twitter:1").await;
assert!(
cache
@@ -269,8 +341,8 @@ mod tests {
let cache = LinkCache::new(
crate::db::open_store(dir.path().join("c.db").to_str().unwrap()).unwrap(),
);
cache.put("twitter:1", &entry()).await;
cache.put("pixiv:2", &entry()).await;
cache.put("twitter:1", &cached_photo()).await;
cache.put("pixiv:2", &cached_photo()).await;
// By key: only the matching row is removed.
assert_eq!(cache.clear(Some("twitter:1")).await, 1);
assert!(
+344 -33
View File
@@ -1,4 +1,5 @@
use dotenv::dotenv;
use std::time::Duration;
use teloxide::dptree::endpoint;
use teloxide::prelude::*;
use teloxide::stop::StopToken;
@@ -8,6 +9,7 @@ use tokio::sync::watch;
use x_media::site;
mod config;
mod ctx;
mod db;
mod handlers;
mod link_cache;
@@ -18,6 +20,7 @@ mod rate_limit;
mod send;
mod state;
use ctx::CONTEXT;
use handlers::{CHAT_STORE, CONFIG, LINK_CACHE, TASK_QUEUE};
/// Docker `stop` / `compose down` delivers SIGTERM, which teloxide's ctrlc
@@ -38,12 +41,85 @@ fn spawn_sigterm_handler(stop_token: StopToken) {
#[cfg(not(unix))]
fn spawn_sigterm_handler(_stop_token: StopToken) {}
/// A leftover temp file must be at least this old before the startup sweep
/// touches it. Orphans come from a *previous* run; anything younger could
/// belong to a second instance sharing the temp directory (a misconfiguration,
/// but one that must not cost it its in-flight download).
const ORPHAN_TEMP_AGE: Duration = Duration::from_secs(3600);
/// Removes this project's own leftover temp entries (`x_media::TEMP_FILE_PREFIX`)
/// from `dir` once they are older than `older_than`. Returns how many were
/// removed. Entries that are not ours, or are too young, or cannot be dated,
/// are left alone: the OS temp directory is shared, and the marker prefix plus
/// the age gate are the only two things that make deleting here safe.
fn sweep_temp_dir(dir: &std::path::Path, older_than: Duration) -> usize {
let Ok(entries) = std::fs::read_dir(dir) else {
return 0;
};
let cutoff = std::time::SystemTime::now() - older_than;
let mut removed = 0;
for entry in entries.flatten() {
let name = entry.file_name();
if !name
.to_string_lossy()
.starts_with(x_media::TEMP_FILE_PREFIX)
{
continue;
}
let old_enough = entry
.metadata()
.and_then(|meta| meta.modified())
.is_ok_and(|modified| modified < cutoff);
if !old_enough {
continue;
}
let path = entry.path();
let result = if entry.file_type().is_ok_and(|kind| kind.is_dir()) {
std::fs::remove_dir_all(&path)
} else {
std::fs::remove_file(&path)
};
match result {
Ok(()) => removed += 1,
// Not worth a warning per entry: a file another process removed
// first (or one we may not delete) is not a problem here.
Err(e) => log::debug!("could not remove orphaned temp entry {path:?}: {e}"),
}
}
removed
}
#[tokio::main]
async fn main() {
dotenv().ok();
pretty_env_logger::init();
// Without RUST_LOG nothing at all was logged (env_logger falls back to
// `error`), so a deployment that forgot the variable looked like a bot
// with no logs; and at `debug` the HTTP client's own lines (hyper_util,
// reqwest) outnumbered the bot's by two to one. The timed builder adds
// the timestamp the plain `init` omitted, so a line can be compared with
// a user's report. An explicit RUST_LOG still wins outright — but a blank
// one (`RUST_LOG=` in `.env`, which is not "unset") must not silence the
// log the way its absence used to.
let filter = std::env::var("RUST_LOG")
.ok()
.filter(|value| !value.trim().is_empty())
.unwrap_or_else(|| "info,hyper_util=warn,reqwest=warn".to_string());
pretty_env_logger::formatted_timed_builder()
.parse_filters(&filter)
.init();
log::info!("Starting bot");
// Temp media (downloaded files, ugoira/remux dirs) is cleaned up by
// `TempDir`/`NamedTempFile` on drop — which a killed process never runs.
// Without this sweep every hard restart left its downloads behind (up to
// hundreds of MB each) and nothing could tell them apart from a live
// process's files or from anything else in the OS temp dir. See
// [`sweep_temp_dir`] for why the age gate makes that safe.
let orphans = sweep_temp_dir(&std::env::temp_dir(), ORPHAN_TEMP_AGE);
if orphans > 0 {
log::info!("swept {orphans} orphaned temp file(s) from a previous run");
}
let bot = Bot::from_env();
// Force the queue workers' shared Bot to initialize now so a missing
// token fails at startup, not on the first queued task.
@@ -54,16 +130,46 @@ async fn main() {
log::warn!("failed to register commands: {e}");
}
// The effective tunables, so an operator can see what the process actually
// resolved (a mistyped DATA_DIR or a forgotten TTL override is otherwise
// invisible until it bites). The proxy URL is never printed — it may embed
// credentials — and admin ids are chat identifiers, so they stay at debug.
let quote_chars = match CONFIG.caption_quote_text_chars {
0 => "off".to_string(),
n => format!("{n} chars"),
};
log::info!(
"config: {} admin(s), edit-message TTL {}s",
"config: {} admin(s), state {}, edit-message TTL {}s, link cache TTL {}s, caption quote {quote_chars}, proxy={}",
CONFIG.admin_ids.len(),
CONFIG.edit_message_ttl.as_secs()
crate::handlers::db_path().display(),
CONFIG.edit_message_ttl.as_secs(),
CONFIG.link_cache_ttl.as_secs(),
if std::env::var("TELOXIDE_PROXY").is_ok() {
"yes"
} else {
"no"
}
);
log::debug!("config: admin ids {:?}", CONFIG.admin_ids);
// Startup repair, before any worker runs: a queued retry whose media was a
// local file (ugoira MP4, bsky remux, a downloaded temp file) can never
// succeed after a restart — the registry that kept those files alive is in
// memory — so those rows are re-fetched from their post instead of
// dead-lettering the user's link.
let repaired = handlers::repair_lost_local_media(&CONTEXT).await;
if repaired > 0 {
log::info!("startup repair: re-fetched {repaired} queued task(s)");
}
// Queue worker: handles typed tasks, dead-letters failed sends to the
// task's chat.
// task's chat. Both closures use the shared context (the queue requires
// 'static handlers, and the statics are process-wide anyway).
TASK_QUEUE
.start(send::handle_task, send::dead_letter_notify)
.start(
|payload| send::handle_task(&CONTEXT, payload),
|payload, message| send::dead_letter_notify(&CONTEXT, payload, message),
)
.await;
log::info!("task queue worker started");
@@ -87,41 +193,25 @@ async fn main() {
}
}
// Edit-expiry sweep: clears the prompt's buttons once the record expires.
// Background sweep: expires the edit prompts and prunes what has aged out.
log::info!(
"edit-expiry sweep: every 300s, ttl {}",
"edit-expiry sweep: every {}s, ttl {}",
SWEEP_INTERVAL.as_secs(),
CONFIG.edit_message_ttl.as_secs()
);
let (stop_tx, stop_rx) = watch::channel(false);
{
let bot = bot.clone();
let mut stop_rx = stop_rx;
tokio::spawn(async move {
loop {
tokio::select! {
_ = stop_rx.changed() => break,
_ = tokio::time::sleep(std::time::Duration::from_secs(300)) => {}
}
let ttl = CONFIG.edit_message_ttl;
let removed = CHAT_STORE.prune_expired(ttl).await;
let pruned = LINK_CACHE.prune(CONFIG.link_cache_ttl).await;
if pruned > 0 {
log::info!("link cache: pruned {pruned} expired entr(ies)");
}
for (chat_id, prompt_message_id) in removed {
// If the prompt was already deleted, this fails with a
// 400 "message to edit not found" — log and ignore.
if let Err(e) = bot
.edit_message_reply_markup(
ChatId(chat_id),
MessageId(prompt_message_id as i32),
)
.await
{
log::info!("edit-expiry sweep: prompt message gone: {e}");
}
}
}
periodic_sweep(
&bot,
&CHAT_STORE,
&LINK_CACHE,
&TASK_QUEUE,
&CONFIG,
stop_rx,
)
.await;
});
}
@@ -201,3 +291,224 @@ async fn main() {
log::info!("Bot stopped");
}
}
/// How often [`periodic_sweep`] runs.
const SWEEP_INTERVAL: Duration = Duration::from_secs(300);
/// The background sweep: rewrites the expired edit prompts in place, prunes the
/// link cache, the idle rate-limit buckets and the idle inline-query entries,
/// and reports the queue only when it is not empty.
///
/// Takes its collaborators instead of reaching for the statics so a test can
/// drive a tick with a paused clock: a sleeping task nothing drives is how the
/// queue's own sweep kept a missing worker wake-up.
async fn periodic_sweep(
sender: &dyn crate::media_sender::MediaSender,
chat_store: &crate::state::ChatStore,
link_cache: &crate::link_cache::LinkCache,
task_queue: &crate::queue::PersistentTaskQueue,
config: &crate::config::Config,
mut stop: watch::Receiver<bool>,
) {
loop {
tokio::select! {
_ = stop.changed() => break,
_ = tokio::time::sleep(SWEEP_INTERVAL) => {}
}
let removed = chat_store.prune_expired(config.edit_message_ttl).await;
let pruned = link_cache.prune(config.link_cache_ttl).await;
if pruned > 0 {
log::info!("link cache: pruned {pruned} expired entr(ies)");
}
let idle_limiters = crate::rate_limit::prune_idle();
if idle_limiters > 0 {
log::debug!("rate limiter: dropped {idle_limiters} idle bucket(s)");
}
// Entries past Telegram's own inline cache window: a repeat is sent to
// the bot again anyway, so keeping them would suppress a fetch the user
// is waiting for (and the map grew one entry per user, forever).
let idle_inline = handlers::prune_idle_states();
if idle_inline > 0 {
log::debug!("inline queries: dropped {idle_inline} idle entry(ies)");
}
// Only speaks up when the queue is not empty: a healthy bot has nothing
// to report, and a periodic "0 pending" line is noise that hides the
// lines that matter.
if let Some((pending, oldest_run_after)) = task_queue.pending_backlog().await {
let overdue = crate::db::now_f64() - oldest_run_after;
if overdue >= 0.0 {
log::info!("queue: {pending} pending task(s), oldest {overdue:.0}s overdue");
} else {
log::info!(
"queue: {pending} pending task(s), oldest retry in {:.0}s",
-overdue
);
}
}
for (chat_id, prompt_message_id) in removed {
// Rewritten in place, not announced: the sweep is a background
// timer, and a fresh message would wake the chat up to a full TTL
// later about a prompt the user already walked away from. The edit
// drops the buttons too. If the prompt was already deleted this
// fails with a 400 "message to edit not found" — log and ignore.
if let Err(e) = sender
.edit_message_text(
ChatId(chat_id),
MessageId(prompt_message_id as i32),
send::EDIT_PROMPT_EXPIRED_TEXT.to_string(),
)
.await
{
log::info!("edit-expiry sweep: prompt message gone: {e}");
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn sweep_removes_only_our_old_temp_entries() {
let dir = tempfile::tempdir().unwrap();
let old = std::time::SystemTime::now() - Duration::from_secs(7200);
let make = |name: &str, aged: bool| {
let path = dir.path().join(name);
std::fs::write(&path, b"x").unwrap();
if aged {
let file = std::fs::File::options().write(true).open(&path).unwrap();
file.set_modified(old).unwrap();
}
path
};
let ours_old = make(&format!("{}photo-old.jpg", x_media::TEMP_FILE_PREFIX), true);
let ours_fresh = make(
&format!("{}photo-new.jpg", x_media::TEMP_FILE_PREFIX),
false,
);
let theirs = make("someone-elses-file", true);
assert_eq!(sweep_temp_dir(dir.path(), Duration::from_secs(3600)), 1);
assert!(!ours_old.exists(), "an old leftover of ours is removed");
assert!(ours_fresh.exists(), "a fresh file may belong to a live run");
assert!(
theirs.exists(),
"files without our prefix are never touched"
);
// A caller with no age gate also reaches the directory branch (aging a
// *directory* is not portable, so the gate is what the first half
// above proves): the fresh dir and file go, the unrelated file stays.
let leftover_dir = dir
.path()
.join(format!("{}ugoira", x_media::TEMP_FILE_PREFIX));
std::fs::create_dir(&leftover_dir).unwrap();
std::fs::write(leftover_dir.join("frame.png"), b"x").unwrap();
assert_eq!(sweep_temp_dir(dir.path(), Duration::ZERO), 2);
assert!(
!leftover_dir.exists(),
"leftover dirs go with their contents"
);
assert!(!ours_fresh.exists(), "no age gate: ours, however fresh");
assert!(theirs.exists());
}
/// The sweep's tick: an expired prompt is rewritten in place (buttons
/// dropped) while a live one is left alone. Driven through the loop's own
/// timer on a paused clock — the loop is what a hand-called helper would
/// leave untested, which is how the queue's sweep kept a missing wake-up.
#[tokio::test(start_paused = true)]
async fn the_sweep_expires_only_the_prompts_past_their_ttl() {
use crate::ctx::test_support::{
FORWARDED_ID, PROMPT_ID, TestStores, api_error, seed_prompt,
};
use crate::media_sender::test_support::MockSender;
use crate::state::EditMessage;
// The interval is pinned here because no assertion on the edits can see
// it: a shorter interval produces the same single edit (the record is
// gone after the first tick), and the paused clock can jump past the
// boundary while a tick's DB work is in flight.
assert_eq!(SWEEP_INTERVAL, Duration::from_secs(300));
let config = crate::config::Config::load();
let stores = TestStores::new();
let sender = MockSender::scripted(vec![], || {
api_error("Bad Request: message to edit not found")
});
let ctx = stores.ctx(&sender);
// Chat 1 holds a prompt past its ttl; chat 2 a live one.
let stale = crate::db::unix_now() - config.edit_message_ttl.as_secs() as i64 - 1;
seed_prompt(&ctx, "", stale).await;
stores
.chat_store()
.update(2, |data| {
data.edit_message.insert(
PROMPT_ID,
EditMessage {
url: "https://x.com/u/status/1".into(),
chat_id: 2,
forward_message_ids: vec![FORWARDED_ID],
template: String::new(),
created_at: crate::db::unix_now(),
},
);
})
.await;
let (stop_tx, stop_rx) = watch::channel(false);
let sweep = periodic_sweep(
&sender,
stores.chat_store(),
stores.link_cache(),
stores.task_queue(),
&config,
stop_rx,
);
tokio::pin!(sweep);
// One second short of the interval: nothing has been touched. The
// select is what polls the loop (a pinned future nobody awaits never
// runs), and the paused clock makes this the loop's own timer.
tokio::select! {
_ = &mut sweep => unreachable!("the sweep only returns on stop"),
_ = tokio::time::sleep(SWEEP_INTERVAL - Duration::from_secs(1)) => {}
}
assert!(
sender.edited_texts().is_empty(),
"the sweep ran before its interval"
);
// The second that crosses the interval: the tick fires.
tokio::select! {
_ = &mut sweep => unreachable!("the sweep only returns on stop"),
_ = tokio::time::sleep(Duration::from_secs(2)) => {}
}
assert_eq!(
sender.edited_texts(),
vec![(1, PROMPT_ID, send::EDIT_PROMPT_EXPIRED_TEXT.to_string())],
"exactly the expired prompt, rewritten in place"
);
assert!(
!ctx.chat_store
.get(1)
.await
.edit_message
.contains_key(&PROMPT_ID),
"the expired record is gone"
);
assert!(
ctx.chat_store
.get(2)
.await
.edit_message
.contains_key(&PROMPT_ID),
"a live prompt keeps its record and its buttons"
);
stop_tx.send(true).unwrap();
sweep.await;
}
}
+418 -16
View File
@@ -8,8 +8,8 @@ use teloxide::RequestError;
use teloxide::prelude::Requester;
use teloxide::prelude::*;
use teloxide::types::{
ChatAction, ChatId, InlineKeyboardMarkup, InputFile, InputMedia, Message, MessageId, ParseMode,
ReplyParameters,
CallbackQueryId, ChatAction, ChatId, InlineKeyboardMarkup, InputFile, InputMedia, Message,
MessageId, ParseMode, ReplyParameters,
};
/// Boxed, `Send` future returned by a [`MediaSender`] method (`async fn` in
@@ -48,14 +48,52 @@ pub trait MediaSender: Send + Sync {
) -> BoxFuture<'_, Result<Vec<MessageId>, RequestError>>;
/// Sends a plain text message, optionally replying to `reply_to` and
/// attaching `reply_markup`.
/// attaching `reply_markup`. Returns the sent message's id: the bot only
/// ever needs that (the edit-before-forward prompt's record is keyed by
/// it), and returning the whole `Message` would force every test mock to
/// construct one.
fn send_message(
&self,
chat_id: ChatId,
text: String,
reply_to: Option<MessageId>,
reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<Message, RequestError>>;
) -> BoxFuture<'_, Result<i64, RequestError>>;
/// Answers a callback query, optionally with a toast `text` shown to the
/// user who pressed the button.
fn answer_callback_query(
&self,
id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Rewrites a message's text and drops its inline keyboard: the
/// edit-expiry sweep rewriting a prompt whose record expired (a button left
/// behind could only answer "Expired").
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Rewrites a message's caption, always with HTML parse mode (every caller
/// in this bot renders escaped HTML: templates and edit-before-forward
/// links).
fn edit_message_caption(
&self,
chat_id: ChatId,
message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Deletes a message (the edit-before-forward prompt after a forward).
fn delete_message(
&self,
chat_id: ChatId,
message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>>;
/// Sets the chat's "typing / uploading …" indicator (cosmetic).
fn send_chat_action(
@@ -78,6 +116,9 @@ impl MediaSender for Bot {
crate::rate_limit::limiter_for(chat_id.0)
.acquire(items.len() as f64)
.await;
// Same spend against the bot-wide budget: a fan-out over chats is
// invisible to the per-chat buckets.
crate::rate_limit::acquire_global(items.len() as f64).await;
// `<Bot as Requester>::` disambiguates from this trait's same-named
// method (teloxide's API lives in the `Requester` trait).
<Bot as Requester>::send_media_group(self, chat_id, items)
@@ -96,6 +137,7 @@ impl MediaSender for Bot {
) -> BoxFuture<'a, Result<Message, RequestError>> {
Box::pin(async move {
crate::rate_limit::limiter_for(chat_id.0).acquire(1.0).await;
crate::rate_limit::acquire_global(1.0).await;
let mut request = <Bot as Requester>::send_animation(self, chat_id, file)
.caption(caption)
.parse_mode(ParseMode::Html)
@@ -119,6 +161,7 @@ impl MediaSender for Bot {
crate::rate_limit::limiter_for(to.0)
.acquire(ids.len() as f64)
.await;
crate::rate_limit::acquire_global(ids.len() as f64).await;
<Bot as Requester>::copy_messages(self, to, from, ids).await
})
}
@@ -129,7 +172,7 @@ impl MediaSender for Bot {
text: String,
reply_to: Option<MessageId>,
reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<Message, RequestError>> {
) -> BoxFuture<'_, Result<i64, RequestError>> {
Box::pin(async move {
let mut request = <Bot as Requester>::send_message(self, chat_id, text);
if let Some(reply_to) = reply_to {
@@ -139,7 +182,62 @@ impl MediaSender for Bot {
if let Some(markup) = reply_markup {
request = request.reply_markup(markup);
}
request.await
request.await.map(|message| message.id.0 as i64)
})
}
fn answer_callback_query(
&self,
id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
let mut request = <Bot as Requester>::answer_callback_query(self, id);
if let Some(text) = text {
request = request.text(text);
}
request.await.map(|_| ())
})
}
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
<Bot as Requester>::edit_message_text(self, chat_id, message_id, text)
.reply_markup(InlineKeyboardMarkup::default())
.await
.map(|_| ())
})
}
fn edit_message_caption(
&self,
chat_id: ChatId,
message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
<Bot as Requester>::edit_message_caption(self, chat_id, message_id)
.caption(caption)
.parse_mode(ParseMode::Html)
.await
.map(|_| ())
})
}
fn delete_message(
&self,
chat_id: ChatId,
message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
<Bot as Requester>::delete_message(self, chat_id, message_id)
.await
.map(|_| ())
})
}
@@ -162,7 +260,7 @@ impl MediaSender for Bot {
#[cfg(test)]
pub(crate) mod test_support {
use super::*;
use std::sync::Mutex;
use parking_lot::Mutex;
/// One scripted outcome, consumed front-to-back; the last entry repeats
/// for further calls of the same method kind.
@@ -176,19 +274,226 @@ pub(crate) mod test_support {
/// An error from `send_message` (replies are fire-and-forget, so an
/// error is fine for tests).
MessageErr,
/// A successful `send_message`, returning message id [`MockSender::SENT_ID`].
MessageOk,
EditOk,
EditErr,
}
/// Replays a script and records the method names that were called.
/// A stand-in for `api.telegram.org` for the tests that must drive a real
/// `Bot` — its request building, the per-chat limiter, the bot-wide budget
/// — which the scripted mock bypasses entirely. Records every call and
/// answers the smallest result each method needs.
pub(crate) mod fake_api {
use parking_lot::Mutex;
use std::sync::Arc;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
pub(crate) struct FakeApi {
url: url::Url,
calls: Arc<Mutex<Vec<(String, serde_json::Value)>>>,
server: tokio::task::JoinHandle<()>,
}
impl FakeApi {
/// Binds an ephemeral port and serves until dropped.
pub(crate) async fn start() -> FakeApi {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let calls = Arc::new(Mutex::new(Vec::new()));
let recorded = Arc::clone(&calls);
let server = tokio::spawn(async move {
while let Ok((mut socket, _)) = listener.accept().await {
let recorded = Arc::clone(&recorded);
tokio::spawn(async move {
let Some((method, body)) = read_request(&mut socket).await else {
return;
};
recorded.lock().push((method.clone(), body));
let payload = serde_json::json!({
"ok": true,
"result": canned_result(&method),
})
.to_string();
let response = format!(
"HTTP/1.1 200 OK\r\ncontent-type: application/json\r\n\
content-length: {}\r\nconnection: close\r\n\r\n{}",
payload.len(),
payload
);
let _ = socket.write_all(response.as_bytes()).await;
let _ = socket.flush().await;
});
}
});
FakeApi {
// Trailing slash: teloxide appends `bot<token>/<method>`.
url: url::Url::parse(&format!("http://{addr}/")).unwrap(),
calls,
server,
}
}
/// Where to point a `Bot`: `Bot::new(token).set_api_url(api.url())`.
pub(crate) fn url(&self) -> url::Url {
self.url.clone()
}
/// Method names in call order.
pub(crate) fn methods(&self) -> Vec<String> {
self.calls.lock().iter().map(|(m, _)| m.clone()).collect()
}
/// The JSON body of the first call to `method` (`Null` for a body
/// that is not JSON, i.e. a multipart upload).
pub(crate) fn body(&self, method: &str) -> serde_json::Value {
self.calls
.lock()
.iter()
.find(|(m, _)| m == method)
.map(|(_, body)| body.clone())
.unwrap_or(serde_json::Value::Null)
}
}
impl Drop for FakeApi {
fn drop(&mut self) {
self.server.abort();
}
}
/// The smallest result teloxide can deserialize for a method. The names
/// arrive as the payload type's own — `SendMediaGroup`, not
/// `sendMediaGroup`: teloxide builds the URL from that, and the Bot API
/// accepts the spelling.
fn canned_result(method: &str) -> serde_json::Value {
match method {
"CopyMessages" => serde_json::json!([{ "message_id": 11 }]),
"SendMediaGroup" => serde_json::json!([minimal_message()]),
"SendMessage" | "SendAnimation" | "EditMessageCaption" => minimal_message(),
_ => serde_json::Value::Bool(true),
}
}
fn minimal_message() -> serde_json::Value {
serde_json::json!({
"message_id": 1,
"date": 0,
"chat": { "id": 1, "type": "private" },
})
}
/// One HTTP/1.1 request: the head up to the blank line, then
/// `content-length` bytes of body — JSON for most methods, multipart
/// for the media ones (teloxide sends `SendMediaGroup` that way).
async fn read_request(socket: &mut TcpStream) -> Option<(String, serde_json::Value)> {
let mut buf = Vec::new();
let mut chunk = [0u8; 4096];
loop {
let n = socket.read(&mut chunk).await.ok()?;
if n == 0 {
return None;
}
buf.extend_from_slice(&chunk[..n]);
let Some(headers_end) = find(&buf, b"\r\n\r\n") else {
continue;
};
let head = String::from_utf8_lossy(&buf[..headers_end]).to_string();
let length: usize = head
.lines()
.find_map(|line| {
line.to_ascii_lowercase()
.strip_prefix("content-length:")
.and_then(|v| v.trim().parse().ok())
})
.unwrap_or(0);
let body_start = headers_end + 4;
while buf.len() < body_start + length {
let n = socket.read(&mut chunk).await.ok()?;
if n == 0 {
break;
}
buf.extend_from_slice(&chunk[..n]);
}
let method = head
.lines()
.next()
// `POST /bot<token>/<method>`
.and_then(|line| line.split(' ').nth(1))
.and_then(|path| path.rsplit('/').next())
.unwrap_or_default()
.to_string();
let body = parse_body(&buf[body_start..], &head);
return Some((method, body));
}
}
/// The request body as JSON: either the JSON body itself, or a
/// multipart form flattened into an object (each part's value parsed as
/// JSON when it is one, so `media` comes back as its array).
fn parse_body(body: &[u8], head: &str) -> serde_json::Value {
let content_type = head
.lines()
.find(|line| line.to_ascii_lowercase().starts_with("content-type:"))
.unwrap_or_default()
.to_ascii_lowercase();
let Some(boundary) = content_type
.split("boundary=")
.nth(1)
.map(|b| b.trim().trim_matches('"').to_string())
else {
return serde_json::from_slice(body).unwrap_or_default();
};
let text = String::from_utf8_lossy(body);
let mut fields = serde_json::Map::new();
for part in text.split(&format!("--{boundary}")).skip(1) {
let Some((part_head, value)) = part.split_once("\r\n\r\n") else {
continue;
};
let Some(name) = part_head
.split("name=\"")
.nth(1)
.and_then(|rest| rest.split('"').next())
else {
continue;
};
let value = value.trim_end_matches("\r\n");
fields.insert(
name.to_string(),
serde_json::from_str(value).unwrap_or_else(|_| value.into()),
);
}
serde_json::Value::Object(fields)
}
fn find(haystack: &[u8], needle: &[u8]) -> Option<usize> {
haystack
.windows(needle.len())
.position(|window| window == needle)
}
}
/// Replays a script and records what was sent, so tests can assert the
/// user-visible text a path produced.
pub(crate) struct MockSender {
script: Mutex<Vec<Outcome>>,
cursor: Mutex<usize>,
calls: Mutex<Vec<&'static str>>,
messages: Mutex<Vec<String>>,
captions: Mutex<Vec<String>>,
answers: Mutex<Vec<Option<String>>>,
/// `(chat, message, text)` of every text rewrite, in order.
edited_texts: Mutex<Vec<(i64, i64, String)>>,
/// Builds the error every `*Err` outcome returns (RequestError is not
/// cloneable, so the factory recreates it per call).
error: Box<dyn Fn() -> RequestError + Send + Sync>,
}
impl MockSender {
/// The message id a successful `send_message` reports.
pub(crate) const SENT_ID: i64 = 1;
pub(crate) fn scripted(
script: Vec<Outcome>,
error: impl Fn() -> RequestError + Send + Sync + 'static,
@@ -197,6 +502,10 @@ pub(crate) mod test_support {
script: Mutex::new(script),
cursor: Mutex::new(0),
calls: Mutex::new(Vec::new()),
messages: Mutex::new(Vec::new()),
captions: Mutex::new(Vec::new()),
answers: Mutex::new(Vec::new()),
edited_texts: Mutex::new(Vec::new()),
error: Box::new(error),
}
}
@@ -204,13 +513,33 @@ pub(crate) mod test_support {
/// Method names in call order (e.g. `["send_media_group",
/// "send_media_group"]` proves the fallback re-sent).
pub(crate) fn calls(&self) -> Vec<&'static str> {
self.calls.lock().unwrap().clone()
self.calls.lock().clone()
}
/// Texts of the plain messages sent, in order.
pub(crate) fn messages(&self) -> Vec<String> {
self.messages.lock().clone()
}
/// Captions passed to `edit_message_caption`, in order.
pub(crate) fn captions(&self) -> Vec<String> {
self.captions.lock().clone()
}
/// Toast texts of the answered callback queries, in order.
pub(crate) fn answers(&self) -> Vec<Option<String>> {
self.answers.lock().clone()
}
/// `(chat, message, text)` of every `edit_message_text`, in order.
pub(crate) fn edited_texts(&self) -> Vec<(i64, i64, String)> {
self.edited_texts.lock().clone()
}
fn next(&self, kind: &'static str) -> Outcome {
self.calls.lock().unwrap().push(kind);
let script = self.script.lock().unwrap();
let mut cursor = self.cursor.lock().unwrap();
self.calls.lock().push(kind);
let script = self.script.lock();
let mut cursor = self.cursor.lock();
if script.is_empty() {
panic!("mock script exhausted: {kind}");
}
@@ -229,9 +558,20 @@ pub(crate) mod test_support {
&self,
_chat_id: ChatId,
_reply_to: MessageId,
_items: Vec<InputMedia>,
items: Vec<InputMedia>,
) -> BoxFuture<'_, Result<Vec<Message>, RequestError>> {
Box::pin(async move {
// Record the captions exactly as Telegram receives them (only
// the first item of a group carries one), so tests can assert
// what a recipient sees.
self.captions
.lock()
.extend(items.iter().filter_map(|item| match item {
InputMedia::Photo(photo) => photo.caption.clone(),
InputMedia::Video(video) => video.caption.clone(),
InputMedia::Animation(animation) => animation.caption.clone(),
_ => None,
}));
match self.next("send_media_group") {
Outcome::GroupOk => Ok(Vec::new()),
Outcome::GroupErr => Err(self.error()),
@@ -274,25 +614,87 @@ pub(crate) mod test_support {
fn send_message(
&self,
_chat_id: ChatId,
_text: String,
text: String,
_reply_to: Option<MessageId>,
_reply_markup: Option<InlineKeyboardMarkup>,
) -> BoxFuture<'_, Result<Message, RequestError>> {
) -> BoxFuture<'_, Result<i64, RequestError>> {
Box::pin(async move {
self.messages.lock().push(text);
match self.next("send_message") {
Outcome::MessageOk => Ok(MockSender::SENT_ID),
Outcome::MessageErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for send_message"),
}
})
}
fn answer_callback_query(
&self,
_id: CallbackQueryId,
text: Option<String>,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Always succeeds: the toast is cosmetic, so the script stays
// focused on the outcomes a test cares about.
Box::pin(async move {
self.calls.lock().push("answer_callback_query");
self.answers.lock().push(text);
Ok(())
})
}
fn edit_message_text(
&self,
chat_id: ChatId,
message_id: MessageId,
text: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Always succeeds: the only caller is the expiry sweep, which
// tolerates a failure (a prompt the user already deleted), so the
// script stays free for the call the test is about.
Box::pin(async move {
self.calls.lock().push("edit_message_text");
self.edited_texts
.lock()
.push((chat_id.0, message_id.0 as i64, text));
Ok(())
})
}
fn edit_message_caption(
&self,
_chat_id: ChatId,
_message_id: MessageId,
caption: String,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
self.captions.lock().push(caption);
match self.next("edit_message_caption") {
Outcome::EditOk => Ok(()),
Outcome::EditErr => Err(self.error()),
other => panic!("unexpected outcome {other:?} for edit_message_caption"),
}
})
}
fn delete_message(
&self,
_chat_id: ChatId,
_message_id: MessageId,
) -> BoxFuture<'_, Result<(), RequestError>> {
// Deletion is fire-and-forget in every caller; always succeeds.
Box::pin(async move {
self.calls.lock().push("delete_message");
Ok(())
})
}
fn send_chat_action(
&self,
_chat_id: ChatId,
_action: ChatAction,
) -> BoxFuture<'_, Result<(), RequestError>> {
Box::pin(async move {
self.calls.lock().unwrap().push("send_chat_action");
self.calls.lock().push("send_chat_action");
Ok(())
})
}
+205 -7
View File
@@ -12,6 +12,7 @@
//! and 24-bit RGB have no alpha channel).
use std::io::Write;
use std::sync::LazyLock;
use fast_image_resize as fir;
use tempfile::NamedTempFile;
@@ -26,9 +27,100 @@ pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900;
/// to a smaller media URL instead.
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024;
/// Decode budget (bytes): a larger intermediate buffer is not worth the peak
/// memory; the photo degrades to the smaller URL instead. Also the cap for
/// downloading photos in the send fallback (they must be downloaded whole).
/// memory; the photo degrades to the smaller URL instead.
pub(crate) const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
/// Cap for *downloading* a photo in the send fallback, kept separate from the
/// decode budget above: the whole body is buffered before it is processed, once
/// per download slot in flight, while the decode budget is about a single
/// buffer. Telegram's upload cap is 10 MiB, so a photo this large can only be
/// sent after a downscale that its reduced variant serves just as well — over
/// the cap the item degrades to the smaller URL
/// (`FallbackError::MediaTooLarge`), it is never an error.
pub(crate) const MAX_PHOTO_DOWNLOAD_BYTES: u64 = 32 * 1024 * 1024;
/// Size of one memory-budget unit. Small enough that ordinary photos do not
/// queue behind each other, coarse enough that the semaphore is not a counter
/// per megabyte.
const MEMORY_UNIT_BYTES: u64 = 64 * 1024 * 1024;
/// Process-wide memory budget for photo preparation, in [`MEMORY_UNIT_BYTES`]
/// units: 512 MiB. `PREP_SLOTS` bounds how many items are prepared at once but
/// not how much memory they hold — one photo's decode buffer can be up to
/// [`MAX_DECODE_BYTES`] (512 MiB), and the guard that refuses a bigger one is
/// per photo, so six concurrent photos could peak near 3 GiB on a host sized
/// for a fraction of that. Each item charges what it actually holds (its
/// downloaded bytes plus the decode buffer its header predicts), so a 10-image
/// album of ordinary photos still runs several at a time while huge ones
/// serialize.
const MEMORY_UNITS: u32 = 8;
static MEMORY_BUDGET: LazyLock<std::sync::Arc<tokio::sync::Semaphore>> =
LazyLock::new(|| std::sync::Arc::new(tokio::sync::Semaphore::new(MEMORY_UNITS as usize)));
/// The buffer `w`×`h` needs in `channels` output channels — the one number the
/// per-photo guards and the reservation below both use, so they cannot drift.
fn decode_bytes(w: u32, h: u32, channels: usize) -> u64 {
(w as u64) * (h as u64) * channels as u64
}
/// Units to charge for `bytes`, clamped to the whole budget: an item must never
/// ask for more than exists, or it would wait for itself forever.
fn memory_units(bytes: u64) -> u32 {
bytes
.div_ceil(MEMORY_UNIT_BYTES)
.clamp(1, MEMORY_UNITS as u64) as u32
}
/// Reserves `bytes` of the preparation budget until the returned permit drops.
pub(crate) async fn reserve_memory(bytes: u64) -> tokio::sync::OwnedSemaphorePermit {
reserve(std::sync::Arc::clone(&MEMORY_BUDGET), bytes).await
}
/// [`reserve_memory`] against a caller-chosen budget; the tests pass their own
/// so they do not fight over the process-wide one.
async fn reserve(
budget: std::sync::Arc<tokio::sync::Semaphore>,
bytes: u64,
) -> tokio::sync::OwnedSemaphorePermit {
budget
.acquire_many_owned(memory_units(bytes))
.await
.expect("memory budget semaphore closed")
}
/// The decode buffer a downloaded photo will allocate, from its header alone —
/// zero when it is already within Telegram's limits and is uploaded as-is, zero
/// for a format [`prepare_photo`] does not decode. Mirrors the early return and
/// the guard of the two branches below.
pub(crate) fn decode_budget_bytes(bytes: &[u8]) -> u64 {
if let Some((w, h, _depth, color)) = parse_png_header(bytes) {
if within_limits(w, h, bytes) {
return 0;
}
return decode_bytes(w, h, output_channels(color));
}
if let Some((w, h)) = jpeg_dims(bytes) {
if within_limits(w, h, bytes) {
return 0;
}
return decode_bytes(w, h, 3);
}
0
}
/// Whether a photo is uploaded untouched (Telegram's dimension sum, and the
/// upload cap its bytes are compared against).
fn within_limits(w: u32, h: u32, bytes: &[u8]) -> bool {
w + h <= PHOTO_MAX_DIMENSION_SUM && bytes.len() as u64 <= MAX_UPLOAD_BYTES
}
/// JPEG dimensions from the headers, without decoding any pixels.
fn jpeg_dims(bytes: &[u8]) -> Option<(u32, u32)> {
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(bytes));
decoder.decode_headers().ok()?;
let info = decoder.info()?;
Some((info.width as u32, info.height as u32))
}
/// JPEG output quality (1-100).
const JPEG_QUALITY: u8 = 90;
@@ -199,6 +291,7 @@ fn encode_jpeg(pix: &PixBuf, w: u32, h: u32) -> Result<Vec<u8>, String> {
fn write_temp(bytes: &[u8], ext: &str) -> Result<NamedTempFile, String> {
let mut file = tempfile::Builder::new()
.prefix(x_media::TEMP_FILE_PREFIX)
.suffix(&format!(".{ext}"))
.tempfile()
.map_err(|e| format!("temp file failed: {e}"))?;
@@ -231,7 +324,7 @@ fn prepare_png(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String> {
);
let channels = output_channels(color_type);
if (w as u64) * (h as u64) * channels as u64 > MAX_DECODE_BYTES {
if decode_bytes(w, h, channels) > MAX_DECODE_BYTES {
log::warn!("photo decode buffer exceeds the memory budget; falling back to smaller media");
return Ok(PhotoPrep::UseFallback);
}
@@ -302,7 +395,7 @@ fn prepare_jpeg(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String>
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
return Ok(PhotoPrep::Upload(file));
}
if (w as u64) * (h as u64) * 3 > MAX_DECODE_BYTES {
if decode_bytes(w, h, 3) > MAX_DECODE_BYTES {
log::warn!("photo decode buffer exceeds the memory budget; falling back to smaller media");
return Ok(PhotoPrep::UseFallback);
}
@@ -326,6 +419,7 @@ fn prepare_jpeg(file: NamedTempFile, bytes: &[u8]) -> Result<PhotoPrep, String>
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
fn png_header(w: u32, h: u32, depth: u8, color: u8) -> Vec<u8> {
let mut bytes = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR".to_vec();
@@ -335,6 +429,100 @@ mod tests {
bytes
}
/// The budget is a *process-wide* memory bound: `PREP_SLOTS` (6) caps how
/// many photos are prepared at once, but six max-size photos would still
/// hold six decode buffers of up to 512 MiB each.
#[tokio::test]
async fn huge_decodes_cannot_overlap_but_do_run_alone() {
let budget = std::sync::Arc::new(tokio::sync::Semaphore::new(MEMORY_UNITS as usize));
let max_photo = MAX_DECODE_BYTES + MAX_PHOTO_DOWNLOAD_BYTES;
// One max-size photo fits (clamped to the whole budget), so it can
// never wait for budget that cannot exist.
let first = tokio::time::timeout(
Duration::from_millis(50),
reserve(budget.clone(), max_photo),
)
.await
.expect("a max-size photo must not wait");
// A second one of the same size has to wait for the first to finish.
assert!(
tokio::time::timeout(
Duration::from_millis(50),
reserve(budget.clone(), max_photo)
)
.await
.is_err(),
"two max-size decodes overlapped"
);
drop(first);
assert!(
tokio::time::timeout(
Duration::from_millis(50),
reserve(budget.clone(), max_photo)
)
.await
.is_ok(),
"the budget was not released"
);
}
/// A 10-image album of ordinary photos must not serialize: they charge
/// their real (small) buffers, not a fixed heavyweight slot.
#[tokio::test]
async fn ordinary_photos_share_the_budget() {
let budget = std::sync::Arc::new(tokio::sync::Semaphore::new(MEMORY_UNITS as usize));
// A 4 MiB photo that decodes to ~36 MiB (4000x3000 RGB).
let ordinary = 4 * 1024 * 1024 + 36 * 1024 * 1024;
let mut held = Vec::new();
for i in 0..MEMORY_UNITS {
held.push(
tokio::time::timeout(Duration::from_millis(50), reserve(budget.clone(), ordinary))
.await
.unwrap_or_else(|_| panic!("ordinary photo {i} waited for budget")),
);
}
}
#[test]
fn memory_units_round_up_and_clamp() {
assert_eq!(memory_units(1), 1);
assert_eq!(memory_units(MEMORY_UNIT_BYTES), 1);
assert_eq!(memory_units(MEMORY_UNIT_BYTES + 1), 2);
// Never more than exists, or the item waits for itself forever.
assert_eq!(memory_units(u64::MAX), MEMORY_UNITS);
// One item's worst case (a max download plus a max decode) takes the
// whole budget by itself.
assert_eq!(
memory_units(MAX_DECODE_BYTES + MAX_PHOTO_DOWNLOAD_BYTES),
MEMORY_UNITS
);
}
/// What the reservation is charged is decided by the header, and it has to
/// agree with what the pipeline does: a photo uploaded as-is costs nothing,
/// one that gets processed costs its decoded buffer.
#[test]
fn decode_budget_follows_the_processing_decision() {
// 9999x2 (sum 10001) is over the dimension cap → processed → charged.
let oversized = png_header(9999, 2, 8, 2); // 8-bit RGB
assert_eq!(decode_budget_bytes(&oversized), 9999 * 2 * 3);
// Inside the limits (dimensions *and* bytes) → uploaded as-is.
let small = png_header(100, 100, 8, 2);
assert_eq!(decode_budget_bytes(&small), 0);
// A format the pipeline does not decode costs nothing either.
assert_eq!(decode_budget_bytes(b"GIF89a not a photo"), 0);
// JPEG: 9999x2 is over the cap, so its RGB decode buffer is charged.
let (w, h) = (9999u16, 2u16);
let rgb = vec![90u8; w as usize * h as usize * 3];
let mut bytes = Vec::new();
jpeg_encoder::Encoder::new(&mut bytes, 90)
.encode(&rgb, w, h, jpeg_encoder::ColorType::Rgb)
.unwrap();
assert_eq!(decode_budget_bytes(&bytes), 9999 * 2 * 3);
}
#[test]
fn parses_png_header() {
let bytes = png_header(8979, 5316, 16, 6); // 16-bit RGBA
@@ -459,7 +647,10 @@ mod tests {
#[test]
fn pipeline_resizes_oversized_jpeg() {
// Build a small over-dimension JPEG with jpeg-encoder.
// Build a small over-dimension JPEG with jpeg-encoder: 9999x2 sums to
// one over the cap. The output's own headers are what must show the
// resize — a copy-through is a perfectly valid JPEG, so magic bytes
// and a non-empty buffer used to pass for nothing.
let (w, h) = (9999u16, 2u16);
let rgb = vec![90u8; (w as usize) * (h as usize) * 3];
let mut bytes = Vec::new();
@@ -475,8 +666,15 @@ mod tests {
PhotoPrep::Upload(file) => {
let out = std::fs::read(file.path()).unwrap();
assert!(out.starts_with(&[0xFF, 0xD8]), "output must stay jpeg");
// 9999x2 downscaled: the buffer length tells the new dims.
assert!(out.len() > 100);
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(out.as_slice()));
decoder.decode_headers().unwrap();
let info = decoder.info().unwrap();
let (nw, nh) = (info.width as u32, info.height as u32);
assert!(
nw + nh <= PHOTO_MAX_DIMENSION_SUM,
"still over the cap: {nw}x{nh}"
);
assert_ne!((nw, nh), (w as u32, h as u32), "output was not resized");
}
PhotoPrep::UseFallback => panic!("over-dimension JPEG should have been resized"),
}
+570 -73
View File
@@ -17,6 +17,17 @@ use tokio::sync::Notify;
use tokio::task::JoinHandle;
pub const MAX_RETRIES: u32 = 2;
/// Attempts for a *terminal* row write (delete / reschedule). These are not
/// like a task retry: failing them leaves the row in `in_progress`, where the
/// expiry sweep can re-run a task that already ran, so a contended DB gets a
/// few quick chances before the caller falls back to a terminal state.
const TERMINAL_WRITE_ATTEMPTS: u32 = 3;
/// 100ms, 200ms, … between terminal write attempts.
fn terminal_write_backoff(attempt: u32) -> Duration {
Duration::from_millis(100 * (1u64 << attempt.min(4)))
}
pub const LOCK_TTL_SECONDS: f64 = 120.0;
/// Number of concurrent worker loops. Tasks are independent (retries and
@@ -41,7 +52,13 @@ type DeadLetter = dyn Fn(Value, String) -> BoxFuture<'static, ()> + Send + Sync;
pub struct PersistentTaskQueue {
pool: std::sync::Arc<crate::db::DbPool>,
/// Wakes the workers when a row becomes leasable. `notify_one` stores a
/// permit, so nothing else may share it: a waiter that is not a worker
/// (the sweep) can consume the permit and leave the due row pending until
/// the next enqueue.
notify: Arc<Notify>,
/// Wakes the lease-expiry sweep; `stop` is the only producer.
sweep_notify: Arc<Notify>,
stop: Arc<AtomicBool>,
worker: Mutex<Vec<JoinHandle<()>>>,
counter: AtomicU64,
@@ -51,8 +68,19 @@ struct LeasedRow {
id: String,
payload: String,
attempts: i32,
/// Random token for *this* lease. Every write-back the worker makes is
/// guarded by it, so a lease that expired (heartbeat starved, host
/// suspended) and was re-leased by another worker cannot be written by
/// its former holder.
lease_token: String,
}
/// The row is no longer ours: its lease expired and another worker took it.
/// The former holder must not write anything back — a `delete` would erase the
/// new holder's row (or a `reschedule` would overwrite its retry state) — so
/// the attempt stops at the next heartbeat instead.
struct LeaseLost;
/// Owned worker state so the spawned loop does not borrow the queue handle.
#[derive(Clone)]
struct QueueWorker {
@@ -64,21 +92,39 @@ struct QueueWorker {
}
/// Resets rows left `in_progress` with an expired lock TTL back to `pending`
/// so they can be leased again (crash/panic recovery).
fn recover_update(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
/// so they can be leased again (crash/panic recovery). Returns how many rows
/// came back, which is what decides whether a worker needs waking.
fn recover_update(conn: &rusqlite::Connection) -> rusqlite::Result<usize> {
conn.execute(
"UPDATE tasks SET status='pending', locked_until=0 WHERE status='in_progress' AND locked_until < ?1",
params![now_f64()],
)?;
Ok(())
)
}
/// Runs [`recover_update`] and wakes a worker when something actually came
/// back. A recovered row is due immediately, but every worker may be parked on
/// `notify` — with no pending row there is no `earliest_run_after` to sleep on
/// — so without this the recovered task waits for the next unrelated enqueue.
/// Same permit semantics as `enqueue`: `notify_one` stores a permit when no
/// worker is registered.
async fn recover_expired(pool: &std::sync::Arc<crate::db::DbPool>, notify: &Notify) {
match pool.with_conn(move |conn| recover_update(conn)).await {
Ok(recovered) if recovered > 0 => {
log::warn!("queue: recovered {recovered} row(s) from an expired lease");
notify.notify_one();
}
Ok(_) => {}
Err(e) => log::error!("queue recovery failed: {e}"),
}
}
/// Base delay × 2^attempts (attempts = retries already done), capped at 300s.
/// Applied at the queue layer so the attempt count actually reaches the
/// backoff computation; Telegram `RetryAfter` delays get the same treatment
/// (conservatively larger wait, no API change needed).
/// backoff computation. The cap only ever scales *up*: a delay the server
/// asked for (Telegram `RetryAfter`) must not be shortened, retrying earlier
/// than allowed just re-triggers the flood control it came from.
fn scaled_retry_delay(base: f64, attempts: i32) -> f64 {
(base * 2f64.powi(attempts)).min(300.0)
(base * 2f64.powi(attempts)).min(300.0).max(base)
}
impl PersistentTaskQueue {
@@ -88,6 +134,7 @@ impl PersistentTaskQueue {
Self {
pool,
notify: Arc::new(Notify::new()),
sweep_notify: Arc::new(Notify::new()),
stop: Arc::new(AtomicBool::new(false)),
worker: Mutex::new(Vec::new()),
counter: AtomicU64::new(0),
@@ -106,7 +153,7 @@ impl PersistentTaskQueue {
let handler: Arc<Handler> = Arc::new(move |payload| Box::pin(handler(payload)));
let dead_letter: Arc<DeadLetter> =
Arc::new(move |payload, message| Box::pin(dead_letter(payload, message)));
self.recover_stale().await;
recover_expired(&self.pool, &self.notify).await;
let mut handles = Vec::with_capacity(QUEUE_WORKERS + 1);
for _ in 0..QUEUE_WORKERS {
let worker = QueueWorker {
@@ -119,13 +166,14 @@ impl PersistentTaskQueue {
handles.push(tokio::spawn(worker.run_loop_supervised()));
}
// Periodic lease-expiry sweep: recovers rows a crashed/panicked
// worker left `in_progress` (the lock TTL bounds the wait). Woken by
// the same notify as the workers, so enqueue and stop interrupt the
// sleep; the first interval tick fires immediately (harmless extra
// recovery at startup).
// worker left `in_progress` (the lock TTL bounds the wait). Its own
// notify (not the workers'): sharing that one let this task consume a
// `notify_one` permit meant for a worker, which then slept through a
// due row until some later event. Only `stop` wakes it.
let sweep_pool = std::sync::Arc::clone(&self.pool);
let sweep_notify = Arc::clone(&self.notify);
let sweep_notify = Arc::clone(&self.sweep_notify);
let sweep_stop = Arc::clone(&self.stop);
let sweep_workers = Arc::clone(&self.notify);
handles.push(tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(30));
loop {
@@ -138,10 +186,7 @@ impl PersistentTaskQueue {
if sweep_stop.load(Ordering::Relaxed) {
break;
}
let result = sweep_pool.with_conn(move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue sweep failed: {e}");
}
recover_expired(&sweep_pool, &sweep_workers).await;
}
}));
*self.worker.lock() = handles;
@@ -150,6 +195,7 @@ impl PersistentTaskQueue {
pub async fn stop(&self) {
self.stop.store(true, Ordering::Relaxed);
self.notify.notify_waiters();
self.sweep_notify.notify_waiters();
let handles = std::mem::take(&mut *self.worker.lock());
for handle in handles {
let _ = handle.await;
@@ -184,16 +230,140 @@ impl PersistentTaskQueue {
Ok(())
}
async fn recover_stale(&self) {
self.recover_sweep().await;
/// Pending task count and the oldest `run_after`, for the periodic sweep's
/// health line. Deliberately separate from the worker's own
/// `earliest_run_after`: that one runs on every idle worker cycle and must
/// stay a single indexed `MIN`, while the count is only asked for once per
/// sweep.
/// `(id, payload)` of every row that can still run (`pending`,
/// `in_progress`). The startup repair reads these before the workers start:
/// with no worker running, no row can be leased while it writes.
pub async fn runnable_rows(&self) -> Vec<(String, String)> {
let result = self
.pool
.with_conn(|conn| {
let mut stmt = conn.prepare(
"SELECT id, payload FROM tasks WHERE status IN ('pending', 'in_progress') ORDER BY run_after",
)?;
let rows = stmt.query_map([], |row| Ok((row.get(0)?, row.get(1)?)))?;
rows.collect::<rusqlite::Result<Vec<(String, String)>>>()
})
.await;
match result {
Ok(rows) => rows,
Err(e) => {
log::error!("queue row scan failed: {e}");
Vec::new()
}
}
}
async fn recover_sweep(&self) {
let result = self.pool.with_conn(move |conn| recover_update(conn)).await;
if let Err(e) = result {
log::error!("queue recovery failed: {e}");
/// Replaces a runnable row's payload and restarts its attempt budget: the
/// new payload is a fresh delivery of the same task, so the retries it has
/// already spent do not carry over. Startup repair only — a worker's
/// write-back is lease-token guarded instead (`replace_payload` cannot race
/// one: it runs before any worker does).
pub async fn replace_payload(&self, id: &str, payload: &Value) -> bool {
let logged_id = id.to_string();
let (id, payload) = (id.to_string(), payload.to_string());
let result = self
.pool
.with_conn(move |conn| {
let affected = conn.execute(
"UPDATE tasks SET payload=?1, attempts=0, run_after=?2, status='pending', locked_until=0 \
WHERE id=?3 AND status IN ('pending', 'in_progress')",
params![payload, now_f64(), id],
)?;
Ok(affected == 1)
})
.await;
match result {
Ok(true) => true,
Ok(false) => {
log::warn!("queue: row {logged_id} vanished before its payload could be replaced");
false
}
Err(e) => {
log::error!("queue payload replace failed for {logged_id}: {e}");
false
}
}
}
pub async fn pending_backlog(&self) -> Option<(i64, f64)> {
let result = self
.pool
.with_conn(|conn| {
let mut stmt = conn
.prepare("SELECT COUNT(*), MIN(run_after) FROM tasks WHERE status='pending'")?;
let mut rows = stmt.query([])?;
match rows.next()? {
Some(row) => {
let count = row.get::<_, i64>(0)?;
match row.get::<_, Option<f64>>(1)? {
Some(oldest) if count > 0 => Ok(Some((count, oldest))),
_ => Ok(None),
}
}
None => Ok(None),
}
})
.await;
match result {
Ok(v) => v,
Err(e) => {
log::error!("queue backlog query failed: {e}");
None
}
}
}
}
/// Last-resort terminal state for a row whose `DELETE` would not go through:
/// `done` is invisible to `lease_next` (`status='pending'`), to the expiry
/// sweep (`status='in_progress'`) and to the backlog line, so a task that
/// already ran cannot be leased and run again. Token-guarded like every other
/// write-back: `Ok(false)` means the row was re-leased and is not ours to
/// tombstone.
async fn mark_done(
pool: &std::sync::Arc<crate::db::DbPool>,
id: &str,
lease_token: &str,
) -> rusqlite::Result<bool> {
let id = id.to_string();
let lease_token = lease_token.to_string();
pool.with_conn(move |conn| {
let affected = conn.execute(
"UPDATE tasks SET status='done', locked_until=0 WHERE id = ?1 AND lease_token = ?2",
params![id, lease_token],
)?;
Ok(affected == 1)
})
.await
}
/// Which task a lease/retry/dead-letter line is about: the chat from the
/// stored payload, plus the post's normalized cache key when the payload
/// carries one (`ForwardMessages` has no source URL). Without these a queue
/// line named only a row id, which is useless to whoever reads the log — the
/// row id is assigned at insert time and appears nowhere else.
///
/// Built only when the line is actually logged (log arguments are lazy).
fn row_fields(payload: &Value) -> String {
let chat = payload
.get("chat_id")
.or_else(|| payload.get("from_chat_id"))
.and_then(Value::as_i64);
let key = payload
.get("source_url")
.and_then(Value::as_str)
.map(crate::handlers::log_key);
match (chat, key) {
(Some(chat), Some(key)) => format!("chat={chat} [key={key}]"),
(Some(chat), None) => format!("chat={chat}"),
(None, Some(key)) => format!("[key={key}]"),
(None, None) => String::new(),
}
}
impl QueueWorker {
@@ -271,15 +441,17 @@ impl QueueWorker {
}
Err(e) => return Err(e),
};
let lease_token = format!("{:016x}", rand::random::<u64>());
tx.execute(
"UPDATE tasks SET status='in_progress', locked_until=?1 WHERE id=?2",
params![now + LOCK_TTL_SECONDS, id],
"UPDATE tasks SET status='in_progress', locked_until=?1, lease_token=?2 WHERE id=?3",
params![now + LOCK_TTL_SECONDS, lease_token, id],
)?;
tx.commit()?;
Ok(Some(LeasedRow {
id,
payload,
attempts,
lease_token,
}))
})
.await
@@ -317,41 +489,72 @@ impl QueueWorker {
Ok(value) => value,
Err(e) => {
log::error!("queue: unparseable payload for {}: {e}", row.id);
self.delete_row(&row.id).await;
self.delete_row(&row.id, &row.lease_token).await;
(self.dead_letter)(Value::Null, format!("invalid stored payload: {e}")).await;
return;
}
};
log::debug!("processing {} (attempt {})", row.id, row.attempts + 1);
let outcome = self.run_with_lease(&row.id, payload).await;
let fields = row_fields(&payload);
log::debug!(
"processing {} {fields} (attempt {})",
row.id,
row.attempts + 1
);
let attempt_started = std::time::Instant::now();
let outcome = match self
.run_with_lease(&row.id, &row.lease_token, payload)
.await
{
Ok(outcome) => outcome,
Err(LeaseLost) => {
// Another worker owns this row now and is delivering the same
// task: write nothing (no delete, no reschedule, no
// dead-letter) and leave it to them.
log::warn!(
"queue: lost the lease on {} {fields} (attempt {}); abandoning this attempt",
row.id,
row.attempts + 1
);
return;
}
};
let attempt_ms = attempt_started.elapsed().as_millis();
match outcome {
Ok(()) => {
log::debug!("task {} completed", row.id);
self.delete_row(&row.id).await;
log::debug!("task {} {fields} completed in {attempt_ms}ms", row.id);
self.delete_row(&row.id, &row.lease_token).await;
}
Err(QueueError::Retryable {
delay_seconds,
payload,
}) => {
if row.attempts as u32 >= MAX_RETRIES {
let message = format!("task failed after {MAX_RETRIES} retries");
log::error!("dead-lettering {}: {message}", row.id);
self.delete_row(&row.id).await;
// The queue keeps only the payload, not the last error, so
// the cause of an exhausted retry is just that: exhausted.
// (The dead-letter message is read by the user, so it must
// not restate its own wrapper — see `failure_text`.)
let message = "retries exhausted".to_string();
log::error!(
"dead-lettering {} {fields}: {message} after {} attempt(s)",
row.id,
row.attempts + 1
);
self.delete_row(&row.id, &row.lease_token).await;
(self.dead_letter)(payload, message).await;
} else {
let delay = scaled_retry_delay(delay_seconds, row.attempts);
log::debug!(
"task {} rescheduled in {delay:.1}s (attempt {})",
"task {} {fields} attempt {} took {attempt_ms}ms, rescheduled in {delay:.1}s",
row.id,
row.attempts + 1
);
self.reschedule(&row.id, payload, delay, row.attempts + 1)
self.reschedule(&row.id, &row.lease_token, payload, delay, row.attempts + 1)
.await;
}
}
Err(QueueError::Permanent { message, payload }) => {
log::error!("dead-lettering {}: {message}", row.id);
self.delete_row(&row.id).await;
log::error!("dead-lettering {} {fields}: {message}", row.id);
self.delete_row(&row.id, &row.lease_token).await;
(self.dead_letter)(payload, message).await;
}
}
@@ -362,7 +565,12 @@ impl QueueWorker {
/// The heartbeat is part of this future, not a separate spawned task: if
/// the worker task dies (panic) the heartbeat dies with it and the sweep
/// recovers the row exactly as before.
async fn run_with_lease(&self, id: &str, payload: Value) -> Result<(), QueueError> {
async fn run_with_lease(
&self,
id: &str,
lease_token: &str,
payload: Value,
) -> Result<Result<(), QueueError>, LeaseLost> {
let fut = (self.handler)(payload);
tokio::pin!(fut);
let mut interval = tokio::time::interval(Duration::from_secs(30));
@@ -370,59 +578,152 @@ impl QueueWorker {
// just set by lease_next).
interval.tick().await;
let id_owned = id.to_string();
let token_owned = lease_token.to_string();
loop {
tokio::select! {
result = &mut fut => return result,
result = &mut fut => return Ok(result),
_ = interval.tick() => {
let now = now_f64();
let id = id_owned.clone();
let token = token_owned.clone();
let result = self
.pool
.with_conn(move |conn| {
conn.execute(
"UPDATE tasks SET locked_until=?1 WHERE id=?2 AND status='in_progress'",
params![now + LOCK_TTL_SECONDS, id],
"UPDATE tasks SET locked_until=?1 \
WHERE id=?2 AND status='in_progress' AND lease_token=?3",
params![now + LOCK_TTL_SECONDS, id, token],
)
})
.await;
if let Err(e) = result {
log::error!("queue lease heartbeat failed: {e}");
match result {
// Still ours: the lease is extended.
Ok(1) => {}
// The row is no longer leased to us (another worker
// re-leased it, or it is gone): dropping the handler
// future here stops this attempt instead of racing the
// new holder through the same send.
Ok(_) => return Err(LeaseLost),
Err(e) => log::error!("queue lease heartbeat failed: {e}"),
}
}
}
}
}
async fn delete_row(&self, id: &str) {
let id = id.to_string();
let result = self
.pool
.with_conn(move |conn| {
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
Ok(())
})
.await;
if let Err(e) = result {
log::error!("queue delete failed: {e}");
/// Deletes a finished row. A failure here is not cosmetic: the row would
/// stay `in_progress` with a live lease, the next sweep would flip it back
/// to `pending`, and the *completed* task would run again — a second album,
/// a second edit prompt, a second channel copy. So the delete is retried
/// (a busy/contended DB is the usual cause and clears), and if the DB still
/// refuses, the row is marked `done` — a status neither the lease query
/// (`pending`) nor the sweep (`in_progress`) looks at — so a task that
/// already ran can never be re-leased. Both writes failing is logged at
/// error level with the row id, since that is the one case where a
/// duplicate send stays possible.
async fn delete_row(&self, id: &str, lease_token: &str) {
for attempt in 0..TERMINAL_WRITE_ATTEMPTS {
match self.try_delete_row(id, lease_token).await {
Ok(true) => return,
// The row is not ours any more (re-leased while we worked):
// leaving it alone *is* the clean outcome — retrying or
// tombstoning here would erase the new holder's work.
Ok(false) => {
log::warn!("queue: row {id} was re-leased; not deleting it");
return;
}
Err(e) => {
log::error!("queue delete failed (attempt {}): {e}", attempt + 1);
tokio::time::sleep(terminal_write_backoff(attempt)).await;
}
}
}
match mark_done(&self.pool, id, lease_token).await {
Ok(true) => log::warn!("queue: row {id} marked done instead of deleted"),
Ok(false) => log::warn!("queue: row {id} was re-leased; nothing to tombstone"),
Err(e) => log::error!(
"queue: row {id} could not be deleted or marked done ({e}); \
the expiry sweep may run this finished task again"
),
}
}
async fn reschedule(&self, id: &str, payload: Value, delay_seconds: f64, attempts: i32) {
/// `Ok(false)` when the `WHERE` matched no row — the lease is not ours.
async fn try_delete_row(&self, id: &str, lease_token: &str) -> rusqlite::Result<bool> {
let id = id.to_string();
let lease_token = lease_token.to_string();
self.pool
.with_conn(move |conn| {
let affected = conn.execute(
"DELETE FROM tasks WHERE id = ?1 AND lease_token = ?2 AND status='in_progress'",
params![id, lease_token],
)?;
Ok(affected == 1)
})
.await
}
/// Writes back a retryable attempt's state. A failure is retried: the row
/// would otherwise stay `in_progress`, and the expiry sweep would re-run
/// the attempt from its *previous* payload — re-sending batches the last
/// attempt had already delivered. Unlike [`Self::delete_row`] there is no
/// safe terminal fallback here (marking it done would drop the retry
/// without telling anyone), so a persistent failure is logged loudly and
/// the sweep's re-run — at-least-once, the documented trade — is named.
async fn reschedule(
&self,
id: &str,
lease_token: &str,
payload: Value,
delay_seconds: f64,
attempts: i32,
) {
let row_id = id.to_string();
let lease_token = lease_token.to_string();
let payload = payload.to_string();
let result = self.pool.with_conn(move |conn| {
conn.execute(
"UPDATE tasks SET payload=?1, run_after=?2, attempts=?3, status='pending', locked_until=0 WHERE id=?4",
params![payload, now_f64() + delay_seconds, attempts, id],
)?;
Ok(())
})
.await;
if let Err(e) = result {
log::error!("queue reschedule failed: {e}");
let run_after = now_f64() + delay_seconds;
let mut last_error = None;
for attempt in 0..TERMINAL_WRITE_ATTEMPTS {
let id = row_id.clone();
let lease_token = lease_token.clone();
let payload = payload.clone();
let result = self
.pool
.with_conn(move |conn| {
let affected = conn.execute(
"UPDATE tasks SET payload=?1, run_after=?2, attempts=?3, status='pending', locked_until=0 \
WHERE id=?4 AND lease_token=?5 AND status='in_progress'",
params![payload, run_after, attempts, id, lease_token],
)?;
Ok(affected == 1)
})
.await;
match result {
Ok(true) => {
// Same permit semantics as enqueue: never lose the wakeup.
self.notify.notify_one();
return;
}
// Re-leased while we worked: the new holder owns the row and
// its retry, so writing our payload would overwrite progress.
Ok(false) => {
log::warn!(
"queue: row {row_id} was re-leased; not rescheduling it (the new holder decides)"
);
return;
}
Err(e) => {
log::error!("queue reschedule failed (attempt {}): {e}", attempt + 1);
last_error = Some(e.to_string());
tokio::time::sleep(terminal_write_backoff(attempt)).await;
}
}
}
// Same permit semantics as enqueue: never lose the wakeup.
self.notify.notify_one();
log::error!(
"queue: row {row_id} could not be rescheduled ({}); the expiry sweep will \
re-run this attempt from its previous state",
last_error.unwrap_or_default()
);
}
}
@@ -439,6 +740,26 @@ mod tests {
assert_eq!(scaled_retry_delay(1.5, 1), 3.0);
assert_eq!(scaled_retry_delay(1.0, 10), 300.0, "capped at 300s");
assert_eq!(scaled_retry_delay(300.0, 0), 300.0);
// A server-asked delay above the cap is honoured, not truncated: a
// 1800s flood-control wait used to become 300s and earn another 429.
assert_eq!(scaled_retry_delay(1800.0, 0), 1800.0);
assert_eq!(scaled_retry_delay(1800.0, 1), 1800.0);
}
/// Puts a row into the state a worker holds while running it.
async fn set_lease(queue: &PersistentTaskQueue, id: &str, token: &str) {
let (id, token) = (id.to_string(), token.to_string());
queue
.pool
.with_conn(move |conn| {
conn.execute(
"UPDATE tasks SET status='in_progress', lease_token=?1 WHERE id=?2",
params![token, id],
)?;
Ok(())
})
.await
.unwrap();
}
async fn new_queue() -> (PersistentTaskQueue, tempfile::TempDir) {
@@ -473,6 +794,122 @@ mod tests {
queue.stop().await;
}
#[test]
fn row_fields_name_the_chat_and_the_post() {
// The payload shapes the three task variants store.
assert_eq!(
row_fields(&serde_json::json!({
"chat_id": 111,
"source_url": "https://x.com/u/status/1"
})),
"chat=111 [key=twitter:1]"
);
// A forward has no source URL; a chat id alone must still name the line.
assert_eq!(
row_fields(&serde_json::json!({"from_chat_id": 111, "to_chat_id": 222})),
"chat=111"
);
// Garbage in the payload must not panic a log line.
assert_eq!(row_fields(&serde_json::json!({"chat_id": "111"})), "");
assert_eq!(row_fields(&serde_json::Value::Null), "");
}
/// The row a leaked deletion would resurrect: `done` is invisible to the
/// lease query, so a task that already ran cannot be run again.
#[tokio::test]
async fn done_rows_are_never_leased() {
let (queue, _dir) = new_queue().await;
let runs = Arc::new(AtomicUsize::new(0));
queue
.enqueue(serde_json::json!({"chat_id": 1}), now_f64())
.await
.unwrap();
let id: String = queue
.pool
.with_conn(|conn| conn.query_row("SELECT id FROM tasks", [], |r| r.get(0)))
.await
.unwrap();
// A token that is not the row's is refused: only the lease holder can
// write the row back.
assert!(
!mark_done(&queue.pool, &id, "someone-elses-token")
.await
.unwrap(),
"a foreign lease must not be able to tombstone the row"
);
set_lease(&queue, &id, "ours").await;
assert!(mark_done(&queue.pool, &id, "ours").await.unwrap());
assert_eq!(
queue.pending_backlog().await,
None,
"a done row is not pending work"
);
let runs_worker = runs.clone();
queue
.start(
move |_payload| {
runs_worker.fetch_add(1, AtomicOrdering::SeqCst);
async { Ok(()) }
},
|_payload, _message| async {},
)
.await;
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(
runs.load(AtomicOrdering::SeqCst),
0,
"the finished row must not run again"
);
queue.stop().await;
}
#[tokio::test]
async fn pending_backlog_counts_only_unleased_rows() {
let (queue, _dir) = new_queue().await;
assert_eq!(queue.pending_backlog().await, None, "empty queue");
let due = now_f64();
queue
.enqueue(serde_json::json!({"chat_id": 1}), due)
.await
.unwrap();
queue
.enqueue(serde_json::json!({"chat_id": 2}), due + 600.0)
.await
.unwrap();
// Hold the first row in the handler so it is leased, not pending: a
// health line that reported work already in flight as backlog would be
// lying about the queue.
let release = Arc::new(tokio::sync::Notify::new());
let held = release.clone();
queue
.start(
move |_payload| {
let held = held.clone();
async move {
held.notified().await;
Ok(())
}
},
|_payload, _message| async {},
)
.await;
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(
queue.pending_backlog().await.map(|(n, _)| n),
Some(1),
"the leased row is not pending"
);
let (_, oldest) = queue.pending_backlog().await.unwrap();
assert!(
(oldest - (due + 600.0)).abs() < 1.0,
"oldest is the earliest run_after: {oldest}"
);
release.notify_one();
queue.stop().await;
}
#[tokio::test]
async fn retryable_reschedules_then_dead_letters() {
let (queue, _dir) = new_queue().await;
@@ -512,6 +949,51 @@ mod tests {
queue.stop().await;
}
#[tokio::test]
async fn runnable_rows_and_payload_replacement() {
let (queue, _dir) = new_queue().await;
queue
.enqueue(serde_json::json!({"s": 1}), now_f64())
.await
.unwrap();
let rows = queue.runnable_rows().await;
assert_eq!(rows.len(), 1);
let (id, payload) = rows[0].clone();
assert_eq!(payload, "{\"s\":1}");
// A replacement restarts the attempt budget (the new payload is a fresh
// delivery, not the continuation of the old one).
{
let id_owned = id.clone();
queue
.pool
.with_conn(move |conn| {
conn.execute("UPDATE tasks SET attempts=2 WHERE id=?1", params![id_owned])?;
Ok(())
})
.await
.unwrap();
}
assert!(
queue
.replace_payload(&id, &serde_json::json!({"s": 2}))
.await
);
let rows = queue.runnable_rows().await;
assert_eq!(rows[0].1, "{\"s\":2}");
assert_eq!(
queue.pending_backlog().await.map(|(n, _)| n),
Some(1),
"a repaired row is pending work again"
);
// A row that is gone (or done) is not rewritten.
assert!(
!queue
.replace_payload("task_missing", &serde_json::json!({}))
.await
);
}
#[tokio::test]
async fn permanent_error_dead_letters_immediately() {
let (queue, _dir) = new_queue().await;
@@ -582,7 +1064,12 @@ mod tests {
queue.stop().await;
}
#[tokio::test]
/// A row that goes stale *after* startup is picked up by the periodic
/// sweep — the spawned task, its 30 s interval included — and the worker
/// that sweeps it gets woken. The paused clock is what makes this the real
/// test: this used to call the recovery by hand, which proved the SQL but
/// left the wiring free to be deleted.
#[tokio::test(start_paused = true)]
async fn runtime_sweep_recovers_expired_lease() {
let (queue, _dir) = new_queue().await;
let calls = Arc::new(AtomicUsize::new(0));
@@ -597,8 +1084,12 @@ mod tests {
|_payload, _message| async {},
)
.await;
// Insert a stale leased row AFTER startup: without a runtime sweep it
// would stay `in_progress` forever (only start() used to recover).
// Let every worker park and the sweep consume its immediate first tick,
// so only a later tick can see the row. The clock is paused: yields do
// not advance it, and the sleep below does.
for _ in 0..16 {
tokio::task::yield_now().await;
}
{
let conn = rusqlite::Connection::open(queue.pool.path()).unwrap();
conn.execute(
@@ -608,12 +1099,18 @@ mod tests {
)
.unwrap();
}
queue.recover_sweep().await;
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(
calls.load(AtomicOrdering::SeqCst),
0,
"the row is stale but no sweep has run since it appeared"
);
tokio::time::sleep(Duration::from_secs(31)).await;
assert_eq!(
calls.load(AtomicOrdering::SeqCst),
1,
"expired lease must be recovered and processed exactly once"
"the periodic sweep must recover the row and wake a worker"
);
queue.stop().await;
}
+105 -15
View File
@@ -1,12 +1,13 @@
//! Per-chat token-bucket rate limiting.
//!
//! Telegram throttles bots that burst past a chat's message budget
//! (roughly 20 messages/min for channels/groups); today the bot absorbs
//! those 429s with queue retries. This limiter smooths the burst *before*
//! it reaches the API: media sends to a chat consume one token per
//! message, refilled at [`REFILL_PER_SEC`], so a batch forward paces itself
//! instead of tripping flood control. The queue retry stays as the safety
//! net for limits this bucket does not model (global per-bot limits etc.).
//! Telegram throttles bots on two budgets: one per chat (roughly 20
//! messages/min for channels/groups) and a bot-wide one (~30 messages per
//! second). Both are smoothed here *before* the burst reaches the API — the
//! per-chat bucket charges one token per message, and [`acquire_global`]
//! charges the same spend against the bot-wide budget, which no per-chat
//! bucket can see (a forward fanned out over many chats spends one token in
//! each and nothing anywhere). The queue retry stays as the safety net for
//! whatever neither bucket models.
use parking_lot::Mutex;
use std::collections::HashMap;
@@ -19,6 +20,12 @@ const CAPACITY: f64 = 20.0;
/// Sustained refill: ~20 messages per minute.
const REFILL_PER_SEC: f64 = 20.0 / 60.0;
/// The bot-wide budget: Telegram allows roughly 30 messages per second for a
/// bot in total, independently of the per-chat limits. Set to the documented
/// ceiling, so it only ever binds on a cross-chat burst.
const GLOBAL_CAPACITY: f64 = 30.0;
const GLOBAL_REFILL_PER_SEC: f64 = 30.0;
struct State {
/// Current token balance; may go negative (debt from an acquire larger
/// than the capacity, repaid by subsequent refills).
@@ -48,6 +55,19 @@ impl TokenBucket {
}
}
/// Applies the elapsed refill to `state`. Shared by [`Self::acquire`] and
/// the idle check so the two cannot drift apart.
fn refill(&self, state: &mut State) {
let now = tokio::time::Instant::now();
let elapsed = now
.saturating_duration_since(state.last_refill)
.as_secs_f64();
// Refill up to the capacity; a debt (negative balance) is repaid
// before any surplus accumulates.
state.tokens = (state.tokens + elapsed * self.refill_per_sec).min(self.capacity);
state.last_refill = now;
}
/// Waits until `n` tokens are available, consuming them. The wait is
/// bounded: the deficit is committed as debt and repaid over time, so a
/// large acquire returns once its share of the refill budget has passed.
@@ -57,14 +77,7 @@ impl TokenBucket {
// would make the future !Send).
let wait = {
let mut state = self.state.lock();
let now = tokio::time::Instant::now();
let elapsed = now
.saturating_duration_since(state.last_refill)
.as_secs_f64();
// Refill up to the capacity; a debt (negative balance) is repaid
// before any surplus accumulates.
state.tokens = (state.tokens + elapsed * self.refill_per_sec).min(self.capacity);
state.last_refill = now;
self.refill(&mut state);
if state.tokens >= n {
state.tokens -= n;
return;
@@ -77,6 +90,23 @@ impl TokenBucket {
};
tokio::time::sleep(Duration::from_secs_f64(wait)).await;
}
/// Current balance, for the tests that assert a call site charged the
/// bucket (a charge is otherwise only observable as a delay).
#[cfg(test)]
pub(crate) fn tokens(&self) -> f64 {
let mut state = self.state.lock();
self.refill(&mut state);
state.tokens
}
/// True when the bucket has refilled to capacity: no debt outstanding, so
/// the chat has not sent anything recently.
fn is_idle(&self) -> bool {
let mut state = self.state.lock();
self.refill(&mut state);
state.tokens >= self.capacity
}
}
/// One limiter per chat, created on first use. Per-chat so one chat's burst
@@ -93,6 +123,29 @@ pub fn limiter_for(chat_id: i64) -> Arc<TokenBucket> {
.clone()
}
/// The one bucket every chat shares: Telegram's bot-wide budget.
static GLOBAL_LIMITER: LazyLock<TokenBucket> =
LazyLock::new(|| TokenBucket::new(GLOBAL_CAPACITY, GLOBAL_REFILL_PER_SEC));
/// Waits for `n` messages' worth of the bot-wide budget. Called by the send
/// paths next to their per-chat [`limiter_for`]: at ~30/s it does not bind on
/// a single chat, but a batch fanned out over many chats has no other guard.
pub async fn acquire_global(n: f64) {
GLOBAL_LIMITER.acquire(n).await;
}
/// Drops limiters that are idle (refilled to capacity, so the chat has not
/// sent recently) and are not still held by an in-flight sender. The map
/// would otherwise keep one bucket per chat that ever sent media, forever.
/// Called from the periodic sweep; returns how many were dropped.
pub fn prune_idle() -> usize {
let mut limiters = LIMITERS.lock();
let before = limiters.len();
// Lock order map → bucket, the only order taken anywhere.
limiters.retain(|_, bucket| Arc::strong_count(bucket) > 1 || !bucket.is_idle());
before - limiters.len()
}
#[cfg(test)]
mod tests {
use super::*;
@@ -133,4 +186,41 @@ mod tests {
start.elapsed()
);
}
#[tokio::test(start_paused = true)]
async fn the_global_budget_is_paced_and_shared() {
// Drain the process-wide budget (no other test touches it: the send
// paths that use it are mocked), then prove the next message waits for
// the refill instead of going out instantly.
acquire_global(GLOBAL_CAPACITY).await;
let start = tokio::time::Instant::now();
acquire_global(1.0).await;
assert!(
start.elapsed() >= Duration::from_secs_f64(1.0 / GLOBAL_REFILL_PER_SEC),
"a fanned-out burst must be paced: elapsed {:?}",
start.elapsed()
);
}
#[tokio::test(start_paused = true)]
async fn prune_idle_drops_full_unheld_buckets_only() {
// Held by this task: kept even at full capacity, a sender has it.
let held = limiter_for(9_001);
assert!(held.is_idle(), "a fresh bucket is full");
// Only the map holds this one and it is full → dropped.
limiter_for(9_002);
// Mid-debt (an acquire larger than the capacity): kept.
{
let bucket = Arc::new(TokenBucket::new(CAPACITY, REFILL_PER_SEC));
bucket.state.lock().tokens = -1.0;
LIMITERS.lock().insert(9_003, bucket);
}
assert!(prune_idle() >= 1);
let limiters = LIMITERS.lock();
assert!(limiters.contains_key(&9_001), "held bucket pruned");
assert!(!limiters.contains_key(&9_002), "idle unheld bucket kept");
assert!(limiters.contains_key(&9_003), "indebted bucket pruned");
}
}
+128
View File
@@ -0,0 +1,128 @@
//! Payload → Telegram input types: `InputFile` selection (cached file id /
//! URL / local path), the per-kind `InputMedia` builders and the media-group
//! assembly with its caption rule.
use super::MediaItemPayload;
use teloxide::types::{
InputFile, InputMedia, InputMediaAnimation, InputMediaPhoto, InputMediaVideo, ParseMode,
};
fn parse_media_url(s: &str) -> Result<url::Url, String> {
url::Url::parse(s).map_err(|e| format!("invalid media URL: {e}"))
}
pub(super) fn item_url(item: &MediaItemPayload) -> &str {
match item {
MediaItemPayload::Photo { media, .. }
| MediaItemPayload::Video { media, .. }
| MediaItemPayload::Animation { media, .. } => media,
}
}
/// Remote http(s) URLs are handed to Telegram to fetch; everything else
/// (e.g. a locally encoded ugoira MP4) is uploaded directly.
pub(super) fn input_file_for(media: &str) -> Result<InputFile, String> {
if media.starts_with("http://") || media.starts_with("https://") {
Ok(InputFile::url(parse_media_url(media)?))
} else if !std::path::Path::new(media).exists() {
// A retried task may reference a temp file the original send's
// TempDir already cleaned up; fail fast and permanent instead of
// burning retries on a file that can never come back.
Err(format!("local media file missing: {media}"))
} else {
Ok(InputFile::file(media))
}
}
impl MediaItemPayload {
/// The input for a send: a cached file id goes out as `InputFile::file_id`
/// (no fetch, no upload), URLs go to Telegram, anything else is a local
/// path (transient upload fallback).
fn input_file(&self) -> Result<InputFile, String> {
match self {
MediaItemPayload::Photo {
media,
file_id: true,
..
}
| MediaItemPayload::Video {
media,
file_id: true,
..
}
| MediaItemPayload::Animation {
media,
file_id: true,
..
} => Ok(InputFile::file_id(media.clone().into())),
_ => input_file_for(item_url(self)),
}
}
}
pub(super) fn photo_media(file: InputFile, caption: Option<&str>, spoiler: bool) -> InputMedia {
let mut photo = InputMediaPhoto::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
photo = photo.caption(caption);
}
if spoiler {
photo = photo.spoiler();
}
InputMedia::Photo(photo)
}
pub(super) fn video_media(file: InputFile, caption: Option<&str>, spoiler: bool) -> InputMedia {
let mut video = InputMediaVideo::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
video = video.caption(caption);
}
if spoiler {
video = video.spoiler();
}
InputMedia::Video(video)
}
pub(super) fn animation_media(file: InputFile, caption: Option<&str>, spoiler: bool) -> InputMedia {
let mut animation = InputMediaAnimation::new(file).parse_mode(ParseMode::Html);
if let Some(caption) = caption {
animation = animation.caption(caption);
}
if spoiler {
animation = animation.spoiler();
}
InputMedia::Animation(animation)
}
/// Builds a media group from payloads; only the first item of the batch gets
/// the caption (Telegram rejects captions on later items).
pub(super) fn build_media_group(
batch: &[MediaItemPayload],
caption: Option<&str>,
) -> Result<Vec<InputMedia>, String> {
batch
.iter()
.enumerate()
.map(|(i, item)| {
let item_caption = if i == 0 { caption } else { None };
Ok(match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(item.input_file()?, item_caption, *has_spoiler)
}
MediaItemPayload::Video {
has_spoiler,
thumbnail,
..
} => {
let mut video = video_media(item.input_file()?, item_caption, *has_spoiler);
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut video) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
video
}
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(item.input_file()?, item_caption, *has_spoiler)
}
})
})
.collect()
}
File diff suppressed because it is too large Load Diff
+545
View File
@@ -0,0 +1,545 @@
//! Everything around a send: the link-cache write that follows one, the
//! keep-alive registry for locally produced media, task settlement, the
//! post-send actions (edit prompt / channel forward) and the queue entry
//! points.
use super::{SendError, Task, forward_messages, send_animation, send_media_sequence};
use crate::ctx::AppContext;
use crate::db::{now_f64, unix_now};
use crate::handlers::log_key;
use crate::link_cache::{CachedMedia, CachedMediaKind};
use crate::media_sender::MediaSender;
use crate::queue::{PersistentTaskQueue, QueueError};
use crate::state::EditMessage;
use std::collections::HashMap;
use std::sync::LazyLock;
use teloxide::types::{ChatId, InlineKeyboardButton, InlineKeyboardMarkup, Message, MessageId};
/// Persists a successful send under the post's cache key. Skips a send that was
/// served from the cache — its entry already holds the file ids the next repeat
/// wants — *unless* the entry was degraded (no file ids left, see
/// `invalidate_cache`): then the ids this send just produced are written back,
/// which is what returns a degraded entry to the fast path instead of leaving
/// it to re-upload the media on every repeat.
pub(super) async fn cache_sent_task(ctx: &AppContext<'_>, task: &Task, media: Vec<CachedMedia>) {
let Some(cache_data) = task.cache_data() else {
return;
};
if cache_data.media.iter().any(|m| !m.file_id.is_empty()) || media.is_empty() {
return;
}
let mut post = cache_data.clone();
post.media = media;
if let Some(key) = x_media::site::cache_key(&post.url) {
ctx.link_cache.put(&key, &post).await;
log::debug!("cached send for [key={}]", log_key(&post.url));
}
}
/// Persists a lone animation send under the post's cache key.
pub(super) async fn cache_animation_send(
ctx: &AppContext<'_>,
task: &Task,
message: &Message,
source_url: &str,
) {
if let Some(file_id) = message.animation().map(|a| a.file.id.to_string()) {
cache_sent_task(
ctx,
task,
vec![CachedMedia {
kind: CachedMediaKind::Animation,
file_id,
url: source_url.to_string(),
}],
)
.await;
}
}
/// How a task ended. The two states differ only in whether a link-cache entry
/// may still be holding the (now unusable) media.
pub(crate) enum Settled {
Sent,
Failed,
}
/// Every path that ends a task's life — sent, permanently failed, or
/// dead-lettered after the last retry — funnels through here, so the cleanup a
/// settled task owes cannot be forgotten by a new path: release the keep-alive
/// temp media (retryable tasks keep it, they will be resent) and deal with the
/// link-cache entry a failed send's stale file ids would keep poisoning
/// (degraded to its source URLs, dropped once those fail too).
pub(crate) async fn settle_task(ctx: &AppContext<'_>, task: &Task, outcome: Settled) {
if matches!(outcome, Settled::Failed) {
invalidate_cache(ctx, task).await;
}
release_keep_alive(task);
}
/// A cached Telegram file id failed permanently (stale/expired). The media
/// itself is usually fine, so the entry is *degraded* rather than dropped: its
/// file ids go away and the source URLs stay, and the next request re-sends the
/// post from those — no source request, no ugoira encode, no HLS remux — with
/// the media fetched by Telegram (or by the upload fallback). An entry that is
/// already degraded, or whose older rows carry no URLs, is removed instead: its
/// URLs did not work either, and the next request should fetch the post again
/// and report what the source says.
async fn invalidate_cache(ctx: &AppContext<'_>, task: &Task) {
if !task.is_cached_send() {
return;
}
let Some(url) = task.source_url() else {
return;
};
let Some(key) = x_media::site::cache_key(url) else {
return;
};
let Some(mut entry) = ctx.link_cache.get(&key, ctx.config.link_cache_ttl).await else {
return;
};
let degradable = entry.media.iter().all(|m| !m.url.is_empty())
&& entry.media.iter().any(|m| !m.file_id.is_empty());
if !degradable {
log::debug!("removing stale link cache entry for [key={}]", log_key(url));
ctx.link_cache.remove(&key).await;
return;
}
log::debug!(
"degrading stale link cache entry to its source URLs for [key={}]",
log_key(url)
);
for media in &mut entry.media {
media.file_id.clear();
}
ctx.link_cache.put(&key, &entry).await;
}
/// Locally produced media files (ugoira MP4, bsky remux MP4) whose temp dirs
/// must stay alive while their task may be retried by the queue. The fetch
/// pipeline hands a reference here via [`x_media::site::Fetched::keep_alive`]
/// before that [`x_media::site::Fetched`] is dropped; a queued retry runs after
/// that drop, so without this the local file would be gone by the time the
/// retry sends it. `Arc` because one fetch can serve several tasks (a
/// concurrent duplicate of the same link shares it): each holder keeps the
/// directory alive until its own task settles. Entries are removed when the
/// task settles (see [`release_keep_alive`]).
pub(crate) static KEEP_ALIVE: LazyLock<parking_lot::Mutex<Vec<std::sync::Arc<tempfile::TempDir>>>> =
LazyLock::new(|| parking_lot::Mutex::new(Vec::new()));
/// Drops the keep-alive temp dirs holding media referenced by `task` (matched
/// by path prefix). Called once a task settles — sent or permanently failed —
/// so retry-only temp files do not leak; retryable tasks keep them alive.
pub(crate) fn release_keep_alive(task: &Task) {
let paths = task.local_media_paths();
if paths.is_empty() {
return;
}
let mut alive = KEEP_ALIVE.lock();
alive.retain(|dir| {
let dir_path = dir.path();
!paths.iter().any(|p| p.starts_with(dir_path))
});
}
/// The edit-before-forward prompt's text. It names both controls and the TTL,
/// because the buttons alone left users waiting for a forward that never came
/// (nothing is forwarded until Confirm).
pub(super) fn edit_prompt_text(ttl: std::time::Duration) -> String {
format!(
"Reply to edit the caption, or tap a template, then ↩️ Confirm to forward. \
Expires in {}. Nothing is forwarded until you confirm.",
coarsest_unit(ttl)
)
}
/// Text the prompt is rewritten to once its record expires. The sweep edits
/// the prompt in place (see `main`): announcing the expiry with a new message
/// would wake the chat up to a full TTL later about a prompt nobody is
/// waiting on.
pub(crate) const EDIT_PROMPT_EXPIRED_TEXT: &str = "⌛ Expired — nothing was forwarded.";
/// `24h` / `90m` / `45s`: the coarsest whole unit, so the prompt stays short.
fn coarsest_unit(ttl: std::time::Duration) -> String {
let secs = ttl.as_secs();
if secs >= 3600 {
format!("{}h", secs / 3600)
} else if secs >= 60 {
format!("{}m", secs / 60)
} else {
format!("{secs}s")
}
}
/// Templates per keyboard row. Telegram rejects a keyboard with more than 100
/// buttons *outright*, which would silently drop the whole prompt, so the
/// names are folded and capped rather than listed one per row.
pub(super) const TEMPLATE_BUTTONS_PER_ROW: usize = 3;
/// Hard cap on template buttons; the prompt text names the ones not shown.
pub(super) const MAX_TEMPLATE_BUTTONS: usize = 60;
/// Template buttons ([`TEMPLATE_BUTTONS_PER_ROW`] per row, at most
/// [`MAX_TEMPLATE_BUTTONS`]), then the confirm/skip pair. Sorted by name: the
/// templates live in a `HashMap`, so an unsorted walk would reshuffle the
/// buttons between prompts.
pub(super) fn build_edit_markup(templates: &HashMap<String, String>) -> InlineKeyboardMarkup {
let mut names: Vec<&String> = templates.keys().collect();
names.sort();
let shown = names.len().min(MAX_TEMPLATE_BUTTONS);
let mut rows = Vec::with_capacity(shown / TEMPLATE_BUTTONS_PER_ROW + 2);
for chunk in names[..shown].chunks(TEMPLATE_BUTTONS_PER_ROW) {
rows.push(
chunk
.iter()
.map(|name| {
InlineKeyboardButton::callback(name.as_str(), format!("template|{name}"))
})
.collect(),
);
}
// Skip exists because the prompt holds the forward hostage until Confirm:
// without it the only escape was deleting the message and waiting out the
// TTL for a forward that then never happens.
rows.push(vec![
InlineKeyboardButton::callback("↩️ Confirm", "forward"),
InlineKeyboardButton::callback("🛑 Skip", "skip"),
]);
InlineKeyboardMarkup::new(rows)
}
/// How many templates the markup could not fit, for the prompt text.
pub(super) fn hidden_template_count(templates: &HashMap<String, String>) -> usize {
templates.len().saturating_sub(MAX_TEMPLATE_BUTTONS)
}
/// Notifies a chat about a dead-lettered task (skips when `notify_chat_id` is
/// absent).
pub(crate) async fn notify_failure(
sender: &dyn MediaSender,
chat_id: Option<i64>,
message_id: Option<i64>,
message: &str,
) {
let Some(chat_id) = chat_id else { return };
let reply_to = message_id.map(|id| MessageId(id as i32));
if let Err(e) = sender
.send_message(ChatId(chat_id), message.to_string(), reply_to, None)
.await
{
log::error!("failed to notify about failed task: {e}");
}
}
/// After a successful send: either open the edit-before-forward prompt or
/// forward to the configured channel (with retry/queue handling).
pub(crate) async fn post_send_actions(ctx: &AppContext<'_>, task: &Task, message_ids: Vec<i64>) {
let (
chat_id,
reply_to,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
) = match task {
Task::SendMediaSequence {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
}
| Task::SendAnimation {
chat_id,
reply_to_message_id,
source_url,
edit_before_forward,
forward_channel_id,
notify_chat_id,
notify_message_id,
..
} => (
*chat_id,
*reply_to_message_id,
source_url.clone(),
*edit_before_forward,
*forward_channel_id,
*notify_chat_id,
*notify_message_id,
),
Task::ForwardMessages { .. } => return,
};
if edit_before_forward {
let templates = ctx.chat_store.get(chat_id).await.template;
let keyboard = build_edit_markup(&templates);
let mut text = edit_prompt_text(ctx.config.edit_message_ttl);
let hidden = hidden_template_count(&templates);
if hidden > 0 {
// The keyboard is capped; say so instead of silently hiding them.
text.push_str(&format!(
"\n({hidden} more templates not shown — /remove_template to prune.)"
));
}
let prompt = ctx
.sender
.send_message(
ChatId(chat_id),
text,
Some(MessageId(reply_to as i32)),
Some(keyboard),
)
.await;
match prompt {
Ok(prompt_id) => {
log::info!(
"edit-before-forward prompt {prompt_id} opened for {} message(s) [key={}] chat={chat_id}",
message_ids.len(),
log_key(&source_url)
);
let source_url = source_url.clone();
ctx.chat_store
.update(chat_id, move |data| {
data.edit_message.insert(
prompt_id,
EditMessage {
url: source_url,
chat_id,
forward_message_ids: message_ids,
template: String::new(),
created_at: unix_now(),
},
);
})
.await;
}
Err(e) => {
log::error!("failed to send edit prompt: {e}");
// Nothing is forwarded until the prompt is confirmed, so a
// prompt that never arrived means this post is never forwarded.
// Tell the chat instead of letting it wait for a prompt that
// will not come.
notify_failure(
ctx.sender,
notify_chat_id,
notify_message_id,
"Could not open the edit-before-forward prompt — nothing was forwarded.",
)
.await;
}
}
return;
}
if let Some(channel_id) = forward_channel_id {
log::info!(
"forwarding {} message(s) to channel {channel_id} from chat {chat_id} [key={}]",
message_ids.len(),
log_key(&source_url)
);
let forward_task = Task::ForwardMessages {
from_chat_id: chat_id,
to_chat_id: channel_id,
message_ids,
notify_chat_id,
notify_message_id,
};
match forward_messages(ctx, &forward_task).await {
Ok(()) => {}
Err(SendError::Retryable {
delay_seconds,
task,
}) => {
// The forward is already committed from the user's side; if it
// cannot be queued, say so rather than going quiet.
if !enqueue_retry(ctx.task_queue, &task, delay_seconds).await {
notify_failure(
ctx.sender,
notify_chat_id,
notify_message_id,
&failure_text(task.source_url(), "retry could not be queued"),
)
.await;
}
}
Err(SendError::Permanent { message, .. }) => {
notify_failure(
ctx.sender,
notify_chat_id,
notify_message_id,
&failure_text(None, &message),
)
.await;
}
}
}
}
/// Enqueues a task for a later attempt (retry / forward resume). Returns
/// whether the retry is actually persisted: when the enqueue itself fails the
/// task can never run again, so its keep-alive temp media is released instead
/// of leaking until process exit — and the caller must not tell the user a
/// retry is coming (nothing would ever deliver it).
pub(crate) async fn enqueue_retry(
queue: &PersistentTaskQueue,
task: &Task,
delay_seconds: f64,
) -> bool {
let payload = serde_json::to_value(task).expect("task serializes");
let run_after = now_f64() + delay_seconds;
if let Err(e) = queue.enqueue(payload, run_after).await {
log::error!("failed to enqueue retry: {e}");
release_keep_alive(task);
return false;
}
true
}
/// Queue entry point: parses the stored task and dispatches.
pub(crate) async fn handle_task(
ctx: &AppContext<'_>,
payload: serde_json::Value,
) -> Result<(), QueueError> {
let task: Task = match serde_json::from_value(payload.clone()) {
Ok(task) => task,
Err(e) => {
return Err(QueueError::Permanent {
message: format!("invalid task payload: {e}"),
payload,
});
}
};
match task {
Task::SendMediaSequence { .. } | Task::SendAnimation { .. } => {
let message_ids = match send_media_or_animation(ctx, &task).await {
Ok(ids) => ids,
Err(SendError::Retryable {
delay_seconds,
task,
}) => {
return Err(QueueError::Retryable {
delay_seconds,
payload: serde_json::to_value(task).expect("task serializes"),
});
}
Err(SendError::Permanent { message, task }) => {
settle_task(ctx, &task, Settled::Failed).await;
return Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
});
}
};
// A task only reaches the queue after a failed send, so this
// successful run is the first time post_send_actions can fire —
// the fresh attempt failed before it ever got here. Run it
// unconditionally: `post_send_actions` executes once, after the
// whole sequence (every batch) completed, so the channel forward
// and the edit-before-forward prompt must not be lost just
// because the send needed a retry.
post_send_actions(ctx, &task, message_ids).await;
settle_task(ctx, &task, Settled::Sent).await;
Ok(())
}
Task::ForwardMessages { .. } => match forward_messages(ctx, &task).await {
Ok(()) => Ok(()),
Err(SendError::Retryable {
delay_seconds,
task,
}) => Err(QueueError::Retryable {
delay_seconds,
payload: serde_json::to_value(task).expect("task serializes"),
}),
Err(SendError::Permanent { message, task }) => {
settle_task(ctx, &task, Settled::Failed).await;
Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
})
}
},
}
}
async fn send_media_or_animation(ctx: &AppContext<'_>, task: &Task) -> Result<Vec<i64>, SendError> {
match task {
Task::SendMediaSequence { .. } => send_media_sequence(ctx, task).await,
Task::SendAnimation { .. } => send_animation(ctx, task).await,
Task::ForwardMessages { .. } => unreachable!(),
}
}
/// User-facing text for a task that will never run again: which link died and
/// why. The raw error alone left the user guessing which post it was about.
pub(super) fn failure_text(source_url: Option<&str>, message: &str) -> String {
match source_url.map(log_key) {
Some(key) => format!("Send failed permanently for {key}: {message}"),
// `ForwardMessages` carries no source URL (and neither does an
// unparsable payload): that failure is about the channel copy, not
// about a post.
None => format!("Forward failed permanently: {message}"),
}
}
/// The post a stored payload is about, without parsing it into a [`Task`]:
/// used when the payload no longer deserializes (written by an older version,
/// or corrupted) but its identity fields are still readable.
fn payload_source_url(payload: &serde_json::Value) -> Option<&str> {
payload.get("source_url").and_then(|v| v.as_str())
}
/// Whether a stored payload was a *cached* send (see `Task::is_cached_send`),
/// read straight off the JSON — the unparsable case still has to know whether
/// a link-cache entry may be holding the media that failed.
fn payload_is_cached_send(payload: &serde_json::Value) -> bool {
payload
.get("cache_data")
.and_then(|data| data.get("media"))
.and_then(|media| media.as_array())
.is_some_and(|media| !media.is_empty())
}
/// Dead-letter callback wired to the queue in main: settles the task and
/// notifies its chat.
pub(crate) async fn dead_letter_notify(
ctx: &AppContext<'_>,
payload: serde_json::Value,
message: String,
) {
// A dead-lettered task never runs again, and the queue dead-letters retry
// exhaustion itself (the handler is not called again), so this is the only
// place that sees the final payload.
let task = serde_json::from_value::<Task>(payload.clone()).ok();
if let Some(task) = &task {
settle_task(ctx, task, Settled::Failed).await;
} else {
// A payload that no longer parses (an older version's row shape, a
// corrupted one) still says which post it was about: drop the stale
// cache entry the same way, instead of leaving a bad file id to be
// re-sent forever — and name the post in the notification rather than
// reporting a *forward* failure for a send task.
if payload_is_cached_send(&payload)
&& let Some(key) = payload_source_url(&payload).and_then(x_media::site::cache_key)
{
log::debug!("removing stale link cache entry for [key={key}]");
ctx.link_cache.remove(&key).await;
}
}
let notify_chat_id = payload.get("notify_chat_id").and_then(|v| v.as_i64());
let notify_message_id = payload.get("notify_message_id").and_then(|v| v.as_i64());
notify_failure(
ctx.sender,
notify_chat_id,
notify_message_id,
&failure_text(
task.as_ref()
.and_then(|task| task.source_url())
.or_else(|| payload_source_url(&payload)),
&message,
),
)
.await;
}
+401
View File
@@ -0,0 +1,401 @@
//! Download-and-reupload fallback: when Telegram cannot fetch a media URL
//! itself (hotlink protection), the bot downloads the file, shrinks photos
//! that exceed Telegram's limits and uploads the batch via multipart.
use super::input_media::{animation_media, input_file_for, item_url, photo_media, video_media};
use super::{MediaItemPayload, SendError, Task, classify_to_send_error, retry_delay_seconds};
use crate::media_sender::MediaSender;
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
use std::sync::LazyLock;
use teloxide::prelude::*;
use teloxide::types::{ChatId, InputFile, InputMedia, MessageId};
use tempfile::NamedTempFile;
use x_media::site::FetchError;
/// How many fallback items may be downloaded and processed at once, across the
/// whole process. A per-batch bound is not a memory bound: `URL_WORKERS` (8)
/// and the queue's workers (4) can each be inside a batch, so a per-batch three
/// allowed two dozen downloads in flight, each buffering a whole photo
/// (up to [`photo::MAX_PHOTO_DOWNLOAD_BYTES`]) before it is processed. This is
/// the only admission control on the media path; the send itself is paced by
/// the rate limiter.
const PREP_CONCURRENCY: usize = 6;
static PREP_SLOTS: LazyLock<tokio::sync::Semaphore> =
LazyLock::new(|| tokio::sync::Semaphore::new(PREP_CONCURRENCY));
/// Infers a file extension from magic bytes so Telegram detects the mime type
/// on multipart uploads.
pub(super) fn sniff_ext(bytes: &[u8]) -> &'static str {
if bytes.starts_with(&[0xFF, 0xD8]) {
"jpg"
} else if bytes.starts_with(b"\x89PNG") {
"png"
} else if bytes.starts_with(b"RIFF") && bytes.len() >= 12 && &bytes[8..12] == b"WEBP" {
"webp"
} else if bytes.starts_with(b"GIF8") {
"gif"
} else if bytes.len() >= 12 && &bytes[4..8] == b"ftyp" {
"mp4"
} else {
"bin"
}
}
pub(super) enum FallbackError {
Retryable {
delay_seconds: f64,
},
Permanent {
message: String,
},
/// The downloaded file exceeds the upload cap; the caller falls back to
/// the item's smaller URL.
MediaTooLarge,
}
/// Brings a downloaded photo within Telegram's limits via the pure-Rust
/// chain in [`crate::photo`] (no ffmpeg): dimension cap / upload cap
/// exceeded photos are decoded, downscaled with Lanczos3, PNG bit depth
/// reduced (>24-bit → 24-bit RGB, ≤24-bit untouched) and transcoded to JPEG
/// only if still too big. Anything that cannot be fixed falls back to the
/// item's smaller URL.
///
/// Downloads one media item to a temp file (deleted on drop), returning the
/// file plus the downloaded bytes (photos keep the bytes for
/// [`photo::prepare_photo`] — re-reading the file would double the I/O).
/// Network errors are retryable; size over the upload cap and other download
/// errors are not.
async fn download_to_temp(
item: &MediaItemPayload,
) -> Result<(NamedTempFile, bytes::Bytes), FallbackError> {
let media_url = match item {
MediaItemPayload::Photo { media, .. }
| MediaItemPayload::Video { media, .. }
| MediaItemPayload::Animation { media, .. } => media,
};
// Photos are downloaded even over the upload cap so `prepare_photo` can
// downscale / transcode them, up to their own download cap; videos and
// animations are refused as soon as the declared size crosses the upload
// cap. The limit is that cap, not `cap + 1`: a file of exactly the cap is
// admitted (`len > max_bytes` is false), and one byte over is not — the
// same boundary the size probe this replaced drew.
let limit = if matches!(item, MediaItemPayload::Photo { .. }) {
photo::MAX_PHOTO_DOWNLOAD_BYTES
} else {
MAX_UPLOAD_BYTES
};
let bytes = match x_media::site::download_media_limited(media_url, limit).await {
Ok(bytes) => bytes,
Err(e) => return Err(classify_download_error(e)),
};
let ext = sniff_ext(&bytes);
let mut file = tempfile::Builder::new()
.prefix(x_media::TEMP_FILE_PREFIX)
.suffix(&format!(".{ext}"))
.tempfile()
.map_err(|e| FallbackError::Permanent {
message: format!("temp file failed: {e}"),
})?;
use std::io::Write;
// A write failure is resource exhaustion far more often than a broken temp
// dir (ENOSPC / EDQUOT), and that clears on its own — worth an attempt
// instead of dropping the post on the first try. Creating the file (above)
// stays permanent: a temp dir that cannot be created at all is a
// deployment fault that should fail loudly and immediately. `Retryable`
// carries no message, so the cause is logged here.
file.as_file_mut().write_all(&bytes).map_err(|e| {
log::error!("temp file write failed: {e}");
FallbackError::Retryable {
delay_seconds: retry_delay_seconds(0),
}
})?;
Ok((file, bytes))
}
/// Which failure class a media download belongs to. Transport errors and
/// server-side hiccups (429/5xx, see `download_media_limited`) are worth
/// another attempt; a 4xx means the media itself is gone or refused, and a
/// retry could only ask the same URL again.
fn classify_download_error(err: FetchError) -> FallbackError {
match err {
FetchError::Http(_) | FetchError::Transient(_) => FallbackError::Retryable {
delay_seconds: retry_delay_seconds(0),
},
FetchError::TooLarge => FallbackError::MediaTooLarge,
e => FallbackError::Permanent {
message: format!("download failed: {e}"),
},
}
}
/// Builds the media group item from an uploaded file.
fn media_from_file(
item: &MediaItemPayload,
path: std::path::PathBuf,
caption: Option<&str>,
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(InputFile::file(path), caption, *has_spoiler)
}
MediaItemPayload::Video { has_spoiler, .. } => {
video_media(InputFile::file(path), caption, *has_spoiler)
}
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(InputFile::file(path), caption, *has_spoiler)
}
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
}
/// Builds the media group item from a (smaller) URL.
fn media_from_url(
item: &MediaItemPayload,
url: &str,
caption: Option<&str>,
thumbnail: Option<&str>,
) -> Result<InputMedia, String> {
let mut media = match item {
MediaItemPayload::Photo { has_spoiler, .. } => {
photo_media(input_file_for(url)?, caption, *has_spoiler)
}
MediaItemPayload::Video { has_spoiler, .. } => {
video_media(input_file_for(url)?, caption, *has_spoiler)
}
MediaItemPayload::Animation { has_spoiler, .. } => {
animation_media(input_file_for(url)?, caption, *has_spoiler)
}
};
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut media) {
*v = v.clone().thumbnail(input_file_for(thumb)?);
}
Ok(media)
}
/// One item prepared for the upload fallback: the ready-to-send media plus
/// the temp file that must stay on disk until the group request completes.
pub(super) struct PreparedItem {
/// Original position in the batch (concurrent prep completes out of order).
pub(super) index: usize,
pub(super) media: InputMedia,
pub(super) keep_alive: Option<NamedTempFile>,
}
/// Downloads / processes one media item for the upload fallback (see
/// [`send_batch_via_upload`]). Local files are uploaded directly; oversized
/// items fall back to their smaller URL; photos are downscaled/transcoded.
pub(super) async fn prepare_upload_item(
item: MediaItemPayload,
index: usize,
caption: Option<&str>,
) -> Result<PreparedItem, FallbackError> {
// Locally produced files (ugoira / bsky remux MP4): nothing to download
// or shrink — upload the file directly. The send is a multipart upload,
// so the only remaining failure is an upload-cap error, which is
// permanent (a video cannot be re-encoded here).
let media_url = item_url(&item);
if !media_url.starts_with("http://") && !media_url.starts_with("https://") {
let media = media_from_file(
&item,
std::path::PathBuf::from(media_url),
caption,
item.thumbnail_url(),
)
.map_err(|message| FallbackError::Permanent { message })?;
return Ok(PreparedItem {
index,
media,
keep_alive: None,
});
}
// Whether a file is over the cap is settled by the download itself:
// `download_media_limited` reads the declared Content-Length before any
// body byte and aborts with `FetchError::TooLarge`, which arrives here as
// `FallbackError::MediaTooLarge` — turned into the item's smaller URL by
// the match below. A separate size probe used to issue a second GET of the
// same URL for an answer this path already has (and issued it for photos,
// whose answer was discarded one line later).
match download_to_temp(&item).await {
Ok((file, bytes)) => {
if matches!(item, MediaItemPayload::Photo { .. }) {
// Telegram rejects photos wider+taller than 10000 px combined
// (PHOTO_INVALID_DIMENSIONS): downscale the downloaded file
// before uploading; photos that cannot be brought within the
// limits degrade to the smaller URL. CPU-heavy work runs off
// the async executor thread.
//
// The header decides what that will cost in memory, so the
// probe travels with the downloaded bytes (both stay alive
// through the decode) and the reservation covers their sum:
// `PREP_SLOTS` bounds how many photos are prepared at once,
// this bounds what they hold between them — 512 MiB, whatever
// the batch looks like.
let (bytes, decode) = tokio::task::spawn_blocking(move || {
let decode = photo::decode_budget_bytes(&bytes);
(bytes, decode)
})
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
})?;
let _budget = photo::reserve_memory(bytes.len() as u64 + decode).await;
let prep = tokio::task::spawn_blocking(move || photo::prepare_photo(file, &bytes))
.await
.map_err(|e| FallbackError::Permanent {
message: format!("photo worker panicked: {e}"),
})?
.map_err(|message| FallbackError::Permanent { message })?;
match prep {
PhotoPrep::Upload(upload) => {
let path = upload.path().to_path_buf();
let media = media_from_file(&item, path, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: Some(upload),
})
}
PhotoPrep::UseFallback => {
let url = item.fallback_url().ok_or_else(|| FallbackError::Permanent {
message: "photo dimensions exceed Telegram limits and no smaller variant is available"
.into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: None,
})
}
}
} else {
let path = file.path().to_path_buf();
let media = media_from_file(&item, path, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: Some(file),
})
}
}
Err(FallbackError::MediaTooLarge) => {
let url = item
.fallback_url()
.ok_or_else(|| FallbackError::Permanent {
message: "media too large".into(),
})?;
let media = media_from_url(&item, url, caption, item.thumbnail_url())
.map_err(|message| FallbackError::Permanent { message })?;
Ok(PreparedItem {
index,
media,
keep_alive: None,
})
}
Err(e) => Err(e),
}
}
/// Download-and-reupload fallback for one media batch. Files over the upload
/// cap are not downloaded/uploaded; the item falls back to its smaller URL
/// (which Telegram fetches itself). Items are prepared concurrently because the
/// downloads are network-bound, under one process-wide bound ([`PREP_SLOTS`] —
/// the URL and queue workers can each be inside a batch, so a per-batch bound
/// would multiply); the batch is then uploaded in its original order. Returns
/// the fallback-error without the task attached; callers wrap it with the
/// updated task state.
pub(super) async fn send_batch_via_upload(
sender: &dyn MediaSender,
chat_id: i64,
reply_to: i64,
batch: &[MediaItemPayload],
caption: Option<&str>,
task: Task,
) -> Result<Vec<Message>, SendError> {
let mut set = tokio::task::JoinSet::new();
for (i, item) in batch.iter().enumerate() {
let item_caption = if i == 0 {
caption.map(str::to_string)
} else {
None
};
let item = item.clone();
set.spawn(async move {
let _permit = PREP_SLOTS.acquire().await.expect("upload semaphore closed");
prepare_upload_item(item, i, item_caption.as_deref()).await
});
}
let mut prepared: Vec<Option<InputMedia>> = (0..batch.len()).map(|_| None).collect();
let mut keep_alive: Vec<NamedTempFile> = Vec::new();
while let Some(joined) = set.join_next().await {
let item = match joined {
Ok(Ok(item)) => item,
// Dropping the JoinSet aborts the remaining prep tasks; their
// temp files are cleaned up on drop (short-circuit like before).
Ok(Err(e)) => return Err(SendError::from_fallback(e, task.clone())),
Err(e) => {
return Err(SendError::Permanent {
message: format!("upload worker panicked: {e}"),
task: Box::new(task),
});
}
};
let PreparedItem {
index,
media,
keep_alive: file_opt,
} = item;
if let Some(file) = file_opt {
keep_alive.push(file);
}
prepared[index] = Some(media);
}
let items: Vec<InputMedia> = prepared
.into_iter()
.map(|m| m.expect("every upload item was prepared"))
.collect();
// `keep_alive` holds the temp files until the group request completes.
let result = sender
.send_media_group(ChatId(chat_id), MessageId(reply_to as i32), items)
.await;
drop(keep_alive);
match result {
Ok(messages) => Ok(messages),
Err(e) => Err(classify_to_send_error(&e, task, "upload failed")),
}
}
#[cfg(test)]
mod download_class_tests {
use super::*;
#[test]
fn download_errors_split_by_whether_a_retry_can_help() {
// Transport failure and a server-side hiccup: try again.
assert!(matches!(
classify_download_error(FetchError::Transient("media status 503".into())),
FallbackError::Retryable { .. }
));
// The media is gone / the host refuses us: a retry repeats the 4xx.
assert!(matches!(
classify_download_error(FetchError::NotFound),
FallbackError::Permanent { .. }
));
assert!(matches!(
classify_download_error(FetchError::Blocked),
FallbackError::Permanent { .. }
));
// Over the cap: degrade to the smaller URL, never retry.
assert!(matches!(
classify_download_error(FetchError::TooLarge),
FallbackError::MediaTooLarge
));
}
}
+172 -47
View File
@@ -17,8 +17,8 @@ pub struct ChatData {
pub edit_message: HashMap<i64, EditMessage>,
/// name -> HTML template containing "[]"
pub template: HashMap<String, String>,
/// site name (twitter/bsky/pixiv) -> user-supplied caption format with
/// {url} {author} {author_url} {title} {tags} placeholders.
/// site name (twitter/bsky/misskey/pixiv/bilibili) -> user-supplied caption format
/// with {url} {author} {author_url} {title} {content} {tags} placeholders.
pub message_format: HashMap<String, String>,
}
@@ -73,7 +73,7 @@ impl ChatStore {
})
.await
.unwrap_or_else(|e| {
log::error!("chat_state read failed: {e}");
log::warn!("chat_state read failed: {e}");
None
})
.unwrap_or_default();
@@ -98,23 +98,26 @@ impl ChatStore {
})
.await;
if let Err(e) = result {
log::error!("chat_state write failed: {e}");
log::warn!("chat_state write failed: {e}");
}
}
/// The per-chat async lock serializing get→mutate→set cycles.
fn lock_for(&self, chat_id: i64) -> Arc<tokio::sync::Mutex<()>> {
self.locks
.lock()
.entry(chat_id)
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
}
/// Serializes a get→mutate→set cycle per chat: concurrent handler tasks
/// (the batch-forward design spawns several per chat) each snapshot the
/// same `ChatData` and last-writer-wins would silently drop mutations,
/// e.g. a second `edit_message` record. The per-chat lock makes the
/// cycle atomic. Returns the closure's result.
pub async fn update<R>(&self, chat_id: i64, f: impl FnOnce(&mut ChatData) -> R) -> R {
let lock = {
let mut locks = self.locks.lock();
locks
.entry(chat_id)
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
};
let lock = self.lock_for(chat_id);
let _guard = lock.lock().await;
let mut data = self.get(chat_id).await;
let r = f(&mut data);
@@ -128,43 +131,51 @@ impl ChatStore {
pub async fn prune_expired(&self, ttl: Duration) -> Vec<(i64, i64)> {
let now = unix_now();
let ttl_secs = ttl.as_secs() as i64;
let mut removed = Vec::new();
// Chats with no live edit records: evicted from the cache (and their
// per-chat lock) so the cache stays bounded to active prompts. The DB
// keeps the row; the next get() reloads it.
let mut evicted_chats = Vec::new();
let changed: Vec<(i64, ChatData)> = {
let mut cache = self.cache.lock();
let mut out = Vec::new();
for (chat_id, data) in cache.iter_mut() {
let keys: Vec<i64> = data.edit_message.keys().copied().collect();
let mut kept = HashMap::new();
for key in keys {
if let Some(entry) = data.edit_message.get(&key) {
if entry.created_at + ttl_secs > now {
kept.insert(key, entry.clone());
} else {
removed.push((*chat_id, key));
}
}
}
if kept.len() != data.edit_message.len() {
// Persist the pruned row (removes expired records from
// the DB too, not just the cache).
data.edit_message = kept;
out.push((*chat_id, data.clone()));
}
if data.edit_message.is_empty() {
evicted_chats.push(*chat_id);
}
}
// Lock order: update() takes the per-chat lock before the cache
// lock, so prune must not hold the cache lock while taking locks.
drop(cache);
out
// Chats worth looking at, from a cache snapshot: the ones with an
// expired record, plus the ones holding no record at all. The latter
// used to be left alone for the process lifetime — every chat that ever
// sent a message or ran a command stayed in the cache and in the
// per-chat lock map — even though a chat with no live prompt is exactly
// what the eviction below is for. The pruning itself re-reads and
// writes under the per-chat lock below; taking no lock here means a
// chat appearing later is simply picked up by the next sweep.
let candidates: Vec<i64> = {
let cache = self.cache.lock();
cache
.iter()
.filter(|(_, data)| {
data.edit_message.is_empty()
|| data
.edit_message
.values()
.any(|entry| entry.created_at + ttl_secs <= now)
})
.map(|(chat_id, _)| *chat_id)
.collect()
};
for (chat_id, data) in changed {
self.set(chat_id, &data).await;
let mut removed = Vec::new();
let mut evicted_chats = Vec::new();
for chat_id in candidates {
let lock = self.lock_for(chat_id);
let _guard = lock.lock().await;
let mut data = self.get(chat_id).await;
let before = data.edit_message.len();
data.edit_message.retain(|key, entry| {
if entry.created_at + ttl_secs > now {
return true;
}
removed.push((chat_id, *key));
false
});
if data.edit_message.len() != before {
self.set(chat_id, &data).await;
}
// Chats with no live edit records: evicted from the cache (and
// their per-chat lock) so the cache stays bounded to active
// prompts. The DB keeps the row; the next get() reloads it.
if data.edit_message.is_empty() {
evicted_chats.push(chat_id);
}
}
if !evicted_chats.is_empty() {
let mut cache = self.cache.lock();
@@ -223,4 +234,118 @@ mod tests {
"concurrent get→mutate→set must not drop records"
);
}
fn edit_entry(chat_id: i64, created_at: i64) -> EditMessage {
EditMessage {
url: "https://x.com/u/status/1".into(),
chat_id,
forward_message_ids: vec![9],
template: String::new(),
created_at,
}
}
#[tokio::test]
async fn prune_removes_only_expired_records() {
let dir = tempfile::tempdir().unwrap();
let pool = crate::db::open_store(dir.path().join("p.db").to_str().unwrap()).unwrap();
let store = ChatStore::new(pool);
let now = unix_now();
store
.update(7, |data| {
data.template.insert("t".into(), "[]".into());
data.edit_message.insert(1, edit_entry(7, now - 3600));
data.edit_message.insert(2, edit_entry(7, now));
})
.await;
let removed = store.prune_expired(Duration::from_secs(60)).await;
assert_eq!(removed, vec![(7, 1)]);
let data = store.get(7).await;
assert!(data.edit_message.contains_key(&2), "live record pruned");
assert_eq!(
data.template.get("t").map(String::as_str),
Some("[]"),
"unrelated state lost by the prune"
);
}
#[tokio::test]
async fn an_idle_chat_is_evicted_and_its_state_reloads() {
let dir = tempfile::tempdir().unwrap();
let pool = crate::db::open_store(dir.path().join("e.db").to_str().unwrap()).unwrap();
let store = ChatStore::new(pool);
// Durable settings and no prompt at all: this chat used to sit in the
// cache (and in the per-chat lock map) for the process lifetime,
// because the sweep only ever looked at chats with an *expired* record.
store
.update(9, |data| {
data.forward_channel_id = Some(-100);
data.message_format.insert("twitter".into(), "{url}".into());
})
.await;
assert!(store.cache.lock().contains_key(&9));
let removed = store.prune_expired(Duration::from_secs(60)).await;
assert!(removed.is_empty(), "nothing had expired");
assert!(
!store.cache.lock().contains_key(&9),
"a chat with no live prompt must leave the cache"
);
assert!(!store.locks.lock().contains_key(&9), "…and its lock");
// The DB kept the row, so the next use reloads everything it held.
let data = store.get(9).await;
assert_eq!(data.forward_channel_id, Some(-100));
assert_eq!(
data.message_format.get("twitter").map(String::as_str),
Some("{url}")
);
}
#[tokio::test]
async fn a_live_prompt_keeps_its_chat_cached() {
let dir = tempfile::tempdir().unwrap();
let pool = crate::db::open_store(dir.path().join("k.db").to_str().unwrap()).unwrap();
let store = ChatStore::new(pool);
store
.update(10, |data| {
data.edit_message.insert(1, edit_entry(10, unix_now()));
})
.await;
store.prune_expired(Duration::from_secs(3600)).await;
assert!(
store.cache.lock().contains_key(&10),
"a live prompt holds its chat in the cache"
);
}
#[tokio::test]
async fn prune_eviction_keeps_the_persisted_state() {
// Every record expires → the chat is evicted from the cache; the
// pruned state must already be in the DB when that happens.
let dir = tempfile::tempdir().unwrap();
let pool = crate::db::open_store(dir.path().join("p.db").to_str().unwrap()).unwrap();
let store = ChatStore::new(pool);
store
.update(8, |data| {
data.template.insert("keep".into(), "[]".into());
data.edit_message.insert(1, edit_entry(8, 0));
})
.await;
let removed = store.prune_expired(Duration::from_secs(60)).await;
assert_eq!(removed, vec![(8, 1)]);
let data = store.get(8).await;
assert!(data.edit_message.is_empty());
assert_eq!(
data.template.get("keep").map(String::as_str),
Some("[]"),
"eviction dropped state the DB never received"
);
}
}
+108
View File
@@ -0,0 +1,108 @@
# Deployment reference for the Docker Hub image. Instance values (token, admins,
# site credentials, domain) live in `.env` next to this file — `docker compose`
# substitutes every `${VAR}` from it automatically — so this file stays in the
# repository unmodified. A variable that is not listed here is not passed into
# the container at all.
#
# JSON-file logs grow without limit by default: a long-running bot (and the
# proxy in front of it) will fill the disk. One cap, applied to every service
# below via the anchor.
x-logging: &default-logging
driver: json-file
options:
max-size: '10m'
max-file: '3'
services:
nginx-proxy:
image: nginxproxy/nginx-proxy:1.11.6-alpine
restart: always
environment:
# Routes requests with an unknown Host (i.e. plain IP access) here; set
# DEFAULT_HOST in .env to use it.
DEFAULT_HOST: '${DEFAULT_HOST:-}'
ports:
- '80:80'
- '443:443'
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- certs:/etc/nginx/certs:ro
- html:/usr/share/nginx/html:ro
networks: [proxy]
labels:
- 'com.github.nginx-proxy.nginx'
container_name: nginx-proxy
logging: *default-logging
acme-companion:
image: nginxproxy/acme-companion
restart: always
environment:
DEFAULT_EMAIL: '${DEFAULT_EMAIL:-}'
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- certs:/etc/nginx/certs:rw
- html:/usr/share/nginx/html:rw
- acme:/etc/acme.sh
networks: [proxy]
container_name: acme-companion
depends_on:
- nginx-proxy
logging: *default-logging
tgxmb:
image: yoursfunny/telegram-twitter-media-bot:latest
restart: always
environment:
# From .env (the instance's own values; see the env table in README.md).
TELOXIDE_TOKEN: '${TELOXIDE_TOKEN:-}'
BOT_ADMIN: '${BOT_ADMIN:-}'
PIXIV_REFRESH_TOKEN: '${PIXIV_REFRESH_TOKEN:-}'
TWITTER_AUTH_TOKEN: '${TWITTER_AUTH_TOKEN:-}'
BILIBILI_COOKIE: '${BILIBILI_COOKIE:-}'
VIRTUAL_HOST: '${VIRTUAL_HOST:-}'
WEBHOOK_URL: '${WEBHOOK_URL:-}'
WEBHOOK_SECRET_TOKEN: '${WEBHOOK_SECRET_TOKEN:-}'
# Defaults, listed so they are discoverable; override in .env when needed.
LOCAL_USER_ID: '${LOCAL_USER_ID:-1000}'
RUST_LOG: '${RUST_LOG:-info}'
EDIT_MESSAGE_TTL_SECONDS: '${EDIT_MESSAGE_TTL_SECONDS:-86400}'
LINK_CACHE_TTL_SECONDS: '${LINK_CACHE_TTL_SECONDS:-604800}'
CAPTION_QUOTE_TEXT_CHARS: '${CAPTION_QUOTE_TEXT_CHARS:-200}'
VIRTUAL_PORT: '${VIRTUAL_PORT:-8443}'
WEBHOOK: '${WEBHOOK:-true}'
WEBHOOK_LISTEN: '${WEBHOOK_LISTEN:-0.0.0.0}'
WEBHOOK_PORT: '${WEBHOOK_PORT:-8443}'
# For a certificate on a bare IP: uncomment and set ACME_HOST in .env.
# ACME_HOST: '${ACME_HOST:-}'
#
# Not listed on purpose: TELOXIDE_PROXY. Docker Desktop reaches a host
# proxy through host.docker.internal (a loopback address inside the
# container is the container itself), and teloxide panics on an *empty*
# value, so add the line deliberately when this deployment needs one:
# TELOXIDE_PROXY: '${TELOXIDE_PROXY}'
volumes:
- ./data:/app/data
networks: [proxy]
depends_on:
- nginx-proxy
container_name: tgxmb
logging: *default-logging
# Webhook mode only (in polling mode there is no listener, so drop this
# block or set WEBHOOK=true): the bot listens on WEBHOOK_PORT; nginx-proxy
# shows 502s while this is down, so surface it to the orchestrator.
healthcheck:
test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/${WEBHOOK_PORT:-8443}'"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
volumes:
certs:
html:
acme:
networks:
proxy:
name: proxy
-74
View File
@@ -1,74 +0,0 @@
services:
nginx-proxy:
image: nginxproxy/nginx-proxy:1.11.6-alpine
restart: always
ports:
- '80:80'
- '443:443'
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- certs:/etc/nginx/certs:ro
- html:/usr/share/nginx/html:ro
networks: [proxy]
labels:
- 'com.github.nginx-proxy.nginx'
container_name: nginx-proxy
acme-companion:
image: nginxproxy/acme-companion
restart: always
environment:
DEFAULT_EMAIL: ''
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- certs:/etc/nginx/certs:rw
- html:/usr/share/nginx/html:rw
- acme:/etc/acme.sh
networks: [proxy]
container_name: acme-companion
depends_on:
- nginx-proxy
tgxmb:
image: yoursfunny/telegram-twitter-media-bot:latest
restart: always
environment:
LOCAL_USER_ID: '1000'
TELOXIDE_TOKEN: ''
BOT_ADMIN: ''
PIXIV_REFRESH_TOKEN: ''
TWITTER_AUTH_TOKEN: ''
EDIT_MESSAGE_TTL_SECONDS: '86400'
LINK_CACHE_TTL_SECONDS: '604800'
RUST_LOG: 'info'
VIRTUAL_HOST: '<YOUR_DOMAIN>'
VIRTUAL_PORT: '8443'
# ACME_HOST: 'your.domain.com'
WEBHOOK: 'true'
WEBHOOK_LISTEN: '0.0.0.0'
WEBHOOK_PORT: '8443'
WEBHOOK_URL: 'https://<YOUR_DOMAIN>/'
WEBHOOK_SECRET_TOKEN: ''
volumes:
- ./data:/app/data
networks: [proxy]
depends_on:
- nginx-proxy
container_name: tgxmb
# Webhook mode only: the bot listens on WEBHOOK_PORT; nginx-proxy shows
# 502s while this is down, so surface it to the orchestrator.
healthcheck:
test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/8443'"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
volumes:
certs:
html:
acme:
networks:
proxy:
name: proxy
+168
View File
@@ -0,0 +1,168 @@
# Bilibili 动态支持:研究与实现记录
状态:已实现(`crates/x-media/src/site/bilibili/`)。本文记录上游调研、实测数据与最终设计;
长期契约以 `AGENTS.md` 为准。
范围:**只发动态里的图片与动图**。动态内嵌视频不发流,降级为封面图;`b23.tv` 短链不匹配;
视频页 / 番剧 / 直播间 / 专栏 / 音频均不支持。
---
## 1. 上游实现研究
### 1.1 nazurin`nazurin/sites/bilibili/`4 个文件 ~6 KB
- 入口正则:`t\.bilibili\.com/(\d+)``t\.bilibili\.com/h5/dynamic/detail/(\d+)``bilibili\.com/opus/(\d+)`
- 请求:`GET https://api.bilibili.com/x/polymer/web-dynamic/v1/detail?id={id}`,仅加 `Referer: https://t.bilibili.com/{id}`
**无 cookie、无 WBI 签名、无 `build` 参数**
- 错误:`code == 4101147` → not found`code != 0` 或缺 `data` → 报错。
- 媒体:只取 `item.modules.module_dynamic.major.draw.items[].src`;缩略图 `src + "@518w.jpg"`
`size` 字段单位是 **KB**`major` 为空或 `draw.items` 为空 → "No image found"。
**忽略视频、转发(forward)与纯文字动态**
- caption`"#" + module_author.name` + `module_dynamic.desc.text`,链接写死 `https://www.bilibili.com/opus/{id}`
### 1.2 telegram-bili-feed-helper`biliparser/provider/bilibili/`9 个文件 ~57 KB
- 9 个策略类(Video/Opus/Live/Audio/Read + Feed 基类 + Credential + api 工具):门禁正则
`bilibili\.com|b23\.tv|BV\w{10}|av\d+`,再分流,兜底 `client.head(url)` 跟随重定向后按子串分流。
- 动态:`GET /x/polymer/web-dynamic/desktop/v1/detail?id={id}&build=11605`**单条,无分页**);
客户端带桌面 UA、随机 `buvid3={uuid}infoc`;登录态用 `bilibili-api-python``Credential`
Redis 持久化 `SESSDATA/bili_jct/buvid3/buvid4/ac_time_value/DedeUserID`,扫码登录)。
- **同样没有 WBI 签名 / appkey 签名**playurl 用的是非 WBI 的 `/x/player/playurl`
- 媒体:`major.type` 分派 —— DRAW 取全部 `items[].src`ARCHIVE/PGC/ARTICLE/MUSIC/COMMON/LIVE
只取一张 `cover`;FORWARD 取原动态作者/正文并递归进 `orig` 找媒体。
- 视频:仅独立 video 策略解析(`qn` 720P→480P→360P 试 durl,再退 DASH + ffmpeg 合并);
**动态内嵌视频只发封面**
- 错误:要求 `status==200 && code==0`;风控 `-352`/`-412` 无特殊处理。
### 1.3 取舍
| 维度 | nazurin | bff | 本仓库 |
|---|---|---|---|
| 接口 | `v1/detail?id=` | `desktop/v1/detail?id=&build=` | `v1/detail?id=`(实测可用) |
| 认证 | 无 | buvid3 + SESSDATA | 默认匿名;可选 `BILIBILI_COOKIE` |
| WBI | 无 | 无 | 不实现(无需求) |
| 图片 | `major.draw.items` | 同 + forward 递归 | 同,加 `orig` 递归、`http→https``.gif → Animated` |
| 视频 | 完全忽略 | 动态内嵌视频发封面 | 发封面(不发流) |
| 短链 | 不匹配 | 跟随重定向 | 不匹配(多数短链是视频,会让"静默忽略"变成失败提示) |
---
## 2. 实测验证(2026-09-17,真实请求)
| 验证项 | 结果 |
|---|---|
| `v1/detail?id=`(无 cookie、UA `Mozilla/5.0`、带 Referer | `200 {"code":0}` ✅ |
| 同上,不带 cookie 也不带 Referer | `200 {"code":0}` ✅(无强制鉴权) |
| bff 的 `bilibili_pc/…Electron/22.3.27` UA | `code:-352` ❌ → **不要抄它的 UA** |
| `desktop/v1/detail?build=11605` | `code:-352` ❌ |
| `feed/space?host_mid=`(用户时间线) | 首次成功、随后 `-352`,也见过 HTTP 412 → **不碰** |
| 不存在 / 已删除的动态 | `code:500` "Cannot read property 'only_fans' of undefined"nazurin 的 4101147 已失效) |
| 非数字 id | `code:-400` param parsing failed |
| 图片 `i0.hdslb.com/bfs/new_dyn/*.jpg` | `HEAD 200 image/jpeg`,带/不带 Referer 均可;`+@518w.jpg` → 2542 KB ✅ |
| `t.bilibili.com/h5/dynamic/detail/<id>` | `200` ✅ |
| `m.bilibili.com/dynamic/<id>` | `302 → t.bilibili.com/<id>` ✅ |
| `www.bilibili.com/opus/<id>` | `200`,转发动态 `302 → t.bilibili.com/<id>` ✅ |
| `b23.tv/BV1JTtt6JEZu` | `302 → www.bilibili.com/video/BV…`(视频) |
| `b23.tv/<无效码>` | **HTTP 200** + `{"code":-404}` ⚠️ 短链判定不能只看状态码 |
| `playurl`(仅调研用,未采用) | `fnval=1` 匿名给 durl720P=9.18 MiB / 360P=2.97 MiB`fnval=4048` 匿名 DASH 上限仅 480P |
| `dyn_archive` 字段 | 有 `aid/bvid/cover/title/duration_text`**没有 `cid`**(所以发流要再来一次 `view` 请求) |
| **风控阶梯(同一 IP 连续请求后实测)** | ① 无 cookie → `-352`;② 仅 `buvid3` → 仍 `-352`;③ `buvid3`+`buvid4`(取自匿名 `/x/frontend/finger/spi`)→ **`code:0` 恢复**;④ 继续高频请求后 → 连同 buvid 一起 `-352`(此时只有登录 cookie 或换 IP) |
| **正文位置(24 条真实动态逐条审计)** | 有正文的动态都在 `module_dynamic.desc.text`(图文/转发/纯文字,含 34–193 字样本);**AV(视频投稿)动态 `desc` 恒为 `null`**,内容在 `major.archive.title` / `.desc` 卡片里 → 已做 title 回退 |
| **`features=itemOpusStyle` 的效果** | 同一端点带此参数后,图文帖改为 `major.opus` 形态:`pics[]`(图,key 是 `url`)、`summary.text`(正文,未截断,实测 307 字整段)、`title`(可选标题);不带参数则是 legacy `major.draw` + `desc`,而 **opus 图文帖的 `desc` 为 `null`、正文与标题完全丢失**`opus/1248857553488576532`legacy `desc:null`,带参数 `summary.text="[doge_金箍]黑白搭配"`)。AV / 转发帖不受该参数影响 → 适配器改为请求时带参数,并保留 legacy 形态兜底 |
| feed 与 detail 的差异 | `feed/space` 的 item 会把 `desc.text` 挖空,**只有 detail 有正文** → 排查时不要用 feed 数据判断正文缺失 |
| 不存在的 19 位 id | `4101105 请求数据发生错误`(提示可重试,但只出现在不可能存在的 id 上)→ 仍归入永久错误,见 `code_error` 注释 |
测试样本(live 测试用):
| 样本 | id | 期望 |
|---|---|---|
| 图片动态(2 图 + 话题) | `1245284537985925159` | 2 个 `Illustration``{tags}` = `ALin出道20周年快乐` |
| 转发动态 | `1248982077447077907` | 媒体来自 `orig`1 图),正文可含 `//@` |
| 视频动态 | `1248717597691609105` | 封面 1 张 `Illustration` |
| 纯文字动态 | `1246767523595026450` | `media` 为空 |
关键字段路径:
```
data.item.id_str
data.item.modules.module_author.{name,mid}
data.item.modules.module_dynamic.desc.text
data.item.modules.module_dynamic.topic.{id,name} # 单话题,{tags} 来源
data.item.modules.module_dynamic.major.{draw.items[].src, archive.cover}
data.item.orig # 转发时存在,结构与 item 相同
```
---
## 3. 实现
```
crates/x-media/src/site/bilibili/mod.rs # re-export
crates/x-media/src/site/bilibili/interface.rs # PATTERN / cache_key / enabled / is_retryable /
# media_headers / BilibiliSite / fetch / code_error /
# From<Item> for Fetched / caption / 12 单测 + 2 live
crates/x-media/src/site/bilibili/model.rs # 纯 Deserialize DTO(全 Option
```
- **正则**(同时用于分发、抽 id、缓存键,一个正则三用):
`^(?:https?://)?(?:www|t|m)\.bilibili\.com/(?:opus/|dynamic/|h5/dynamic/detail/)?(\d+)`
- **缓存键**`bilibili:<动态 id>``source_url` 统一 `https://www.bilibili.com/opus/{id}`
- **请求**`GET /x/polymer/web-dynamic/v1/detail?id=` + `Referer: https://www.bilibili.com/`
`Cookie` 头按优先级取:`BILIBILI_COOKIE` → 缓存的设备 cookie`GET /x/frontend/finger/spi``buvid3`/`buvid4`
进程内缓存一次;取不到就不带 cookie,仅 debug 日志)→ 无。指纹接口本身失败**不**让抓取失败。
走共享 `CLIENT`UA `Mozilla/5.0`30s 超时,`TELOXIDE_PROXY` 透传)。
- **错误映射**`0` → 成功;`-352/-412` 与 HTTP 412 → `Transient`(可重试,队列退避;首次记一条 warn 提示
`BILIBILI_COOKIE`);`500`/`4101147``NotFound`(永久);其他 code → `Site`(永久)。
- **媒体**
- `major.opus.pics[]`(带 `features=itemOpusStyle` 时的图文帖形态,字段名是 `url`)→ 每张一张图;
其次 `major.draw.items[]`legacy,字段名 `src`)→ 同样逐张;`http://` / `//``https://`,非 https 开头直接丢弃。
`.gif``Media::Animated``thumbnail_url` 留空,Telegram 自己取首帧——`@518w.jpg` 只对 jpg/webp 实测过),
其余 → `Media::Illustration``thumbnail_url = url + "@518w.jpg"`,兼作超大时的降级 URL)。
- `major.archive.cover` → 1 张 `Illustration`(视频不发流)。
- 转发且自身无媒体 → 递归取 `orig` 的媒体;正文拼 `//@{原作者}:\n{原文}`
- 其他 majorPGC/ARTICLE/MUSIC/LIVE/COMMON)不建模 → 无媒体,走既有 "No media found"。
- **正文 / title**(按信息量从多到少回退):`major.opus.title` + `major.opus.summary.text`
`module_dynamic.desc.text``major.archive.title`。三者分别对应:图文文档(标题+正文)、
legacy/转发帖正文、视频投稿卡片标题。开头结尾空白做 trim;整体再由既有 `truncate_caption` 截断。
- **caption**(与 misskey 同形):`{opus 链接}\n<a href="space.bilibili.com/{mid}">{name}</a>: {正文}`
`RenderData``{tags}` 来自话题名;正文由既有 `truncate_caption` 截断。
- **注册表**`SITES` 末尾追加 → `/set_format` 白名单、链接缓存、启动校验、日志前缀全部自动生效。
- **bot 侧仅文案**`handlers/commands.rs` 三处站点清单字符串 + `state.rs`/`handlers/mod.rs` 注释。
### 与原计划的偏差(及原因)
| 原计划 | 实际 | 原因 |
|---|---|---|
| `x/web-interface/view` + `playurl` 发视频 | 不做 | 需求收窄为图片/动图;视频只发封面 |
| `site/mod.rs``MAX_MEDIA_UPLOAD_BYTES` 常量 | 不加 | 没有视频尺寸决策就不需要该常量,避免跨 crate 耦合 |
| `b23.tv` 短链(跟随重定向) | 不匹配 | 多数短链指向视频,匹配后会把"静默忽略"变成用户的 "Failed to fetch media" |
| `validate()` 校验 cookie | 不做 | 匿名可用,cookie 失效不致命;校验要额外请求一个端点,收益低 |
| `media_headers` 给 hdslb 加 Referer | 返回 `None` | 实测图片与 durl 均无需 Referer(注释里记了这条验证) |
| 计划阶段认为设备 cookie 是 YAGNI,不实现 | **实现**`buvid3`+`buvid4`) | 计划之后做了对照实验:同一 IP 上"无 cookie → -352、只有 buvid3 → -352、buvid3+buvid4 → code:0",说明这是对本适配器主要失败模式的直接修复,而不是冗余保险 |
| 只用不带参数的 `v1/detail` | 加 `features=itemOpusStyle` | 用户实测反馈"有内容的动态没有 title":不带参数时 opus 图文帖返回 legacy 形态,`desc``null`,正文与标题整个丢失。带参数后同一 ID 返回 `major.opus.summary.text` / `title` / `pics`。AV / 转发帖不受影响,legacy 形态仍保留为兜底 |
---
## 4. 测试与验证
- 单元(13):正则匹配/拒绝/忽略短链、缓存键归一、图片映射(https 归一 + 缩略图 + `.gif → Animated`)、
封面、转发取 `orig` 媒体与正文拼接、纯文字无媒体、caption 转义、业务 code 分类(可重试性)、URL 归一、
设备 cookie 拼装。
- live3`#[ignore = "live network: …"]`):设备 cookie 可取、图片动态 2 图、纯文字动态无媒体。
CI 的 `live` job 已覆盖。动态接口被风控时这两条 live 测试打印 `skipping:` 并提前返回(与 pixiv 的
token 门控同款约定),设备 cookie 那条仍会真实执行。
- 实测命令:
`cargo run -p x-media --example fetch -- https://www.bilibili.com/opus/1245284537985925159`
(输出 2 张 `https://i0.hdslb.com/…jpg` + `@518w.jpg` 缩略图 + 话题 tags)。
- 全套:`cargo fmt --check``cargo clippy --workspace --all-targets -- -D warnings``cargo test --workspace` 全绿。
## 5. 已知限制
- 风控按 IP/请求量漂移,阶梯见 §2 最后一行:轻度靠设备 cookie 自愈,重度需 `BILIBILI_COOKIE` 或换 IP。
被拦时按**可重试**失败处理(队列退避)+ 一条 warn,不会静默丢帖。
- 接口 schema 会漂移(`module_dynamic.major` 实测可为 `null` 而正文留在 `desc`);DTO 全 `Option`
未知形态降级为"无媒体",不 panic。
- 动态内嵌视频只发封面图(与 bff 同策略),不下载流。
- 纯文字动态复用既有 "No media found" 回复。
- `b23.tv` 短链不被匹配(见上表)。
+4
View File
@@ -10,6 +10,10 @@
## 1. 现状摩擦清单
> ⚠️ 本节记录的是**重构前**的现状:其中的行号、以及 `site/mod.rs` 里的
> `fetch_once` 分派函数(当时的实现)都已不存在,仅作历史记录。当前形态见
> `site/mod.rs``SITES` 注册表——新增站点 = 新模块 + 注册一行。
以现有三站(twitter / bsky / pixiv)为基线,新增第 4 个站点(代号 `example`
今天需要触碰的位置: