The per-request CLIENT timeout and the per-chunk DOWNLOAD_IDLE_TIMEOUT both reset on progress, so neither bounded the fetch as a whole: a URL that keeps dripping (a trickle the idle timeout reads as life) could hold one of the eight FETCH_SLOTS for effectively ever, and the queue behind the slots is what then stalls. The attempts loop now runs inside tokio::time::timeout(900 s) — generous for a genuinely large ugoira zip on a honest slow link (minutes), fatal for a drip — and answers Transient with the cache key, so the retry and its jittered backoff take over. The permit is taken outside the timeout: the slot is released on either path. Audit low finding 'url_workers long tail' (process-wide total-deadline option).
Two gaps in fetch_with_attempts: the sleep was the bare 1 << attempt, so every worker that failed together (a source coming back, a shared proxy blip) also recovered on the same tick and re-stamped the source; and a429 arrived as an anonymous Transient, its Retry-After header — the one place a source tells you exactly how long it wants silence — dropped on the floor.
retry_wait(attempt, roll, err) is the pure decision: doubling base plus a random slice of itself ([base, 2x base)), floored at a named Retry-After. status_error now takes the response, reads the seconds-form header and returns the new FetchError::RateLimited { site, retry_after_secs } (HTTP-date parses to None and stays transient); the pure table moved to classify_status so tests still build it from bare status codes. The delay is capped at MAX_RETRY_AFTER_SECS = 60 — the header is server-supplied and must not park one of the eight fetch slots.
Everywhere a Transient meant 'retryable' the new variant joins: the Site trait default, pixiv's override (plus its ugoira-zip mapping), bsky's HLS second attempt, the download classifier, and the user-facing message arm. Tests pin the429 rows (with and without the header, cap included) and retry_wait's math; AGENTS' retries bullet and variant list follow.
apply_media_headers had exactly one caller — media_request, the documented 'one choke point' every download already goes through — so the site-header walk sits inside that choke point now instead of one indirection past it, and the two doc comments merge into the one that describes both rules (guard + headers).
Every adapter built RenderData { url } from the exact value it then moved into Fetched.source_url — a copy of the same canonical URL held in two places, feeding one placeholder. caption_with reads {url} from self.source_url now and the field is gone from the struct (bsky's and twitter's halves of this change landed with their caption commits; their render builders no longer set it either).
Tweet::caption duplicated site::caption byte for byte for non-empty text (and dropped the shared empty-text rule, leaving a dangling ': ' behind the author link when a tweet's text comes out empty after link expansion). It delegates now, like bsky's Post::caption a commit earlier and bilibili/misskey before them; encode_double_quoted_attribute loses its last twitter user.
Post::caption hand-formatted exactly the string site::caption produces — same template, same three escapes (the shared fn even handles the empty-text case these two lines did not: no dangling ': ' after the author link). One line now, matching what bilibili and misskey already do; the encode_double_quoted_attribute import loses its last bsky user.
ffmpeg_available() and log_once_ffmpeg_missing() only ever appeared together (both encoders, three lines each: check, log, return), and calling one without the other was a bug in waiting — a probe that never logged, or a log that never gated. ffmpeg_missing() is the single gate: false when the binary is there, otherwise log once per process and say yes. Both call sites shrink to one condition.
find_site walked the registry with "enabled && matches" and disabled_site walked it again with "!enabled && matches" — two passes re-running the same regexes to answer one question. matching_site finds the first match and the fetch decides what a disabled site means; the patterns are disjoint (one domain each), so "first enabled match" and "first match, then check" never disagree.
site_id_from_key split the prefix out and then walked the registry to confirm the prefix was a registered site id — a round trip over a value cache_key itself produced from that registry: the unknown and no-colon branches were unreachable for any key the bot makes. The single caller (the link-cache hit path) splits the prefix directly; the registry-echo test and the misskey assertion of it go with it.
Three holes in the live job. (1) It ran -p x-media only, on the comment that everything network-gated lives there — false: xmedia-bot carries two #[ignore]d live tests (repair refetch, text-only link) that CI never executed; both crates run now, still behind the 'live' name filter, and the bot crate's tests need no token (their sends go through MockSender). (2) BILIBILI_COOKIE was never passed, so the bilibili live tests always hit the risk-control early return. (3) That early return — like the pixiv download test's token gate — printed an indistinguishable 'skipping:' line that --show-output never showed, so with continue-on-error a fully-skipped weekly run read exactly like full coverage; skips now print a 'SKIP ' prefix, the steps tee with --show-output, and a summary step lists every skip in the run summary plus a workflow warning. The redundant 'Run token-gated tests' step (the whole non-ignored suite, a rerun of the test job) goes away with the comment that justified it.
The pixiv download test was token-gated but not #[ignore]d, so a shell with PIXIV_REFRESH_TOKEN exported made every local cargo test --workspace hit i.pximg.net — where it demonstrably flaked (twice in one session: TLS EOF mid-body) — and CI's non-ignored step ran it as a side effect of having a secret. It carries the convention's #[ignore = "live network: …"] now and is renamed with the live_ prefix so the live job's --ignored live name filter actually picks it up (the old name did not contain 'live' and would have been filtered out). Its skip prints the SKIP prefix the next commit's live job greps, and AGENTS.md stops claiming a local workspace run is not fully offline.
The URL workers bound only their own path (8 workers, bounded channel); inline queries, /debug and /test spawned fetches straight from handler tasks with no limit at all — and the expensive part runs inside the fetch (ugoira: up to 512 MiB plus ffmpeg; bsky: an HLS remux), so N users could mean N concurrent encodes. fetch_with_attempts now takes one permit from an 8-slot gate (the count matching URL_WORKERS, so the bot's own pipeline keeps its full width), which covers every entry at once: message links, queue-side repair, inline, /debug, /test, and retries. Unsupported and disabled links answer before the gate, and the shared-fetch dedup already waits outside x-media, so nothing double-counts.
download_media_limited had one hard-coded total (600s) for every caller, and its heaviest caller — the bot's upload fallback — holds a PREP slot (and its memory reservation) for the whole transfer: six slow-but-alive downloads (a byte every 29s satisfies the idle window) could stall the fallback chain for ten minutes, queue retries included. The budget is a parameter now: the fallback passes 300s of its own (50 MiB in 300s ≈ 1.4 Mbit/s; a slower link is better served by retrying toward the item's smaller URL than by pinning a slot), while bsky's in-fetch HLS segments keep the generous 600s DOWNLOAD_TOTAL_TIMEOUT, now pub(crate) and re-exported for them. download_too_slow reports whichever budget it got.
download_media_to_file took &mut std::fs::File and wrote every network chunk with a sync write_all on the executor thread — for a ugoira frame zip (up to 512 MiB) that is the whole download stalling a runtime worker, while bsky's remux had already been moved to tokio::fs for exactly this reason (its comment: a multi-megabyte std::fs::write blocks the executor thread). The signature takes &mut tokio::fs::File now; the single caller (pixiv's ugoira path) clones the NamedTempFile's handle — the clone shares the file offset, so the ZipArchive extraction in spawn_blocking reads what the download wrote — and drops it after the download hands the bytes to the OS.
status_error's catch-all called every unlisted status Transient, so a 400/405/418/451 got three retries per link before the same answer (twitter syndication's broken-token 400 being the live example), and download_status_error plus misskey's and bilibili's local fallbacks each carried their own copy of the table — bilibili and misskey classifying a 404 as Transient while the center classified it NotFound. The center now makes any client error except 408/429 a refusal (permanent), the media path delegates to it as status_error("media", ...) and its duplicated fn is deleted, and misskey/bilibili fall through to the center after their own special statuses (misskey's 400 body, bilibili's 412). A table test pins every class.
download_media_to_file's non-transport errors all folded into PixivError::Api, and Api is permanent in pixiv_error_is_retryable — so one 429/5xx, stalled transfer or temp-file write failure while fetching the ugoira frame zip permanently failed the whole post, while the bot's own upload downloads classify the very same classes as retryable (classify_download_error). A new PixivError::Transient carries those classes into the existing retry policy (and into startup validation's 'stays enabled' branch); Http keeps its arm and everything else stays permanent. The retryable table and the startup-validation loop pin both halves.
docker-compose injects PIXIV_REFRESH_TOKEN='' for a blank .env value and enabled() only checked that the variable exists, so a default deployment read pixiv as configured, sent an empty refresh token to OAuth and failed login validation on every boot (error line + admin DM) — and it disagreed with the empty-as-unset gating the tests already use. Filter empty strings in PIXIV_CLIENT and enabled(), like twitter and bilibili; a unit test pins empty == unset.
A page whose `original` the API left out (the restricted ones) was dropped
whole by the multi-page branch's filter_map — a missing picture in the album
while `large` sat right there — and the single-page branch produced no media
at all when both places it looks for an original were empty. Both now share
one builder that falls back to `large` (the same picture, lower resolution).
The two tests that pinned the old behaviour now pin the fallback.
site/mod.rs held the registry, the Site trait, Fetched, the caption helpers,
FetchError and, at the end, the whole media-download stack: two HTTP clients
with different timeouts, the guard that refuses a URL inside the host's own
network (start URL and every redirect hop), and the two streaming entry
points. That stack is its own reason to change and moves whole, with its
tests; CLIENT and the download fns stay re-exported at site::… so no adapter
or bot call site moved. SITES becomes pub(crate) for the header rule.
bilibili and misskey each had a byte-identical caption() (same escaping,
same empty-text early return, differing only in a named format argument).
site::caption holds it once; both adapters call it.
The one-photo MediaItemPayload literal was written out at six sites across
the urls and send test modules; ctx::test_support::photo_item holds it once.
permanent_error now delegates to api_error (it stays a fn pointer because
that is what MockSender::scripted takes), misskey's x_media_site_id wrapper
is gone in favour of the function it renamed, and PixivAPI is no longer
re-exported -- nothing outside pixiv/api.rs names it.
TypeModel's Illust and Manga variants existed only so serde would accept
those values -- both call sites ask a single question (is this ugoira?), so
the field is a String and the check is a comparison. Same for a type the API
adds later: it no longer fails the whole parse.
It existed only to pass u64::MAX to download_media_limited and was called
from this file's tests alone; the bot's own download path already calls
download_media_limited directly. The tests now exercise that function, with
the same uncapped bound.
site_name() only returned the public site_id field; the five callers (the
/debug report, the caption-format lookup on both fetch paths, the per-site
override in urls.rs) read the field now.
Every adapter set title (mostly None) on all three Media variants and
nothing ever read it: the bot's CachedMedia carries kind/file_id/url, and the
one read was misskey's own test. Removing it also drops misskey's
DriveFile.name, which existed only to feed it. Fetched::title (the post's
own headline, which captions do use) is untouched.
twitter, bsky, misskey and bilibili each carried enabled() -> true,
media_headers(url) -> None and is_retryable(err) -> the trait's own default,
with no caller outside their tests (the adapters never override those
methods, so the default was already the production policy). The tests that
only restated the default are gone; the two that pin site-specific classes
(bsky's MediaPrep, bilibili's risk-control codes) now ask the Site impl, and
site/mod.rs keeps one assertion of the shared retry policy. The live
verification notes (no Referer needed for hdslb/twimg) survive as comments.
twitter, twitter auth and bsky carried the same 404/410 -> NotFound,
401/403 -> Blocked, else Transient block (comments included).
site::status_error holds it once. bilibili and misskey keep their own
matches: neither maps 404/410 and each has a status the others do not
(412 risk control, 400 + NO_SUCH_NOTE), so routing them through the shared
block would have reclassified those statuses for the user.
`MEDIA_CLIENT` deliberately has no reqwest total timeout (a 30s cap made a
hundreds-of-MB ugoira zip impossible to deliver), and the idle window only
covers *silence*: a server that drips a chunk every 29 s keeps the download
alive indefinitely. On the bot's side each such download holds one of the
process-wide upload-prep slots (`send::upload`'s `PREP_SLOTS`, 6), so a
handful of trickling sources can take the whole fallback path out of service
without ever looking broken.
`DOWNLOAD_TOTAL_TIMEOUT` (600s) bounds the whole transfer, checked between
chunks — a transfer that completes just over the budget is kept rather than
thrown away, and a genuinely slow link (the case the cap was removed for)
stays far inside it. Reported as `Transient` like the idle-window stall: the
transfer may simply have been unlucky, and a retry restarts it.
Verified with a local trickling server (a 1 KiB chunk every 1.2s, chunked so
the client cannot see the total up front): with the budget temporarily
lowered to 2s the download aborted after 2425ms with
`transient: download exceeded 2s` — two chunks in, the server seeing the
client go away — proving the budget and not the 30s idle window ended it.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
`link_cache` is only written *after* a send succeeds, so two chats posting
the same link at the same moment each ran a full fetch: two sets of source
requests, and for an ugoira or a bsky video two ffmpeg encodes of the same
post — minutes of CPU for the second one. The same applies to a batch
forward racing a queued retry. Within one message `dedupe_urls` already
handled the duplicates; across calls nothing did.
`fetch_shared` keys an in-flight fetch by the post's cache key: the first
caller runs it, the rest subscribe and take its result. The entry is removed
by a guard when the fetch settles (cancellation included), so this dedupes
what is *concurrent* and never answers from an old result — a repeat later
fetches again, and a failure is deliberately not cached: the user is told to
try again, and a cached failure would answer that retry from a stale state.
Two hazards that shape the code, each with a test: a broadcast channel lives
while *any* sender does, so the waiter drops its own clone of the sender
before waiting — otherwise a cancelled sharer would leave it waiting forever
— and a waiter whose sharer vanished fetches for itself instead of failing a
link that is perfectly fetchable.
One fetched post can now serve several sends, which the temp files behind
`Fetched::keep_alive` had to support: the field is `Arc<TempDir>` and the
accessor hands out references (the bot's `KEEP_ALIVE` registry holds the
same), so the ugoira/bsky MP4 stays on disk until the *last* task settles
rather than the first. `take_keep_alive` is gone — a `take` could only ever
serve one of the senders.
Verified: 116 bot tests (three new sharing tests, including the cancelled
sharer) and 91 x-media tests (a new one pinning the keep-alive refcount)
pass, plus a live check that two concurrent `fetch_shared` calls for one
tweet return the very same `Arc` after one fetch's worth of wall time.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
x-media's reqwest had exactly `["json", "rustls-tls"]` — no decompression, no
HTTP/2 — so every adapter fetched its JSON as identity over HTTP/1.1. The
site APIs compress: twitter's syndication body measures 4469 bytes identity
against 1066 gzipped (4.2x) for a single-tweet response, and the fetch is on
every twitter link. The CDNs all negotiate h2 (the shared client reports
HTTP/2.0 against cdn.syndication.twimg.com, api.bilibili.com and
public.api.bsky.app), which also multiplexes the concurrent media downloads
that used to open a connection each.
Both features are one crate-level switch, so the pages' compression is what
the APIs answer with and requires no adapter change: reqwest adds
`accept-encoding: gzip` and decompresses transparently. `download_media_limited`
keeps a correct size cap either way — it checks the accumulated *body* while
streaming, not the declared Content-Length, which for a compressed response
is the compressed size.
Verified: a local echo server recorded `accept-encoding: gzip` on a request
from the shared client (both clients come from `build_client`), and a probe
over the shared `CLIENT` reported `HTTP/2.0` for the three site hosts above.
`cargo test -p x-media -- --ignored live` (14 passed) covers the real
endpoints with the new transport.
Cargo.lock gains async-compression (+codecs/core), fnv and h2.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
`cookie()` cached the device cookies only on success: the `Option<String>`
it held could not tell "not fetched yet" from "the fetch failed", so a failed
fingerprint meant *every* later post re-asked the SPI endpoint — one extra
round trip per post, forever, exactly the case the fingerprint exists for (a
flagged IP, where that endpoint is the thing answering `-352`/412).
The cache is now `Option<Option<String>>`: the outer level is "an attempt has
been made", the inner one is the cookie it produced, so a failure is
remembered as no-cookie and reaches the request path unchanged. The guard is
also dropped before the request instead of being held across it, which had
serialized every concurrent bilibili fetch behind that one round trip.
A racing first pair still costs a duplicate fingerprint call
(`get_or_insert`, first writer wins) — never a wrong cookie.
Verified live: the 5 bilibili tests (`--ignored`, including
`live_fingerprint_yields_device_cookies` and the four dynamic fetches that
send the cookie) pass. The failure path itself has no unit test: `SPI_URL` is
a const, so there is no seam to make the endpoint fail on demand.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
A long bluesky video can be 500 segments, and the remux downloaded them
strictly one at a time: the user waited for every round trip in turn, which
is the dominant cost of the whole remux (the ffmpeg concat itself is local).
Each segment's multi-megabyte body also went to disk through a blocking
`std::fs::write` on an executor thread.
Segments now download and write under a small bound (`SEGMENT_CONCURRENCY`,
4 — a segment can be 20 MiB and the playlist is capped at 256 MiB, so this is
also what bounds the remux's peak memory) and the write goes through
`tokio::fs`. Concurrent downloads complete in completion order, and ffmpeg
concatenates the list in whatever order it holds — an out-of-order list is a
*silently* scrambled video, not an error — so `concat_list` sorts by segment
index and carries its own test.
x-media's own tokio features gain `rt` (JoinSet) and `fs`: the library
already used `spawn_blocking` on the strength of the bot crate's features.
Verified against a local HLS fixture — 12 one-second segments of solid
red/green/blue, each served with a 150 ms delay, reached through a name that
resolves to loopback (the guard refuses a literal 127.0.0.1) — with a
temporary in-module test: all 12 sampled frames come back in the right colour
order, and the server recorded a peak of 4 requests in flight, where the
serial version showed 1.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
`fetch` classified the response with `parse_syndication_body` — which parses
the whole body into a `serde_json::Value` — and then threw that value away
and re-parsed the same text into a `SyndicationTweet`. Two full JSON scans
and two allocations of every string in the body (text, entities, media
details) per tweet.
`Tweet::from_syndication_value` takes the value classification already
built and deserializes from that: `from_value` moves the strings out of the
tree instead of allocating copies, so the response text is scanned once.
The auth fallback had the mirror image of the same waste — it built the
syndication shape as a `Value` and then serialized it back to a string for
a parse — and the 8 test call sites lose their `.to_string()` with it.
Verified live: `cargo test -p x-media -- --ignored live` (14 passed),
including the 5 twitter fetches that exercise this path.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
A failed bsky video remux was reported as `FetchError::Transient`, so the
fetch loop retried the *whole* adapter — master playlist, variant playlist
and up to 500 segments again (256 MiB of cap each time), for a failure that
happened near the end of the work the retry was about to redo. The message
had to stay honest (returning `Ok` with no media reads as "this post has no
media"), so it needed a class of its own.
`FetchError::MediaPrep` is that class: post fetched, media could not be
prepared locally, not retryable (the per-site `is_retryable` whitelist
excludes it by construction), with its own user-facing text — the generic
"failed to fetch" would have hidden that the download or encode was what
broke. The retry itself is not lost, it moved: `fetch_hls` retries the
request that actually failed, once, for the classes a retry can change
(transport, 429/5xx).
Checked the sibling sites before widening the change: pixiv already degrades
to no media when the ugoira encode fails (`Ok(None)`), and its frame-zip
download failure is the last step so a re-fetch re-does only that; twitter's
auth leg replays two cheap metadata GETs and a GraphQL 5xx *is* worth
retrying. Neither needed the new class.
Verified with a throwaway proxy harness: a 503 answered twice-in-a-row path
costs 2 requests and succeeds, a 404 costs exactly 1 and fails (no pointless
retry). `site::bsky::interface::tests::media_prep_failure_is_not_retried`
pins the classification.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
The upload fallback asked `x_media::site::media_size` for every remote item
before downloading it. That call is a real GET (not a HEAD) on the *un-
guarded* `CLIENT` — so every fallback item cost two requests where one would
do, the response body was never consumed (the connection cannot return to
the pool), and for photos the answer was discarded outright
(`too_large && !matches!(item, Photo { .. })` still fired the request). It
bypassed `media_request`'s private-network guard as well, the one choke
point every other egress goes through.
For videos the probe was redundant twice over: `download_media_limited`
reads the declared Content-Length before any body byte and aborts with
`FetchError::TooLarge`, which the call site already turns into the item's
smaller URL (`FallbackError::MediaTooLarge` → `fallback_url`).
`media_size` is deleted (no other caller) and the download's own cap is the
only size gate. The video cap is now exactly `MAX_UPLOAD_BYTES` instead of
`MAX_UPLOAD_BYTES + 1`, so the boundary the probe drew survives byte for
byte: a file of exactly the cap is admitted (`len > max_bytes` is false),
one byte over degrades to the smaller URL.
Verified with a throwaway harness (a local HTTP server reached through
`TELOXIDE_PROXY`, the one LAN egress the guard allows): a small video, a
photo and an oversized video each cost 1 request where the probe made it 2,
and the oversized one still lands on `/fallback.mp4` without fetching it.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
The repo runs the ignored network tests with `cargo test -- --ignored
live`, which matches on the test name; the new redirect test had no
`live_` prefix and would have been skipped by it.
Every test went through `MockSender`, so `media_sender`'s `Bot`
implementation — the URL it builds, the multipart it sends, the per-chat
limiter and the bot-wide budget it charges — was never exercised, and neither
was any handler reached from a real update. The two things that made that hard
are gone:
- `media_sender::test_support::fake_api::FakeApi` is a stand-in for
`api.telegram.org`: a `tokio` TCP listener that reads one HTTP/1.1 request
(JSON or multipart), records it and answers the smallest result the method
needs. No new dependency, and `Bot::new(token).set_api_url(api.url())`
points a real `Bot` at it. Note for future tests: teloxide keys methods by
payload type, so the path is `SendMediaGroup`, not `sendMediaGroup`.
- `message_handler` built its own `AppContext::from_statics` internally, so no
test could reach its branches; its body is now `handle_message(ctx, bot,
message)` with `message_handler` as the thin `dptree` entry.
Tests: a media group through the real `Bot` (asserting the multipart fields —
chat, media URL, caption — and that the send charged the chat's limiter), the
forward button through the real callback path (`CopyMessages`,
`DeleteMessage`, `AnswerCallbackQuery` with the prompt's ids and the toast
text), and `handle_message` twice (a prompt reply becoming an
`EditMessageCaption`, and a supported link in a group producing the one
explanatory `SendMessage`).
Also closes the redirect-hop gap left open by the download guard: the live
`a_redirect_into_the_hosts_network_is_refused` follows a public redirector to
`169.254.169.254` and asserts the policy refuses the hop (verified against
httpbin.org here, and by mutation — disabling the hop check fails it).
Docs: AGENTS.md's testing conventions and untested-modules list (the Bot
implementation and the handler branches are covered now; `main.rs`'s
startup/shutdown and its `dptree` tree still are not).
`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (201 passed, 16 ignored) clean.
The media URLs the bot fetches come from a site's own API response (media
URLs, `fallback_url`, thumbnails), `build_client` left reqwest's default
redirect policy in place (up to 10 hops, any host), and the downloaded bytes
are uploaded to Telegram — so a response pointing at a cloud metadata
endpoint would read it back into a chat.
`media_request` is now the one choke point both download paths go through:
http(s) only, and a host that is no address or name of the host's own network
(`blocked_ip` covers loopback, private, link-local, unspecified, broadcast,
documentation, multicast, IPv6 unique-local/link-local, IPv4-mapped, plus
CGA-NAT and benchmarking ranges; `is_local_name` covers `localhost` and
`*.local`). A refusal is `FetchError::Blocked` — permanent, so the send path
does not retry a URL that would be refused again (a connection error used to
be retryable and burned attempts). The same guard runs on every redirect hop
through a custom redirect policy, keeping reqwest's 10-hop cap.
Deliberate gap, documented at the function: DNS rebinding (a name the site
controls resolving to a private address) needs a `reqwest::dns::Resolve`
wrapper, which would also resolve the operator's own proxy host — and
`TELOXIDE_PROXY` is routinely a LAN address — so it would take down working
deployments to block a much less likely attack.
Verified: three offline tests (the address table, the URL table, and a
refusal that holds with nothing listening at the metadata endpoint), both
mutations confirmed to fail them (guard disabled → the download test fails;
link-local dropped from `blocked_ip` → 169.254.169.254 is accepted), and the
live `download_media_pixiv_original_with_referer` still fetches from
i.pximg.net through the guarded client, so real media downloads are
unaffected. `cargo fmt`, `cargo clippy --workspace --all-targets --locked --
-D warnings`, `cargo test --workspace --locked` (198 passed, 15 ignored)
clean.
Three gaps the last audit list named, all in the "resource growth, background
timers and limits nobody watches" class.
**Idle inline-query entries are pruned.** `DebounceStates` had no eviction at
all: one entry per user who ever used inline mode, forever, while the rate
limiter's buckets and the chat store both prune in the 300s sweep. Entries
now carry a `last_seen` stamp and `prune_idle_states()` drops the ones idle
past 300s — the window Telegram caches an inline answer for
(`cache_time(300)`), after which a repeat reaches the bot again and has to be
answered fresh, so the entry would only suppress a fetch the user is waiting
for. The boundary is tested through `prune_idle_at(now, idle_for)` so it does
not depend on ageing a monotonic clock.
**The 300s sweep is a function, and tested.** It was an inline `tokio::spawn`
block: the expiry edit (the only part that talks to Telegram) had no test at
all. It is now `periodic_sweep(sender, chat_store, link_cache, task_queue,
config, stop)`, which also prunes the inline entries, driven in a test with
`start_paused` — the loop's own timer fires the tick, exactly one expired
prompt is rewritten in place, a live one keeps its record and buttons. The
interval is pinned as a constant because no assertion on the edits can see it
(a shorter one produces the same single edit; the paused clock can jump past
the boundary while a tick's DB work is in flight). To make the edit reachable
at all, `edit_message_text` joined the `MediaSender` trait (Bot impl + mock
recording), which is also what keeps `main.rs`'s remaining `Bot` calls
unambiguous. `main.rs` leaves the "untested modules" list except for
startup/shutdown and the dispatcher tree.
**The bot-wide send budget exists.** Telegram throttles a bot in total
(~30 msg/s) as well as per chat; only the per-chat bucket existed, so a batch
forward fanned out over many chats was unguarded and earned 429s the queue
then retried. `acquire_global` charges the same spend against a single shared
bucket at the three paced sites (`send_media_group`, `send_animation`,
`copy_messages`). The unpaced ones (`send_message`, the edits, the toasts) stay
unpaced on purpose: they are one call per action, far below the ceiling, and
pacing a user-visible reply would delay it. Not covered: that the send paths
call it (they need a real `Bot`), which is the same structural gap as the
dispatcher tree.
Also: the startup token-exchange decision is now `startup_validation(result)`
instead of living inside the `Site::validate` future, so "a 5xx while the
container comes up must not disable pixiv" is asserted as a decision — the
message the admin gets plus `enabled()` unchanged. The rejected-credential
half is deliberately not exercised: it calls `disable()`, a process-wide flag
with no reset, and a test touching it would order-couple every other pixiv
test.
Verified: `cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (184 passed, 14 ignored) — plus
mutations, each confirmed to fail the relevant test: the sweep not being
driven on its timer, the interval shortened to 60s, and (earlier) the queue
sweep's missing wake-up. Dropped an empty leftover `crates/x-media/tests/`
directory while there (never tracked by git).
Audit of all 205 tests (five read-only passes plus a line-by-line
re-check). Ten test functions were removed or merged and eight
subsumed assertion blocks trimmed; the suite is down to 180 tests with
no loss of mutation coverage, and four tests that were passing for
nothing now fail when the code they name is broken.
Redundant (deleted or merged):
- twitter: `syndication_text_only_has_no_media` (re-asserts its own
empty fixture), `..._keeps_multibyte_text` (both transforms are
no-ops for that text), `..._strips_trailing_short_link_without_entities`
(same branch as `..._media_short_link`, which now also covers the
real multibyte tweet), `..._regardless_of_index_units` (its
`display_text_range` rationale outlived the function it described).
- pixiv: `test_fetch` (a bare `is_ok()` on the illustration
`download_media_pixiv_original_with_referer` already asserts and
downloads, and the only network touch in a plain `cargo test`),
`startup_validation_only_disables_on_a_definitive_failure` (four rows
that are a subset of the retry-policy table; the `validate()` branch
it was named for is not asserted at all).
- bilibili: `from_item_legacy_draw_shape_still_parses` (its fixture is
the same legacy `draw` shape `from_item_maps_draw_images_and_topic`
builds, with a subset of its assertions).
- site/mod.rs: two `Ok(None)` cases merged into one test.
- urls.rs: `cache_hit_success_keeps_the_cache_entry` (the `/test` test
asserts the same two things under stricter settings), plus a
`assert_ne!` loop that re-states the mapping assertions above it.
- send/mod.rs: `media_group_success_and_forward_ok` (the forward half is
covered by `post_send_forwards_immediately_when_configured`; the
`is_ok()` half cannot see the returned file ids), and two boundary
rows implied by the constant they sit next to.
- commands.rs: the parse tail that `every_documented_invocation_parses`
already covers per README form, and three `debug_report` rows the
escaping test pins with stronger input.
Passing for nothing (now real):
- `truncate_caption_does_not_split_an_html_entity` — the cut lands
inside the entity, so `!contains("&")` never fired; it now asserts
the exact output in both directions and fails when the guard in
`truncate_caption` is deleted (verified).
- `pipeline_resizes_oversized_jpeg` — magic bytes and a non-empty buffer
pass for a copy-through; it now decodes the output's headers and
fails when the JPEG branch skips the resize (verified).
- `live_validate_with_bogus_token_fails` — expected `PixivError::Api`,
which the status check before the body read made unreachable; a bogus
token is a 4xx. Confirmed against the live endpoint: the old
assertion fails with `got Err(Status(400))`, the new one passes.
- bsky `live_fetch_with_photos` — its URL is a text-only post and it had
a byte-identical twin, so no live test pinned media; it now points at a
labelled post with photos and asserts media + the label (live-verified).
Also fixed, found by turning the runtime-sweep test into a real one:
the 30 s lease-expiry sweep recovered crashed rows but never woke a
worker, so a recovered task waited for the next unrelated enqueue (every
worker is parked on `notify` when no row is pending). `recover_update`
now reports its count, `recover_expired` wakes a worker when it changed
something, and `runtime_sweep_recovers_expired_lease` drives the spawned
loop with a paused clock instead of calling the recovery by hand — it
fails on both the missing wake-up and a sweep that recovers nothing.
Verified: `cargo fmt --check`, `cargo clippy --workspace --all-targets
--locked -- -D warnings`, `cargo test --workspace --locked` (180 passed,
14 ignored) and `cargo test -p x-media -- --ignored live` (13 passed).
P2 (hardening) of the retry audit, closing the report's remaining findings.
- Lease fencing. `lease_next` now stamps a random `lease_token`, and every
write-back a worker makes (the 30s heartbeat, `delete_row`, `reschedule`,
`mark_done`) is guarded by it. A lease that expired while its holder was
stalled and was then re-leased used to let *both* holders write the same row:
one duplicated the send, the other silently discarded the new holder's retry
(a 0-row update was not even logged). Now a worker that no longer holds the
lease drops its attempt at the next heartbeat and writes nothing. Reaching
existing databases needed a migration chain, which `db.rs` had been
pre-committed to: `MIGRATIONS` + `migrate` track `PRAGMA user_version`, with
`schema_init` as the version-0 baseline. Verified on a database created
before this change: user_version 0 -> 1, column added, rows intact.
- Dead-letter notifications no longer mislabel an unparsable payload. A row
whose payload no longer deserializes as a `Task` (an older version's shape,
corruption) used to skip the cache invalidation *and* report "Forward failed
permanently" for a send task, because both were derived from the parsed
value. The identity now comes off the raw JSON, so the stale link-cache entry
is dropped and the message names the post.
- Temp files are marked and swept. Every temp file/dir the project creates now
carries `x_media::TEMP_FILE_PREFIX`, and startup removes entries with that
prefix older than an hour — a killed process leaves its downloads (up to
hundreds of MB) behind because no destructor runs, and the age gate keeps the
sweep away from a second instance's in-flight files. Verified live: the log
reports the sweep, an aged leftover goes, a fresh prefixed file and an
unrelated file stay.
P1 of the retry audit, from the report's "reliability and diagnosis" batch.
- Media downloads no longer share the 30s *total* timeout of metadata
fetches. The size caps allowed 10 MiB (reupload fallback) and 512 MiB
(ugoira frame zip) while the clock allowed 30s, so a slow link made those
posts impossible: `MEDIA_CLIENT` has no total timeout and instead bounds
the response head and every chunk with a 30s *idle* window, which keeps the
stalled-connection protection. Verified against a local probe: the old
policy aborts a 40s download at 30.0s, the new one completes it (2 MiB,
40.1s), and a body that stops delivering still fails after exactly 30s.
- A finished row's write-back is no longer best-effort. `delete_row` failing
left the row `in_progress` with a live lease, so the next sweep flipped it
back to `pending` and re-ran a completed task — a second album, a second
prompt, a second channel copy. Both terminal writes are now retried, and a
delete that still fails falls back to a `done` tombstone that neither the
lease query nor the sweep looks at; reschedule (no safe tombstone: marking
it done would drop the retry silently) logs what the sweep will do.
- bsky and pixiv no longer present a *failed* video conversion as a post with
no media: the remux/ugoira error propagates (pixiv keeps its retry class,
bsky reports Transient), so the user sees the real cause and `fetch` gets
its retries. bsky's "no ffmpeg" case stays a degradation — retrying a
deployment gap cannot help.
- pixiv's token exchange checks the HTTP status before parsing the body, so a
429/5xx from the OAuth endpoint stays retryable instead of becoming a
permanent Api/Json error (via the shared `pixiv_error_is_retryable`), and
startup validation only disables pixiv for a rejected credential — one 503
while the container came up used to turn every later pixiv link into
"pixiv support is disabled".
P0 of the retry audit. The main finding: a Telegram 5xx was classified
Permanent, so one Telegram-side blip dead-lettered the post.
- `classify_request_error`: a server error is retryable again. teloxide sleeps
10s on a 5xx and then parses the body, so the HTTP status is gone by the
time the error arrives; it is recognised by shape instead — a JSON
server-error description, or an `InvalidJson` whose raw body is not JSON
(a proxy/error page). A JSON body of the wrong shape stays permanent, since
retrying a type mismatch cannot help. Reproduced end to end: with the old
classification a fake 502 (HTML body) logged "failed permanently" and
dead-lettered; now it logs "queued for retry" and the retry delivers.
- The same class of mistake elsewhere: `is_media_fetch_failure` was missing
`failed to get HTTP url content`, the description single-media URL sends
answer with, so hotlink-rejected media failed permanently instead of going
through the reupload fallback.
- `enqueue_retry` now reports whether the row was written, and the callers
only promise a retry when it was — a failed enqueue (DB write) used to tell
the user "retrying in Ns" and then deliver nothing, ever.
- A forward that fails retryably now settles the prompt instead of leaving it
live: the queued row carries the message ids itself, and a live prompt let
a second Confirm copy the same messages to the channel twice and let Skip
answer "nothing was forwarded" while the row still delivered.
- A prompt that could not be sent no longer swallows the gated forward
silently: the chat is told, since nothing would ever forward.
- `scaled_retry_delay` only scales up, so a server-asked `retry_after` above
the 300s cap is honoured instead of retried early (which earned another 429
and then dead-lettered the post).
- Download classification: a 4xx media download is permanent (the media is
gone or refused) while transport errors and 429/5xx retry — previously every
download error counted as retryable and burned the whole budget. A temp-file
*write* failure retries too (resource exhaustion clears; a temp dir that
cannot be created stays permanent).
- Site status mapping: 401/403 are `Blocked` (permanent) rather than
`Transient`, so a refusal is reported at once instead of after three
wasted attempts; and a twitter 200 that is not a tweet is no longer
reported as withheld content (the empty `{}` withheld shape keeps
`Sensitive`, which is what triggers the auth fallback).
P0 (foundation) + P1 (diagnostic depth) of the logging plan:
- main.rs initializes the timed builder with a default filter of
`info,hyper_util=warn,reqwest=warn`. Without RUST_LOG nothing was logged at
all (env_logger falls back to `error`), so `docker run --env-file .env` was
silent, and the plain `init` had no timestamps.
- info-and-above lines stop printing user URLs (fetch/send failures, inline
fetch, bsky's remux warnings). The full URL, the message text and the inline
query move to `trace`, so a `debug` log can be handed to someone else.
- Lifecycle lines name the chat and the post: sent/failed/queued plus the
total `ms`, the edit prompt, the channel forward, and every queue line
(`chat=` + `[key=…]` + per-attempt `ms`, dead-letters included).
- Queue work is visible: `x-media`'s fetch line carries its duration (ugoira
encode and HLS remux included), and the 300s sweep reports the pending count
and how overdue the oldest task is — only when the queue is non-empty.
- URL workers are supervised like the queue workers: a panicking worker used
to die silently and shrink the pool for the rest of the process.
- Degradations that still serve the user (cache/state write or read failures,
a failed chat action) are `warn`, not `error`.
Verified against the scripted fake-API harness: unset RUST_LOG logs info with
timestamps, `debug` carries no user URL, `trace` does, a cache-hit send logs
`chat=111 in 5ms`, a failing send queues and dead-letters with chat+key, and
the sweep reports the pending retry.
`/start` was "Hello!" and `/help` was the bare command list teloxide can
render — no argument syntax, no caption placeholders, no mention that
links only work in private chats. Both now carry that guidance, and the
bot's profile description / short description are set at startup so a
shared link says what the bot does.
`/settings` reports what this chat is configured to do (forward channel,
edit-before-forward, per-site formats, saved templates) to anyone in the
chat — `/bot_dict` is a raw admin-only dump. Templates can be removed
(`/remove_template`, listing the live names on a typo) and the prompt's
keyboard folds 3 per row with a cap: Telegram rejects a keyboard over 100
buttons outright, which would silently drop the whole prompt.
Inline results hand URLs to Telegram, which fetches them without any
site headers — pixiv's pximg.net answers 403 to that, so those items are
skipped instead of shipped broken. `needs_media_headers` answers that
question from the same per-site rule the downloader uses.
Dead-letter and retry notices name the failing post and the cause
(`failure_text`), since "Task failed after retries: task failed after 2
retries" said neither which link it was nor what happened.
Four ways a user could get silence are closed: a registered-but-disabled
site (pixiv without a token) now answers instead of being dropped as an
unsupported link, `/test` on such a link replies instead of doing nothing,
a supported link posted in a group gets a one-line hint (channels stay
silent), and fetch failures name their cause — gone / withheld / source
risk control / site disabled / source down — instead of one generic
sentence. `FetchError::Disabled` carries the "matched but switched off"
answer, which `find_site` used to fold into `Ok(None)`.
A withheld tweet no longer degrades to "no media": without
`TWITTER_AUTH_TOKEN` it stays `Sensitive` so the reply says the media is
age-restricted, and a failed authenticated fallback propagates its own
class instead of masquerading as an empty post (`empty_fetched` is gone).
Long jobs stop looking stalled: `run_with_chat_action` re-sends the chat
action every 4s while the pipeline is pending and the hint switches from
typing to send-photo/video once the media kinds are known. Media groups
go from 9 to Telegram's 10.
`/set_format` rejects unknown `{…}` placeholders (a typo used to be
published verbatim in every caption) and resets with `-`. The
edit-before-forward prompt states its TTL and that Confirm is required,
gains a Skip button, and is rewritten in place to "expired" by the sweep
— an edit, never a new message, so a background timer cannot wake a chat.
YoursFunny