Three places disagreed with AGENTS' own level rules: get_chat/get_chat_administrators failures during /set_channel printed at error even though the usual cause is a mistyped channel name the user fixes on the spot (error is reserved for dead letters and the like — now warn, same as other user-served degradations); the edit-before-forward callback arrival line logged at info while the sibling arrival lines for commands and inline queries log at debug (now debug, still carries the escaped payload); and the two upload-fallback lines logged at info while photo.rs's parallel degradations log at warn (now warn both — Telegram failing to fetch our media is a degradation that leaves the user served). AGENTS' convention sentence is now explicit about all three instead of the ambiguous '(fallback, retry enqueue, dead-letter is error)'.
Two gaps in fetch_with_attempts: the sleep was the bare 1 << attempt, so every worker that failed together (a source coming back, a shared proxy blip) also recovered on the same tick and re-stamped the source; and a429 arrived as an anonymous Transient, its Retry-After header — the one place a source tells you exactly how long it wants silence — dropped on the floor.
retry_wait(attempt, roll, err) is the pure decision: doubling base plus a random slice of itself ([base, 2x base)), floored at a named Retry-After. status_error now takes the response, reads the seconds-form header and returns the new FetchError::RateLimited { site, retry_after_secs } (HTTP-date parses to None and stays transient); the pure table moved to classify_status so tests still build it from bare status codes. The delay is capped at MAX_RETRY_AFTER_SECS = 60 — the header is server-supplied and must not park one of the eight fetch slots.
Everywhere a Transient meant 'retryable' the new variant joins: the Site trait default, pixiv's override (plus its ugoira-zip mapping), bsky's HLS second attempt, the download classifier, and the user-facing message arm. Tests pin the429 rows (with and without the header, cap included) and retry_wait's math; AGENTS' retries bullet and variant list follow.
A button press on an edit-before-forward prompt already dropped records past EDIT_MESSAGE_TTL lazily (before the 300 s sweep clears them); a text reply to the same prompt did not — it rewrote the caption from a record the sweep was about to treat as dead, so the two paths disagreed about what 'expired' means. The reply path now applies the same rule: drop the stale record, return false, and let the message flow on as if no prompt existed. The test seeds a prompt90000 s old (past the TTL under any config a parallel test can hold) and pins all three effects: no edit consumed, no API call, record gone.
set()/update() logged a failed DB write and returned nothing: the cache already held the new value, so /set_format answered 'Format set' for a change that vanishes on the next restart — a promise the retry queue's enqueue path is already forbidden from making. Both now return whether the write landed, and /set_format's two save paths (set and reset) answer 'in memory only … lost on restart' when it did not. The flag travels as the second tuple element; callers that only relay the value destructure it, callers that ignore it are untouched. A false cannot be forced in a test without a pool-failure seam, so the honest path is pinned by the code, not a fixture.
A Telegram display name, callback payload or channel handle is attacker-controlled text, and it went straight into info/error lines: one embedded newline forged a second log entry, and control characters could hide inside a line (log injection, audit SEC-006). handlers::log_escape replaces newlines, carriage returns and other control characters with visible escapes — clean input borrows, so logging allocates nothing extra — and every site that prints such a value uses it: the callback arrival line, the set-forward-channel info line (name + handle) and both get_chat error lines. The test pins that no raw newline can survive; AGENTS' logging convention names the helper.
InlineKind mirrored CachedMediaKind variant for variant (Photo/Video/Gif) and existed only to feed url_result, with two conversions kept in step: Media → InlineKind at the fetch call and CachedMediaKind → InlineKind at the cache call (plus the enum and the mapper). url_result takes CachedMediaKind now — the type the cache path already carries and the fetch path maps to with the same three-arm match — and the degraded-video skip collapses to a guard before the call. One enum fewer between a media kind and its Telegram result.
site_id_from_key split the prefix out and then walked the registry to confirm the prefix was a registered site id — a round trip over a value cache_key itself produced from that registry: the unknown and no-colon branches were unreachable for any key the bot makes. The single caller (the link-cache hit path) splits the prefix directly; the registry-echo test and the misskey assertion of it go with it.
debug_report wrapped every caption in <blockquote>, but preview_caption for a post over CAPTION_QUOTE_TEXT_CHARS already carries quote_long_caption's expandable blockquote — and the API rejects nested entities, so /debug on any long post answered 400. The wrapper now skips a caption that already carries a blockquote, exactly the rule quote_long_caption itself applies; the new test pins that the quoted caption passes through with a single blockquote while the existing test keeps pinning the wrap for ordinary captions.
Inline results were URL-only: Telegram fetches an inline result's URL itself
and cannot send site headers, so every pixiv item (and every locally encoded
ugoira/bsky MP4) was skipped and such a query answered empty. A post that is
already in the link cache now answers with InlineQueryResultCached* built
from its Telegram file ids — no fetch, no upload, and the hotlink-protected
case simply works. A degraded entry (file ids gone) falls back to URLs, and
there a video with no poster is skipped (Telegram would try to render the
mp4 as its own thumbnail).
The answer call moves onto MediaSender (answer_inline_query, mirroring the
other user-flow methods), which is what makes the path testable at all: the
two new tests drive the cache answer and the degraded/empty answer through
TestStores + MockSender, which recorded nothing about inline before.
The result builders are shared by both paths now (url_result/cached_result +
inline_kind), so the fetch path's behaviour is unchanged.
A post with no media was answered with "No media found or media type is not
supported.", throwing away text the fetch had already parsed, escaped and
built a caption for (the per-site format and the long-post quoting
included). It now goes out as a message through the same caption the media
path would attach — the senders' own quoting is applied here, since there is
no sender to do it. No queue entry: there is no Task shape for text and a
post with nothing to download is cheap to paste again, so a failure is
reported (send::send_text_post) rather than retried.
Proven live: live_a_text_only_link_is_sent_as_text fetches a real text-only
tweet through url_media and asserts one send_message carrying the post link
and no media send.
MediaItemPayload carried one String field with two meanings and a file_id
bool beside it to say which, in all three variants; every reader re-checked
the flag (input_file had a three-arm pattern just to find the file-id case).
MediaRef::Source/FileId says it once, and the readers now match on it —
local_media_paths, item_url, input_file and the download path each shrank to
the one branch they care about.
Fixes a real failure that the flag was hiding: send_animation built its
InputFile with input_file_for(media_url), which read a cached file id as a
local path and answered "local media file missing" — permanent. So the
second request for a single-gif post always failed (the third worked, from
the degraded entry). It now uses the payload's own input_file, and
a_cached_animation_sends_by_file_id fails without that line.
Wire shape: a queued row from before this change no longer parses, and the
queue already handles that shape (handle_task dead-letters it as an invalid
payload, and dead_letter_notify still names the post and drops the stale
cache entry).
urls.rs carried five reasons to change: the job channel and its worker pool,
the single-flight fetch, URL parsing, the per-URL pipeline, and the startup
repair of queued retries. The two with their own lifecycle move out:
- url_workers.rs: the bounded channel, its supervised pool and
start/stop_url_workers (the pipeline stays in urls.rs, which the workers
call).
- repair.rs: needs_refetch/apply_refresh/refetch/repair_lost_local_media with
their tests, moved whole (the live one keeps its #[ignore]).
Also folds the two byte-identical render_fields -> CachedPost mappings
(urls.rs and repair.rs) into urls::cached_snapshot, and moves the shared
permanent_error test fixture into ctx::test_support.
execute_command reached for CHAT_STORE/LINK_CACHE/CONFIG in 16 places, which
is why the one handler body with no test could not have one: those statics
point at the real $DATA_DIR/task_queue.db, so any test would have written to
the developer's state. It now takes &AppContext (the shape handle_message
uses) and answers through the given sender; bot stays for what the
MediaSender surface does not carry (channel admin lookups, the HTML report).
Three tests cover what that unlocks: the settings/set_format round-trip
against the chat's own store (including the refused placeholder), the admin
gate on both admin-only commands, and /debug answering without sending.
Replacing is_group(kind) with teloxide's predicates dropped the parentheses:
&& binds tighter than ||, so the branch read as is_group() OR (is_supergroup()
AND has-link) -- a plain group got the hint for any link, supported or not.
The e2e test only covered a group with a supported link (which is true either
way); it now also covers an unsupported link in a group.
/bot_dict and /clear_cache each resolved the sender id and replied "Admin
only." on their own; the helper returns the id or answers the refusal, so
both arms read as one line.
/bot_dict's dump and /debug's report each cut a string at a byte boundary
and appended an ellipsis, with slightly different bookkeeping (one left a
byte for the ellipsis, one did not). cap_text is the version that keeps the
result within the cap, with a test covering the exact-fit, truncating and
multi-byte cases.
Four call sites (cache hit, fresh fetch, startup refetch, /debug preview)
spelled out the same message_format.get(...).cloned().unwrap_or_default();
the accessor names the lookup and keeps the empty-format contract in one
place.
is_group hand-rolled what Chat::is_group/is_supergroup/is_private already
answer (verified in teloxide-core's chat.rs), and the private check was a
ChatKind pattern match. The unit test that exercised the helper asserted
teloxide's semantics; the channel case it guarded (a channel must not get the
group hint) is now asserted through handle_message instead, next to the group
case.
The one-photo MediaItemPayload literal was written out at six sites across
the urls and send test modules; ctx::test_support::photo_item holds it once.
permanent_error now delegates to api_error (it stays a fn pointer because
that is what MockSender::scripted takes), misskey's x_media_site_id wrapper
is gone in favour of the function it renamed, and PixivAPI is no longer
re-exported -- nothing outside pixiv/api.rs names it.
site_name() only returned the public site_id field; the five callers (the
/debug report, the caption-format lookup on both fetch paths, the per-site
override in urls.rs) read the field now.
Every adapter set title (mostly None) on all three Media variants and
nothing ever read it: the bot's CachedMedia carries kind/file_id/url, and the
one read was misskey's own test. Removing it also drops misskey's
DriveFile.name, which existed only to feed it. Fetched::title (the post's
own headline, which captions do use) is untouched.
caption_from_fields already returns truncate_caption(built_in) for an empty
format, which is exactly what the if-branch did; passing cached.caption as
built_in makes both paths one call. The format path is unmoved -- built_in is
read only when the format is empty.
build_send_task and apply_refresh each wrote the same animation-vs-sequence
branch and the same two 13-field literals; Task::from_items takes the
delivery envelope (chat, reply, forward/edit settings, notify targets) once.
The two callers keep computing that envelope from their own source -- chat
settings vs. the queued row being replaced -- so the repair path's delivery
semantics are unchanged.
`link_cache` exists so a repeat link costs nothing: no source request, no
download, no upload. It was written only on a *successful* send, and a send
that failed permanently deleted the entry — so the user's immediate retry, the
one case where they are most likely to try again, re-fetched everything:
site requests, a download, and for a ugoira or a bsky video a full ffmpeg
encode. Invalidation is right about the cause (the cached Telegram file id is
what went stale) and wrong about the cure (the media and its URLs are usually
fine).
Cached media now carries the source URL it was sent from, and a permanent
failure *degrades* the entry: the file ids are cleared, the URLs and the
caption fields stay, and the next request sends from those URLs — Telegram
fetches the media (or the upload fallback does) with no source round trip.
That is the same media a fresh fetch would have produced (site CDN URLs are
stable per post), and it is bounded: an entry that is already degraded, or one
from before this field existed, is removed instead, so a dead post still ends
up re-fetched and reported rather than retried forever.
Verified: a cached send that fails permanently leaves the entry with its URL
and no file id, a second failure drops it, and a degraded entry sends the
media with no fetch at all (the mock records no reply, which is what the
fetch-error path would have produced). 124 bot tests + 91 x-media tests pass,
including a direct test of the two payload shapes.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
Every item was skipped — `needs_media_headers` for pixiv's pximg.net, or a
local ugoira/bsky MP4 that does not parse as a URL — and the function fell
through to `Ok(false)`, which the debounce reads as "retry this query". The
client got no answer at all (a spinner with nothing behind it) and every
keystroke re-ran the fetch, while an unsatisfiable link is the *permanent*
truth for that query: Telegram fetches inline result URLs itself and never
sends a Referer.
It now answers the empty result set with a 300 s window, so the client stops
spinning and the same query is served from Telegram's cache and from the
debounce (which only releases on a failure). A *failed* fetch still releases,
so a retry is not answered from a stale empty answer.
Verified with a real pixiv link through a real `Bot` against the stand-in
API: `AnswerInlineQuery` with `results: []` and `cache_time: 300` (and
`Ok(true)`, so no release).
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
`link_cache` is only written *after* a send succeeds, so two chats posting
the same link at the same moment each ran a full fetch: two sets of source
requests, and for an ugoira or a bsky video two ffmpeg encodes of the same
post — minutes of CPU for the second one. The same applies to a batch
forward racing a queued retry. Within one message `dedupe_urls` already
handled the duplicates; across calls nothing did.
`fetch_shared` keys an in-flight fetch by the post's cache key: the first
caller runs it, the rest subscribe and take its result. The entry is removed
by a guard when the fetch settles (cancellation included), so this dedupes
what is *concurrent* and never answers from an old result — a repeat later
fetches again, and a failure is deliberately not cached: the user is told to
try again, and a cached failure would answer that retry from a stale state.
Two hazards that shape the code, each with a test: a broadcast channel lives
while *any* sender does, so the waiter drops its own clone of the sender
before waiting — otherwise a cancelled sharer would leave it waiting forever
— and a waiter whose sharer vanished fetches for itself instead of failing a
link that is perfectly fetchable.
One fetched post can now serve several sends, which the temp files behind
`Fetched::keep_alive` had to support: the field is `Arc<TempDir>` and the
accessor hands out references (the bot's `KEEP_ALIVE` registry holds the
same), so the ugoira/bsky MP4 stays on disk until the *last* task settles
rather than the first. `take_keep_alive` is gone — a `take` could only ever
serve one of the senders.
Verified: 116 bot tests (three new sharing tests, including the cancelled
sharer) and 91 x-media tests (a new one pinning the keep-alive refcount)
pass, plus a live check that two concurrent `fetch_shared` calls for one
tweet return the very same `Arc` after one fetch's worth of wall time.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
Two calls per link went out that could not affect anything:
- `run_with_chat_action` awaited the opening `send_chat_action` to completion
before the pipeline was polled at all, and again inside the loop on every
`ACTION_REFRESH`. Telegram round trips are hundreds of ms: the first delay
came out of the user's wait for every link, and each refresh suspended the
fetch (an ugoira encode or HLS remux runs for seconds) by the same amount.
- `handle_message` enqueued *every* URL a private chat posted, including
links no site adapter claims. Those cost a queue slot, a worker wake-up,
a `Message` clone and (through the action above) one Telegram call, only
for `url_media_inner` to conclude there was nothing to send. The group
branch has always made the `cache_key(url).is_some()` test before it acts;
the private branch now makes it before it enqueues.
The in-flight action is held (`Option<BoxFuture>` — the sender surface is
already type-erased, so it is `Unpin`) and polled as its own `select!`
branch: still polled *before* the pipeline, so the indicator is on screen
before the first send, but a slow Telegram response can no longer delay the
pipeline, and none of the branch bodies ever awaits one. One action is in
flight at a time; a refresh while one is unanswered is skipped rather than
dropping the request mid-flight. Note that `select!` evaluates every branch's
future expression eagerly, so the `None` case is an `async` block whose
`unwrap` only runs when the branch is polled (the eager form panicked).
Behavior pinned by the existing tests, unchanged: the opening action precedes
the first send, a 12s pipeline still sees exactly three actions
(`a_long_pipeline_keeps_the_chat_action_alive`), and an unsupported URL
reaching `url_media` still gets the one indicator before the pipeline settles
— in production it no longer reaches `url_media` at all.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
A failed bsky video remux was reported as `FetchError::Transient`, so the
fetch loop retried the *whole* adapter — master playlist, variant playlist
and up to 500 segments again (256 MiB of cap each time), for a failure that
happened near the end of the work the retry was about to redo. The message
had to stay honest (returning `Ok` with no media reads as "this post has no
media"), so it needed a class of its own.
`FetchError::MediaPrep` is that class: post fetched, media could not be
prepared locally, not retryable (the per-site `is_retryable` whitelist
excludes it by construction), with its own user-facing text — the generic
"failed to fetch" would have hidden that the download or encode was what
broke. The retry itself is not lost, it moved: `fetch_hls` retries the
request that actually failed, once, for the classes a retry can change
(transport, 429/5xx).
Checked the sibling sites before widening the change: pixiv already degrades
to no media when the ugoira encode fails (`Ok(None)`), and its frame-zip
download failure is the last step so a re-fetch re-does only that; twitter's
auth leg replays two cheap metadata GETs and a GraphQL 5xx *is* worth
retrying. Neither needed the new class.
Verified with a throwaway proxy harness: a 503 answered twice-in-a-row path
costs 2 requests and succeeds, a 404 costs exactly 1 and fails (no pointless
retry). `site::bsky::interface::tests::media_prep_failure_is_not_retried`
pins the classification.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
Every test went through `MockSender`, so `media_sender`'s `Bot`
implementation — the URL it builds, the multipart it sends, the per-chat
limiter and the bot-wide budget it charges — was never exercised, and neither
was any handler reached from a real update. The two things that made that hard
are gone:
- `media_sender::test_support::fake_api::FakeApi` is a stand-in for
`api.telegram.org`: a `tokio` TCP listener that reads one HTTP/1.1 request
(JSON or multipart), records it and answers the smallest result the method
needs. No new dependency, and `Bot::new(token).set_api_url(api.url())`
points a real `Bot` at it. Note for future tests: teloxide keys methods by
payload type, so the path is `SendMediaGroup`, not `sendMediaGroup`.
- `message_handler` built its own `AppContext::from_statics` internally, so no
test could reach its branches; its body is now `handle_message(ctx, bot,
message)` with `message_handler` as the thin `dptree` entry.
Tests: a media group through the real `Bot` (asserting the multipart fields —
chat, media URL, caption — and that the send charged the chat's limiter), the
forward button through the real callback path (`CopyMessages`,
`DeleteMessage`, `AnswerCallbackQuery` with the prompt's ids and the toast
text), and `handle_message` twice (a prompt reply becoming an
`EditMessageCaption`, and a supported link in a group producing the one
explanatory `SendMessage`).
Also closes the redirect-hop gap left open by the download guard: the live
`a_redirect_into_the_hosts_network_is_refused` follows a public redirector to
`169.254.169.254` and asserts the policy refuses the hop (verified against
httpbin.org here, and by mutation — disabling the hop check fails it).
Docs: AGENTS.md's testing conventions and untested-modules list (the Bot
implementation and the handler branches are covered now; `main.rs`'s
startup/shutdown and its `dptree` tree still are not).
`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (201 passed, 16 ignored) clean.
`extract_urls` decides whether a pasted link is seen at all — a bug there is
silence for the user, which is the complaint this bot's UX work keeps coming
back to — and it had no test: it takes a teloxide `Message`, which is not
worth building by hand.
Split it into the two decisions that are ours and keep the offset work
(teloxide's `parse_entities` turning entities into slices of the message
text) in the thin `extract_urls` composition:
- `url_of(kind, text)`: a bare `Url` entity is its own text, a `TextLink`
keeps its target (not the words the user sees), everything else carries
none.
- `dedupe_urls`: first occurrence wins, deduped by normalized post id — so
`/status/1`, `/status/1/photo/1` and a text link to the same post are one
entry — and by exact text for URLs no site claims.
Verified by mutation, each confirmed to fail the new tests: dropping the
`TextLink` branch, and deduping by raw text (`left: [.../status/1,
.../status/1/photo/1]`).
`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` clean.
`db.rs`'s two rules — `schema_init` is the version-0 baseline and never gains
a column, `MIGRATIONS` is append-only and never edited — were enforced by
comments only. Both have a silent failure mode across releases, and the worst
one (a baseline edit) makes `open_store` fail with `duplicate column name`,
i.e. a fresh deployment that will not start.
Three tests, no production change:
- A pre-migration database (the historical DDL written out literally, so an
edit to the baseline shows up here instead of being followed) upgrades
through `open_store`: version at the latest, exactly the migrated column
set, rows intact.
- The shipped migration text is frozen and compared entry by entry; the
assertion names the rule when it fires. Appending still passes — that is
the one allowed change.
- A fresh database lands at the latest version, so a deployment that only
ever saw fresh databases is on the same schema as an upgraded one, and
re-opening the same file is a no-op.
Verified by mutation, both confirmed to fail the new tests: editing the
shipped migration (`shipped_migrations_are_frozen`, with the rule in the
message) and adding the column to `schema_init` instead of a migration
(`a_fresh_database_lands_at_the_latest_version`, `duplicate column name:
lease_token`).
Docs synced for this and the previous two items: `main.rs` (startup repair),
`queue.rs` (`runnable_rows`/`replace_payload`), `handlers/` (`urls.rs`'s
repair, `mod.rs`'s `apply_caption_edit`), `db.rs` (the migration tests) and
the untested-modules list (`db.rs` now covered for migrations; the bot-side
live test renamed to match the repo's `--ignored live` filter).
`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (193 passed, 15 ignored) and
`cargo test -p x-media -- --ignored live` (13) clean.
Both caption-edit paths logged the error and carried on as if they had worked:
`edit_message_handler` consumed the user's reply (`let _ =`), and the
template button updated the prompt record and dismissed its toast with no
text. Since the edit surface is not rate limited, a 429 or a transient
failure meant the caption never changed and the user got no hint — the text
they sent was simply gone.
`apply_caption_edit` (handlers/mod.rs) is now the one place that applies a
caption edit and reports the outcome:
- A failure the API calls worth retrying (`classify_request_error` →
`Retryable`) is retried once when the delay is at most 2 s — a reply or a
button press has already been consumed by then, so a long flood-control
wait must not stall the chat's update queue behind it.
- Otherwise the caller reports it: the reply path answers the user ("Could
not update the caption (…). Send it again to retry."), the template path
puts it in the callback toast and leaves the record alone — a swap that
never happened must not be recorded as the prompt's template.
Tests: the swallowed-failure test now asserts the notice (it pinned the old
silent behaviour), plus a short `RetryAfter` that is retried and lands, a
60 s one that is not retried and is reported instead, and the template
button's failure toast with the record left unchanged.
`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (190 passed, 15 ignored) clean.
One note: the first full run tripped `download_media_pixiv_original_with_
referer`, the token-gated pixiv CDN download test that AGENTS already
documents as a local-network flake; it passes in isolation and on the rerun.
A queued retry that holds a local file — the ugoira MP4, a bsky remux, or a
temp file the reupload fallback downloaded — could never succeed after a
restart: those files live in the system temp dir and `send::KEEP_ALIVE`, the
registry that keeps them alive for the retry, is in memory. The row retried
into an upload error, said nothing about why, and dead-lettered the user's
link even though the payload carries the `source_url`.
`handlers::repair_lost_local_media` now runs in `main` before any worker
starts (so no row can be leased while it writes payloads, which is why it can
replace them without the lease guard a worker's write-back carries):
- `Task::local_media_paths` decides which rows are affected: any local path
that is gone. A partially delivered album is left alone — its remaining
batches cannot be reconciled with a fresh media list without risking a
second copy of what the user already received.
- The post is re-fetched from `source_url` through the ordinary `site::fetch`,
so a repaired task looks like a first send: fresh media, the chat's caption
format, a fresh link-cache snapshot, and a new keep-alive entry when the
re-fetch produced another local file.
- The delivery envelope (chat, reply, forward/edit settings, notify targets) is
kept, the attempt budget restarts, and nothing counts as sent.
- A post that cannot be fetched again (gone, withheld, site down) notifies the
user with that reason instead of letting the retry die on a missing file.
New queue plumbing: `runnable_rows()` (pending + in-progress rows, read before
the workers exist) and `replace_payload()` (rewrites the payload, resets
`attempts`, marks the row pending).
Verified: 5 new offline tests (the two decisions above against a real temp
file, the queue scan/replace, and the envelope-preserving rewrite) plus
`a_lost_local_media_row_is_refetched_from_its_post`, a live test that seeds a
row pointing at a missing file with a real bsky post as its source and asserts
the row now carries http(s) media and that nothing was sent — run against the
live API here. `cargo fmt`, `cargo clippy --workspace --all-targets --locked --
-D warnings` and `cargo test --workspace --locked` (187 passed, 15 ignored)
are clean.
Three gaps the last audit list named, all in the "resource growth, background
timers and limits nobody watches" class.
**Idle inline-query entries are pruned.** `DebounceStates` had no eviction at
all: one entry per user who ever used inline mode, forever, while the rate
limiter's buckets and the chat store both prune in the 300s sweep. Entries
now carry a `last_seen` stamp and `prune_idle_states()` drops the ones idle
past 300s — the window Telegram caches an inline answer for
(`cache_time(300)`), after which a repeat reaches the bot again and has to be
answered fresh, so the entry would only suppress a fetch the user is waiting
for. The boundary is tested through `prune_idle_at(now, idle_for)` so it does
not depend on ageing a monotonic clock.
**The 300s sweep is a function, and tested.** It was an inline `tokio::spawn`
block: the expiry edit (the only part that talks to Telegram) had no test at
all. It is now `periodic_sweep(sender, chat_store, link_cache, task_queue,
config, stop)`, which also prunes the inline entries, driven in a test with
`start_paused` — the loop's own timer fires the tick, exactly one expired
prompt is rewritten in place, a live one keeps its record and buttons. The
interval is pinned as a constant because no assertion on the edits can see it
(a shorter one produces the same single edit; the paused clock can jump past
the boundary while a tick's DB work is in flight). To make the edit reachable
at all, `edit_message_text` joined the `MediaSender` trait (Bot impl + mock
recording), which is also what keeps `main.rs`'s remaining `Bot` calls
unambiguous. `main.rs` leaves the "untested modules" list except for
startup/shutdown and the dispatcher tree.
**The bot-wide send budget exists.** Telegram throttles a bot in total
(~30 msg/s) as well as per chat; only the per-chat bucket existed, so a batch
forward fanned out over many chats was unguarded and earned 429s the queue
then retried. `acquire_global` charges the same spend against a single shared
bucket at the three paced sites (`send_media_group`, `send_animation`,
`copy_messages`). The unpaced ones (`send_message`, the edits, the toasts) stay
unpaced on purpose: they are one call per action, far below the ceiling, and
pacing a user-visible reply would delay it. Not covered: that the send paths
call it (they need a real `Bot`), which is the same structural gap as the
dispatcher tree.
Also: the startup token-exchange decision is now `startup_validation(result)`
instead of living inside the `Site::validate` future, so "a 5xx while the
container comes up must not disable pixiv" is asserted as a decision — the
message the admin gets plus `enabled()` unchanged. The rejected-credential
half is deliberately not exercised: it calls `disable()`, a process-wide flag
with no reset, and a test touching it would order-couple every other pixiv
test.
Verified: `cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (184 passed, 14 ignored) — plus
mutations, each confirmed to fail the relevant test: the sweep not being
driven on its timer, the interval shortened to 60s, and (earlier) the queue
sweep's missing wake-up. Dropped an empty leftover `crates/x-media/tests/`
directory while there (never tracked by git).
The same fixtures were rebuilt in five test modules: a `CachedPost`
literal in `link_cache.rs`, `handlers/urls.rs` and twice in
`send/mod.rs`, the edit-before-forward prompt in `handlers/mod.rs` and
`handlers/callback.rs`, and a scripted API error in both handler
modules. They now live in `ctx::test_support` next to `TestStores`:
- `cached_photo()` — the canonical cached post (photo + file id at
`https://x.com/u/status/1`, key `twitter:1`); tests mutate the fields
they care about, as the caption-quote test already did.
- `seed_prompt(template, created_at)` + `PROMPT_ID`/`FORWARDED_ID` —
the prompt record, the chat template and the bound forward channel.
The two former copies differed only in which knob the caller set (the
callback tests backdate it for the expiry cases, the reply tests pick
the template), so the union is one helper.
- `api_error(message)` — construction only; each test module keeps its
own message constant, because the wording is what that module's path
answers with (`chat not found` vs `message not found`).
`send/mod.rs`'s `cached_sequence_cache_data()` (which re-extracted the
post out of the task it had just built) is gone: the two settle tests
seed the cache from the same builder the task uses.
No behaviour change: the values are the ones the tests used except
`file_id` (`AgAC-file-id` everywhere, asserted in the link-cache
round-trip) and `sensitive` (the unasserted `true` in the link-cache
fixture), and every test still passes unchanged.
Verified: `cargo fmt`, `cargo clippy --workspace --all-targets --locked
-- -D warnings` and `cargo test --workspace --locked` (180 passed, 14
ignored).
Audit of all 205 tests (five read-only passes plus a line-by-line
re-check). Ten test functions were removed or merged and eight
subsumed assertion blocks trimmed; the suite is down to 180 tests with
no loss of mutation coverage, and four tests that were passing for
nothing now fail when the code they name is broken.
Redundant (deleted or merged):
- twitter: `syndication_text_only_has_no_media` (re-asserts its own
empty fixture), `..._keeps_multibyte_text` (both transforms are
no-ops for that text), `..._strips_trailing_short_link_without_entities`
(same branch as `..._media_short_link`, which now also covers the
real multibyte tweet), `..._regardless_of_index_units` (its
`display_text_range` rationale outlived the function it described).
- pixiv: `test_fetch` (a bare `is_ok()` on the illustration
`download_media_pixiv_original_with_referer` already asserts and
downloads, and the only network touch in a plain `cargo test`),
`startup_validation_only_disables_on_a_definitive_failure` (four rows
that are a subset of the retry-policy table; the `validate()` branch
it was named for is not asserted at all).
- bilibili: `from_item_legacy_draw_shape_still_parses` (its fixture is
the same legacy `draw` shape `from_item_maps_draw_images_and_topic`
builds, with a subset of its assertions).
- site/mod.rs: two `Ok(None)` cases merged into one test.
- urls.rs: `cache_hit_success_keeps_the_cache_entry` (the `/test` test
asserts the same two things under stricter settings), plus a
`assert_ne!` loop that re-states the mapping assertions above it.
- send/mod.rs: `media_group_success_and_forward_ok` (the forward half is
covered by `post_send_forwards_immediately_when_configured`; the
`is_ok()` half cannot see the returned file ids), and two boundary
rows implied by the constant they sit next to.
- commands.rs: the parse tail that `every_documented_invocation_parses`
already covers per README form, and three `debug_report` rows the
escaping test pins with stronger input.
Passing for nothing (now real):
- `truncate_caption_does_not_split_an_html_entity` — the cut lands
inside the entity, so `!contains("&")` never fired; it now asserts
the exact output in both directions and fails when the guard in
`truncate_caption` is deleted (verified).
- `pipeline_resizes_oversized_jpeg` — magic bytes and a non-empty buffer
pass for a copy-through; it now decodes the output's headers and
fails when the JPEG branch skips the resize (verified).
- `live_validate_with_bogus_token_fails` — expected `PixivError::Api`,
which the status check before the body read made unreachable; a bogus
token is a 4xx. Confirmed against the live endpoint: the old
assertion fails with `got Err(Status(400))`, the new one passes.
- bsky `live_fetch_with_photos` — its URL is a text-only post and it had
a byte-identical twin, so no live test pinned media; it now points at a
labelled post with photos and asserts media + the label (live-verified).
Also fixed, found by turning the runtime-sweep test into a real one:
the 30 s lease-expiry sweep recovered crashed rows but never woke a
worker, so a recovered task waited for the next unrelated enqueue (every
worker is parked on `notify` when no row is pending). `recover_update`
now reports its count, `recover_expired` wakes a worker when it changed
something, and `runtime_sweep_recovers_expired_lease` drives the spawned
loop with a paused clock instead of calling the recovery by hand — it
fails on both the missing wake-up and a sweep that recovers nothing.
Verified: `cargo fmt --check`, `cargo clippy --workspace --all-targets
--locked -- -D warnings`, `cargo test --workspace --locked` (180 passed,
14 ignored) and `cargo test -p x-media -- --ignored live` (13 passed).
P0 of the retry audit. The main finding: a Telegram 5xx was classified
Permanent, so one Telegram-side blip dead-lettered the post.
- `classify_request_error`: a server error is retryable again. teloxide sleeps
10s on a 5xx and then parses the body, so the HTTP status is gone by the
time the error arrives; it is recognised by shape instead — a JSON
server-error description, or an `InvalidJson` whose raw body is not JSON
(a proxy/error page). A JSON body of the wrong shape stays permanent, since
retrying a type mismatch cannot help. Reproduced end to end: with the old
classification a fake 502 (HTML body) logged "failed permanently" and
dead-lettered; now it logs "queued for retry" and the retry delivers.
- The same class of mistake elsewhere: `is_media_fetch_failure` was missing
`failed to get HTTP url content`, the description single-media URL sends
answer with, so hotlink-rejected media failed permanently instead of going
through the reupload fallback.
- `enqueue_retry` now reports whether the row was written, and the callers
only promise a retry when it was — a failed enqueue (DB write) used to tell
the user "retrying in Ns" and then deliver nothing, ever.
- A forward that fails retryably now settles the prompt instead of leaving it
live: the queued row carries the message ids itself, and a live prompt let
a second Confirm copy the same messages to the channel twice and let Skip
answer "nothing was forwarded" while the row still delivered.
- A prompt that could not be sent no longer swallows the gated forward
silently: the chat is told, since nothing would ever forward.
- `scaled_retry_delay` only scales up, so a server-asked `retry_after` above
the 300s cap is honoured instead of retried early (which earned another 429
and then dead-lettered the post).
- Download classification: a 4xx media download is permanent (the media is
gone or refused) while transport errors and 429/5xx retry — previously every
download error counted as retryable and burned the whole budget. A temp-file
*write* failure retries too (resource exhaustion clears; a temp dir that
cannot be created stays permanent).
- Site status mapping: 401/403 are `Blocked` (permanent) rather than
`Transient`, so a refusal is reported at once instead of after three
wasted attempts; and a twitter 200 that is not a tweet is no longer
reported as withheld content (the empty `{}` withheld shape keeps
`Sensitive`, which is what triggers the auth fallback).
P2 of the logging plan (the README recipe landed with the code change):
- `docker-compose.yml.example`: one `x-logging` anchor applied to all three
services. json-file grows without limit by default, so a long-running bot
and the proxy in front of it fill the disk; capped at 10m × 3 files.
- The startup `config:` line now reports what the process actually resolved —
the state DB path (a mistyped `DATA_DIR` or a surprising CWD was invisible
until it bit), both TTLs, the caption-quote setting (`off` rather than a bare
`0`) and whether a proxy is configured. The proxy URL is never printed (it
may embed credentials) and admin ids — chat identifiers — stay at `debug`.
Verified: `docker compose config -q` accepts the file, and a scripted fake-API
run shows `caption quote off` / `link cache TTL 3600s` under overrides,
`proxy=yes` with no credential in any line, and the ids at `debug` only.
P0 (foundation) + P1 (diagnostic depth) of the logging plan:
- main.rs initializes the timed builder with a default filter of
`info,hyper_util=warn,reqwest=warn`. Without RUST_LOG nothing was logged at
all (env_logger falls back to `error`), so `docker run --env-file .env` was
silent, and the plain `init` had no timestamps.
- info-and-above lines stop printing user URLs (fetch/send failures, inline
fetch, bsky's remux warnings). The full URL, the message text and the inline
query move to `trace`, so a `debug` log can be handed to someone else.
- Lifecycle lines name the chat and the post: sent/failed/queued plus the
total `ms`, the edit prompt, the channel forward, and every queue line
(`chat=` + `[key=…]` + per-attempt `ms`, dead-letters included).
- Queue work is visible: `x-media`'s fetch line carries its duration (ugoira
encode and HLS remux included), and the 300s sweep reports the pending count
and how overdue the oldest task is — only when the queue is non-empty.
- URL workers are supervised like the queue workers: a panicking worker used
to die silently and shrink the pool for the rest of the process.
- Degradations that still serve the user (cache/state write or read failures,
a failed chat action) are `warn`, not `error`.
Verified against the scripted fake-API harness: unset RUST_LOG logs info with
timestamps, `debug` carries no user URL, `trace` does, a cache-hit send logs
`chat=111 in 5ms`, a failing send queues and dead-letters with chat+key, and
the sweep reports the pending retry.
`/debug` reported `Fetched::caption`, the site's built-in caption, so a
chat's `/set_format` override never showed up in the preview — the command
looked like a no-op, and the `/set_format` success reply now tells users to
preview with `/debug`, which made that advice wrong.
`preview_caption` mirrors the send paths instead: the per-site format
override (`caption_from_fields`, empty → built-in) plus the long-post
quoting. Reported from a live run against the test bot.
teloxide's `split` parser takes exactly one space-separated token per
field, so `/set_format <site> <format>` — a two-token command — never
parsed: `Command::parse` failed, `message_handler` fell through to the URL
flow, and the user got silence. `/clear_cache` without its optional link
failed the same way ("too few arguments"), so clearing everything was
unreachable. Both now use the crate's `parse_arg_remainder` (whole
remainder, trimmed), which is what their executors were already written
against (`split_once(char::is_whitespace)`).
Found by driving the real binary against a scripted fake Bot API: the
`/set_format` replies never appeared while `/set_template` and the other
single-token commands did. `every_documented_invocation_parses` now pins
every documented form, which is what should have caught it.
The placeholder validation and `-` reset added earlier only work now that
the command reaches its executor at all.
`/start` was "Hello!" and `/help` was the bare command list teloxide can
render — no argument syntax, no caption placeholders, no mention that
links only work in private chats. Both now carry that guidance, and the
bot's profile description / short description are set at startup so a
shared link says what the bot does.
`/settings` reports what this chat is configured to do (forward channel,
edit-before-forward, per-site formats, saved templates) to anyone in the
chat — `/bot_dict` is a raw admin-only dump. Templates can be removed
(`/remove_template`, listing the live names on a typo) and the prompt's
keyboard folds 3 per row with a cap: Telegram rejects a keyboard over 100
buttons outright, which would silently drop the whole prompt.
Inline results hand URLs to Telegram, which fetches them without any
site headers — pixiv's pximg.net answers 403 to that, so those items are
skipped instead of shipped broken. `needs_media_headers` answers that
question from the same per-site rule the downloader uses.
Dead-letter and retry notices name the failing post and the cause
(`failure_text`), since "Task failed after retries: task failed after 2
retries" said neither which link it was nor what happened.
Four ways a user could get silence are closed: a registered-but-disabled
site (pixiv without a token) now answers instead of being dropped as an
unsupported link, `/test` on such a link replies instead of doing nothing,
a supported link posted in a group gets a one-line hint (channels stay
silent), and fetch failures name their cause — gone / withheld / source
risk control / site disabled / source down — instead of one generic
sentence. `FetchError::Disabled` carries the "matched but switched off"
answer, which `find_site` used to fold into `Ok(None)`.
A withheld tweet no longer degrades to "no media": without
`TWITTER_AUTH_TOKEN` it stays `Sensitive` so the reply says the media is
age-restricted, and a failed authenticated fallback propagates its own
class instead of masquerading as an empty post (`empty_fetched` is gone).
Long jobs stop looking stalled: `run_with_chat_action` re-sends the chat
action every 4s while the pipeline is pending and the hint switches from
typing to send-photo/video once the media kinds are known. Media groups
go from 9 to Telegram's 10.
`/set_format` rejects unknown `{…}` placeholders (a typo used to be
published verbatim in every caption) and resets with `-`. The
edit-before-forward prompt states its TTL and that Confirm is required,
gains a Skip button, and is rewritten in place to "expired" by the sweep
— an edit, never a new message, so a background timer cannot wake a chat.
A post whose text (the split `title` plus `content`, joined by
`site::compose_text`) reaches `CAPTION_QUOTE_TEXT_CHARS` — default 200,
`0` disables — now has that text wrapped in `<blockquote expandable>`
inside its caption, leaving the URL and author line outside the quote.
Applied at the send boundary (`send_media_sequence`, `send_animation` and
the inline answers), where the caption is already truncated and the same
cache snapshot supplies the text, so a fresh send, a link-cache resend
and a queued retry all decide identically. The text is located as what
follows the author link, with the visible prefix accepted as a match
because `truncate_caption` may cut inside it — that keeps the longest
posts, the ones that most need folding, quoted. Captions whose layout
moves the text elsewhere (pixiv's title-inside-a-link, a `/set_format`
that puts `{title}`/`{content}` first) stay unquoted rather than risking
a blockquote nested in a tag, and a caption that already carries one is
never wrapped again.
Telegram measures a caption *after entities parsing*, so the tags cost no
length and the 1024-character limit cannot be breached; retries replay
the unwrapped caption, so a threshold change takes effect immediately.
The edit-before-forward rewrite stays unquoted by design.
Verified against Telegram: a media-group caption built this way comes
back with `caption_entities` `url` @0, `text_link` @50,
`expandable_blockquote` @56 — the quote starts after the author line.
`Fetched.title` carried whatever text the platform had — a tweet's body,
a bilibili dynamic's body, a pixiv artwork's title — which was enough
while x/twitter (no title at all) set the shape. The platforms actually
disagree: pixiv has a title *and* a description, bilibili has an opus
headline *and* a body. Posts now carry both:
- `title`: the platform's title (a pixiv artwork title, a bilibili opus
headline or video card title), empty on text-only platforms;
- `content`: the body (tweet / bsky / misskey text, bilibili dynamic
body, and pixiv's description — fetched for the first time here and
flattened from the app API's HTML to plain text).
`{content}` joins the caption-format placeholders, so a custom
`/set_format` can include a pixiv description. The built-in captions keep
producing byte-identical output: `compose_text` joins the two fields the
same way the single field already was, and bilibili's forward marker
(`//@author:`) now lands in `content` behind the head line's `title`.
`CachedPost.content` is `#[serde(default)]`, so link-cache entries and
queued task payloads written before the split still parse, their text
living in `title`.
YoursFunny