Two calls per link went out that could not affect anything:
- `run_with_chat_action` awaited the opening `send_chat_action` to completion
before the pipeline was polled at all, and again inside the loop on every
`ACTION_REFRESH`. Telegram round trips are hundreds of ms: the first delay
came out of the user's wait for every link, and each refresh suspended the
fetch (an ugoira encode or HLS remux runs for seconds) by the same amount.
- `handle_message` enqueued *every* URL a private chat posted, including
links no site adapter claims. Those cost a queue slot, a worker wake-up,
a `Message` clone and (through the action above) one Telegram call, only
for `url_media_inner` to conclude there was nothing to send. The group
branch has always made the `cache_key(url).is_some()` test before it acts;
the private branch now makes it before it enqueues.
The in-flight action is held (`Option<BoxFuture>` — the sender surface is
already type-erased, so it is `Unpin`) and polled as its own `select!`
branch: still polled *before* the pipeline, so the indicator is on screen
before the first send, but a slow Telegram response can no longer delay the
pipeline, and none of the branch bodies ever awaits one. One action is in
flight at a time; a refresh while one is unanswered is skipped rather than
dropping the request mid-flight. Note that `select!` evaluates every branch's
future expression eagerly, so the `None` case is an `async` block whose
`unwrap` only runs when the branch is polled (the eager form panicked).
Behavior pinned by the existing tests, unchanged: the opening action precedes
the first send, a 12s pipeline still sees exactly three actions
(`a_long_pipeline_keeps_the_chat_action_alive`), and an unsupported URL
reaching `url_media` still gets the one indicator before the pipeline settles
— in production it no longer reaches `url_media` at all.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
Every test went through `MockSender`, so `media_sender`'s `Bot`
implementation — the URL it builds, the multipart it sends, the per-chat
limiter and the bot-wide budget it charges — was never exercised, and neither
was any handler reached from a real update. The two things that made that hard
are gone:
- `media_sender::test_support::fake_api::FakeApi` is a stand-in for
`api.telegram.org`: a `tokio` TCP listener that reads one HTTP/1.1 request
(JSON or multipart), records it and answers the smallest result the method
needs. No new dependency, and `Bot::new(token).set_api_url(api.url())`
points a real `Bot` at it. Note for future tests: teloxide keys methods by
payload type, so the path is `SendMediaGroup`, not `sendMediaGroup`.
- `message_handler` built its own `AppContext::from_statics` internally, so no
test could reach its branches; its body is now `handle_message(ctx, bot,
message)` with `message_handler` as the thin `dptree` entry.
Tests: a media group through the real `Bot` (asserting the multipart fields —
chat, media URL, caption — and that the send charged the chat's limiter), the
forward button through the real callback path (`CopyMessages`,
`DeleteMessage`, `AnswerCallbackQuery` with the prompt's ids and the toast
text), and `handle_message` twice (a prompt reply becoming an
`EditMessageCaption`, and a supported link in a group producing the one
explanatory `SendMessage`).
Also closes the redirect-hop gap left open by the download guard: the live
`a_redirect_into_the_hosts_network_is_refused` follows a public redirector to
`169.254.169.254` and asserts the policy refuses the hop (verified against
httpbin.org here, and by mutation — disabling the hop check fails it).
Docs: AGENTS.md's testing conventions and untested-modules list (the Bot
implementation and the handler branches are covered now; `main.rs`'s
startup/shutdown and its `dptree` tree still are not).
`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (201 passed, 16 ignored) clean.
Both caption-edit paths logged the error and carried on as if they had worked:
`edit_message_handler` consumed the user's reply (`let _ =`), and the
template button updated the prompt record and dismissed its toast with no
text. Since the edit surface is not rate limited, a 429 or a transient
failure meant the caption never changed and the user got no hint — the text
they sent was simply gone.
`apply_caption_edit` (handlers/mod.rs) is now the one place that applies a
caption edit and reports the outcome:
- A failure the API calls worth retrying (`classify_request_error` →
`Retryable`) is retried once when the delay is at most 2 s — a reply or a
button press has already been consumed by then, so a long flood-control
wait must not stall the chat's update queue behind it.
- Otherwise the caller reports it: the reply path answers the user ("Could
not update the caption (…). Send it again to retry."), the template path
puts it in the callback toast and leaves the record alone — a swap that
never happened must not be recorded as the prompt's template.
Tests: the swallowed-failure test now asserts the notice (it pinned the old
silent behaviour), plus a short `RetryAfter` that is retried and lands, a
60 s one that is not retried and is reported instead, and the template
button's failure toast with the record left unchanged.
`cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (190 passed, 15 ignored) clean.
One note: the first full run tripped `download_media_pixiv_original_with_
referer`, the token-gated pixiv CDN download test that AGENTS already
documents as a local-network flake; it passes in isolation and on the rerun.
A queued retry that holds a local file — the ugoira MP4, a bsky remux, or a
temp file the reupload fallback downloaded — could never succeed after a
restart: those files live in the system temp dir and `send::KEEP_ALIVE`, the
registry that keeps them alive for the retry, is in memory. The row retried
into an upload error, said nothing about why, and dead-lettered the user's
link even though the payload carries the `source_url`.
`handlers::repair_lost_local_media` now runs in `main` before any worker
starts (so no row can be leased while it writes payloads, which is why it can
replace them without the lease guard a worker's write-back carries):
- `Task::local_media_paths` decides which rows are affected: any local path
that is gone. A partially delivered album is left alone — its remaining
batches cannot be reconciled with a fresh media list without risking a
second copy of what the user already received.
- The post is re-fetched from `source_url` through the ordinary `site::fetch`,
so a repaired task looks like a first send: fresh media, the chat's caption
format, a fresh link-cache snapshot, and a new keep-alive entry when the
re-fetch produced another local file.
- The delivery envelope (chat, reply, forward/edit settings, notify targets) is
kept, the attempt budget restarts, and nothing counts as sent.
- A post that cannot be fetched again (gone, withheld, site down) notifies the
user with that reason instead of letting the retry die on a missing file.
New queue plumbing: `runnable_rows()` (pending + in-progress rows, read before
the workers exist) and `replace_payload()` (rewrites the payload, resets
`attempts`, marks the row pending).
Verified: 5 new offline tests (the two decisions above against a real temp
file, the queue scan/replace, and the envelope-preserving rewrite) plus
`a_lost_local_media_row_is_refetched_from_its_post`, a live test that seeds a
row pointing at a missing file with a real bsky post as its source and asserts
the row now carries http(s) media and that nothing was sent — run against the
live API here. `cargo fmt`, `cargo clippy --workspace --all-targets --locked --
-D warnings` and `cargo test --workspace --locked` (187 passed, 15 ignored)
are clean.
Three gaps the last audit list named, all in the "resource growth, background
timers and limits nobody watches" class.
**Idle inline-query entries are pruned.** `DebounceStates` had no eviction at
all: one entry per user who ever used inline mode, forever, while the rate
limiter's buckets and the chat store both prune in the 300s sweep. Entries
now carry a `last_seen` stamp and `prune_idle_states()` drops the ones idle
past 300s — the window Telegram caches an inline answer for
(`cache_time(300)`), after which a repeat reaches the bot again and has to be
answered fresh, so the entry would only suppress a fetch the user is waiting
for. The boundary is tested through `prune_idle_at(now, idle_for)` so it does
not depend on ageing a monotonic clock.
**The 300s sweep is a function, and tested.** It was an inline `tokio::spawn`
block: the expiry edit (the only part that talks to Telegram) had no test at
all. It is now `periodic_sweep(sender, chat_store, link_cache, task_queue,
config, stop)`, which also prunes the inline entries, driven in a test with
`start_paused` — the loop's own timer fires the tick, exactly one expired
prompt is rewritten in place, a live one keeps its record and buttons. The
interval is pinned as a constant because no assertion on the edits can see it
(a shorter one produces the same single edit; the paused clock can jump past
the boundary while a tick's DB work is in flight). To make the edit reachable
at all, `edit_message_text` joined the `MediaSender` trait (Bot impl + mock
recording), which is also what keeps `main.rs`'s remaining `Bot` calls
unambiguous. `main.rs` leaves the "untested modules" list except for
startup/shutdown and the dispatcher tree.
**The bot-wide send budget exists.** Telegram throttles a bot in total
(~30 msg/s) as well as per chat; only the per-chat bucket existed, so a batch
forward fanned out over many chats was unguarded and earned 429s the queue
then retried. `acquire_global` charges the same spend against a single shared
bucket at the three paced sites (`send_media_group`, `send_animation`,
`copy_messages`). The unpaced ones (`send_message`, the edits, the toasts) stay
unpaced on purpose: they are one call per action, far below the ceiling, and
pacing a user-visible reply would delay it. Not covered: that the send paths
call it (they need a real `Bot`), which is the same structural gap as the
dispatcher tree.
Also: the startup token-exchange decision is now `startup_validation(result)`
instead of living inside the `Site::validate` future, so "a 5xx while the
container comes up must not disable pixiv" is asserted as a decision — the
message the admin gets plus `enabled()` unchanged. The rejected-credential
half is deliberately not exercised: it calls `disable()`, a process-wide flag
with no reset, and a test touching it would order-couple every other pixiv
test.
Verified: `cargo fmt`, `cargo clippy --workspace --all-targets --locked -- -D
warnings`, `cargo test --workspace --locked` (184 passed, 14 ignored) — plus
mutations, each confirmed to fail the relevant test: the sweep not being
driven on its timer, the interval shortened to 60s, and (earlier) the queue
sweep's missing wake-up. Dropped an empty leftover `crates/x-media/tests/`
directory while there (never tracked by git).
The same fixtures were rebuilt in five test modules: a `CachedPost`
literal in `link_cache.rs`, `handlers/urls.rs` and twice in
`send/mod.rs`, the edit-before-forward prompt in `handlers/mod.rs` and
`handlers/callback.rs`, and a scripted API error in both handler
modules. They now live in `ctx::test_support` next to `TestStores`:
- `cached_photo()` — the canonical cached post (photo + file id at
`https://x.com/u/status/1`, key `twitter:1`); tests mutate the fields
they care about, as the caption-quote test already did.
- `seed_prompt(template, created_at)` + `PROMPT_ID`/`FORWARDED_ID` —
the prompt record, the chat template and the bound forward channel.
The two former copies differed only in which knob the caller set (the
callback tests backdate it for the expiry cases, the reply tests pick
the template), so the union is one helper.
- `api_error(message)` — construction only; each test module keeps its
own message constant, because the wording is what that module's path
answers with (`chat not found` vs `message not found`).
`send/mod.rs`'s `cached_sequence_cache_data()` (which re-extracted the
post out of the task it had just built) is gone: the two settle tests
seed the cache from the same builder the task uses.
No behaviour change: the values are the ones the tests used except
`file_id` (`AgAC-file-id` everywhere, asserted in the link-cache
round-trip) and `sensitive` (the unasserted `true` in the link-cache
fixture), and every test still passes unchanged.
Verified: `cargo fmt`, `cargo clippy --workspace --all-targets --locked
-- -D warnings` and `cargo test --workspace --locked` (180 passed, 14
ignored).
P2 of the logging plan (the README recipe landed with the code change):
- `docker-compose.yml.example`: one `x-logging` anchor applied to all three
services. json-file grows without limit by default, so a long-running bot
and the proxy in front of it fill the disk; capped at 10m × 3 files.
- The startup `config:` line now reports what the process actually resolved —
the state DB path (a mistyped `DATA_DIR` or a surprising CWD was invisible
until it bit), both TTLs, the caption-quote setting (`off` rather than a bare
`0`) and whether a proxy is configured. The proxy URL is never printed (it
may embed credentials) and admin ids — chat identifiers — stay at `debug`.
Verified: `docker compose config -q` accepts the file, and a scripted fake-API
run shows `caption quote off` / `link cache TTL 3600s` under overrides,
`proxy=yes` with no credential in any line, and the ids at `debug` only.
P0 (foundation) + P1 (diagnostic depth) of the logging plan:
- main.rs initializes the timed builder with a default filter of
`info,hyper_util=warn,reqwest=warn`. Without RUST_LOG nothing was logged at
all (env_logger falls back to `error`), so `docker run --env-file .env` was
silent, and the plain `init` had no timestamps.
- info-and-above lines stop printing user URLs (fetch/send failures, inline
fetch, bsky's remux warnings). The full URL, the message text and the inline
query move to `trace`, so a `debug` log can be handed to someone else.
- Lifecycle lines name the chat and the post: sent/failed/queued plus the
total `ms`, the edit prompt, the channel forward, and every queue line
(`chat=` + `[key=…]` + per-attempt `ms`, dead-letters included).
- Queue work is visible: `x-media`'s fetch line carries its duration (ugoira
encode and HLS remux included), and the 300s sweep reports the pending count
and how overdue the oldest task is — only when the queue is non-empty.
- URL workers are supervised like the queue workers: a panicking worker used
to die silently and shrink the pool for the rest of the process.
- Degradations that still serve the user (cache/state write or read failures,
a failed chat action) are `warn`, not `error`.
Verified against the scripted fake-API harness: unset RUST_LOG logs info with
timestamps, `debug` carries no user URL, `trace` does, a cache-hit send logs
`chat=111 in 5ms`, a failing send queues and dead-letters with chat+key, and
the sweep reports the pending retry.
Four ways a user could get silence are closed: a registered-but-disabled
site (pixiv without a token) now answers instead of being dropped as an
unsupported link, `/test` on such a link replies instead of doing nothing,
a supported link posted in a group gets a one-line hint (channels stay
silent), and fetch failures name their cause — gone / withheld / source
risk control / site disabled / source down — instead of one generic
sentence. `FetchError::Disabled` carries the "matched but switched off"
answer, which `find_site` used to fold into `Ok(None)`.
A withheld tweet no longer degrades to "no media": without
`TWITTER_AUTH_TOKEN` it stays `Sensitive` so the reply says the media is
age-restricted, and a failed authenticated fallback propagates its own
class instead of masquerading as an empty post (`empty_fetched` is gone).
Long jobs stop looking stalled: `run_with_chat_action` re-sends the chat
action every 4s while the pipeline is pending and the hint switches from
typing to send-photo/video once the media kinds are known. Media groups
go from 9 to Telegram's 10.
`/set_format` rejects unknown `{…}` placeholders (a typo used to be
published verbatim in every caption) and resets with `-`. The
edit-before-forward prompt states its TTL and that Confirm is required,
gains a Skip button, and is rewritten in place to "expired" by the sweep
— an edit, never a new message, so a background timer cannot wake a chat.
Fetch `t.bilibili.com/<id>`, `www.bilibili.com/opus/<id>`,
`t.bilibili.com/h5/dynamic/detail/<id>` and `m.bilibili.com/dynamic/<id>`
through the anonymous `/x/polymer/web-dynamic/v1/detail` endpoint (no
cookie, no WBI signature; the site adds the device cookies
`/x/frontend/finger/spi` hands out, which is what lifts bilibili's
`-352` risk control).
Media: the `major.draw` grid (`.gif` sources become animations, the rest
photos with a downscaled `@518w.jpg` thumbnail used both as preview and
as the oversized fallback), an attached video's cover, and the quoted
dynamic's media for forwards. The video stream itself is not resolved;
`b23.tv` short links stay unmatched (they mostly point at videos, so
matching them would turn a silently ignored link into a failure reply).
`-352`/`-412` map to a retryable error so the queue backs off instead of
dropping the post; a removed dynamic (`500`) is permanent.
Registry-driven, so no bot-side code changes beyond the site lists in the
command replies; found while researching nazurin and
telegram-bili-feed-helper (see BILIBILI_PLAN.md).
docs/architecture-refactor.md §3 sketched the trait with "按需扩展：
edit_message_caption / delete_message / answer_callback_query …", but only the
five send methods landed, so `callback.rs` and the edit-before-forward caption
swap were stuck on the concrete `Bot` and remained untested (AGENTS.md still
lists callback.rs as untestable).
- `MediaSender` gains `answer_callback_query`, `edit_message_caption` (HTML
parse mode baked in, every caller uses it) and `delete_message`; the mock
records call order plus the texts, captions and answer toasts, so tests can
assert what the user saw.
- `send_message` now returns the sent message id instead of the whole
`Message`: the only consumer of the value is the edit-before-forward prompt
(which keys its record by it), and returning a `Message` forced every mock
to build a teloxide type. `reply`/`reply_html` follow.
- `callback.rs`: the dptree entry only unpacks the update; `handle_callback`
takes plain values + `&AppContext`. `handlers/mod.rs::edit_message_handler`
likewise takes the values the reply carries. Admin/setup APIs
(`get_chat`, `get_chat_administrators`, `get_me`, `set_my_commands`) stay on
the concrete `Bot`: they are not user flows worth a trait.
- The scripted mock moves to `parking_lot::Mutex` (no poisoning unwraps).
Tests: +11 (template button, forward ok/no-channel/retryable, expired+unknown
prompt, caption swap via template, escaping of user text into the caption,
failed swap still consuming the reply, prompt record written by post_send).
fmt/clippy clean, 70 + 69 tests pass.
- commands: /bot_dict dumped the whole chat state to any member of the chat
and could exceed Telegram's 4096-char message limit (the send then failed
and bubbled up as a handler error). It is now admin-only and capped at
MAX_DEBUG_DUMP_CHARS; README, README.en and the /help description updated.
- send: the edit-before-forward template buttons were built from a HashMap
walk, so their order changed between prompts. Now sorted by name.
- link_cache: an unparseable payload (older schema) was reported as a miss
but left in place, re-failing the parse on every later hit; the row is
dropped on read.
- handlers: a link handed to the URL workers after the channel closed
(shutdown) was discarded silently; it is now logged.
- rate_limit: LIMITERS kept one bucket per chat that ever sent media,
forever. The periodic sweep now drops buckets that are idle (refilled to
capacity) and not held by an in-flight sender; acquire's refill was
factored into a shared helper used by the idle check.
Tests: +3 (corrupted row dropped, sorted markup, idle-bucket pruning); the
cache one was verified to fail before the fix. fmt/clippy clean, 55 + 69.
Replaces the strip-tags plain-text rendering: the /test reply is now an
HTML message (reply_html helper with ParseMode::Html). Raw fields (url,
source_url, title, author_url, media urls) are escaped, the pre-escaped
render fields are embedded as-is, and the caption is wrapped in
<blockquote>...</blockquote> so the report shows it exactly as it will
render in the sent media caption — escaped text and clickable links
included, no literal &/</> and no raw markup.
YoursFunny