Two things the 300 s sweep did the hard way:
- The link cache is pruned by `created_at` (`DELETE FROM link_cache WHERE
created_at < ?`) and had no index on it, so every sweep scanned the whole
table — every post sent inside the TTL window, which is up to a week of
them — while the `url` primary key served none of it. A new migration
(appended; migration 1 is frozen and already shipped) creates the index, and
the upgrade test now asserts it exists after an upgrade.
- `ChatStore::prune_expired` only ever *looked* at chats that had an expired
edit-before-forward record, so a chat with no prompt at all — the common
case: every chat that ever sent a message or ran a command — stayed in the
cache and in the per-chat lock map for the process lifetime. The candidate
set now includes chats holding no records, which is what the eviction below
was written for; the DB keeps the row, so the next use costs one SELECT
(pinned by a new test that also shows the durable settings come back).
Deliberately *not* done: skipping the write in `ChatStore::set` when the state
is unchanged. Comparing against the cached copy would skip a serialize plus a
blocking DB round trip for a no-op update — but every one of the 13 `update`
callers mutates something, so the no-op case is a user repeating an identical
command, and the same comparison would also skip the write that repairs a row
whose earlier write failed. A rare saving against a rare repair, and the write
is what makes the cache a cache rather than a source of truth.
Verified: the new eviction test fails without the candidate change (checked by
reverting it) and passes with it; 118 bot tests and 91 x-media tests pass.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
P0 (foundation) + P1 (diagnostic depth) of the logging plan:
- main.rs initializes the timed builder with a default filter of
`info,hyper_util=warn,reqwest=warn`. Without RUST_LOG nothing was logged at
all (env_logger falls back to `error`), so `docker run --env-file .env` was
silent, and the plain `init` had no timestamps.
- info-and-above lines stop printing user URLs (fetch/send failures, inline
fetch, bsky's remux warnings). The full URL, the message text and the inline
query move to `trace`, so a `debug` log can be handed to someone else.
- Lifecycle lines name the chat and the post: sent/failed/queued plus the
total `ms`, the edit prompt, the channel forward, and every queue line
(`chat=` + `[key=…]` + per-attempt `ms`, dead-letters included).
- Queue work is visible: `x-media`'s fetch line carries its duration (ugoira
encode and HLS remux included), and the 300s sweep reports the pending count
and how overdue the oldest task is — only when the queue is non-empty.
- URL workers are supervised like the queue workers: a panicking worker used
to die silently and shrink the pool for the rest of the process.
- Degradations that still serve the user (cache/state write or read failures,
a failed chat action) are `warn`, not `error`.
Verified against the scripted fake-API harness: unset RUST_LOG logs info with
timestamps, `debug` carries no user URL, `trace` does, a cache-hit send logs
`chat=111 in 5ms`, a failing send queues and dead-letters with chat+key, and
the sweep reports the pending retry.
`Fetched.title` carried whatever text the platform had — a tweet's body,
a bilibili dynamic's body, a pixiv artwork's title — which was enough
while x/twitter (no title at all) set the shape. The platforms actually
disagree: pixiv has a title *and* a description, bilibili has an opus
headline *and* a body. Posts now carry both:
- `title`: the platform's title (a pixiv artwork title, a bilibili opus
headline or video card title), empty on text-only platforms;
- `content`: the body (tweet / bsky / misskey text, bilibili dynamic
body, and pixiv's description — fetched for the first time here and
flattened from the app API's HTML to plain text).
`{content}` joins the caption-format placeholders, so a custom
`/set_format` can include a pixiv description. The built-in captions keep
producing byte-identical output: `compose_text` joins the two fields the
same way the single field already was, and bilibili's forward marker
(`//@author:`) now lands in `content` behind the head line's `title`.
`CachedPost.content` is `#[serde(default)]`, so link-cache entries and
queued task payloads written before the split still parse, their text
living in `title`.
Fetch `t.bilibili.com/<id>`, `www.bilibili.com/opus/<id>`,
`t.bilibili.com/h5/dynamic/detail/<id>` and `m.bilibili.com/dynamic/<id>`
through the anonymous `/x/polymer/web-dynamic/v1/detail` endpoint (no
cookie, no WBI signature; the site adds the device cookies
`/x/frontend/finger/spi` hands out, which is what lifts bilibili's
`-352` risk control).
Media: the `major.draw` grid (`.gif` sources become animations, the rest
photos with a downscaled `@518w.jpg` thumbnail used both as preview and
as the oversized fallback), an attached video's cover, and the quoted
dynamic's media for forwards. The video stream itself is not resolved;
`b23.tv` short links stay unmatched (they mostly point at videos, so
matching them would turn a silently ignored link into a failure reply).
`-352`/`-412` map to a retryable error so the queue backs off instead of
dropping the post; a removed dynamic (`500`) is permanent.
Registry-driven, so no bot-side code changes beyond the site lists in the
command replies; found while researching nazurin and
telegram-bili-feed-helper (see BILIBILI_PLAN.md).
AGENTS.md:
- db.rs row claimed a per-store connection pool; there is one shared pool for
all three tables (statics.rs builds it once).
- retry enqueue moved to send.rs, noted in both handlers rows.
- queue row now names both notifies (workers' + the sweep's).
- Retries bullet documents fetch vs fetch_once.
- test count ~125 -> ~135, the untested-files list no longer claims state.rs
and handlers.rs are untested, and the live-test inventory mentions the
token-gated, not-#[ignore]d pixiv download test that makes a local
`cargo test --workspace` hit the network.
- /bot_dict is admin-only now.
Code docs:
- site/mod.rs: the module doc pointed new sites at `fetch_once` (a name that
did not exist then and now means a single-attempt fetch) -> `SITES`; the
cache_key/SITES/Site docs still said "twitter -> bsky -> pixiv" (misskey
is registered third); RenderData now documents which fields are escaped
and why url/author_url are not.
- state.rs, callback.rs: drop the pre-misskey site list and the `<name>`
that rustdoc read as an HTML tag.
- Fixed the remaining rustdoc links/warnings: `cargo doc --workspace
--no-deps` is now warning-free (was 8).
- docs/site-registry-refactor.md: §1 describes the pre-refactor state; said so.
No behavior change. fmt/clippy clean, 55 + 68 tests pass (the live pixiv
download test flaked on a CDN body timeout, as before).
- send: the PHOTO_INVALID_DIMENSIONS marker never matched (the description is
lower-cased, the marker was not), so oversized photos sent by URL were
classified Permanent instead of taking the download-and-downscale fallback.
- inline: the debounce state was one global slot, so a second user's query
cancelled the first user's pending answer entirely; it is now per user.
- state: prune_expired wrote back a stale snapshot without the per-chat lock,
clobbering a concurrent update() (lost edit-message record -> "Expired");
it now re-reads and prunes under the same lock update() uses.
- send/queue: a task dead-lettered on retry exhaustion kept its keep-alive
temp media (ugoira MP4) alive until process exit; dead_letter_notify now
releases it, and enqueue_retry releases when the enqueue itself fails.
Also folds the duplicated retry enqueue in post_send_actions into
send::enqueue_retry (single clock source, single place that releases).
Tests: +6 (marker, per-user debounce x3, keep-alive release, prune contract
x2); the marker and keep-alive cases were verified to fail before the fix.
cargo fmt/clippy clean, 52 + 69 tests pass.
- updated_sequence_task: clone the Task and mutate the two fields
instead of rebuilding all 12 by hand (-22 lines; new fields no
longer need a sync here)
- unify unix_now with db::now_f64 (unix_now() = now_f64() as i64),
moved to db.rs next to its clock source
- classify_to_send_error takes the MediaFetchFailure label, folding
the duplicated inline match in send_batch_via_upload (-8 lines)
Every DB operation (queue lease/enqueue, chat_state get/set, link_cache
read/write) used to open a fresh connection — including the busy timeout
and WAL pragma — then close it, on every message, URL job and callback.
Replace with DbPool: a tiny pool (4 connections max, semaphore-bounded
concurrency for backpressure) whose with_conn() method runs the closure on
a pooled connection inside spawn_blocking. Steady-state cost of an
operation is a list pop + semaphore acquire instead of a connection open.
prune_expired only shrank edit_message maps, so the cache kept one
ChatData per chat forever (a leak proportional to chat count). Chats
without live edit records are now dropped from the cache and their
per-chat lock (DB row persists; get() reloads). Lock order kept safe:
prune never holds the cache lock while taking the per-chat locks.
Concurrent handler tasks (the batch-forward design spawns several per
chat) each snapshotted the same ChatData and last-writer-wins silently
dropped mutations — e.g. a second edit_message record, leaving one
prompt's Forward button dead. All write cycles now run under a per-chat
async lock; read-only callers keep get().
Converge the duplicated open_db (open + busy_timeout) and the
spawn_blocking + expect ceremony that every table access repeated
into one db.rs module. ChatStore no longer creates the tasks table
(schema ownership: queue.rs owns tasks, state.rs chat_state,
link_cache.rs link_cache). No schema or behavior change - all
CREATE TABLE statements are byte-identical, IF NOT EXISTS stays
idempotent, so existing data/task_queue.db files need no migration.
YoursFunny